Monday, January 19, 2015

Logstash Examples with Data Generators

Logstash is a pretty cool application that generically takes some input, does something with the data, and spew it out somewhere else.

It's commonly used in conjunction with Elasticsearch, a Lucene based search service, and Kibana, a dashboard UI for Elasticsearch.  You can easily consider this as a "Splunk without super awesome ad hoc query capabilities" but the software itself is FREE so there's that.

The documentation for Logstash is pretty straightfoward, but I thought it would be nice to have some hard examples to work off of that involved the whole ELK stack:

Right now it just has two examples: file and log4j.

  • Write to a file
  • Logstash monitors the specific file and does some grokking before passing it to Elasticsearch
  • Load the provided kibana dashboard.
  • Spew log4j stuff out with SocketAppender
  • Set up Logstash to monitor a port for log4j messages and grok before passing it to Elasticsearch
  • load the provided Kibana dashboard
So yeah I hope this serves as a quick and easy way for people to observe the awesomeness that is Logstash.  Feel free to add your own examples to this guy as well.

Saturday, January 10, 2015


Wife:  "How do you think I've been doing with my makeup lately?"


Wife:  "... IT'S A TRAP!  No, seriously.  You have to answer."

Tuesday, December 23, 2014

Read the Freaking Documentation to Couchbase!

Just wanted to point out a pretty awesome product with pretty awesome documentation:


We're using it for a pretty specific use case and for that, it's handling things extremely well.  However, we noticed that it was using a ton of swap space to the point I was thinking we were on Windows boxes.

Buuuut lo and behold we didn't read all the documentation on how our nodes should be set up and here it is for v2.2 that we're using.

Actually that screenshot is further down in the issues section, complete with impact and JIRA issues!

It's actually pretty awesome to be able to track something down this easily.  Obviously, other people have encountered this that has lead to issues but it's great to see the trail and the resulting recommendation.

Sunday, December 07, 2014

Searching Pull Requests in Stash - You Don't

A pretty popular JIRA Issue for the Stash project is this guy, STASH-3856:

Specifically, the only reason why I'm searching for a pull request after the fact is to see if there were any conversation regarding a particular design decision or if there were any explicit things that were abstained from for later work.

Also, it's a super quick way to see when a given feature was merged into a particular branch like Master.

With that said, chances are, you're using JIRA alongside Stash.  Let's be honest, the only reason to use Stash over other solutions is for its JIRA integrations.  If your issues are directly associated with your Pull Requests, you should then be able to easily search your JIRA issues using its robust search features.

JIRA Development Panel
So unfortunately, if you're eagerly awaiting STASH-3856 to be implemented, I highly doubt that they're going to accomplish this any time soon.  There are definitely some Stash specific searches that would be nice, such as when a pull request's been merged or dismissed and when it's been updated.  However, until then searching for the JIRA issue can resolve the vast majority of people's use cases.

Tuesday, December 02, 2014

JIRA - Old School Greenhopper and Milestone Management with Versions

I wouldn't be surprised if there are still a lot of organizations that are still using the Classic Boards in JIRA Agile to manage their medium to long term plans in JIRA.  I want to just go over really quickly how one can manage these in the Classic Board before I talk about a much more complex scenario in a future post and I also want to go over the limitations of this approach.

First, you'd establish Versions in your Project that represented different Milestones with dates associated with them.  From there, you'd have your actual release Versions that are directly associated with the releases that you're cutting.
With the Classic Planning Board, using the SubVersion feature, you can readily see which Release versions that build up to Milestones that build up to broader Milestones.  You'd associate stories to your release versions and from there, you can track the progress of your Parent Milestone versions.

I wanted to work this particular feature in here as I still find it to be extremely useful today that few people pay attention to.  That is the Merge Version feature.  Essentially, if you already established a Patch release and realized that this guy isn't going to be released until the next Minor version, you just merge the Version into that Minor version.  From a Git Flow perspective, this would be comparable to the idea that you had a Release Branch prepared but never merged it to Master, but rather, just merged changes (if any) back to Develop.

So anyway, the biggest limitation to this approach are the following:

  • FixVersion in JIRA suddenly becomes an overloaded Term.
  • This method is limited for a Single JIRA Project.
Specifically, Atlassian realized that using FixVersion for so many different things, especially sprints, was not going to be a good fit for organizations that require some flexibility.  That is one of the core reasons why the Sprint field is a new core field for JIRA Agile.  Also, when a field starts to mean multiple things, 

Organizations can establish JIRA projects in a variety of ways for a variety of reasons.  However, when you have two projects gearing towards a common milestone, a lot of organizations start getting into the habit of having versions of the same name in multiple projects and identify them as the same milestone.

This can lead to a lot of confusion and inflexibility in how a new project can be established.  Also, you're shoehorning an association between two fields that the system doesn't natively support, you're probably using that field wrong.

Soapbox Time:  It's pretty hilarious how we, as developers of software for others to use, really suck at using the tools as they are originally intended.  Shoehorning functionality that doesn't exist "just because we want it that way" either means that we're just using the wrong tool or we honest don't know how we want our processes to function.  

Anyway, if you are still using Versions in this manner in JIRA Agile, I'd be very interested in what you like or don't like about using this method.  Next time, I'm going to cover the more recent JIRA Agile functionality that's be introduced circa 2011 that provide some more flexibility, but still have a lot of limitations.

Tracking Multiple Sprint Teams with a Common Goal

... or rather, a goal that YOU have that you're depending on other teams to accomplish for you...

So one thing I feel that people using JIRA Agile should get used to is the notion of building and breaking down Agile boards for targeted efforts to allow them to track something very specific and eventually just get rid of the board when they're done with it.

Here's a query:

"Epic Link" in (EPIC-1, EPIC-2, EPIC-3, EPIC-4) OR issue in (EPIC-1, EPIC-2, EPIC-3, EPIC-4) OR labels = myEffortLabel ORDER BY Rank ASC

What this query essentially means is "Give me these Epics, give me the Issues in these Epics, and give me any issue that I happen to label it.

This gives you a quick and hopefully, targeted view of a set of milestones in the form of Epics that you and other teams are driving towards. My personal observation is that the majority of teams pretty much ignore epics and epic links anyway, so establishing this linkage to the actual stories that they are working on should have minimal impact.

At the very least, you can keep an eye on the general progress of a concerted effort.

Just remember to clean up after yourself and delete the agile board when it's all said and done!

Friday, July 11, 2014

New Job, New City... so Let's Talk St. Louis

So how long does it take for one to find apply, interview, accept a job, prepare a house for sale, move, and settle down for a weekend?

About 3 months, apparently.

BTW, Google "days since april 17, 2014" and you get to a nifty app that tells you how many days.

So anyway, I don't usually talk about St. Louis on here as that's what's Yelp's been for.  Aside from our friends in the area, I'd like to take some time to point out some things that I enjoyed in my 10 years there.

La Pizza / Protzel's Deli / Bob's Seafood


These are the 3 establishments that I will miss the most.  Not because of what they offered, but the people who worked there.  Heck, the La Pizza guys convinced the Bob's Seafood guy to let me pick up 60 lbs of live crawfish on a day they were closed.  That's how awesome they are.  I think I'll be hard pressed to find places comparable.  On top of that, these places were within 5 miles of each other, which leads me to my next point:

Regional Convenience

Sure, you could talk about what other cities have that St. Louis doesn't have, but if it were in St. Louis, it was dead simple to get there.  We lived in a spot where we could get to just about anywhere in the region in 30 minutes or less.  Also, 30 minutes typically means ~20 miles away.  Seriously.  The other day the wife was looking for things within a mile away from our place in Chicago and she was like "that's FAR!"

There's no Mitsuwa in St. Louis, but it might as well be because I have no idea how often we're going to get out that far into the Chicago burbs from the city.

Ok, I'm lying.  We have some good friends who live nearby and we'll likely use them as an excuse to go to Mitsuwa at least once a month or so.

Hometown Pride

To be honest, I think just about every single midwest city suffers from some form of inferiority complex.  Chicago included.  St. Louis is definitely not an exception, but the passion for working towards improving the area is infectious.  The atmosphere in the more urban neighborhoods is leaps and bounds more lively and optimistic than they were even a couple years ago.  The Loop, the Grove, Cherokee, even Downtown, which many left for dead multiple times in the past couple of decades, all have some pretty significant work going on that I was pretty gosh darned excited for.  

View from the top of the City Museum
STL is one of those areas that is still transitioning from manufacturing to technology and sciences.  They definitely have some big trials coming up on the jobs front but there're enough skilled passionate people who actually have loyalty to the area itself to establish themselves and make it work.  Square, Woot, and Riot presences past or present were as much personal decisions as business decisions.

St. Louis has been good to us and I have nothing but fond memories of the area.  It will forever be the first place I've been an "adult" at and has definitely shaped my outlook on things.  See you around.

Sunday, April 27, 2014

Microservices! Rorschach Reactions.

+Martin Fowler and James Lewis of ThoughtWorks have a really in-depth and thoroughly thought out article talking about Microservices.  There's particularly a nice blurb in it comparing the term to the more generic SOA.

The context of this article is kept at a more technical execution level.  I'd like to bring up some thoughts on the requirements and communication that can be involved.  In reality, this is more of a Rorschach reaction to the pretty awesome images that they had on their article.

Generally, three things a software development team thinks about are:
  • What is my Deliverable?
  • How is it going to change over time?  What are my requirements?
  • Who are impacted of my changes?  How do I coordinate with them?
However, there's another question that, surprisingly, a lot of IT organizations are still extremely immature at handling:
  • What deliverables need to be changed for my needs?  How do I get those changes to happen?
Here's a pic from the article discussing a functionally siloed organization and the corresponding Architecture that tends to occur.

Because of Conway's law, the architecture reflects the org structure, and this is also a representation of how requirements flow.  The UI folks have some user-facing use case and they need a service change from the middleware folks, who in turn, need something of the DBA guys.  One common disadvantage to this is the further away you are from the original requirement, the more you get some odd telephone game situation going where the API isn't exactly what the UI guys are looking for or the DBAs decide to ignore the middleware guy's model design.  This can lead to a lot of churn and rework.

With a cross functional team, you have teams that are more focused on the use-facing use case, rather than entire groups of people who are only focusing on their immediate deliverable.  One excellent point that is also made in the article is that large monolithic architectures can be organized "too many contexts."  The article posits that this tends to be a result of cross-functional teams in a monolithic architecture, but I think this happens regardless of how your teams are organized.  This is a big reason why we can get a lot of spaghetti logic, where a module may be trying to satisfy too many things at once, as opposed to being split into separate modules.

Something that these pictures also accurately depict is that your cross-functional teams would have fewer people of each function focusing on particular business logic and can likely be working on a particular microservice.  Looking at this from a requirements flow perspective, you can potentially get resource constraints (or managers imposing them), which is one of the main reasons why functionally siloed organizations exist to begin with.

This comes back to the questions:
  • What microservices need to be changed for my needs?  How do I get those changes to happen?
Do you make requests to a microservice's backlog to be prioritized?  How do you deal with that team's resource constraints?  How do your requested changes impact other teams consuming that microservice?  These questions aren't specific to microservices, but with more granular teams, this amount of "red tape" may seem daunting to more traditionally managed organizations.

In addition,  with that many more deliverables in the form of microservices, ensuring that the consumers of your deliverables are prepared for the changes you're making.

In the left side for monoliths, consumers of modules have the option to choose which version of the module to include in their process at build/deploy time.  On the right side, the consumers have the option to choose which module to utilize by communicating with the appropriate service at run-time.  Fundementally, there isn't too big a difference in these dependencies, but operationally, in terms of identifying your dependencies and working with your sys admins on deployments, it is pretty different in terms of how you're communicating changes, dependencies, and to whom you're communicating to.

It all boils down to how well your organization can communicate across your teams.  For a small organization, physically talking or chatting on email or a chat service works out.  However, with large geographically distributed teams with potentially different cultures and whatnot, this gets unwieldly very quickly.

This post is getting long so I'm going to cut it here.  I'm not just talking about microservices, I'm talking about any organization that has multiple internal deliverables across teams.  How can a large geographically distributed organization, consisting of multiple teams, meet customer expectations while ensuring smooth operation from build, to test, to deploy?  I'll run through some thoughts on the challenges and solutions that may fit your given culture.