30 August 2017

Not Just A Song

Patience is not just a song. 

Personally I struggle mightily with patience, but it is a virtue well worth developing. Patience as a teacher, mentor, collaborator, and friend is essential to success. If you can remain calm and keep your communication open and clear you can make a lot more progress than if you lose your patience and do something rash.

Remember, Deep Breaths.

28 August 2017

Over Come By Events

Sometimes we take on too much. I've talked about taking a break and about meeting objectives. I've even talked about commitment. But what happens when it is physically impossible to meet an objective -- when you are over come by events.

You need to reset your commitment, and you need to do so publicly and firmly. 

Over the last 40 years I don't know how many times I've had to appeal to someone else's grace because I've failed to deliver something on time. From homework assignments in school to project deliverables or time-sheets. There are any number of things I've failed to deliver on time.

I think I've mentioned, I'm an early-is-on-time, on-time-is-too-late kinda guy. I hate being late. I will burn the candle at both ends to be on-time. Even with that attitude there are times where the laws of physics get in the way.

So, when I find that I'm not going to make a commitment, as soon as I know, I contact the person I've made a commitment to and let them know. I then recommit and get back to it. 

Simple examples of this are, I'm going to be late to a meeting. I call the organizer and tell them when I will be there. I'm not going to finish my story card by the end of sprint, I contact my tech-lead and product owner and inform them of when I will be finished. A deploy is not going to get delivered on time. I contact my Delivery Lead and Product Owner and inform them of when it will happen. 

It is a terrible feeling to not get something done when you've made a commitment. That said, it is inevitable that you will fail at some point in your career. So when that happens, you have to buck-up and let the person you committed to know that you are aware of the issue and then recommit to them when you will complete the work. 

25 August 2017

Taking a Break

It is important to remember to take a break every once and a while. Give your mind a chance to recover and it will do amazing things. One thing I know about myself is that I'm very persistent; once I dive into a problem I have a hard time giving up or stepping away. 

The other thing I know is that I can get somewhat scope locked. I get committed to a possible solution and then I don't let go. That's often times a bad thing. So what I've learned to do is, take a break every once in a while. 

If I feel stuck I will go to the restroom and wash my hands. I imitate TV surgeons, washing up my forearms and scrubbing my fingers etc. If I really concentrate on what I'm doing, I push the active thinking about the problem out of my mind. When I finish washing and try to resume my thinking about the problem at hand I often have a revelation of some sort that unsticks me.

I don't think there is magic in washing your hands (though its good practice) however, the momentary distraction from the current thinking unblocks your mind by creating space between your conscious thought and the cage you have subconsciously places around the problem space. 

23 August 2017

Asking Why?

I've been thinking lately about how frequently (or infrequently) we ask why? That is, I think maybe we don't do it often enough. 

Today in standup there was a question about adding some markers to cards to indicate something or other. I finally remembered to ask "Why?". The answer was clear enough, so we'd have some big-visible reminder of something, but only the requester had a feeling that that would be useful and he wasn't really committed to it.

If we had stopped and discussed the merits of the proposed change, then gone about making that change, and spent time over the next week or two reminding ourselves of this process change it would have been a bit wasteful. Wasteful because nobody really saw the benefit of doing it. One person even suggested it was redundant because the same answer was inferred by other data that we already track. 

Asking "Why?" can save you some unnecessary work.

21 August 2017

Only the Living

So in consideration of source control, you may have heard, only the live/active code should be present in your repository. There are tons of discussions in the coderverse about comments, commented code, config files, certificates, and other things that should never be put into source control. An there are things in the codeverse that exist as part of your deployment but also aren't in source control (cache files, log files, etc.)

What do you do about the dead stuff. Dead code should not even exist in your project to start with, but lets say it does for some reason. You should ruthlessly track down all that dead code and delete it with extreme prejudice.

Here's why. All that dead code leads to confusion later down the line. For example, you have a module of code with dead functionality in it. You go on vacation to Aruba. While you are gone the team discovers an issue and needs to fix and redeploy that module of code. Their change breaks some tests for the dead code. Now what? Do we fix the dead code? Is it supposed to be dead? How come these things are touching? There should be no touching!

Thats a simple example about code but what about dead Jenkins jobs, dead SALT states, dead scripts. But wait, theres more! What about those little dribbles of scripts and log files we leave behind on DEV and QA systems? Say, stuff we wrote to help us diagnose an issue. Or copies of things we made to avoid calamity. That's dead too. DELETE IT!

These things (backups of log files, extra scripts, temp files, etc.) cause just as much confusion if not more than the dead code. This is especially true over time.  A tmp folder created today might look important to someone else next month when PROD is down. It is important to clean these things up immediately after they have served their purpose.

To summarize, if it isn't part of a prod deploy it doesn't really count. If it counts, it should be part of the prod deploy and therefore in source control. If it is dead, get rid of it rather than let it stink up the joint. Only the living code, configuration, files, should exist in your system.

18 August 2017

Fooling Yourself.

I'm working in a code base now where we frequently see lists of objects pushed into a creational method that often times passes delegates to a persistence method somewhere else. Typically this is some kind of convert JSON strings into database rows activity. 

Yesterday I came across some tests that claimed to validate the insertions. To my sorrow, the validation was bogus. I then spent several hours scanning code looking for this pattern and discovered that our codebase is riddled with this mistake. I then went about fixing these issues. Fortunately I didn't discover any new issues once I had the tests fixed. 


We have a SqlAlchemy data object named Foo. Foo has two attributes, id and name. Then there is the FooDBWriter that ingests JSON strings and makes instances of the data object. Once that object has been created it gets inserted into the database via the session object.

You can see the entire code suite here. Below I will just show and discuss the code that has problems and how to fix it.

This is a pretty typical pattern in our code base and elsewhere. This is a fine example of an establish micro-pattern in our development environment.

False Positive Tests

Here is a simplification of the issue that I discovered;

The class WorkingFalsePositiveTestOfDBWriter represents the erroneous testing approach. What you can see here is that the developer wanted to check that the session.insert was called once for each model object in the call to write_to_db. For a reason I cannot to fathom, the check is done with mock objects. 

If you aren't familiar with MagicMock, they are part of the Python testing framework and simply put they can be used as programmable mock substitute for any other object that provides reasonable defaults. In this case, __eq__ returns True despite the fact that the objects might not actually be equal to each other. 

So, the WorkingFalsePositiveTestOfDBWriter implementation uses a number of MagicMock instances equal to the number of model objects and checks for equality via the assert_has_calls method. (assert_has_calls is another mocking method that allows you to verify the occurrence of calls on the mock).

The reason this doesn't work is because the MagicMock is always going to return True. So it is possible that the objects being inserted have different values for their fields and the MagicMock will return True in all cases. So that isn't a very complete verification of the behavior.

Authors Intent

I'm guessing at this because the original author does not appear to be available any more. But I think the original approach was something like this;

So you'd hope that an approach like this works. However, our Foo class doesn't define an __eq__ method. I put the test run output in a comment below the assertion, you can see that it claims the results are not equal (despite appearances).

The issue is in how the two lists are compared. Each element of each list is compared with the other by identity, because these aren't the same instances __eq__ returns False. You could get around this by adding an __eq__ method to the data object, but you may have any number of reasons that you don't want to or cannot make that change. [Like an angry goblin that doesn't believe in collective code ownership, or an external library you have no control over]

Working Solution

The following is a more effective means of checking the call list without adding an __eq__ method to the data object. 

This approach uses a matcher class. The matcher class simply wraps the desired object and intercepts the call to __eq__. The matcher's __eq__ method then checks attribute by attribute for equality. (There are some other handy advantages to this approach that I'll address elsewhere). 


So the point here is, be careful with your assumptions when you are writing tests. Understand as much as possible how the comparisons will be made by the testing framework. Validate that your tests really work by tweaking expectations to ensure that the assertions you are making are real. Sometimes you end up fooling yourself; and the consequence is a lack of faith in your test suite.

16 August 2017

Fucking Up

Saw a great post yesterday about fucking up. That is, making mistakes. It simply stated that you should own it. 

It's hard to own it sometimes. Our egos get in the way. But, if you own it and move on with a minimum of drama, thats better than dancing around your errors and making excuses. It requires vastly less energy too.

Everyone fucks up, not everyone is willing to admit it. Have the courage to own your mistakes and you will go farther, faster.

14 August 2017

Pause and check for level

Today I set out on a mission to build a fire pit in my back yard. I used to have one of those stand-alone pits but it rusted out. So I got an oil drum and cut it down to 1/3 it's height and then dug a hole to bury it. Once I had my hole dug I poured 1 1/2 cubic feet of river rock into the hole and leveled it off. I then set the barrel down into the hole. 

It looks level, from my knees.

Before I poured rock inside the barrel I had to get the bags off the wheelbarrow behind me, so I stood up and grabbed a bag. Before I plunged my trusty pocket knife into the bag I looked down at the drum sticking out of the hole and noticed it wasn't level.

I had to go get the level.

I know better than to do this project without the right tools, but I had left the level in the garage. So I went and got it and check the drum. Sure enough it was considerably out of level North to South. So I pulled it back out and shifted the rock around. Checked level and it was spot on.

Then I checked East to West.

East to West I was still off, so out comes the drum, more shifting of rock, etc. etc. Eventually I got it close enough. After all, there will be field stone surrounding my fire-pit and you probably won't notice if I'm off a bit. 

Why am I telling this story.

So when we are working on a story card at work, sometimes we loose sight of the big picture. We think we are doing the right thing because we don't periodically check in on the larger objective. This manifests itself in all kinds of ways; long draw out quests for perfection on throw away features, over built functionality that handles impossible cases, and so on. And sometimes, we can't tell on our own, we have to talk to someone else (go get the level). 

It is important to pause occasionally and check to see that the work we are doing is on target, and when it isn't, correct course. It is also important to check more than one target, because we might be on track in one direction and off in another.

11 August 2017

Winchester's Law

When I was a kid my sister and I would watch M.A.S.H pretty much every day. I can say with some confidence that while I might not recall all the episodes I'm sure I've seen them all, and many of them twice. 

On the topic of focus though, I've often been in a situation where it has been necessary to express the need to do a job well. That is, not to hurry and get a job done so you can move on, but to do it as well as it can be done at the time. 

I call this Winchester's Law in reference to M.A.S.H. Season 6 Episode 1. Thats the one where Frank Burns leaves and Charles Emerson Winchester III becomes permanently attached to the 4077th. It's based on Charles Winchesters comment to Hawkeye as he is explaining his surgical technique. He says,

"I do one thing at a time, I do it very well, and then I move on."

This is, in essence, the road to greatness. Do just one thing as well as you possibly can, then move on. This is in contrast to doing 3 or 4 things reasonably well and maybe not getting it all done or of course, doing nothing. 

09 August 2017


This might seem obvious, but clearly it isn't. Rob Miller and I were pairing on a Friday afternoon and ran across a script in our SALT configuration. In the script Jinja was used to setup various variables before the script body. The Jinja code proceeded the shebang line and so the resulting script placed the shebang several lines into the file, with blank lines proceeding it. As a consequence, the shebang line was not executed. We dutifully fixed this in about six locations in our SALT config. 

We thought we should share our learning/remembering with the world. So we whipped up this simple example on a laptop. 

The following script (x.sh) changes the shell to zsh via the shebang and then prints the zsh version via echo. We then run the script and see that the version is 5.2. After that we create a new script (y.sh) with five blank lines followed by the content of x.sh. When we run that script we get no zsh version. That is because the shebang was ignored, its not the very first line of the file. 

Our point is, the magic number/shebang must be the first two bytes of the file or the shell will ignore it.

07 August 2017

More Thoughts on Time

Michael Gantz was kind enough to comment on my previous post about Early is On Time. He put forth the notion that we should be more honest about our estimates and what they mean. For example, "This project will be done in 3 weeks with a 75% confidence" or alternately "This project will be done in 3 to 5 weeks".

The Effort

I like the idea of making honest estimates and I like the idea of providing confidence levels or ranges of time. Years ago I read a book from Microsoft on project management that provided a estimation confidence guide. I don't recall all the details any more but it suggested something like, at the beginning of a project our estimates are +/- 65%, near the end we are within +/- 1%. I think thats interesting, but I think we put a lot of effort into managing that information when we don't need to. Rather we should be working together, business and technology, to find an optimal solution.

The Old Saw

Traditional agile practices (is that a thing?) tell us to use yesterdays weather to forecast the future. So if we have a know quantity of work and some data about how fast we typically burn through work, we should be able to provide a number in some reasonable range. Thats great. So now we can talk to the business about when we might deliver a quantity of work. 

Imagine a World

Disregarding all the unknowns that tend to throw a project off schedule, let pretend we can forecast work with some reasonable expectation of matching our estimate. 75% of the time we hit the mark on the head, when we are late it is typically by 5% of the work. We won't bother with early.

Early is On Time and On Time is Too Late, part 2

Software development is supposed to be a collaboration. If a development team tells you that they can deliver the product in 5 weeks with a 75% confidence, it is necessary to cooperate with that estimate. That is, throwing more bodies, mandatory overtime, or free pizza at a deadline almost never changes the delivery schedule. Rather, it leads to burn out, cut corners, and tech debt.

In my time I've seen entire departments halted to re-estimate work because the business didn't like the forecast. There is a lack of trust for you. I've also seen 'Saturday Bug Hunt Parties' with all the pizza and pop you can consume in order to bring down the backlog. A fine example of exploitation leading to burn out. 

We might suck at estimating while a team is still forming, but a team that is someplace in the norming arc has a pretty good idea of what they can do. If what they can do doesn't satisfy the business objective there isn't much in the way of effective tactics to solve that problem. 

The Brutal Truth

Facing a missed objective there are really only three avenues left. Eliminate all obstacles to between the team and the goal, change the goal, or replace the team with more reliable/accurate resources. Most any other suggestion is a variation on these three. 

Never mind that we should eliminate obstacles in the first place, this is probably our best hope for success. Given enough latitude a disciplined team can burn down a ton of work if we take everything else out of their way. But, if that isn't going to be enough, cutting scope is your next best option. Take away all non-essential functionality, all the bells and whistles, and get to market. You can always deploy again the next week (or day even). 

Nobody likes to talk about replacing the team, but sometimes that is the answer. Though often times it doesn't help as much as we'd like. I'll write about that situation some other day in more detail, but keep it in the back of your mind, sometimes organization overreach or bad politics or other human issues force you into this situation. 

Back on Point

In the end, business and technology are much better off when they work together to find shared belief about what they can do. That requires getting the actual doers involved in the estimation process. It also requires the people asking for the features to be reasonable and accommodating. It should be a collaboration though, and if it isn't, Michael' suggestion isn't going to help, nor would the most precise estimate in the world. Estimates, after all, are just educated guesses and prone to error.

04 August 2017

Configuration Testing

An often overlooked component of systems development is configuration testing. Time after time systems are built with no tests related to their configuration that later lead to confusion.

Configuration testing is really quite simple. Following the same general guideline of nothing goes to production that wasn't the result of a test; no configuration should exist without tests.

Simply put, if you have a configuration key-value-pair, there should be tests for what the system does when the key-value-pair is present, not present, and variations on the value.

For example, you might have a configuration key for output directory. You should have tests for that key being present and set to a valid value, absent, and some collection of invalid values. Invalid values being nil, non-existent directory, directory to which the application does not have privileges to, etc. 

These kinds of tests act as executable documentation for what a system expects and how it behaves with respect to its configuration and really do act as a means of communicating to yourself and others what the configuration keys do. 

Of course, we should not forget about default values for configuration keys. Ideally all configuration keys have some reasonable default value; when they don't our programs should halt gracefully with a clean message indicating the issue.

* Apologies to Michael Gantz for having not yet responded to his comments on being on time. I hope to formulate a coherent statement soon.

02 August 2017

Early is On Time

and On Time is Too Late.

I first heard that in Sonny Barger's book 'Freedom: Creeds from the Road' and it took me a while to really understand it.  

To be clear, it took a while for me to understand it as it relates to me. Sonny was pretty clear about what it meant to him.

To me its about respect. I don't like to be late because I think its disrespectful. This extends beyond being late to a meeting or a lunch date, it goes to delivery too. If you commit to a delivery, being late shows disrespect.