31 May 2017

Execution Time of Test Suites

Your tests should be fast. Really fast. Like all your unit tests should run in 10 seconds fast. 10 seconds is actually an eternity. That is almost long enough for me to loose focus and walk away. But if there are a lot of tests I can see how 10 seconds is reasonable.

Crazy talk? I think not. There are some caveats of course, but in general, if your system is up configured, the actual execution of your unit test suite should be in the vicinity of 10 seconds. For that mater, your Integration and Acceptance tests should be really fast too. 

In the end, everything should run in 10 minutes. 10 minutes is really too long, more than enough time to get coffee and become distracted by the PingPong table. But 10 minutes will do.

If you can't get your tests running this fast you need to seriously examine what you are doing. Some things you might investigate are;

Are you unit tests really unit tests? Or have they integrated with something?
Is your setup overly complicated? 
Do your integration tests or acceptance tests keep rebuilding the system?
Have you maximized your ability to run test suites in parallel?
Have you failed to modularize your application?

I think there will be several more posts on this topic. But let me lay out this maxim for you; No matter how fast your tests are, you should always be trying to make them faster. 

29 May 2017

Everyone should build a webserver

A great way to get good at writing server side code is to write server side code. I suggest everyone try to create a web-server. You can start by just trying to serve some documents with a simple application that leverages tools like Sinatra or Flask. However, if you really want to learn something, start from the Socket layer and work your way up. It is a lot more difficult than it sounds, but its not so difficult as to be impossible.

What you will gain from an experience like this is a very deep understanding of how most servers work (more or less). The more sophisticated you make your web-server the more your are going to learn. First you have to deal with the basics, like the HTTP protocol, locating files, sending files back to users, etc. Then you need to handle more complicated things like non-ascii content, multiple simultaneous requests, following symlinks, etc. The list of things you would need to compete with a tool like Apache or NGINX is very long. But if you can develop insight in developing those features it will go a very long way to growing your overall skill set as a developer.

If you don't feel like finding your own path on this topic I suggest you read Concurrent Programming in Java the examples result in a simple web-server. 

26 May 2017

Deleting Tests, Part 2

More commentary than anything else today. On the topic of bloated test suites and when is it safe to delete some of them. There are a lot of trite and overly clever answers to this on the internet. I'm going to try to avoid both of those kinds of answers and be practical. 

Pretty much you should never be deleting tests that are still valid. That is, if the system functionality was intended to be something, you should never remove the things that ensure that it is that way. Sorry, not enough pronouns. If your test suite is complete and confirms that all features are present and functioning correctly, then those test should remain in perpetuity.

That might leave you with a nasty problem though. What if you test suite is huge and slow and clumsy and awful. Deleting some of the tests, especially the low level stuff, or the things that just seem redundant, might sound like a great idea. I caution you not to. Rather, fix the clumsy stuff. Remove, carefully, the redundancy, but don't directly delete tests. 

I'm trying to make a distinction here. Deleting tests because they are annoying is not the right call. Deleting tests because they are redundant is an iffy proposition depending on how things are organized. Deleting tests should really only happen because the feature or functionality they are tied to is going away. Now all that said, deleting tests because you replaced them with something better, thats OK. Its sometimes hard to do, but its OK if you feel confident that your replacement test is better than the previously existing one. 

In an upcoming post I'll talk more about redundancy and how we need to be careful about that. 

24 May 2017

Deleting Tests

So after my recent post on Refactoring and TDD I had a request to talk a little about keeping your test suite healthy by deleting things. I love to delete things almost as much as I love to create them so I'm up for this discussion in a big way.
Thanks Tim for the request. After starting on this I discovered it was going to be much longer than planned, so deleting tests will have to span a few posts, I hope thats OK. I'll start with some simple stuff around your Unit Tests.

So in the process of keeping your test suite healthy there comes a time when you need to remove some dead-weight. There are a lot of theories and practices out there discussing this topic. I'm going to share my view point. 

First, don't delete unit tests. Properly written unit test should be blazingly fast, thousands of them should run in a few seconds +/- environment startup. So as long as they are still attached to live code, keep them around, they aren't hurting anyone. That said, if you delete code there should be some corresponding failures in your test suite that should cause you to remove some tests, or parts of some tests. Don't leave the test code behind, delete it along with the source it was attached to; ideally in the same commit so that the changes go together in and out of your source control system. 

That sounds easy, but in practice it can be a bit of a challenge. When production source goes away, it should be going away because some test is telling it to go away. That is, a test that once demanded an action occur, no longer demands that action. 

For example, 

This silly example does three calculations, but lets pretend that in the future we don't care about mass, so we need to remove from the result set, if we're really test driving, what does our code look like?

We could test the resultant structure to ensure that mass no longer appears in the output. First, lets add a test to show the structure of the result of calculate_all;

This test would pass of course, but if we removed the :mass key from the assertion it would start failing. In order to make it pass we would then remove :mass from the result of calculate_all and that would cause the validation of the mass calculation to fail, causing us to remove that test as well.

Our final result might look like this,

The point is, we got rid of some code and we did it in a test driven way. Why would we expend all this extra effort? To ensure that we removed only what we intended to remove and nothing more. 

OK you say, that's easy, but what about more intricate constructs, what about Integration Level Tests, or Acceptance Tests? I submit to you that the process is the same, only there are many more moving parts. 


PS -- I'd like to point out a few things about my super-ultra contrived example. One, I don't generally advocate for static/class methods, but it made this example a little cleaner. Two, I wouldn't necessarily advocate for the implementation of the calculate_all function, it does too many things, but again, for clarity I mashed it all together. Lastly, and probably most interesting to me is, what is the validity of the test_calculate_all_return_structure? Should that test have always been there? Maybe. Probably. I think, in retrospect, if I had taken more time to develop this example I would have put it there in the first place and not introduced it later. That said, the test implies something about the structure of the result that may or may not be important. All of this has me thinking about other topics, like should tests imply something about the structure of a result, especially in dynamic languages like Ruby. I guess we can talk about that soon.

22 May 2017

TDD, Refactoring Tests

So we are all familiar with the cycle Red-Green-Refactor as a critical behavior when applying TDD in practice. Over the years that I've been practicing TDD I've developed a habit of delaying my refactors. So I often do something more like 5xRed-Green, then refactor. This of course depends upon what I'm building and how well I have envisioned my end-state. 

Previously I'd mentioned a two step technique for practicing TDD in which your goal is to not have a vision of the results, but in practice, I often do have some sense of what the resultant code might look like; at least from a public interface perspective. With that in mind, I often find my self tempted to refactor code immediately to make it conform with my expectations, and I resist this temptation. 

I'm more likely to write four or five test cycles (test + code) and then refactor what little I might have. I also try to keep those refactors small. I might tweak a name or dry out a bit of setup, but I don't go whole hog crazy time and change the API without careful consideration. I might let a situation with too many arguments fester for a few iterations.

I do this because experience has shown me that if I start doing the more serious refactors too soon I will often undo those refactors later; either directly or indirectly. Typically, when I find myself backtracking a refactor, especially a large refactor, its because I don't know what I'm doing. That is, I have not thought the situation through completely. Other times it is because I've misremembered some aspect of the system and realize that my change is going to cause a design problem. 

So, while I fully support refactoring during the test cycle, I recommend some prudence with respect to when you start the refactoring step and how far you go with it. In most cases you can always apply the big refactor steps later, but once they are done, you've potentially doubled the labor if you were wrong. So don't speculate to wildly about the refactors, keep them small and simple. 

* On a related note, check out Daedtech's great post on Refactoring as a Development Technique from earlier today. 

19 May 2017

Naming Tests

I'm sure you have heard the old joke about the 2 hardest things in Computer Science. 

0. Naming Things
1. Cache Invalidation
2. Off by One Errors

That said, Naming Things is really hard. Sometimes it feels impossible to find a single word or short phrase that expresses what we intend. That said, putting the effort forth to name things well is very important to the readability of your code. This is even more important in your test suite. 

Lately we've been using Python and the unittest module therein. Therefore all my test methods start with the word test_. But, how do  we name things beyond that? Here is how I start.

First, name the class well. There are two choices here, first, to name the class after the test subject, VolumeCalculator and VolumeCalculatorTest. This is pretty standard practice and it is helpful for finding a tests. For example, if I'm looking at VolumeCalculator.calculate_volume(container) and I want to find it's tests, I can look for VolumeCalculatorTest.py and they are probably in there. Alternately, I could name my tests after the behavior that I'm after, for example CargoCapacityTest.py. I can use both of these techniques simultaneously as well, if for example, CargoCapacity uses VolumeCalculator and WeightLimitLookup, however, that might suggest some loose encapsulation of the overall Cargo Capacity activity (possibly those two are used in higher level method?). The key to any of this though is to be expressive and consistent without being too verbose.

Second, test method names. We've established that in unittest we'll start with test_ because we have to. But what comes after that? I suggest that you create a pattern and stick to it. One pattern might be test_{method}_{behavior}_{ext}. Where {ext} might be things like _fails, _succeeds, _yeilds_{value}. Again, the game here is to express the intent of the test. Some things you will observe are, if the test name is ridiculously long, the behavior might be too complicated, and if you can't figure out what the test should be named, you might not know what your testing -- you need to ask more questions to get this right. 

Third is organization. This isn't so much about naming as it is about namespaces. A lot of test suites can be very flat. However, if we organize our tests in namespaces like we organize our production code we can imply a scope for our tests that makes organizing them more logical to the reader. It makes the tests easy to find, and suggests to the reader something about the independent nature of these components in the suite. 

Of course all these suggestions are for tools like unittest, they would be different using a tool such as Cucumber, Behave, Jasmine, or RSpec. More on that later.

17 May 2017

DRY and the Test Suite

So I mentioned that there are some rules we should follow when developing a test suite. One of those rules that we normally apply to code is to make it DRY (Don't Repeat Yourself). I think test suites, particularly Microtests need to observe a special form of this rule.

Typically when we make code DRY we work very hard to reduce repetition. I've seen developers go to extraordinary lengths to ensure that only the boilerplate gets duplicated and that no module of code exists twice. I've done this myself and it generally pays off well. Though I must admit, sometimes the code becomes somewhat contrived. 

I don't think this is entirely appropriate within a test suite. When I think of a Microtest I think of code that expresses essentially three things. The relevant setup, the particular execution, and a single assertion. Nothing more and nothing less. Its that nothing less that causes us to repeat ourselves. 

Consider the following pair of tests;

That code is not at all dry, but it tells a complete story of what the foo function should do. We could take this code to an extreme when we dry it out though. That might look like this;

So what is wrong here? Well, it is no longer obvious which inputs go with with outputs. I have to read both the setup code and the test code to figure out how each test works. In my contrived example here this is pretty easy, there are only two tests. But when there are dozens of tests this become much more difficult. 

Here is a reasonable compromise;

This code reveals our intent with a minimum of duplication and clearly expresses our the intent of our code. There is just enough duplication that each test will expose its intent without you having to read more code and drys out just enough code that it will be easy to maintain. 

Even if you don't like this particular example (I know, its kinda horrid), finding the balance between expressive and crafted is important when developing a suite of tests in order to ensure that the integrity of each test is maintained without making a nightmare out of your tests by duplicating everything for each test. 

15 May 2017

Care and Feeding of a Test Suite

Consider that your test suite is the actual asset in software development. That is, the tests are more valuable than the code; but the code itself is still essential. 

The test suite is an expression of the desired state of the software. It tells us what the system is supposed to do, but not necessarily how. This is, of course, true at many levels. Microtests tell us specifically what a function does and should encompass all of the behavior of that function. Larger tests should tell us how a collection of functions, or an object should behave. Composed tests should tell us how a collection of objects interact, integration tests how those objects interact with other resources. How the internals work below the level of the test aren't important in terms of the asset. 

Granted, I tend to think of systems of software as a collection of both the deployable software and the test suite that comes with it. So I'm not trying to suggest that the deployed code doesn't matter or shouldn't be well crafted or anything like that. But I am trying to suggest that the test suite is more vital than the code itself in terms of its value to the organization.

The tests are the living breathing embodiment of what is supposed to be true. Without them, the rest of it, no matter how elegant and well polished is a fixed state with no explanation. The tests tell us what is really important. 

As a consequence of the tests importance I think that should guide us in how we develop and maintain test. That is, we should treat the code within the tests with at least as much respect as the code under test. I think there are a few different rules that should be applied to the tests, but we should still attempt to apply a level of craft to the code. By that I mean observing all the same rules we would with our production code and then some. 

I hope to write more about those rules over the next few days.

12 May 2017

I Got Nuthin'

In the sprit of 3 posts a week, this is my post. I got nuthin'. After a long week of tangling with some issues I've managed a big zilch on the blog topics. So I'll pull something together for next week and try to restart my streak of posts. 

Have a happy Friday.

10 May 2017


So I ran my backlog of articles dry and I am struggling to find a topic I want to write about. As a consequence of not being able to pick a topic I found this topic, Failure. I'm sure because I failed to deliver my article by 8am today. 

So, failure is a regular part of our lives. We can't always succeed on everything we do. What do we do with failure? We should learn from it. Failing and not learning is bad, failing and learning is good.

08 May 2017

Clarity > Certainty

Over the weekend a wrote a long and detailed pitch for why Clarity is more important than Certainty in planning (and in general). I decided late last night that that was too straight forward; specifically it provided too much certainty.

So I deleted it and started again.

The details of any plan are more or less a guideline for how to get from point A to point B. We all know and have experienced having to change the plan once we confront reality. There are plenty of historical quotes about this and much has been written on the topic. I summarize all of this by placing Clarity over Certainty. 

If we know where we are heading and staying focused on that objective, more or less the details will work themselves out. The trick is to make absolutely clear what the final state objective is, account for all the conditions, and then move forward.

Clarity is more important than Certainty.

05 May 2017

Consulting, Incentivization, and Your Job

My friend Adam likes to remind me on a regular basis that how you are incentivized drives your behavior. There must be a hundred quips in the world about incentiveization, and a lot of thoughts on motivation in general.  Also don't forget the internally motivated v. externally motivated angle. 

I'm very self driven. I can force my self to do almost anything and do it until it is done, even if it isn't much fun. Income taxes come to mind; I hate them and I always procrastinate, and yet, when the time comes, I always get them done. One thing I've noticed in the professional world is that often times our jobs and our immediate tasks aren't well aligned with our incentives. In some cases the alignment is only very general, do your job and get paid. For some very loose definition of 'your job'. I've seen plenty of 'just show up' and get paid behavior too, but thats a different thing. 

So for consultants there is a very interesting conundrum. I think it might actually be how consulting got a bad name in some parts of the world. Very often the incentives of a consultant are misaligned with the needs of a client. In a strict sense, an employee of an organization is motivated to behave as their incentives indicate; so if you regional manager promise a bonus for the most blog articles published, you tend to spend your time publishing blog articles and not paying attention to your customers. 

I once worked for a consulting organization where my boss would layout quarterly goals for each of us. They were often things like number of articles published, customers visited, answers provided in on-line forums, example/demo applications produced, and occasionally, 7-digit deals closed. Each quarter you could earn as much as $10,000 in bonuses (or possibly more) by making your goals. I took full advantage of this situation by working extra hours to make sure I did all the things on the list if possible. I couldn't control 7-digit deals, but I could write articles, spend hours coming the user-groups and answering questions, and crank out demo apps with the best of them. Needless to say I did well on the bonus receipts. 

What were the consequences of my behavior? Well for one, I frequently dropped the ball, or did less than 110% for our customers. I frequently negotiated with the sales team to avoid going on client calls in order to get my bonus work done. Once, I got a regional VP to have my boss grant my bonus without doing the work, because he wanted to serve the customer more than he didn't want to pay the bonus. 

What I'm getting at is that my behavior was strongly effected by how I was being incentivized. From talking to others I get that their behavior is too. At least I'm not alone. What are the consequences then of poor incentivization in an organization of consultants. What happens when you ask your consultants to also take on matters like internal personnel management or extracurricular teaching, speaking, demoing, and sales work. Typically the customer gets the short end of the stick. I think this problem compounds itself when paired with the 40 hour work week.

For thirteen of the past seventeen years I've traveled for work. In most cases at least four days a week. I had plenty of time to work on extras at night and plenty of time to get things done when the client didn't need me, so I hardly noticed any pressures on my personal life. I also got paid for every hour of it, so the incentives were built right in. Adding in Netflix and DVRs and I really had no excuses to not work 60 hour weeks. 

Enough about me, what about the other people in the world who aren't work-a-holics and who don't want a lifestyle of continuous labor followed by bing watching Mad Men on the weekends? Well, if you are a consultant/contractor you have made a significant commitment to your client and you need to honor that commitment. If you cannot, you need to let the client know that you can't and you need to be very comfortable with there response. That response is likely to be something you don't want by the way, something like, 'We're giving notice on your contract.' So be prepared. 

There is a way out though. You can choose to focus on your client and your commitments to them and push back on the incentives. Point out to the person devising the incentives that they have asked you to ignore the customer. Make sure they are comfortable with that, and make sure they understand the possible consequences. If they are happy, and you can be happy, I guess the customer doesn't matter much. However, if you are like me, you'll push back on your boss hard, and tell them that you aren't serving the customer by doing whatever silly thing they asked for. This of course might put you in a position of having to quit or find a new boss at least, but you (or I'd) be happier serving the customer.

03 May 2017

The Power of the Index

So last week I spent an excess of time thinking through a database modification. In our current system we have a semi-normal database that works but has a bunch of cruft in it. Some poorly named columns, a few incorrect data types and some significant domain abstraction leaks. So, we sat down and created a new domain model. The new model is all nice and clean and shiny. 

Two features we wanted to add to our database structure were versioning and source record tie-backs. That makes each insert pretty complicated. The following diagram is a simplified (and obscured) version of what I've been working with. 

Essentially we have source data tables with a few hundred thousand records and we need to insert them into our highly normalized system. In the real system there are actually multiple source tables representing different sources and types of person, but for this example I'm just including the bare essentials. 

So after playing around with various fancy tricks using CTE inserts I gave up and went to a simple in-line function. We're using Postgres for this experiment. The first test run used 378,755 records from two sources and had a total run time of 6 hours 32 minutes and 14.917 seconds. That is pretty pathetic. 

We put the queries into DataGrip and ran a few SQL Explains and after some tinkering we added six simple indexes to the system. We ended up with something like this;
The end result was amazing!

On the second run, using the same data set, we were able to get the total runtime down to 84.182 seconds. That is a world changing impact for so few indexes. 

ProTip: Before you abandon your experiment with the database, make sure you have appropriate indexes.

01 May 2017

Stoppage Solution Number One

Still on the topic of being stopped. I know one reason we get stopped. It is because we've exhausted all our ideas. We run out of ways to move forward. There is a solution to this. Communication. More often than not, when I'm stopped, if I try to communicate with someone else why I'm stopped, in the process of forming a coherent explanation of what has me stopped I find the 'next thing'. 

That is, the process of communicating, either verbally or in writing, forces me to think through a thorough explanation in such a way that I either see my mistake, see a different way forward, or invent more experiments I could try. I find this to be one of my most effective tools for solving 'stuck'. 

For me, I can't just explain things to my pair partner (if I have one). So when I'm stuck and pairing I still do this process, but I do it with my pair. We write to someone else on the team. I find that my pair partner and I often have the same mindset around a problem, and that helps us stay stuck because was can assume a lot of things together and those assumptions will stay alive in our conversations. 

The underlying key to all of this is to explain the problem completely and to have real clarity about what you are trying to accomplish. This is a lot harder than it might sound at first. So a second part of this is 'explain it and it will go away' approach is to pick your audience. When I try to explain my problem I try to keep in mind that the person I'm speaking with has little or no active knowledge of the problem, the details, or the things we've tried already. I do assume they know something about programming, computers, networks, etc. 

The third part of this is to be an effective communicator. After having written the description of the problem, clean it up, take out all the extra and unnecessary bits. Trim it down to the most important pieces. Create a TL;DR section if you have to. 

When all this is done I often find I have at a minimum gotten unstuck. When I have not, I have a great way to get a conversation started with someone else on what the problem is, and then we can have a conversation that might get me unstuck. 

Remember, don't be stuck.