Configuration Testing

An often overlooked component of systems development is configuration testing. Time after time systems are built with no tests related to their configuration that later lead to confusion.

Configuration testing is really quite simple. Following the same general guideline of nothing goes to production that wasn't the result of a test; no configuration should exist without tests.

Simply put, if you have a configuration key-value-pair, there should be tests for what the system does when the key-value-pair is present, not present, and variations on the value.

For example, you might have a configuration key for output directory. You should have tests for that key being present and set to a valid value, absent, and some collection of invalid values. Invalid values being nil, non-existent directory, directory to which the application does not have privileges to, etc. 

These kinds of tests act as executable documentation for what a system expects and how it behaves with respect to its configuration and really do act as a means of communicating to yourself and others what the configuration keys do. 

Of course, we should not forget about default values for configuration keys. Ideally all configuration keys have some reasonable default value; when they don't our programs should halt gracefully with a clean message indicating the issue.

* Apologies to Michael Gantz for having not yet responded to his comments on being on time. I hope to formulate a coherent statement soon.