I just read this very nice post from Martin Fowler on The Role of an Enterprise Architect in a Lean Enterprise. He has a lot of good thoughts in there.
I submit that there is one aspect lacking emphasis in this post however, the responsibility of the agile team to interact with the architect. That is, agile or not, architecture is still a thing in every organization. At some time, some person made a variety of choices about how systems should be structured, interconnected, and in some cases to be implemented.
Those decisions were not made lightly or in a vacuum and often the consequences of changing them have far reaching consequences that may not be apparent to those without knowledge of the entire enterprise system. Consider that an architects job among other things is to deliver value to the business over an extended period of time. Part soothsayer, part Oracle, an architect should be working to further enable business objectives through technology, not just in the near term, but for the foreseeable future of the organization.
Architects, like everyone else, are fallible human beings and they cannot foresee every circumstance. Any given project may run afoul of gaps in a generalized architecture or may struggle to integrate two technologies due to any number of compatibility factors. These are times when an agile/lean team should be reaching out to the architect for help. As Martin points out, part of the architects job is not to say 'No' but to make sure the team is fully aware of the consequences of deviations from the current technology stack.
Failure to consult with the architect when these decision points are reached is just plain negligence. Furthermore, failure of the architect to respond and cooperate with the team is a gross dereliction of duty. Like everything else in the agile/lean world, the relationship is a partnership with a shared goal and it is incumbent on all parties to participate in that partnership; to cooperate to find solutions.
15 November 2017
06 November 2017
Vague
I spent most of my weekend working on my NaNoWriMo book. If you aren't familiar with NaNoWriMo, the idea is to write 50,000+ words in a month, basically a novel. The degree to which you prepare for this event is entirely up to you. I've been talking about what I would write about since July, but I haven't really developed the details.
So, I'm trying to write about gang members in Mexico, but I don't have a town name, a general description of the town, etc. I haven't even named the gang. Its all pretty vague.
So the problem with that is, nothing in the descriptions is named. Or if it is, it's in general terms. The Pontiac. The Boulevard. The Restaurant. Obviously, those are details I'll need to go back and fill in at some point.
I've witnessed a parallel in project teams. Teams where an idea is discussed, changing the messaging infrastructure for example. The idea is discussed periodically in non-specific terms over an extended period of time, but there are never any details.
Two things tend to happen. Often, because there is nothing concrete, the work never happens; it remains a goal that is unachieved. The other is, the work is attempted, but it is a horrid mess of a change because there wasn't agreement about the 'how and the what'.
Don't get me wrong, I'm all for late binding on the 'how and what', but that ultimately has to happen. There should be a progression of work toward that 'how and what', maybe a series of spikes to prove or disprove things about the intended approach, but those need to be focused a little. Someone, someplace, has to provide a specific objective or set of criteria in order to focus the work.
There are a number of ways to get there. The Architect, as a technical product owner, can specify a set of behaviors required of the system. The Tech Lead could specify an approach, either in how the interaction should work, or even some specifics about the tool. The Director of Engineering might also have requirements about cost, platform selection, etc. Without these hints and nudges a team could drift aimlessly trying to make the change happen.
One last thing that is essential. The work has to get scheduled. It won't just happen in between other cards on the board. The product management team must make a change important and schedule it; set a goal date for completion, or otherwise do something to cause the work to happen. Even if a development team agrees upon the importance of a change, they are beholden to the planned work. If it isn't on the schedule, it just wont happen.
So, I'm trying to write about gang members in Mexico, but I don't have a town name, a general description of the town, etc. I haven't even named the gang. Its all pretty vague.
So the problem with that is, nothing in the descriptions is named. Or if it is, it's in general terms. The Pontiac. The Boulevard. The Restaurant. Obviously, those are details I'll need to go back and fill in at some point.
I've witnessed a parallel in project teams. Teams where an idea is discussed, changing the messaging infrastructure for example. The idea is discussed periodically in non-specific terms over an extended period of time, but there are never any details.
Two things tend to happen. Often, because there is nothing concrete, the work never happens; it remains a goal that is unachieved. The other is, the work is attempted, but it is a horrid mess of a change because there wasn't agreement about the 'how and the what'.
Don't get me wrong, I'm all for late binding on the 'how and what', but that ultimately has to happen. There should be a progression of work toward that 'how and what', maybe a series of spikes to prove or disprove things about the intended approach, but those need to be focused a little. Someone, someplace, has to provide a specific objective or set of criteria in order to focus the work.
There are a number of ways to get there. The Architect, as a technical product owner, can specify a set of behaviors required of the system. The Tech Lead could specify an approach, either in how the interaction should work, or even some specifics about the tool. The Director of Engineering might also have requirements about cost, platform selection, etc. Without these hints and nudges a team could drift aimlessly trying to make the change happen.
One last thing that is essential. The work has to get scheduled. It won't just happen in between other cards on the board. The product management team must make a change important and schedule it; set a goal date for completion, or otherwise do something to cause the work to happen. Even if a development team agrees upon the importance of a change, they are beholden to the planned work. If it isn't on the schedule, it just wont happen.
03 November 2017
Microcosm
I live in a microcosm.
So last week I went to SCNA and talked to a bunch of people there that I don't talk to enough. I've also been having lots of conversations about How it Should Be Done™
So many of the people I've been working with over the past 10 years or so have very similar back grounds, have read the same books and white papers, and worked in the same environments. There is a significant degree of group-think involved.
Recently though, I've been exposed to some new people who don't have those characteristics, or rather, come from a different group. I'm learning a lot about how other people think about problems and I'm learning a lot about how other people think.
I don't agree with some of the things I'm learning; which is to say, I've learned a few new techniques and ways of looking at problems and I don't agree that they are more effective or better than what I was already doing. I'm trying to be careful to avoid a false confirmation on these things of course.
What I find most useful about this experience is that I gives me a change to question how I've gone about solving problems for the last 25+ years. Some days it sucks, because you stand there questioning something you've been doing for 10 years and wondering if you've been 'wrong' for that long. Sometimes it's awesome because you get validation of your technique.
All of us live to some degree in a bubble created by those who surround us. All of us could be fooling ourselves every day in terms of How it Should Be Done™
I highly recommend that you get out, meet some new people (or read some new blogs) and consider that what you think you know as the right thing might be improved upon.
So last week I went to SCNA and talked to a bunch of people there that I don't talk to enough. I've also been having lots of conversations about How it Should Be Done™
So many of the people I've been working with over the past 10 years or so have very similar back grounds, have read the same books and white papers, and worked in the same environments. There is a significant degree of group-think involved.
Recently though, I've been exposed to some new people who don't have those characteristics, or rather, come from a different group. I'm learning a lot about how other people think about problems and I'm learning a lot about how other people think.
I don't agree with some of the things I'm learning; which is to say, I've learned a few new techniques and ways of looking at problems and I don't agree that they are more effective or better than what I was already doing. I'm trying to be careful to avoid a false confirmation on these things of course.
What I find most useful about this experience is that I gives me a change to question how I've gone about solving problems for the last 25+ years. Some days it sucks, because you stand there questioning something you've been doing for 10 years and wondering if you've been 'wrong' for that long. Sometimes it's awesome because you get validation of your technique.
All of us live to some degree in a bubble created by those who surround us. All of us could be fooling ourselves every day in terms of How it Should Be Done™
I highly recommend that you get out, meet some new people (or read some new blogs) and consider that what you think you know as the right thing might be improved upon.
01 November 2017
NaNoWriMo
Today is the first day of National Novel Writing Month. I'm going to be one very busy camper. I hope to be able to get a few posts out each week, but coverage may be spotty. Stick with me though.
Thanks,
.rich
Thanks,
.rich
23 October 2017
Functions as First Class Citizens
So I've been doing a lot of thinking and reading on the notion of functions as a first class citizens. In Python, functions are objects, just like instances of classes are objects. So when do you create a class and when do you create a function?
In my mind its somewhat a matter of style. That is, there is no mandate to use a function in place of a whole class or vise versa. That said, there are certainly times when a class is way too much work and a function will do just fine.
So a function doesn't have state, or at least it shouldn't. That means anything it knows about the world it gets as an explicit or implicit argument. Explicit arguments come in the form of parameters just like you would expect. Clearly implicit arguments should be constants or retrieved values and should never be global variables and the like.
Once you have written your function you can feel free to import it, call it, pass it around. It is pretty handy to not have to instantiate something in order to get some functionality. BUT, is this all necessary?
Again, I think we get back to a style question. I grew up on C/C++/Java with a little Perl thrown in. Discounting structured programming, in my mind everything should be an object and every object should have a class. That is clearly a habit born of Java.
I would not be critical of an application full of objects. I would not criticize a developer for never having a stand alone function. That kind of style question does not register in my lizard brain as a concern.
As I continue to play with the idea of functions as objects and functions standing alone I find myself questioning a lot of what I've learned to believe about good design. The biggest issue that occurs to me is, functions just laying about feels disorderly. I have to remind myself that the Python concept of 'module' as a means of organization has some significance. Think module == classy and it feels a tiny bit better.
Anyway, there is certainly some utility advantage to having free standing functions. For one, no tedious object instantiation. Another benefit in my mind is, you don't have to prefix a function with self. That is one of the least pleasant things in Python in my mind, all this self prefixing (same complaint about OO-Perl etc.). So if typing has somehow become the hinderence to your success, using standalone functions will save you some of that.
One other thing, in the past we've had this issue of 'where do I put utility functions'. Like convert_to_utc(datetime), a function that given a datetime in some timezone will convert it to the UTC timezone. That doesn't need a utility class, we don't need/want to do Utils().convert_to_utc(datetime) and so the function cleans that up, making it just convert_to_utc(datetime). That is one aspect of the function as a standalone/first-class object that I do appreciate. No more 'common' or 'util' classes cluttering things up.
I have some other thoughts I'll share in future posts but for now, chew on the idea that functions can be first-class citizens in your program.
In my mind its somewhat a matter of style. That is, there is no mandate to use a function in place of a whole class or vise versa. That said, there are certainly times when a class is way too much work and a function will do just fine.
So a function doesn't have state, or at least it shouldn't. That means anything it knows about the world it gets as an explicit or implicit argument. Explicit arguments come in the form of parameters just like you would expect. Clearly implicit arguments should be constants or retrieved values and should never be global variables and the like.
Once you have written your function you can feel free to import it, call it, pass it around. It is pretty handy to not have to instantiate something in order to get some functionality. BUT, is this all necessary?
Again, I think we get back to a style question. I grew up on C/C++/Java with a little Perl thrown in. Discounting structured programming, in my mind everything should be an object and every object should have a class. That is clearly a habit born of Java.
I would not be critical of an application full of objects. I would not criticize a developer for never having a stand alone function. That kind of style question does not register in my lizard brain as a concern.
As I continue to play with the idea of functions as objects and functions standing alone I find myself questioning a lot of what I've learned to believe about good design. The biggest issue that occurs to me is, functions just laying about feels disorderly. I have to remind myself that the Python concept of 'module' as a means of organization has some significance. Think module == classy and it feels a tiny bit better.
Anyway, there is certainly some utility advantage to having free standing functions. For one, no tedious object instantiation. Another benefit in my mind is, you don't have to prefix a function with self. That is one of the least pleasant things in Python in my mind, all this self prefixing (same complaint about OO-Perl etc.). So if typing has somehow become the hinderence to your success, using standalone functions will save you some of that.
One other thing, in the past we've had this issue of 'where do I put utility functions'. Like convert_to_utc(datetime), a function that given a datetime in some timezone will convert it to the UTC timezone. That doesn't need a utility class, we don't need/want to do Utils().convert_to_utc(datetime) and so the function cleans that up, making it just convert_to_utc(datetime). That is one aspect of the function as a standalone/first-class object that I do appreciate. No more 'common' or 'util' classes cluttering things up.
I have some other thoughts I'll share in future posts but for now, chew on the idea that functions can be first-class citizens in your program.
20 October 2017
Card Numbers
Ever notice that teams who number cards stop talking about cards and start talking about numbers?
Repeatedly I see this happen. A card is placed in a system like JIRA where it is assigned a number. Lets say AB-123. The card has a concise and meaningful title, a complete description, and all the other goodness that a card should have. Prior to being entered into the card tracking system the card was referred to as 'Spell Check Project Name Field'.
Notice, once it is placed in JIRA it is forever after referred to as AB-123.
The problem that then occurs is that the team stops talking about it as 'Spell Check Project Name Field', it becomes an abstract concept AB-123. As an abstract concept the team stops understanding and communicating it's meaning when they discuss it. As a consequence two things happen; the impact of the story isn't something that can be internalized or reasoned about and a majority of the time the team can't remember what is on AB-123, so when it comes up, nobody really understands what is being discussed.
So here is some advice, stop numbering cards and or don't allow people to talk about cards in terms of the number. Rather, enforce a behavior of discussing cards in terms of what they actually do/mean.
Repeatedly I see this happen. A card is placed in a system like JIRA where it is assigned a number. Lets say AB-123. The card has a concise and meaningful title, a complete description, and all the other goodness that a card should have. Prior to being entered into the card tracking system the card was referred to as 'Spell Check Project Name Field'.
Notice, once it is placed in JIRA it is forever after referred to as AB-123.
The problem that then occurs is that the team stops talking about it as 'Spell Check Project Name Field', it becomes an abstract concept AB-123. As an abstract concept the team stops understanding and communicating it's meaning when they discuss it. As a consequence two things happen; the impact of the story isn't something that can be internalized or reasoned about and a majority of the time the team can't remember what is on AB-123, so when it comes up, nobody really understands what is being discussed.
So here is some advice, stop numbering cards and or don't allow people to talk about cards in terms of the number. Rather, enforce a behavior of discussing cards in terms of what they actually do/mean.
18 October 2017
Preparation
You've probably heard of the 7Ps, Proper Prior Preparation Prevents Piss Poor Performance. What does that mean in the context of an agile team?
I often use the imperfect analogy of an automobile to describe an agile development team. Simply, the tires are the production system, the drive train is dev-ops, and the engine is the development team. The product owner/architect/tech-lead are the fuel delivery system to that team and stakeholders/business people are the gas-tank. When they all work together the car runs soothly down the track.
Here's the thing. If the product owner and architect aren't filtering the fuel before feeding it to the team, they aren't doing all of their job. It isn't just about pumping fuel from the tank to the developers, its also about straining out the gunk that might be in the fuel as well as distributing that work somewhat evenly across the team members. That is, making sure the fuel is clean and that every cylinder gets some of it.
I'm going to just use Product Owner as a proxy for anyone providing requirements directly to the team (Architects, Tech Leads, Engineering Leads, etc. all count as Product Owners in this scenario).
When a product owner shows up with a story, it needs to be fully digested and understood by the product owner. It needs to include answers to questions like For Whom? Why? When? and What? One essential component is of understanding that needs to be present is, what.
What means, what does this story do or accomplish and in understanding that we need some of the How, but not in a technical sense. That is, we don't need the Product Owner to provide the algorithm or a design document, but rather, How do you do the math?
An example; Calculate the percentage of ripe fruit on sale for a given date. Sounds simple enough, look in the inventory for all the fruit available for sale who's status is ripe on a given date and the total count of fruit available and do some math. But in real life that might not be a simple query. For example, we may only have available a total amount of fruit, ever. How do we do the math now? Do we deduct the entire historical record of fruit sold from total count of fruit to get fruit available? Then how do we determine ripe fruit? Same way?
This space of understanding can be solved without the business getting involved. The developers are more than smart enough to figure out a way to get an answer, but it's quite possible it won't be the right one.
So somewhere in the preparation of the card the Product Owner should have discovered this and documented it. This isn't about stealing the thunder of the brilliant developer and it isn't about providing a how; this is about communicating what is expected. That is, what ratio is the business trying to show.
I often use the imperfect analogy of an automobile to describe an agile development team. Simply, the tires are the production system, the drive train is dev-ops, and the engine is the development team. The product owner/architect/tech-lead are the fuel delivery system to that team and stakeholders/business people are the gas-tank. When they all work together the car runs soothly down the track.
Here's the thing. If the product owner and architect aren't filtering the fuel before feeding it to the team, they aren't doing all of their job. It isn't just about pumping fuel from the tank to the developers, its also about straining out the gunk that might be in the fuel as well as distributing that work somewhat evenly across the team members. That is, making sure the fuel is clean and that every cylinder gets some of it.
I'm going to just use Product Owner as a proxy for anyone providing requirements directly to the team (Architects, Tech Leads, Engineering Leads, etc. all count as Product Owners in this scenario).
When a product owner shows up with a story, it needs to be fully digested and understood by the product owner. It needs to include answers to questions like For Whom? Why? When? and What? One essential component is of understanding that needs to be present is, what.
What means, what does this story do or accomplish and in understanding that we need some of the How, but not in a technical sense. That is, we don't need the Product Owner to provide the algorithm or a design document, but rather, How do you do the math?
An example; Calculate the percentage of ripe fruit on sale for a given date. Sounds simple enough, look in the inventory for all the fruit available for sale who's status is ripe on a given date and the total count of fruit available and do some math. But in real life that might not be a simple query. For example, we may only have available a total amount of fruit, ever. How do we do the math now? Do we deduct the entire historical record of fruit sold from total count of fruit to get fruit available? Then how do we determine ripe fruit? Same way?
This space of understanding can be solved without the business getting involved. The developers are more than smart enough to figure out a way to get an answer, but it's quite possible it won't be the right one.
So somewhere in the preparation of the card the Product Owner should have discovered this and documented it. This isn't about stealing the thunder of the brilliant developer and it isn't about providing a how; this is about communicating what is expected. That is, what ratio is the business trying to show.
17 October 2017
Delays
Hello again. I was sick over the weekend and failed to get a post out on Monday. That has taken me on a tangent to thinking about delays.
So one of my projects is now effectively delayed because I was unavailable. I'm sure we will recover, I just need to make up for that keyboard time and we'll be back on track. But this impacts our delivery date. I now need to work some more hours durning the week to make up the difference.
Similarly, on another project we're running into a problem of stakeholder availability. A few of the stories we are working on have come to a halt because we don't have a complete understanding of the business needs or the technical details. Each of those stories has been delayed. An interesting side effect of this is that we are stretching our WIP limit because we're picking up other work to fill the gap.
In the first case we have a pretty simple case of needing to just rearrange the only work schedule because of a lack of resources to do the work. It's a simple solve. Given more staff we could have just swapped in another developer to do that work. Since we don't have more staff, we just shift the schedule.
In the second case we have plenty of worker resources. The issue here is a lack of executable work, leaving worker resources idle. Whats interesting to me is that, as a result of having too many workers and blocked cards do to incomplete specs, we have pulled more work into the system to prevent idleness. As a consequence the load on the team, in terms of things to keep track of/manage is increased, but the productivity is unchanged as best and possible decreased if the cards remain blocked.
So this second case is whats most interesting to me. In an attempt to keep all resources utilized we add work to the In Progress list and mark other work blocked. In particular, because we're waiting on an external resource (Stakeholder) to answer a question. Back in the day we would have asked our product owner, who was sitting in the room, to help us get the answer and we would have actively pursued that answer until it was either found or determined to be unanswerable. In the later case we would discard all the work and move on to another card.
We seem to have adopted a 'wait for the answer' posture, one that is even passive. That to me shouldn't be acceptable. Rather, we should work to engage the stakeholder in a manner that encourages them to give up the goods on demand OR have given it up in advance.
In advance is a problem though. In my Shangri-La the product owner is in the room and ready to answer questions at all times, no advanced planning necessary; but thats not reality. In reality, where we actually live, the product owner/tech-lead team have looked at, though through, and triaged all the cards before the development team sees them. They have at least a minimum amount of information ready and available to answer the teams questions. [I'm not saying that info is necessarily on the cards, just that its readily available]
While this doesn't conform exactly to the 'A card is a promise for a conversation' ideal, if a card could be a proxy for pre-digested (by the PO/TL pair) answers to expected/common questions it will impart efficiency into the process of delivering the story.
I recognize that this is similar to the Definition of Ready that some teams use, but I see a difference in how it is applied. The DoR is typically applied as a gate keeper on the 'Ready' column for the team; if the check boxes aren't checked then the card can't be added. Rarely, if ever, do I see the Product Owner or Tech Lead using the DoR as a check list for preparing work; and when I have it's been an awful disaster.
Somewhere between the DoR and a card with two or three words on it is an place where the Product Owner and the Tech Lead understand enough of the details of a card that they can answer reasonable questions about it without having to wait for an external stakeholder and/or a series of spikes to be completed.
I'm way over my length goal here, so I'll save my thoughts on the realistic solution to this problem for another post.
So one of my projects is now effectively delayed because I was unavailable. I'm sure we will recover, I just need to make up for that keyboard time and we'll be back on track. But this impacts our delivery date. I now need to work some more hours durning the week to make up the difference.
Similarly, on another project we're running into a problem of stakeholder availability. A few of the stories we are working on have come to a halt because we don't have a complete understanding of the business needs or the technical details. Each of those stories has been delayed. An interesting side effect of this is that we are stretching our WIP limit because we're picking up other work to fill the gap.
In the first case we have a pretty simple case of needing to just rearrange the only work schedule because of a lack of resources to do the work. It's a simple solve. Given more staff we could have just swapped in another developer to do that work. Since we don't have more staff, we just shift the schedule.
In the second case we have plenty of worker resources. The issue here is a lack of executable work, leaving worker resources idle. Whats interesting to me is that, as a result of having too many workers and blocked cards do to incomplete specs, we have pulled more work into the system to prevent idleness. As a consequence the load on the team, in terms of things to keep track of/manage is increased, but the productivity is unchanged as best and possible decreased if the cards remain blocked.
So this second case is whats most interesting to me. In an attempt to keep all resources utilized we add work to the In Progress list and mark other work blocked. In particular, because we're waiting on an external resource (Stakeholder) to answer a question. Back in the day we would have asked our product owner, who was sitting in the room, to help us get the answer and we would have actively pursued that answer until it was either found or determined to be unanswerable. In the later case we would discard all the work and move on to another card.
We seem to have adopted a 'wait for the answer' posture, one that is even passive. That to me shouldn't be acceptable. Rather, we should work to engage the stakeholder in a manner that encourages them to give up the goods on demand OR have given it up in advance.
In advance is a problem though. In my Shangri-La the product owner is in the room and ready to answer questions at all times, no advanced planning necessary; but thats not reality. In reality, where we actually live, the product owner/tech-lead team have looked at, though through, and triaged all the cards before the development team sees them. They have at least a minimum amount of information ready and available to answer the teams questions. [I'm not saying that info is necessarily on the cards, just that its readily available]
While this doesn't conform exactly to the 'A card is a promise for a conversation' ideal, if a card could be a proxy for pre-digested (by the PO/TL pair) answers to expected/common questions it will impart efficiency into the process of delivering the story.
I recognize that this is similar to the Definition of Ready that some teams use, but I see a difference in how it is applied. The DoR is typically applied as a gate keeper on the 'Ready' column for the team; if the check boxes aren't checked then the card can't be added. Rarely, if ever, do I see the Product Owner or Tech Lead using the DoR as a check list for preparing work; and when I have it's been an awful disaster.
Somewhere between the DoR and a card with two or three words on it is an place where the Product Owner and the Tech Lead understand enough of the details of a card that they can answer reasonable questions about it without having to wait for an external stakeholder and/or a series of spikes to be completed.
I'm way over my length goal here, so I'll save my thoughts on the realistic solution to this problem for another post.
13 October 2017
Criticism -- You're language is no good
So I was reading an article the other day, suggested by someone I work with, and it had some very disparaging things to say about Java.
It got me thinking, why does Java have a reputation for being overly verbose and full of interfaces and such. C# seems to have a similar reputation.
But heres the thing, it's not the language that's doing all that. Its the frameworks, the books, and the people. The language doesn't care.
I have to agree that Java has some overhead involved just by its very nature, but it is mostly easy to use otherwise. Sure, it's statically typed and has some limitations (though gradually many of those are going away) but the language doesn't force you into creating a bunch of interfaces and having super complex structures to get things done.
So why do we keep beating up on the language? Why, instead, are we not critical of the people who write the code like that? And the designers who create frameworks that force us to write code like that.
I'll admit, I've written some insanely complicated Java in my time, but I learned that I wasn't getting any advantage from doing so and I quit doing it. Now, for the most part, my Java is pretty clean and simple. At least as much as I can make it be within the limitations of the language.
I have read a fair amount of open source code. Not as much as others, but certainly my fair share. I think some of the code out there is way over complicated. Sometimes its just down right ridiculous. And it doesn't matter if it's Java or Python or Ruby or anything else. So why does Java (or C#) get the bad rap?
I'm heading somewhere with this line of thinking, but I'm not sure where yet. I've been trying to get this much out of my head for 3 days. It's hard to do without sounding like a lunatic.
More later I hope.
It got me thinking, why does Java have a reputation for being overly verbose and full of interfaces and such. C# seems to have a similar reputation.
But heres the thing, it's not the language that's doing all that. Its the frameworks, the books, and the people. The language doesn't care.
I have to agree that Java has some overhead involved just by its very nature, but it is mostly easy to use otherwise. Sure, it's statically typed and has some limitations (though gradually many of those are going away) but the language doesn't force you into creating a bunch of interfaces and having super complex structures to get things done.
So why do we keep beating up on the language? Why, instead, are we not critical of the people who write the code like that? And the designers who create frameworks that force us to write code like that.
I'll admit, I've written some insanely complicated Java in my time, but I learned that I wasn't getting any advantage from doing so and I quit doing it. Now, for the most part, my Java is pretty clean and simple. At least as much as I can make it be within the limitations of the language.
I have read a fair amount of open source code. Not as much as others, but certainly my fair share. I think some of the code out there is way over complicated. Sometimes its just down right ridiculous. And it doesn't matter if it's Java or Python or Ruby or anything else. So why does Java (or C#) get the bad rap?
I'm heading somewhere with this line of thinking, but I'm not sure where yet. I've been trying to get this much out of my head for 3 days. It's hard to do without sounding like a lunatic.
More later I hope.
11 October 2017
Dogma
So getting back into the blogging rhythm is proving challenge. I'm going to try to do the 3x a week thing again, but I might be a bit uneven on the delivery while I get back into it. Please be patient, I'm just rearranging my self a bit too much right now, but I'll get there.
--------------
Apropos nothing and everything at the same time, Dogma is BAD.
Blind adherence to a method or practice leads to stagnation and complacence.
Right at the moment I'm working on learning and understanding some subtleties of using functions as first class objects and how that impacts. What I'm looking for is the balance point between too much and too little of this approach.
I'm doing this first because I want to learn something; something I hope will improve my skills as a developer. Secondly, in conversation about exactly this it was difficult to articulate how much of this should and should not be done.
After that discussion a few links to various sources were provided and I started digging into this idea. Coming from a C++/Java background originally I'd bought into the idea that everything is an object, if it isn't an object it shouldn't exist (with all the necessary caveats necessary to digest C++ and Java).
So, if I accept that that isn't necessarily true (reject dogma) then I need to embrace this new approach. First I need to really understand it and how it works, but then I need to really get into the subtle parts so I can speak coherently about it.
This is just one example of the whole dogma thing. You've heard me speak about 'creating cards' and 'you have to because the process says so' etc. elsewhere. The same stuff applies to things like OOD and idiomatic Python or 'Standard Java'.
All of this has inspired a bunch of other thoughts I hope to share soon.
--------------
Apropos nothing and everything at the same time, Dogma is BAD.
Blind adherence to a method or practice leads to stagnation and complacence.
Right at the moment I'm working on learning and understanding some subtleties of using functions as first class objects and how that impacts. What I'm looking for is the balance point between too much and too little of this approach.
I'm doing this first because I want to learn something; something I hope will improve my skills as a developer. Secondly, in conversation about exactly this it was difficult to articulate how much of this should and should not be done.
After that discussion a few links to various sources were provided and I started digging into this idea. Coming from a C++/Java background originally I'd bought into the idea that everything is an object, if it isn't an object it shouldn't exist (with all the necessary caveats necessary to digest C++ and Java).
So, if I accept that that isn't necessarily true (reject dogma) then I need to embrace this new approach. First I need to really understand it and how it works, but then I need to really get into the subtle parts so I can speak coherently about it.
This is just one example of the whole dogma thing. You've heard me speak about 'creating cards' and 'you have to because the process says so' etc. elsewhere. The same stuff applies to things like OOD and idiomatic Python or 'Standard Java'.
All of this has inspired a bunch of other thoughts I hope to share soon.
09 October 2017
Trust
Welcome back! I missed you.
I had to take a short break from blogging. I got a little overwhelmed with too many things and blogging fell to the wayside.
So I've been thinking about trust for a couple days now and its role in the team room and in collaborations in general. I'm not going to role out a bunch of quips about trust here, I want to talk about it's value though.
Without trust in a team there can be some serious consequences. A simple consequence is that when team members don't trust each other they can burn a lot of energy second guessing each others work. Related to that, individuals can burn a lot of energy speculating about what their team mates are doing.
Thats a lot of wasted energy.
So how do you develop trust in a team? I think there are a lot of ways, and I won't claim to know them all. My usual reply is 'drinking!'. But more seriously, it is important to develop a human connection between team members. That can be as easy as asking how someone is doing, what they did over the weekend, or if they need some help. Just being a human being is necessary, and will eventually lead to trust.
There is another critical component to developing trust. You have to listen to each other. When I say listen, I don't mean wait for the other person to stop talking, that's being polite, I mean actually hear and process what they have said.
We don't have to agree all the time, and we don't have to have groupthink, but if we aren't even allowing alternate opinions to be voiced, in addition to showing a lack of respect, we are fostering a lack of trust. Consider the view point of the speaker; when you ignore or shutdown their words you are telling them you don't care about what they have to say -- you are telling them you don't trust them. Now ask, why should they trust you?
I had to take a short break from blogging. I got a little overwhelmed with too many things and blogging fell to the wayside.
So I've been thinking about trust for a couple days now and its role in the team room and in collaborations in general. I'm not going to role out a bunch of quips about trust here, I want to talk about it's value though.
Without trust in a team there can be some serious consequences. A simple consequence is that when team members don't trust each other they can burn a lot of energy second guessing each others work. Related to that, individuals can burn a lot of energy speculating about what their team mates are doing.
Thats a lot of wasted energy.
So how do you develop trust in a team? I think there are a lot of ways, and I won't claim to know them all. My usual reply is 'drinking!'. But more seriously, it is important to develop a human connection between team members. That can be as easy as asking how someone is doing, what they did over the weekend, or if they need some help. Just being a human being is necessary, and will eventually lead to trust.
There is another critical component to developing trust. You have to listen to each other. When I say listen, I don't mean wait for the other person to stop talking, that's being polite, I mean actually hear and process what they have said.
We don't have to agree all the time, and we don't have to have groupthink, but if we aren't even allowing alternate opinions to be voiced, in addition to showing a lack of respect, we are fostering a lack of trust. Consider the view point of the speaker; when you ignore or shutdown their words you are telling them you don't care about what they have to say -- you are telling them you don't trust them. Now ask, why should they trust you?
18 September 2017
TDD is development
Apropos very little, in a recent conversation it was suggested that testing is waste from the perspective of the business. My immediate reaction was visceral opposition to that line of thinking.
After some consideration it occurs to me that the assertion that Testing is Waste must be wholly incorrect. Here's how I got there.
First, if I ask you to build me a Flux Capacitor, how do I know I have one unless I test it? That is, testing is the validation that tells me as a product owner that I have received what I asked for. No waste there.
Second, if I'm making a Flux Capacitor I must have some means of knowing that what I've built does what I want it to do. I'll have to test the device to see if it qualifies as a Flux Capacitor.
So, in order to build the Flux Capacitor, the engineer building it must conduct some tests to ensure that the thing built behaves like a Flux Capacitor. In order to receive a Flux Capacitor, as a product owner, I must conduct some tests to verify that the device I've been given is a Flux Capacitor.
Therefore, testing isn't a waste, it is a necessity.
After some consideration it occurs to me that the assertion that Testing is Waste must be wholly incorrect. Here's how I got there.
First, if I ask you to build me a Flux Capacitor, how do I know I have one unless I test it? That is, testing is the validation that tells me as a product owner that I have received what I asked for. No waste there.
Second, if I'm making a Flux Capacitor I must have some means of knowing that what I've built does what I want it to do. I'll have to test the device to see if it qualifies as a Flux Capacitor.
So, in order to build the Flux Capacitor, the engineer building it must conduct some tests to ensure that the thing built behaves like a Flux Capacitor. In order to receive a Flux Capacitor, as a product owner, I must conduct some tests to verify that the device I've been given is a Flux Capacitor.
Therefore, testing isn't a waste, it is a necessity.
15 September 2017
Idioms, Standards, and Conventions
Finally, a blog with more than a one word title.
Today we had some conversation about standards on our team. I think it was a good conversation. It got me thinking about idiomatic use of platforms.
Did you know (technically) idioms are not standards?
I'm all for idiomatic use of tools, platforms, etc. Doing it like others is a good thing. Convention over configuration, etc. etc.
That said, there are occasions where you need a standard. Just because your agile doesn't mean you can't have some kind of rigor.
It is important for every team to figure out what standards they need, and each team is bound to be a bit different. It's also important to follow the standards if you have them. Having them and not following them is a bit of a waste.
Lastly, like all other things, you should periodically question why you have a standard and if it is still appropriate and/or needs to be changed, updated, or removed. As we learn more about what we're doing our needs usually change and sometimes our standards need to change with them.
Today we had some conversation about standards on our team. I think it was a good conversation. It got me thinking about idiomatic use of platforms.
Did you know (technically) idioms are not standards?
I'm all for idiomatic use of tools, platforms, etc. Doing it like others is a good thing. Convention over configuration, etc. etc.
That said, there are occasions where you need a standard. Just because your agile doesn't mean you can't have some kind of rigor.
It is important for every team to figure out what standards they need, and each team is bound to be a bit different. It's also important to follow the standards if you have them. Having them and not following them is a bit of a waste.
Lastly, like all other things, you should periodically question why you have a standard and if it is still appropriate and/or needs to be changed, updated, or removed. As we learn more about what we're doing our needs usually change and sometimes our standards need to change with them.
13 September 2017
Myopia
I just wrote a rather lengthy post in trying to get to the following post. There was some useful catharsis in that process, but I think the salient part is this;
When we are developing software and we recognize a feature but do not fully understand it in the context of the entire system it is incumbent upon us to learn its meaning in the full context of the system before we discard it as unnecessary. Without this context and understanding the consequences of not implementing the feature are unknowable to us.
Similarly, when we plan and approve work we must keep in mind the larger picture. We cannot fail to implement a feature or approve work that fails to meet a specification out of expedience. Undelivered system features fight for primacy until they are delivered. In considering a feature set in its entirety it is relatively easy to determine which problems to tackle first, however, closer to the metal the issues can become indistinct. We must remember to back off and look at the larger picture before we discard something.
Some features require a large number of individual parts working in concert to achieve a target behavior. Failure to deliver one of those parts leads to a failure to deliver the feature itself.
If we develop a myopic view of our systems we will fail to recognize when we underserve macroscopic features of an overall system.
When we are developing software and we recognize a feature but do not fully understand it in the context of the entire system it is incumbent upon us to learn its meaning in the full context of the system before we discard it as unnecessary. Without this context and understanding the consequences of not implementing the feature are unknowable to us.
Similarly, when we plan and approve work we must keep in mind the larger picture. We cannot fail to implement a feature or approve work that fails to meet a specification out of expedience. Undelivered system features fight for primacy until they are delivered. In considering a feature set in its entirety it is relatively easy to determine which problems to tackle first, however, closer to the metal the issues can become indistinct. We must remember to back off and look at the larger picture before we discard something.
Some features require a large number of individual parts working in concert to achieve a target behavior. Failure to deliver one of those parts leads to a failure to deliver the feature itself.
If we develop a myopic view of our systems we will fail to recognize when we underserve macroscopic features of an overall system.
11 September 2017
Burnout
Stewing in the back of my mind are thoughts about burnout. I have been know to get burned out periodically. Usually a mental-health day is all it takes to get back on track. Sometimes I need two. Typically what I need to do is find some other problem set to work on. Just something to let me know there is another class of problem to work on, assure me that I'm not trapped in Dante's Inferno someplace.
An observation, some people accept burnout. That is, they get burned out and they just keep marching forward. I think this leads to a complacent attitude. It allows you to become dull witted and you loose effectiveness. It's important to guard against that and to find a way to get through it.
An observation, some people accept burnout. That is, they get burned out and they just keep marching forward. I think this leads to a complacent attitude. It allows you to become dull witted and you loose effectiveness. It's important to guard against that and to find a way to get through it.
06 September 2017
Zombies
I was supposed to post about test driving regex today. I failed. I've become very busy, and that is making it hard to get posts with code done. I'll try again for Friday.
While you are waiting, I have something to say about Zombies. I saw a tweet today about Scrum;
I'm not dissing Scrum per se, but what I am saying is that normalization for the sake of normalization is bad. Normalization should be in the eye of the beholder; a team should find a norm that works for them and use that to their advantage. Please don't get hooked on someone else's normal.
Every team should develop practices and procedures that work for _them_ and _not_ use any practices or procedures that _don't_ work for them.
Don't be a Scrum Zombie!
While you are waiting, I have something to say about Zombies. I saw a tweet today about Scrum;
Consider this, by normalizing the social DNA of your team you are creating Zombies. As we all know, zombies are bad. Zombies are scary. Zombies eat your brains!#Scrum normalizes a team's social DNA: builds better accountability, collab, & trust norms w/in team ^ @SocialAgilist pic.twitter.com/3EBOE6kHNO— AgileFortune (@AgileFortune) September 5, 2017
I'm not dissing Scrum per se, but what I am saying is that normalization for the sake of normalization is bad. Normalization should be in the eye of the beholder; a team should find a norm that works for them and use that to their advantage. Please don't get hooked on someone else's normal.
Every team should develop practices and procedures that work for _them_ and _not_ use any practices or procedures that _don't_ work for them.
Don't be a Scrum Zombie!
01 September 2017
So I Was Working On It
I was working on a post about using TDD for Regular Expressions. I've run out of time. I'll work on this over the weekend and hopefully have a good example of how you can get good coverage around a regex.
Considering the Labor Day Weekend, look for a post on Wednesday of next week on the topic of Test Driven Regex.
While you wait, consider that regular expressions are a super powerful way to parse strings. That said, I like to joke that given a parsing problem you can find a regular expression. Now you have three problems.
Regular Expressions can be very challenging to learn and can often be very fragile. That said, they provide a concise means of parsing complicated string data into something you can easily use.
Working on my example of how to TDD a regular expression I came up with this simple (and still somewhat dumb) regex for at least one form of ISO 8601 Date-Time strings.
Using the Python re library, this regex will quickly chop up a date-time string. This is kind of silly since there are plenty of date-time parsers in the world; I picked this example because its still reasonably simple.
In any case you might find yourself in need of a custom string parser and regex is a great way to get there. Doing it with tests can be challenging though. There are lots of funny edges to be considered.
So, if all goes well, I'll have an example for you next week. Until the, enjoy the holiday.
Considering the Labor Day Weekend, look for a post on Wednesday of next week on the topic of Test Driven Regex.
While you wait, consider that regular expressions are a super powerful way to parse strings. That said, I like to joke that given a parsing problem you can find a regular expression. Now you have three problems.
Regular Expressions can be very challenging to learn and can often be very fragile. That said, they provide a concise means of parsing complicated string data into something you can easily use.
Working on my example of how to TDD a regular expression I came up with this simple (and still somewhat dumb) regex for at least one form of ISO 8601 Date-Time strings.
'(\d{1,4})-(\d+)-(\d+)T(\d+):(\d+):(\d+)\.(\d+)(.\d+:\d+)'
Using the Python re library, this regex will quickly chop up a date-time string. This is kind of silly since there are plenty of date-time parsers in the world; I picked this example because its still reasonably simple.
In any case you might find yourself in need of a custom string parser and regex is a great way to get there. Doing it with tests can be challenging though. There are lots of funny edges to be considered.
So, if all goes well, I'll have an example for you next week. Until the, enjoy the holiday.
30 August 2017
Not Just A Song
Patience is not just a song.
Personally I struggle mightily with patience, but it is a virtue well worth developing. Patience as a teacher, mentor, collaborator, and friend is essential to success. If you can remain calm and keep your communication open and clear you can make a lot more progress than if you lose your patience and do something rash.
Remember, Deep Breaths.
Personally I struggle mightily with patience, but it is a virtue well worth developing. Patience as a teacher, mentor, collaborator, and friend is essential to success. If you can remain calm and keep your communication open and clear you can make a lot more progress than if you lose your patience and do something rash.
Remember, Deep Breaths.
28 August 2017
Over Come By Events
Sometimes we take on too much. I've talked about taking a break and about meeting objectives. I've even talked about commitment. But what happens when it is physically impossible to meet an objective -- when you are over come by events.
You need to reset your commitment, and you need to do so publicly and firmly.
Over the last 40 years I don't know how many times I've had to appeal to someone else's grace because I've failed to deliver something on time. From homework assignments in school to project deliverables or time-sheets. There are any number of things I've failed to deliver on time.
I think I've mentioned, I'm an early-is-on-time, on-time-is-too-late kinda guy. I hate being late. I will burn the candle at both ends to be on-time. Even with that attitude there are times where the laws of physics get in the way.
So, when I find that I'm not going to make a commitment, as soon as I know, I contact the person I've made a commitment to and let them know. I then recommit and get back to it.
Simple examples of this are, I'm going to be late to a meeting. I call the organizer and tell them when I will be there. I'm not going to finish my story card by the end of sprint, I contact my tech-lead and product owner and inform them of when I will be finished. A deploy is not going to get delivered on time. I contact my Delivery Lead and Product Owner and inform them of when it will happen.
It is a terrible feeling to not get something done when you've made a commitment. That said, it is inevitable that you will fail at some point in your career. So when that happens, you have to buck-up and let the person you committed to know that you are aware of the issue and then recommit to them when you will complete the work.
You need to reset your commitment, and you need to do so publicly and firmly.
Over the last 40 years I don't know how many times I've had to appeal to someone else's grace because I've failed to deliver something on time. From homework assignments in school to project deliverables or time-sheets. There are any number of things I've failed to deliver on time.
I think I've mentioned, I'm an early-is-on-time, on-time-is-too-late kinda guy. I hate being late. I will burn the candle at both ends to be on-time. Even with that attitude there are times where the laws of physics get in the way.
So, when I find that I'm not going to make a commitment, as soon as I know, I contact the person I've made a commitment to and let them know. I then recommit and get back to it.
Simple examples of this are, I'm going to be late to a meeting. I call the organizer and tell them when I will be there. I'm not going to finish my story card by the end of sprint, I contact my tech-lead and product owner and inform them of when I will be finished. A deploy is not going to get delivered on time. I contact my Delivery Lead and Product Owner and inform them of when it will happen.
It is a terrible feeling to not get something done when you've made a commitment. That said, it is inevitable that you will fail at some point in your career. So when that happens, you have to buck-up and let the person you committed to know that you are aware of the issue and then recommit to them when you will complete the work.
25 August 2017
Taking a Break
It is important to remember to take a break every once and a while. Give your mind a chance to recover and it will do amazing things. One thing I know about myself is that I'm very persistent; once I dive into a problem I have a hard time giving up or stepping away.
The other thing I know is that I can get somewhat scope locked. I get committed to a possible solution and then I don't let go. That's often times a bad thing. So what I've learned to do is, take a break every once in a while.
If I feel stuck I will go to the restroom and wash my hands. I imitate TV surgeons, washing up my forearms and scrubbing my fingers etc. If I really concentrate on what I'm doing, I push the active thinking about the problem out of my mind. When I finish washing and try to resume my thinking about the problem at hand I often have a revelation of some sort that unsticks me.
I don't think there is magic in washing your hands (though its good practice) however, the momentary distraction from the current thinking unblocks your mind by creating space between your conscious thought and the cage you have subconsciously places around the problem space.
The other thing I know is that I can get somewhat scope locked. I get committed to a possible solution and then I don't let go. That's often times a bad thing. So what I've learned to do is, take a break every once in a while.
If I feel stuck I will go to the restroom and wash my hands. I imitate TV surgeons, washing up my forearms and scrubbing my fingers etc. If I really concentrate on what I'm doing, I push the active thinking about the problem out of my mind. When I finish washing and try to resume my thinking about the problem at hand I often have a revelation of some sort that unsticks me.
I don't think there is magic in washing your hands (though its good practice) however, the momentary distraction from the current thinking unblocks your mind by creating space between your conscious thought and the cage you have subconsciously places around the problem space.
23 August 2017
Asking Why?
I've been thinking lately about how frequently (or infrequently) we ask why? That is, I think maybe we don't do it often enough.
Today in standup there was a question about adding some markers to cards to indicate something or other. I finally remembered to ask "Why?". The answer was clear enough, so we'd have some big-visible reminder of something, but only the requester had a feeling that that would be useful and he wasn't really committed to it.
If we had stopped and discussed the merits of the proposed change, then gone about making that change, and spent time over the next week or two reminding ourselves of this process change it would have been a bit wasteful. Wasteful because nobody really saw the benefit of doing it. One person even suggested it was redundant because the same answer was inferred by other data that we already track.
Asking "Why?" can save you some unnecessary work.
Today in standup there was a question about adding some markers to cards to indicate something or other. I finally remembered to ask "Why?". The answer was clear enough, so we'd have some big-visible reminder of something, but only the requester had a feeling that that would be useful and he wasn't really committed to it.
If we had stopped and discussed the merits of the proposed change, then gone about making that change, and spent time over the next week or two reminding ourselves of this process change it would have been a bit wasteful. Wasteful because nobody really saw the benefit of doing it. One person even suggested it was redundant because the same answer was inferred by other data that we already track.
Asking "Why?" can save you some unnecessary work.
21 August 2017
Only the Living
So in consideration of source control, you may have heard, only the live/active code should be present in your repository. There are tons of discussions in the coderverse about comments, commented code, config files, certificates, and other things that should never be put into source control. An there are things in the codeverse that exist as part of your deployment but also aren't in source control (cache files, log files, etc.)
What do you do about the dead stuff. Dead code should not even exist in your project to start with, but lets say it does for some reason. You should ruthlessly track down all that dead code and delete it with extreme prejudice.
Here's why. All that dead code leads to confusion later down the line. For example, you have a module of code with dead functionality in it. You go on vacation to Aruba. While you are gone the team discovers an issue and needs to fix and redeploy that module of code. Their change breaks some tests for the dead code. Now what? Do we fix the dead code? Is it supposed to be dead? How come these things are touching? There should be no touching!
Thats a simple example about code but what about dead Jenkins jobs, dead SALT states, dead scripts. But wait, theres more! What about those little dribbles of scripts and log files we leave behind on DEV and QA systems? Say, stuff we wrote to help us diagnose an issue. Or copies of things we made to avoid calamity. That's dead too. DELETE IT!
These things (backups of log files, extra scripts, temp files, etc.) cause just as much confusion if not more than the dead code. This is especially true over time. A tmp folder created today might look important to someone else next month when PROD is down. It is important to clean these things up immediately after they have served their purpose.
To summarize, if it isn't part of a prod deploy it doesn't really count. If it counts, it should be part of the prod deploy and therefore in source control. If it is dead, get rid of it rather than let it stink up the joint. Only the living code, configuration, files, should exist in your system.
What do you do about the dead stuff. Dead code should not even exist in your project to start with, but lets say it does for some reason. You should ruthlessly track down all that dead code and delete it with extreme prejudice.
Here's why. All that dead code leads to confusion later down the line. For example, you have a module of code with dead functionality in it. You go on vacation to Aruba. While you are gone the team discovers an issue and needs to fix and redeploy that module of code. Their change breaks some tests for the dead code. Now what? Do we fix the dead code? Is it supposed to be dead? How come these things are touching? There should be no touching!
Thats a simple example about code but what about dead Jenkins jobs, dead SALT states, dead scripts. But wait, theres more! What about those little dribbles of scripts and log files we leave behind on DEV and QA systems? Say, stuff we wrote to help us diagnose an issue. Or copies of things we made to avoid calamity. That's dead too. DELETE IT!
These things (backups of log files, extra scripts, temp files, etc.) cause just as much confusion if not more than the dead code. This is especially true over time. A tmp folder created today might look important to someone else next month when PROD is down. It is important to clean these things up immediately after they have served their purpose.
To summarize, if it isn't part of a prod deploy it doesn't really count. If it counts, it should be part of the prod deploy and therefore in source control. If it is dead, get rid of it rather than let it stink up the joint. Only the living code, configuration, files, should exist in your system.
18 August 2017
Fooling Yourself.
I'm working in a code base now where we frequently see lists of objects pushed into a creational method that often times passes delegates to a persistence method somewhere else. Typically this is some kind of convert JSON strings into database rows activity.
Yesterday I came across some tests that claimed to validate the insertions. To my sorrow, the validation was bogus. I then spent several hours scanning code looking for this pattern and discovered that our codebase is riddled with this mistake. I then went about fixing these issues. Fortunately I didn't discover any new issues once I had the tests fixed.
Setup
We have a SqlAlchemy data object named Foo. Foo has two attributes, id and name. Then there is the FooDBWriter that ingests JSON strings and makes instances of the data object. Once that object has been created it gets inserted into the database via the session object.
You can see the entire code suite here. Below I will just show and discuss the code that has problems and how to fix it.
This is a pretty typical pattern in our code base and elsewhere. This is a fine example of an establish micro-pattern in our development environment.
False Positive Tests
Here is a simplification of the issue that I discovered;
The class WorkingFalsePositiveTestOfDBWriter represents the erroneous testing approach. What you can see here is that the developer wanted to check that the session.insert was called once for each model object in the call to write_to_db. For a reason I cannot to fathom, the check is done with mock objects.
If you aren't familiar with MagicMock, they are part of the Python testing framework and simply put they can be used as programmable mock substitute for any other object that provides reasonable defaults. In this case, __eq__ returns True despite the fact that the objects might not actually be equal to each other.
So, the WorkingFalsePositiveTestOfDBWriter implementation uses a number of MagicMock instances equal to the number of model objects and checks for equality via the assert_has_calls method. (assert_has_calls is another mocking method that allows you to verify the occurrence of calls on the mock).
The reason this doesn't work is because the MagicMock is always going to return True. So it is possible that the objects being inserted have different values for their fields and the MagicMock will return True in all cases. So that isn't a very complete verification of the behavior.
Authors Intent
I'm guessing at this because the original author does not appear to be available any more. But I think the original approach was something like this;
So you'd hope that an approach like this works. However, our Foo class doesn't define an __eq__ method. I put the test run output in a comment below the assertion, you can see that it claims the results are not equal (despite appearances).
The issue is in how the two lists are compared. Each element of each list is compared with the other by identity, because these aren't the same instances __eq__ returns False. You could get around this by adding an __eq__ method to the data object, but you may have any number of reasons that you don't want to or cannot make that change. [Like an angry goblin that doesn't believe in collective code ownership, or an external library you have no control over]
Working Solution
The following is a more effective means of checking the call list without adding an __eq__ method to the data object.
This approach uses a matcher class. The matcher class simply wraps the desired object and intercepts the call to __eq__. The matcher's __eq__ method then checks attribute by attribute for equality. (There are some other handy advantages to this approach that I'll address elsewhere).
Conclusion
So the point here is, be careful with your assumptions when you are writing tests. Understand as much as possible how the comparisons will be made by the testing framework. Validate that your tests really work by tweaking expectations to ensure that the assertions you are making are real. Sometimes you end up fooling yourself; and the consequence is a lack of faith in your test suite.
Yesterday I came across some tests that claimed to validate the insertions. To my sorrow, the validation was bogus. I then spent several hours scanning code looking for this pattern and discovered that our codebase is riddled with this mistake. I then went about fixing these issues. Fortunately I didn't discover any new issues once I had the tests fixed.
Setup
We have a SqlAlchemy data object named Foo. Foo has two attributes, id and name. Then there is the FooDBWriter that ingests JSON strings and makes instances of the data object. Once that object has been created it gets inserted into the database via the session object.
You can see the entire code suite here. Below I will just show and discuss the code that has problems and how to fix it.
This is a pretty typical pattern in our code base and elsewhere. This is a fine example of an establish micro-pattern in our development environment.
False Positive Tests
Here is a simplification of the issue that I discovered;
The class WorkingFalsePositiveTestOfDBWriter represents the erroneous testing approach. What you can see here is that the developer wanted to check that the session.insert was called once for each model object in the call to write_to_db. For a reason I cannot to fathom, the check is done with mock objects.
If you aren't familiar with MagicMock, they are part of the Python testing framework and simply put they can be used as programmable mock substitute for any other object that provides reasonable defaults. In this case, __eq__ returns True despite the fact that the objects might not actually be equal to each other.
So, the WorkingFalsePositiveTestOfDBWriter implementation uses a number of MagicMock instances equal to the number of model objects and checks for equality via the assert_has_calls method. (assert_has_calls is another mocking method that allows you to verify the occurrence of calls on the mock).
The reason this doesn't work is because the MagicMock is always going to return True. So it is possible that the objects being inserted have different values for their fields and the MagicMock will return True in all cases. So that isn't a very complete verification of the behavior.
Authors Intent
I'm guessing at this because the original author does not appear to be available any more. But I think the original approach was something like this;
So you'd hope that an approach like this works. However, our Foo class doesn't define an __eq__ method. I put the test run output in a comment below the assertion, you can see that it claims the results are not equal (despite appearances).
The issue is in how the two lists are compared. Each element of each list is compared with the other by identity, because these aren't the same instances __eq__ returns False. You could get around this by adding an __eq__ method to the data object, but you may have any number of reasons that you don't want to or cannot make that change. [Like an angry goblin that doesn't believe in collective code ownership, or an external library you have no control over]
Working Solution
The following is a more effective means of checking the call list without adding an __eq__ method to the data object.
This approach uses a matcher class. The matcher class simply wraps the desired object and intercepts the call to __eq__. The matcher's __eq__ method then checks attribute by attribute for equality. (There are some other handy advantages to this approach that I'll address elsewhere).
Conclusion
So the point here is, be careful with your assumptions when you are writing tests. Understand as much as possible how the comparisons will be made by the testing framework. Validate that your tests really work by tweaking expectations to ensure that the assertions you are making are real. Sometimes you end up fooling yourself; and the consequence is a lack of faith in your test suite.
16 August 2017
Fucking Up
Saw a great post yesterday about fucking up. That is, making mistakes. It simply stated that you should own it.
It's hard to own it sometimes. Our egos get in the way. But, if you own it and move on with a minimum of drama, thats better than dancing around your errors and making excuses. It requires vastly less energy too.
Everyone fucks up, not everyone is willing to admit it. Have the courage to own your mistakes and you will go farther, faster.
It's hard to own it sometimes. Our egos get in the way. But, if you own it and move on with a minimum of drama, thats better than dancing around your errors and making excuses. It requires vastly less energy too.
Everyone fucks up, not everyone is willing to admit it. Have the courage to own your mistakes and you will go farther, faster.
14 August 2017
Pause and check for level
Today I set out on a mission to build a fire pit in my back yard. I used to have one of those stand-alone pits but it rusted out. So I got an oil drum and cut it down to 1/3 it's height and then dug a hole to bury it. Once I had my hole dug I poured 1 1/2 cubic feet of river rock into the hole and leveled it off. I then set the barrel down into the hole.
It looks level, from my knees.
Before I poured rock inside the barrel I had to get the bags off the wheelbarrow behind me, so I stood up and grabbed a bag. Before I plunged my trusty pocket knife into the bag I looked down at the drum sticking out of the hole and noticed it wasn't level.
I had to go get the level.
I know better than to do this project without the right tools, but I had left the level in the garage. So I went and got it and check the drum. Sure enough it was considerably out of level North to South. So I pulled it back out and shifted the rock around. Checked level and it was spot on.
Then I checked East to West.
East to West I was still off, so out comes the drum, more shifting of rock, etc. etc. Eventually I got it close enough. After all, there will be field stone surrounding my fire-pit and you probably won't notice if I'm off a bit.
Why am I telling this story.
So when we are working on a story card at work, sometimes we loose sight of the big picture. We think we are doing the right thing because we don't periodically check in on the larger objective. This manifests itself in all kinds of ways; long draw out quests for perfection on throw away features, over built functionality that handles impossible cases, and so on. And sometimes, we can't tell on our own, we have to talk to someone else (go get the level).
It is important to pause occasionally and check to see that the work we are doing is on target, and when it isn't, correct course. It is also important to check more than one target, because we might be on track in one direction and off in another.
It looks level, from my knees.
Before I poured rock inside the barrel I had to get the bags off the wheelbarrow behind me, so I stood up and grabbed a bag. Before I plunged my trusty pocket knife into the bag I looked down at the drum sticking out of the hole and noticed it wasn't level.
I had to go get the level.
I know better than to do this project without the right tools, but I had left the level in the garage. So I went and got it and check the drum. Sure enough it was considerably out of level North to South. So I pulled it back out and shifted the rock around. Checked level and it was spot on.
Then I checked East to West.
East to West I was still off, so out comes the drum, more shifting of rock, etc. etc. Eventually I got it close enough. After all, there will be field stone surrounding my fire-pit and you probably won't notice if I'm off a bit.
Why am I telling this story.
So when we are working on a story card at work, sometimes we loose sight of the big picture. We think we are doing the right thing because we don't periodically check in on the larger objective. This manifests itself in all kinds of ways; long draw out quests for perfection on throw away features, over built functionality that handles impossible cases, and so on. And sometimes, we can't tell on our own, we have to talk to someone else (go get the level).
It is important to pause occasionally and check to see that the work we are doing is on target, and when it isn't, correct course. It is also important to check more than one target, because we might be on track in one direction and off in another.
11 August 2017
Winchester's Law
When I was a kid my sister and I would watch M.A.S.H pretty much every day. I can say with some confidence that while I might not recall all the episodes I'm sure I've seen them all, and many of them twice.
On the topic of focus though, I've often been in a situation where it has been necessary to express the need to do a job well. That is, not to hurry and get a job done so you can move on, but to do it as well as it can be done at the time.
I call this Winchester's Law in reference to M.A.S.H. Season 6 Episode 1. Thats the one where Frank Burns leaves and Charles Emerson Winchester III becomes permanently attached to the 4077th. It's based on Charles Winchesters comment to Hawkeye as he is explaining his surgical technique. He says,
This is, in essence, the road to greatness. Do just one thing as well as you possibly can, then move on. This is in contrast to doing 3 or 4 things reasonably well and maybe not getting it all done or of course, doing nothing.
On the topic of focus though, I've often been in a situation where it has been necessary to express the need to do a job well. That is, not to hurry and get a job done so you can move on, but to do it as well as it can be done at the time.
I call this Winchester's Law in reference to M.A.S.H. Season 6 Episode 1. Thats the one where Frank Burns leaves and Charles Emerson Winchester III becomes permanently attached to the 4077th. It's based on Charles Winchesters comment to Hawkeye as he is explaining his surgical technique. He says,
"I do one thing at a time, I do it very well, and then I move on."
This is, in essence, the road to greatness. Do just one thing as well as you possibly can, then move on. This is in contrast to doing 3 or 4 things reasonably well and maybe not getting it all done or of course, doing nothing.
09 August 2017
Shebang
This might seem obvious, but clearly it isn't. Rob Miller and I were pairing on a Friday afternoon and ran across a script in our SALT configuration. In the script Jinja was used to setup various variables before the script body. The Jinja code proceeded the shebang line and so the resulting script placed the shebang several lines into the file, with blank lines proceeding it. As a consequence, the shebang line was not executed. We dutifully fixed this in about six locations in our SALT config.
We thought we should share our learning/remembering with the world. So we whipped up this simple example on a laptop.
The following script (x.sh) changes the shell to zsh via the shebang and then prints the zsh version via echo. We then run the script and see that the version is 5.2. After that we create a new script (y.sh) with five blank lines followed by the content of x.sh. When we run that script we get no zsh version. That is because the shebang was ignored, its not the very first line of the file.
Our point is, the magic number/shebang must be the first two bytes of the file or the shell will ignore it.
We thought we should share our learning/remembering with the world. So we whipped up this simple example on a laptop.
The following script (x.sh) changes the shell to zsh via the shebang and then prints the zsh version via echo. We then run the script and see that the version is 5.2. After that we create a new script (y.sh) with five blank lines followed by the content of x.sh. When we run that script we get no zsh version. That is because the shebang was ignored, its not the very first line of the file.
Our point is, the magic number/shebang must be the first two bytes of the file or the shell will ignore it.
07 August 2017
More Thoughts on Time
Michael Gantz was kind enough to comment on my previous post about Early is On Time. He put forth the notion that we should be more honest about our estimates and what they mean. For example, "This project will be done in 3 weeks with a 75% confidence" or alternately "This project will be done in 3 to 5 weeks".
The Effort
I like the idea of making honest estimates and I like the idea of providing confidence levels or ranges of time. Years ago I read a book from Microsoft on project management that provided a estimation confidence guide. I don't recall all the details any more but it suggested something like, at the beginning of a project our estimates are +/- 65%, near the end we are within +/- 1%. I think thats interesting, but I think we put a lot of effort into managing that information when we don't need to. Rather we should be working together, business and technology, to find an optimal solution.
The Old Saw
Traditional agile practices (is that a thing?) tell us to use yesterdays weather to forecast the future. So if we have a know quantity of work and some data about how fast we typically burn through work, we should be able to provide a number in some reasonable range. Thats great. So now we can talk to the business about when we might deliver a quantity of work.
Imagine a World
Disregarding all the unknowns that tend to throw a project off schedule, let pretend we can forecast work with some reasonable expectation of matching our estimate. 75% of the time we hit the mark on the head, when we are late it is typically by 5% of the work. We won't bother with early.
Early is On Time and On Time is Too Late, part 2
Software development is supposed to be a collaboration. If a development team tells you that they can deliver the product in 5 weeks with a 75% confidence, it is necessary to cooperate with that estimate. That is, throwing more bodies, mandatory overtime, or free pizza at a deadline almost never changes the delivery schedule. Rather, it leads to burn out, cut corners, and tech debt.
In my time I've seen entire departments halted to re-estimate work because the business didn't like the forecast. There is a lack of trust for you. I've also seen 'Saturday Bug Hunt Parties' with all the pizza and pop you can consume in order to bring down the backlog. A fine example of exploitation leading to burn out.
We might suck at estimating while a team is still forming, but a team that is someplace in the norming arc has a pretty good idea of what they can do. If what they can do doesn't satisfy the business objective there isn't much in the way of effective tactics to solve that problem.
The Brutal Truth
Facing a missed objective there are really only three avenues left. Eliminate all obstacles to between the team and the goal, change the goal, or replace the team with more reliable/accurate resources. Most any other suggestion is a variation on these three.
Never mind that we should eliminate obstacles in the first place, this is probably our best hope for success. Given enough latitude a disciplined team can burn down a ton of work if we take everything else out of their way. But, if that isn't going to be enough, cutting scope is your next best option. Take away all non-essential functionality, all the bells and whistles, and get to market. You can always deploy again the next week (or day even).
Nobody likes to talk about replacing the team, but sometimes that is the answer. Though often times it doesn't help as much as we'd like. I'll write about that situation some other day in more detail, but keep it in the back of your mind, sometimes organization overreach or bad politics or other human issues force you into this situation.
Back on Point
In the end, business and technology are much better off when they work together to find shared belief about what they can do. That requires getting the actual doers involved in the estimation process. It also requires the people asking for the features to be reasonable and accommodating. It should be a collaboration though, and if it isn't, Michael' suggestion isn't going to help, nor would the most precise estimate in the world. Estimates, after all, are just educated guesses and prone to error.
The Effort
I like the idea of making honest estimates and I like the idea of providing confidence levels or ranges of time. Years ago I read a book from Microsoft on project management that provided a estimation confidence guide. I don't recall all the details any more but it suggested something like, at the beginning of a project our estimates are +/- 65%, near the end we are within +/- 1%. I think thats interesting, but I think we put a lot of effort into managing that information when we don't need to. Rather we should be working together, business and technology, to find an optimal solution.
The Old Saw
Traditional agile practices (is that a thing?) tell us to use yesterdays weather to forecast the future. So if we have a know quantity of work and some data about how fast we typically burn through work, we should be able to provide a number in some reasonable range. Thats great. So now we can talk to the business about when we might deliver a quantity of work.
Imagine a World
Disregarding all the unknowns that tend to throw a project off schedule, let pretend we can forecast work with some reasonable expectation of matching our estimate. 75% of the time we hit the mark on the head, when we are late it is typically by 5% of the work. We won't bother with early.
Early is On Time and On Time is Too Late, part 2
Software development is supposed to be a collaboration. If a development team tells you that they can deliver the product in 5 weeks with a 75% confidence, it is necessary to cooperate with that estimate. That is, throwing more bodies, mandatory overtime, or free pizza at a deadline almost never changes the delivery schedule. Rather, it leads to burn out, cut corners, and tech debt.
In my time I've seen entire departments halted to re-estimate work because the business didn't like the forecast. There is a lack of trust for you. I've also seen 'Saturday Bug Hunt Parties' with all the pizza and pop you can consume in order to bring down the backlog. A fine example of exploitation leading to burn out.
We might suck at estimating while a team is still forming, but a team that is someplace in the norming arc has a pretty good idea of what they can do. If what they can do doesn't satisfy the business objective there isn't much in the way of effective tactics to solve that problem.
The Brutal Truth
Facing a missed objective there are really only three avenues left. Eliminate all obstacles to between the team and the goal, change the goal, or replace the team with more reliable/accurate resources. Most any other suggestion is a variation on these three.
Never mind that we should eliminate obstacles in the first place, this is probably our best hope for success. Given enough latitude a disciplined team can burn down a ton of work if we take everything else out of their way. But, if that isn't going to be enough, cutting scope is your next best option. Take away all non-essential functionality, all the bells and whistles, and get to market. You can always deploy again the next week (or day even).
Nobody likes to talk about replacing the team, but sometimes that is the answer. Though often times it doesn't help as much as we'd like. I'll write about that situation some other day in more detail, but keep it in the back of your mind, sometimes organization overreach or bad politics or other human issues force you into this situation.
Back on Point
In the end, business and technology are much better off when they work together to find shared belief about what they can do. That requires getting the actual doers involved in the estimation process. It also requires the people asking for the features to be reasonable and accommodating. It should be a collaboration though, and if it isn't, Michael' suggestion isn't going to help, nor would the most precise estimate in the world. Estimates, after all, are just educated guesses and prone to error.
04 August 2017
Configuration Testing
An often overlooked component of systems development is configuration testing. Time after time systems are built with no tests related to their configuration that later lead to confusion.
Configuration testing is really quite simple. Following the same general guideline of nothing goes to production that wasn't the result of a test; no configuration should exist without tests.
Simply put, if you have a configuration key-value-pair, there should be tests for what the system does when the key-value-pair is present, not present, and variations on the value.
For example, you might have a configuration key for output directory. You should have tests for that key being present and set to a valid value, absent, and some collection of invalid values. Invalid values being nil, non-existent directory, directory to which the application does not have privileges to, etc.
These kinds of tests act as executable documentation for what a system expects and how it behaves with respect to its configuration and really do act as a means of communicating to yourself and others what the configuration keys do.
Of course, we should not forget about default values for configuration keys. Ideally all configuration keys have some reasonable default value; when they don't our programs should halt gracefully with a clean message indicating the issue.
* Apologies to Michael Gantz for having not yet responded to his comments on being on time. I hope to formulate a coherent statement soon.
Configuration testing is really quite simple. Following the same general guideline of nothing goes to production that wasn't the result of a test; no configuration should exist without tests.
Simply put, if you have a configuration key-value-pair, there should be tests for what the system does when the key-value-pair is present, not present, and variations on the value.
For example, you might have a configuration key for output directory. You should have tests for that key being present and set to a valid value, absent, and some collection of invalid values. Invalid values being nil, non-existent directory, directory to which the application does not have privileges to, etc.
These kinds of tests act as executable documentation for what a system expects and how it behaves with respect to its configuration and really do act as a means of communicating to yourself and others what the configuration keys do.
Of course, we should not forget about default values for configuration keys. Ideally all configuration keys have some reasonable default value; when they don't our programs should halt gracefully with a clean message indicating the issue.
* Apologies to Michael Gantz for having not yet responded to his comments on being on time. I hope to formulate a coherent statement soon.
02 August 2017
Early is On Time
and On Time is Too Late.
I first heard that in Sonny Barger's book 'Freedom: Creeds from the Road' and it took me a while to really understand it.
To be clear, it took a while for me to understand it as it relates to me. Sonny was pretty clear about what it meant to him.
To me its about respect. I don't like to be late because I think its disrespectful. This extends beyond being late to a meeting or a lunch date, it goes to delivery too. If you commit to a delivery, being late shows disrespect.
I first heard that in Sonny Barger's book 'Freedom: Creeds from the Road' and it took me a while to really understand it.
To be clear, it took a while for me to understand it as it relates to me. Sonny was pretty clear about what it meant to him.
To me its about respect. I don't like to be late because I think its disrespectful. This extends beyond being late to a meeting or a lunch date, it goes to delivery too. If you commit to a delivery, being late shows disrespect.
31 July 2017
It's not going to happen
Sometimes we need to realize that 'It's not going to happen'.
There are times when we've committed to a solution or a timeline that we need to realize for any number of reasons, it's not going to happen. We need to recognize when it isn't going to happen and adjust our plans.
A key component of that recognition is communication. If we don't communicate to others that something isn't going to happen they cannot adjust their plans. This can throw a project team into chaos.
Sometimes, it just isn't going to happen.
There are times when we've committed to a solution or a timeline that we need to realize for any number of reasons, it's not going to happen. We need to recognize when it isn't going to happen and adjust our plans.
A key component of that recognition is communication. If we don't communicate to others that something isn't going to happen they cannot adjust their plans. This can throw a project team into chaos.
Sometimes, it just isn't going to happen.
28 July 2017
Side Effects
Side Effects are an often over looked consideration when coding. A side-effect occurs when a function modifies the state of something outside of its own scope. That is, it changes the value of a global, static, or argument. Functions should ideally not do this.
However, here is a conundrum, how do you modify the state of they system otherwise? For example, if your code is updating the value of a transfer object, how do change its value since it is technically outside of your scope?
One technique is to clone the input and modify that input. This prevents the side-effect of having modified the object passed in. In stead you are creating a new object as a copy of the object received and modifying it. If you do this consistently throughout the system there will be no side-effects.
An approach like that has some powerful consequences. For one, in a multithreaded system there is no risk of two threads making concurrent modifications. Thats typically a good thing. Another benefit is that you don't have to worry about a function changing the state of its arguments, so as a caller you can rely on the state of the arguments being unchanged. This also allows you to make all of your objects immutable. Immutability providing you the assurance that an specific instances value/state has not changed.
In the end, this will force you to be very concise in your method declarations however. Typically we don't return complicated structures, if we generally call a function we expect only one value/object back from it. This is also a pretty good design approach.
However, here is a conundrum, how do you modify the state of they system otherwise? For example, if your code is updating the value of a transfer object, how do change its value since it is technically outside of your scope?
One technique is to clone the input and modify that input. This prevents the side-effect of having modified the object passed in. In stead you are creating a new object as a copy of the object received and modifying it. If you do this consistently throughout the system there will be no side-effects.
An approach like that has some powerful consequences. For one, in a multithreaded system there is no risk of two threads making concurrent modifications. Thats typically a good thing. Another benefit is that you don't have to worry about a function changing the state of its arguments, so as a caller you can rely on the state of the arguments being unchanged. This also allows you to make all of your objects immutable. Immutability providing you the assurance that an specific instances value/state has not changed.
In the end, this will force you to be very concise in your method declarations however. Typically we don't return complicated structures, if we generally call a function we expect only one value/object back from it. This is also a pretty good design approach.
26 July 2017
It's too Big
An ongoing topic is the size of a class, function, or test. I think for each of us there is a size limit of some sort, but it varies. What I am clear on is that too big is too big.
For me, functions with more than 2-3 logical branches are too big.
Classes that are more than about 75 lines are too big.
Tests that are more than 10 lines are too big.
The size of things is a feedback mechanism we often don't pay close enough attention to. When we feel that pressure its time to split things up.
For me, functions with more than 2-3 logical branches are too big.
Classes that are more than about 75 lines are too big.
Tests that are more than 10 lines are too big.
The size of things is a feedback mechanism we often don't pay close enough attention to. When we feel that pressure its time to split things up.
Subscribe to:
Posts (Atom)