tldr; Similarities between why custom code takes time and why woodworking is laborious. The purpose is to examine the nature of maker crafts with an eye toward how this work can be done faster. Slowness reasons discussed:
- Lack of precision
- I Don’t Have Everything I Need
- The Context is Different
- The Perpetual Prototype
- Errors Compound
- No Defaults
- Everything Custom Is Slow
Let’s get started:
When doing THE-METHOD coding the resource accessors and the engines takes the longest. The managers are (almost) disposable. I think we all agree on this but I’ve been doing this for years and I am always shocked at how long “long” actually is. I want an engine this afternoon, not in 8 weeks. But the production ready version is much more likely to take weeks. In fact, there’s not even a notion that it could be done in “hours.” That is simply ridiculous.
For their 6th birthday I just built my nephews desks. These are old fashioned flip top desks that would be considered by any woodworker to be “basic.” The project is “simple.” Each desk took me a week to build. I’ve given them to the boys and they’ve picked the colors they would like them painted. They will take me another week to paint. This is crazy and the parallels to why this is are the basis of this article.
As I said in my previous posts, I’ve been doing some woodworking and making observations. In particular what is the nature of speed and components. Woodworking and software programming are crafts and more similar than most developers realize.
An aspect that I’ve been trying to pay attention to is what is it that keeps one from moving fast. That’s not quite right. That gives the impression that I want things to move fast for the sake of speed. Let me try that again.
What are the reasons things take so long. Yes, that’s better. What’s the difference? Well, let’s say I bought something off the shelf and I could use it as is, then that’s probably fastest. But we’re often adding value because there is no “off the shelf” option. So it’s more like, we dug something out of the attic and it might work but we’ve got to fix it first. It’s close but we’ve got to modify it somehow and then it will do what we need.
In software it’s very common to hear, “Yes, we can that. It’s very similar to what we did on <this-other-project/engine/class>.” This is then followed by weeks and weeks of “updates” and re-estimations of the progress. The phrase “harder than we thought” and “we didn’t know that when we said that” will come up dozens of times.
Let’s assume for the moment that you can’t buy what you need. If you can buy it and can afford it, you should buy it. I’m a maker and I’m telling not to build. Especially don’t build it if your concerns are cost and schedule. The reasons to build are many but they do *not* include cost and schedule.
The lumber for my nephew’s desks was over $200 per desk. I will have spent over a month on these desks when I finally deliver them. The value of this project is not time or money. I love those kids and want them to have something I made. I want them to know that someone cares about them enough to build them something. It is *not* a good scheduling or financial decision.
Software is still in the phase of development where a lot of the value is added by building. But developers are willing builders. We barely spend any time looking at buying. I’m doing woodworking because we bought a house and I need to fix some things up. I haven’t done this sort of work since high school. My skills are pretty good, but a storage cabinet, a few stools, and table later I’ll be much better at working on the house. That’s already proving to be true.
Sadly, the thing that has led the “off-the-shelf” moment the most in the last decade has been the availability of software packages. Before packages (NPM, Nuget, etc.) people had to talk management into buying. Buying with real money, involving accounting, budgeting, and purchase orders. It was insanely difficult. Developers usually couldn’t be bothered. Especially when building is “job security.”
The big issue that I have with the package community is that it was led by front-end developers. This is exactly the wrong community. They ended up trying to prematurely move everything to the front-end. I’ve seen many man-years wasted using packages to move things to the front-end that should have stayed on the back-end. Packages are a miracle of productivity. Having the front-end developers lead this movement was a lost opportunity.
I will point out that ServiceModelEx was never released as a Nuget package.
However you look at it we are a build first community.
So, what are the things that slow down the work on my woodworking projects? I’ve taken a look and there are definitely a few that seem to overlap with software development. I’m “righting” this (see what I did there?) because I think the comparison is insightful and useful. I’m hoping that I’ll be able to offer some specific suggestions along with my comparisons.
REASON #1 — Lack of Precision
This is the most common reason for slowness. It also introduces the most slowness.
Let’s say you go to the big box store and by a piece of furniture that you have to assemble yourself. You may find some parts difficult or feel that it took too long, but compared to building it yourself? It’s superfast. Part of the reason is that, in the kits, everything is precut and the cuts are (generally) perfect.
When woodworking I start with lumber and cut it to make the parts. When the cuts are not “square” (at 90 degree angles) or wander away from the lines I’ve marked it’s not the end of the world. Almost anything in woodworking can be fixed. It’s the same with software. If you do a design and the implementation is “off the mark” you can almost always fix it. The number of times I’ve quit or given up on an implementation is essentially zero. I can only recall one or two times.
In woodworking you can plane or chisel away excess wood. If the cut is too short you can start again with a new piece of lumber. Nothing is wasted. The piece cut too short will eventually be used someplace else. But those options cost time and money.
In software the “out of square” parts might be something like getting a DTO or message that isn’t in the best format for your use-case. This forces you to reformat the message before the data can be used. How many times have you set out to use a DTO and then spent hours writing parsers and/or transformers instead? There’s a reason the “adapter” is one of the most common design patterns.
“Cut too long” or “too short” in software might be something like getting lots of compound values and you only need the “bits in the middle.” After hours of parsing you can finally get on with your use-case.
I think the more common case is “too short.” This might be something like getting the key or ID for something you need but not it’s attributes. You find yourself coding a DAO for something you thought was in the original DTO but it was just the ID. You install or code the DAO, look up the data you need, format it, bounds check it, and whatever else. Only then can you proceed with your use-case.
Dealing with NULL falls into this category. In terms of a value null is sort of the ultimate imprecision. You know you need one of these values but all you know about this one is that “we don’t know it’s value.” Dammit!
REASON #2 — I Don’t Have Everything I Need
Every time I realize that I need to “run to Home Depot” I roll my eyes and think, “Dammit!”
In software, this might be that a component is missing. But the most common is missing information. Think of all the times you had to stop, go find someone, get some questions answered. And the information alone is generally not enough. You have to integrate the information into your understanding and design. The information may mean that other design or implementation choices have to change.
How many times has the project been estimated with the assumption that the required information was available? How many times have you refused to provide an estimate until the scope of the available information could be confirmed?
In the services world this happens a lot because people aren’t following THE-METHOD. Many tasks start by someone suggesting that we “reuse a stream” or “this API already has what we need.” But quite often it does not.
Things become much more reusable if your Engines speak in algorithms and your Resource-Accessors speak in atomic-business-verbs. I find that with Engines the thing I’m missing most to reuse an engine is usually context information. There is an old rule-of-thumb in software that says: “Code is not reusable until it has been reused three times.” This is meant to illustrate the need to do the work to expand the context via three separate uses before you can get it to conform to the general case. In my experience there has always been a significant amount of work required during these first three reuses. Despite my best efforts this rule-of-thumb has proven to be frustratingly accurate.
REASON #3 — The Context is Different
Related to “not having what you need” is the ever shifting sands of context.
In woodworking this might slow you down because you have something but not for this context. Take fasteners for example. You might have a shelf full of wood screws, but you need to join a thin piece of veneer and what you need is a finishing nail. You might have epoxy when what the task calls for is superglue. You might have painter’s-tape but what you need is duct tape.
The same is true in software. I was working at a financial house on Market Street a few years ago. The values I got were 32 bit, but our project did everything in 64 bit values. In finance precision matters. You cannot (easily) convert 32 bit values to 64 and visa versa. The concerns and boundary conditions introduced by these conversions are a huge burden.
[Side note: During that project I discovered a bug that had existed for at least 8 years. You cannot go around fixing 8 year old bugs. Years of software had been written with that bug in place. On the other hand I was responsible for my work and my code was aware of the boundary where the bug existed. That’s how I found it in the first place. I had to reimplement (by copying and pasting) a handful of their core functions with the bug fixed in my versions. My code used my versions. Both versions got checked in. I don’t think they ever fixed the bug. The bug containing context of the original code had to stay in place.]
Reusing an Engine and extending it’s scope is often frustrating. To do so requires more context information.
For example, suppose you have an engine that processes things one at a time. You decide to extend it to progress things in batches. As soon as you introduce batches you’ve changed the context. Now the things in the batch are processed in the context of the other things in the batch. If the process fails do they all get rejected? Or do you keep the ones that got processed and do something else with the other ones? You add the notion of summaries and totals to the context. These changes reveal you need even more context information to answer these questions. Oh brother.
REASON #4 — The Perpetual Prototype
Nothing’s ever perfect, nothing is optimal. As I build I learn “how I should have done it.” Whether it’s wood or code everything is perpetually a prototype of the next version.
How many times have you checked in some code and told a colleague, “Okay, it’s checked in. Pull it and start using it.” And then added, “But be aware, I’m still modifying it. I’ll let you know as I do check-ins and I’ll try not to break your code too much.” This is common, particularly for a lead-developer. But it also means that your code is still a prototype.
In woodworking I recently built a small stool and the first leg I did seemed fine, but then I did the second. I noticed it was a bit better than the first. By the time I finished the fourth leg I had myself wondering if I should rebuild the first one. Each had been a prototype for the next.
If you have three engines to write after the second that you have to cut/paste back to the first. You innovated. Now it’s almost unconscionable to not port the work back.
And not all prototype efforts show up in the finished product. With the legs of my stool part of the improvement was better ways of holding the legs. Better techniques at removing the waste. Better ways of making the cuts.
In software you often need to build tests and test data. Maybe the first class you work on there are no tests, but by the time you check it in there is. By the time you tell your colleagues to look at a check-in you’re working on testing the edge case discovered while testing edge cases.
This is great, but it also means you’re perpetually prototyping. I get it. That’s the nature of a craft, be it woodworking, be it software development. But it’s NOT the nature of engineering. To move toward software engineering we’ve got to find ways to make our work less prototyping. We need more specification driven development.
This starts by acknowledging that everything we build by hand is a prototype.
REASON #5 — Errors Compound
Every construction project has errors and compromises. In both software and woodworking the skill of the team member are a limitation. Materials, like the lumber, tools and fasteners don’t match the ideal. Integers are too small or take to big and eat memory.
In software the programming environment limits what is possible. Not every idea is possible in every environment. As a C# developer I find myself doing things in the .NET way. When I was a Java developer I coded in the Java style. But in practice the environment is not the limiting factor.
The main source of compromises in my experience is the skill of the developer. And I’m not (usually) talking about me. I’m talking about the developer that came before me. That’s not a complete slam on past developers. They operated in a context I’ll never know, with deadlines and pressures I don’t understand. But every lead developer can be heard at times shouting (perhaps under their breath), “those bastards! Why the hell did they do _____.”
Two practices that have plagued me the most are “how nulls are handled” and “using exceptions” for messaging. This is especially true for service development.
It’s beyond the scope of this post to go into those issues in detail. But how previous developers handled these issues often compounds over time. The developers that followed those developers often continued the poor practices. With each copy or bad-clone the code became harder and harder to fix and moved further out of spec. This is one of the reasons that Resource-Accessors are so time consuming to write. To code in Atomic-Business-Verbs (ABV) you have to take on these issues. ABV will not allow you to ignore them.
REASON #6 — No Defaults
To a casual observer this issue sounds trivial. Even most experienced developers don’t appreciate this issue. A system without reasonable defaults is not easy to fix. Defaults can be hard coded, they can be retrieved from settings, context sensitive, many certainly dependent upon the runtime context. Defaults sound like simple values. They are not and they are important.
This is such a techno concept. Allow me to make a comparison to tie this to the real world example of woodworking. In woodworking one of the “defaults” is lumber size. In the USA there is no “metric lumber.” If you buy a board in America it will in feet and dimensioned in inches. Period. The default is imperial measurements. If your default measurements metric are metric you will have built in a certain amount of waste. No cut or measurement will ever fit the dimensions of purchased lumber. I watch a lot of Japanese woodworking videos (they’re wonderful) but it’s different. Their stuff is metric back to the source. Metric is far less practical in American woodworking. The core component (lumber) has default values that are Imperial.
Defaults matter — a lot. Changing them after they’ve been ignored is difficult. Take the example above and turn it around.
Supposed I found a source for 3 meter boards that were 30 x 5 centimeters. I can assure you that these boards would be frustratingly close to 2x12 boards. Off by enough that every edge, every depth would need adjustment. The work on your project might be double, quadruple, or more. What a hassle.
The expense would go up and the speed down.
REASON #7 — Everything Custom Is Slow
Said another way, use standard parts, straight lines, common sizes, and right-angles. Carry reasonable default from end to end (do not ignore them). Avoid null like a plague.
Even a simple problem like a security setting can set you back weeks. A few years ago I built a service for tracking entity ids. It was built using a database that used Lucerne search as it’s base. No problem. I had the system up and running quickly. I went to install it in an Azure Container running Linux. It would not start. Turns out there was a setting in Lucerne that requires a security level that Azure Linux Container WILL NOT let the end user have. There is no work around, not post boot fix, nada. I spent days trying to get it to work.
You can work with Azure Containers for years and never find something that won’t deploy to them. But once I did I wasted a ton of time. I was convinced I could work around a simple setting. I should have known this was not true. When MS changed the setting they pulled Lucerne from it’s set of available packages. They also pulled a few other packages based on Lucerne.
I’m building a wooden desk right now. It’s a desk like you had in school. You lift the lid and there’s storage. You take out books or papers and write on the top. At the bottom edge of the lid is a little lip. The lip holds your papers and keeps your books from sliding off the lid and into your lap.
The plans call for gluing a little strip of wood to the surface of the lid. I’m building my desks for twin 5 year old for their 6ths birthday. I am certain that, if I glued a strip of wood to the lid, they would rip it off in short order. They’re 5!
I decided to fit the strip of wood into a mortice. That’s custom. Think of everything that changes with that tiny decision. First, you’ve got to cut a mortice into the lid. If you mess that up you’ll have to rebuild the lid. Second, now that you’re attempting to put a strip of wood into a mortice it needs to be a good fit or it’s going to look bad. A strip glued to the surface looks normal. Even tiny gaps in a morticed strip will look horrible.
I had the strip of wood I was going to use when I made the decision to embed it in the mortice. To glue it to the surface and hold it down with a weight would have taken about 3 minutes. It took me 40 minutes to mortice it in and a mistake might have put the entire lid in place.
After all that what value did I add? I suspect I added the ultimate value. A feature that actually works vs one that will quickly fail. But there’s no way to prove this. You can’t reasonably model the actions of a 5 year old. There are hundreds of ways they can destroy my work that don’t involve the book-stop. Ha.
Comparing two finished desks, one with a glued on book-stop and one with my embedded book-stop, they would look identical.
And therein lies the issue with this entire article.
If you address every issue I’ve mentioned in a reasonable way you will not be “faster or cheaper.” It’s likely the person who ignores these issues will be faster and cheaper. Building things well is hard work. Is all doomed? No.
The effects of quality and automation show up later rather than sooner. Being a hack who ignores important engineering concerns shows benefit immediately. If you can find a way to persist, you should be rewarded but it’s a slog.
If another (let’s say, “less loving”) uncle goes to Walmart and buys some prefab desks, the twin tots still get desks. In fact, I had to “bust my ass” to get mine done by their birthday. That jerk-uncle could have walked into Walmart at anytime. My schedule was stressed. Bad-uncle would have almost certainly paid less too.
Obviously I’m teasing a bit here. But I’m trying to illustrate the point that addressing these issues is challenging. It’s hard work and it’s hard work for which you’re unlikely to be lauded. In fact you’ll have to fight for the schedule and budget. In the long run the twin boys will know they have someone that loves them who spent time and effort to build them something of quality. On the day of their birthday it’s unlikely they would recognize the difference.
And in software it’s worse because you won’t be fighting with an imaginary bad-uncle. The villain in software is the developer who ignore the issues. They made imprecise cuts, left parts off, hard-coded context decisions, shipped prototypes, left compounding errors in the code, ignored defaults and nulls, and shipped ahead of schedule. They’re the hero.
If there’s interest we could explore ways of mitigating these issues in a future post.
