Software != Construction (II - The Differences)
January 4th, 2007 | Leaderware | SDLC | Software | TechnologyThis is the second article in the Software != Construction series. The first article focuses on why it matters:
- With construction metaphor, we assume software has properties of constructions that it does not have, namely
- Construction building effort can be fully parallelized - so can software (this is the mythical man month myth)
- Construction can be done by many cheap labors - so can software (this fuels the offshore movement)
- These thinking lead to the focus on reduction of software development cost, rather than focusing on adding value through software, focusing on the cost will cause outflow of technical skills in the company
- Business in the long run need to compete with technology as enabler, and outflow of technology skills will have drastic impacts in the long run
While the construction metaphor has enabled us to realize effective planning and management increases the likelihood of successfully delivering a complex software project, the problem with using a metaphor is that it draws your attention on the similarities, and conveniently hides away the differences. We can easily see where the similarities between software and building constructions, but many cannot articulate the differences.
The trouble is - these differences matter (shown in the first article). In this article we will focus on seeing the differences. As previously stated, the focus on this series is to provide the non-technical folks (a.k.a. business folks) with an appreciation of what software really is. If you are a techie, while you might already know this, you can benefit by letting the business folks you work with read this article. Because such knowledge only resides in technical folks, we must take the burden to education the business folks to help them understand software so everyone can benefit.
First Difference
First - let’s examine the physical characteristics of buildings and software. This should be obvious, software and buildings have very different physical characteristics - one is tangible and the other is not.
Buildings are tangible. Their physical forms are, of course, buildings. One can see, touch, and live in a building. For us, we have a very good understanding what buildings are, and what they should do (provide us a space to live, work, play, and doing other things).
Software, on the other hand, is intangible. We might see software if it has an user interface, and we also might not. We cannot touch it, except touching through keyboard or mouse, and we cannot easily articulate what software can do, because it can do too many different things. For us, software is amorphous. What is the physical form of software?
If you know anything about computers you know that computer only understand 1’s and 0’s, and software are instructions to the computers. And this means that the physical form of software is 1’s and 0’s.
Separation of Design and Build
Now - let’s review the building construction process - the first stage is design, and the second stage is build (this process can be generalized to AFAIK all manufacturing processes - but that will be the focus of another article).
As previously mentioned, because buildings have substantial material (and labor) costs, it is prohibitive to fix the design once the build phase starts, and hence the best practice is to partition the design and build phase, i.e. design stops, and then build starts, and design provides only one set of input to build. Builders can just take the output from the design phase, a.k.a. blueprints, as INPUT to execute with practically no more input from the designers. And the build phase, no matter how complex, basically can be summed up as "transformation of the full design into its physical form" (we will use TRANFORMATION to mean the build phase going forward to reduce overloading on the word "build").
To sum up, in building constructions,
- Separation of design and TRANSOFORMATION is critical
- No back and forth between two stages
- OUTPUT (i.e. the physical form) is the building
- TRANSFORMATION is the regular build phase
- INPUT is the blueprint
What is the equivalent in software?
INPUT and TRANFORMATION in Software
The output, i.e. the physical form, is of course, 1’s and 0’s. What qualifies as TRANSFORMATION and INPUT?
Remember, as defined above, TRANSFORMATION phase has no back and forth with the previous stages, and that means the regular development and test phase is very shaky as a TRANSFORMATION candidate. We all know that no matter how detailed specs are, there are always questions.
Also, spec is very shaky as the INPUT, because even we implement processes to lock specs, they still change, and for good and important reasons.
Now - some will argue that the reason for needing to ask questions and unable to lock specs are poor practices and processes, but they are overlooking a very fundamental alternative reason because they continue to think in construction metaphor:
Asking questions, going back and forth, and continue to change spec are all characteristics of the design process, not build process.
In other words, what we witness as problems are actually characteristics of a design process. If development and testing naturally fit into the TRANSFORMATION phase, we will not have these problems.
The only process in software development that meets the criteria as defined for TRASNFORMATION is the time when developers compile (and link) their source code into the binary physical form, which coincidentally often colloquially referred to as the "build process"
When one is compiling their source code, source code is the INPUT, and the compilers can take the source code and transform into the corresponding 1’s and 0’s without further input from anyone.
Quick summary for software:
- TRANSFORMATION is the build process, i.e. compiling and linking
- OUTPUT is 1’s and 0’s
- INPUT is the source code
The Implication of Software Build Process
This process is very fast - often seconds or minutes. Sometimes it might take hours, which seem forever, but compared against months in building constructions, all software builds are nearly instantaneous. In other words, it is extremely cheap to build software - it is so cheap that it is almost free. During the course of a project, the source code can be compiled hundreds or thousands of times. Try to do that with building constructions - one have hard time to create even hundreds or thousands of building models, and these software builds are real, i.e. they are not models, although they might not fully work.
But software projects of course, can be very expensive, and if TRANSFORMATION is very cheap, where do all the rest of the cost go? Of course it goes to design 
That means software is very expensive to design.
The above is actually very easy to understand, but difficult to explain and grasp when we continue to live in the construction metaphor, even if one is a developer.
As Jack Reeve’s seminal article What Is Software Design points out: [1]
The overwhelming problem with software development is that everything is part of the design process. Coding is design, testing and debugging are part of design, and what we typically call software design is still part of design. Software may be cheap to build, but it is incredibly expensive to design.
If we think from this perspective, many "problems" can easily be explained:
- Why developers and testers will continue to ask questions even with a very detailed spec - as detailed as specs are, they are not the complete design document and will contain holes. And as the coding phase continues, developers and testers are likely to uncover problems with the spec
- Why we cannot lock spec during the "design" phase - because the design is not done until coding and testing is done. As the code is being written, it is likely to uncover problems with the earlier design that need to be revisited
- Why people do testing and debugging cycle rather than upfront validation - because it often is cheaper to just create a build and test, and also because testing is validation, although not the only validation (Jack’s article has more detailed treatment of this particular topic)
- Why developers cannot create perfect code in first try - because coding is also designing, which is never perfect in the first shot
- Why we cannot do away with testing - because testing is part of the design process - its role is to validate
What Is The Takeaway?
Software is expensive because of design, not build. If you optimize to reduce the cost of the "build phase" as we understand in the construction paradigm, you are reducing the quality of your software (You might believe Offshore will give you advantage on cost without reducing in quality… you are right, under certain circumstances. And if you do not know what the circumstances are, then you are very likely to be proven wrong. We will talk about offshore in a different article).
This means that when you spend software money, you need to think thoroughly on what you get out of it, because it can be a very expensive endeavor. If your focus is on cost saving, there is not a lot of room for that in the world of software to optimize the build, and if you save on design, then you likely get crappy software. Instead, it is better to buy in this case. Let someone else carry the cost of designing the software, and you just pay for a license. You might think it’s expensive to buy, but trust me, it is cheaper than you do it yourself unless you do it for a living.
And if you are focusing on creating value through software, well, I trust you already know what this article is about
[1] Jack’s article basically started the whole agile programming phenomenon, where the mantra is "source code is the documentation". I had once sent that article to my manager, who is "business" and great in taking inputs. He does not understand it. I have other business folks reading Jack’s article too but it just does not seem to jive with them. This article tries to explain some of the semi-assumed point in that article. I hope this article works better with business folks because they really decide the fate of IT projects, but only time will tell.
Digg This!
Reddit!
Del.icio.us!
24 responses so far ↓
When you’ve completed 20 stories of a building, you don’t have someone come over to you and ask “we’d like to move the building to the left by five feet. Is that a lot of work?”
;-)
The first bridge builders may have built plain simple bridges without any specs/tools, and as they learnt and innovated in building those first bridges, they could consider it as design activity.
Having said that, I think any effort to separate between design and coding is subject while not necessary. At the end of the day, you must acknowledge the difficulties of the tasks required to create your product, regardless of whether they are part of designing or building. I maybe wrong and would love to hear reasons why this separation is beneficial for developing better software.
Thanks for the comment - you’ve made some great points, and below are my thoughts:
While I understand what you mean by “design has nothing to do with code or UML or anything”, they are the deliverables of our design effort, same as blueprints, i.e. code is NOT the deliverable of a construction effort. The software is. IMHO the distinction is important.
If spec can generate code, then the spec IS the code. We used to “code” in machine language, then assembly comes along to generate machine language, and then 3rd-gen languages to generate assembly languages and machine languages. If MDA can generate a full rep of 3rd-gen language then we would just “write” in MDA, so MDA would just be THE new language.
30 years ago Fred Brooks already predicts that in the foreseeable future there are no silver bullets for software engineering, and there will not be an order of magnitude of improvements, because most of what’s left are the essence - designs. He’s pretty accurate so far. But of course no one can predict distant future so maybe we will cross a day that we can have true artificial intelligence to do design for us…
Why is the separation essential? Bottom line - People. Great software requires great people in all functions (process & tools only enable). Construction metaphor has the opposite effect - treating technical skills as commodity. IMHO, companies competing in the future need to heavily leverage technology, and that means they need right technical people. Realizing technical skills are not commodities is the first step to create a more hospitable culture to grow necessary technical talents for the future. Many people can speak to why they want to leave places that have no clue about how to grow technical people
Of course - many technical people create their own pigeonholes too. That will be the topic for future articles
Obviously, compilers aren’t people, but if they were, parallelization and cheap labor would be a given. As it is, compilers already are parallized, and they’re the cheapest labor there is.
Architects, like software engineers, can’t be paralleized, and can’t be outsourced.
Brooks’ idea is great and should be read by all engineering managers. I can recall one of my earlier boss once told me that “You guys are very fortunate, Java is damn easy to code with compared with those languages of my time”. He just completely missed the essence in Brooks’ “No Silver Bullet”.
- Why does software estimation suck? Because people usually estimate the specs, not the code
Bravo - you are right. Hope more people see it exactly the way you describe it, instead of the current belief that programming is a construction activity.
Nguyen -
interesting concept. Can you elaborate a bit more? Are you saying that if we are able to estimate from code we would be A LOT more accurate?
In software, we do not estimate construction (while necessary in building construction), we estimate the design - and since software design is hard and takes so much time (we only finish design when we finish coding) while there is usually a pressure on quoting an estimate (e.g. outsourced projects etc.), software estimation will always be a black art.
http://www.amazon.com/Software-Estimation-Demystifying-Practices-Microsoft/dp/0735605351
Probably most fundamentally, the construction workers have an idea of what is going on, and there are people whose sole job it is to wander around the building site, watching the construction workers and ensuring that what is realised is the intent of the engineers, architects and designers - effectively the design is being refined / tweaked / tested and fixed DURING the build. My compiler doesnt do that
Contrived Example time:
A standard door specified to a basement utility room (marked “utility room” on the plans) that wont let any large equipment in? In an office block that is picked up, admittedly usually about the time the workers have finished installing the door and gone for smoke break, but it is almost always picked up DURING the build - rather than say, the day after the building opens. The design was not obviously flawed - the room has a door, it is just not big enough. In hindsight it is obvious, and experienced engineers will look for these things. But it still happens - that is the point of foremans and engineer inspections during construction. “Bill, this is the A/C room right? How are we going to get A/C units in that door? Lets make these doors double.”. Problem solved.
In software, something like:
/* overflow.c - demonstrates a buffer overflow */
#include
#include
int main(int argc, char *argv[])
{
char buffer[10];
if (argc
is an example of something that should be picked up at design time, but often isnt. The compiler should see the buffer might overflow, and ask you politely if you want to change it, or use some of it’s own smarts to fix the problem with it’s own initiative. Admittedly these days many compilers warn of this sort of thing, code optimisers can find these, and so on, but many developers ignore these problems, and there are many more subtle issues that may pass through undetected by today’s cleverest compiler. Especially problems with logic.
Pie in the sky dreaming time: Perhaps, if the code were being compiled by intelligent agents, or at least with intelligence overseeing the process, anytime a potential flaw was found, the compiler could notify the software engineer, modifications be made during the build, and so on… Higher level languages presumably would give the compiler ‘more’ initiative, etc.
Unfortunately this to me sounds so unfeasable as to be practically impossible with today’s technology, and it would probably have dramatic effects on the time of production of software, and still not prevent poor software from being written, as the poor compiler would presumably have to submit to the will of the developer. And that is ofcourse another issue altogether - anyone can write code and compile it, and while anyone can design a building only a select few people can pursuade construction companies to use their designs - usually these are people with architecture or civil engineering degrees. And even then we still get ugly buildings!
/* overflow.c - demonstrates a buffer overflow */
#include
#include
int main(int argc, char *argv[])
{
char buffer[10];
if (argc != 2)
{
fprintf(stderr, “USAGE: %s string\n”, argv[0]);
return 1;
}
strcpy(buffer, argv[1]);
return 0;
}
ripped off from:
http://en.wikipedia.org/wiki/Buffer_overflow
Good point. Yes construction workers are not robots and still uses their intelligences on the job, because there are many construction jobs out there doing something “unique” in the sense that the same building or bridge will not be built twice. In that case - it makes no sense to automate the tasks, because you reap no benefit from the investment.
On the other hand, there are houses built by automation - for example, mobile homes, or manufactured homes. Because the goal is to mass produce, automation makes sense.
What it comes down to is - if it makes economic sense, build CAN be automated for mass production, but not design. And in that case, most if not all human intelligence will be removed from the build effort, and we will have a factory. Yes, the dumb compilers are actually little factories
But as you said, compilers ARE getting smarter. People are definitely motivated to make it happen, so we just have to sit back and relax
thanks for the great comment
1) It is true that a lot of what most engineers have done are routine and has been done before (not necessarily by the same dev before, but I digress)
2) It is true that these works can be considered routine - I have certainly considered them so in the past
3) but that does not mean software is construction work and not design work
Let’s continue to use the same building construction metaphor. Most of the houses/buildings differ from each other in small ways and have a lot in common, even skyscrapers. Building architects employ the same same “design patterns” day-in and day-out and doing a lot of routine stuffs. By the same logic they MUST be construction workers right? Of course NOT, and they will be the first to tell you so
People are forever trying to look for similarities between construction and software engineering that they can no longer see the differences, and it is not surprising the conclusion is that software developers are construction workers because they do a lot of similar stuffs day-in and day-out.
Well, it is true that there are a lot of routine work in software engineering, but so are the other engineering disciplines. Engineers must build on building blocks, but engineering/design is the effort putting into arranging the building blocks to make it work. And that is the case with software developers.
I would agree with you that there are developers out there who are little more than construction workers - they need others to give them the pseudocode in order to do any work, and you will notice you might be better without them.
If we have to use “whether the work is routine” as a measuring stick, then this argument will not get anywhere. 90% of ANY work in this world IS routine. But it is the other 10% that makes the difference.
A better measuring stick is - if you HAVE TO AUTOMATE the effort, CAN you? YES if you are talking about construction work, but not engineering work. That is the difference.
Lastly you are correct that doing the whole analysis/design upfront is a massive undertaking. That is why it makes no sense to treat developers as construction workers. It would be far more economical to have developers participate in designs, than to have the whole effort separated and hire code monkeys who can do no more than trasnliteration of pseudocode - they will get it wrong and you would have to redo the work anyways
For differences, construction is much older, more precise and better organized. Because it is more visible and mistakes are more obvious the state of construction is far better than software development. Software is usually a massive undertaking, but it can be built in smaller phases (iteratively) than construction (but you can build the different wings of a building at different times). Using an iterative design in software, you can start development without knowing how it will end or what you are doing, but you’ll need to accept that you’ll have to toss lots of code, when the direction changes. You can do that in construction, build part of the structure, tear it down, then rebuild it, but your financiers will think you are crazy. Finally, the different professions that work on a construction site probably complain a lot less about their jobs than do the people working on software development.
You said: Both software and construction require a creative architectural phase, and a more routine building phase.
Very few designs in any discipline is world shaking, and most are routine. But just because work is routine, that does not mean it’s construction and not design. Developers have a lot of routine design work, not routine building work. The construction in software is “compilation”, not writing code - that is something most people, even many developers, do not realize, because they are STUCK in the construction metaphor.
Your analogy is well thought out.
I understand the need for analogy, but this analogy IMHO has already stopped being useful. The fact is, there are *more* differences between software and buildings than there are similarities. You can duplicate software as often as you want - try that with buildings. You can morph the software any way you wish - try that with buildings. You can compile ten thousand builds (all real software, not models), try that with buildings. Or for that matter, anything physical. The financiers is smart in this case - you HAVE to separate design from construction for buildings.
The reason you can do so with digital medium is because the construction phase is so short that it is almost *instantaneous*, and the rest is design (even routine). That is a software property many do not appreciate or take advantage, because they are stuck in the construction metaphor. Now that I think of it, most people (especially the biz folks) probably think coding is construction, because compilation is invisible to them.
It used to be one can take years to release software, but that pace has shortened. The more one emphasize on creating the artificial separation between the phases, the more one would lengthen the cycle and/or reduce the functionality delivered, and they are just not taking advantage of the brain power they hired, but they would not know that because they still think in construction metaphor
I think that sense that software can be morphed and that development is speeding up is misleading. Its hard to change the course of development once it gets going, much like the titanic, big projects steer slowly. You can write a small piece of code quickly, but that’s only 10% of getting it out there to be used by people. It must be tested, packaged and distributed. A