Software Pain and Pleasure

Un-writing code

2006-07-23T11:17:00.000-07:00

Recently I got a chance to clean up a real mess in our system-monitoring code. Our application was starting to run very slowly when hundreds of users were connected to a certain service. This was puzzling, as the service was intended to handle thousands of users simultaneously -- why was it bogged down with such a small load?

This had been a nagging problem for several months, and many developers had gone into the system to try to fix it. Some improvements had been made, but we were nowhere near the needed level of performance. The general consensus was that we needed a bigger server -- thow hardware at the problem, as the software was just fine.

I spent a few days reading the code, then talking with developers. After some simple code changes, the system was easily able to handle tens of thousands of users, without having to upgrade the hardware.

What was the problem, and why hadn't it been discovered earlier?

Here's what I found -- you will easily see how the problem developed ...

The first developer to work on this system wrote some prototype code that efficiently handled twelve users. He had designed a simple linear array of user-records, and when a certain operation needed to scan through the list of users, that table got built, sorted (using a bubble sort), and then traversed to make the report.

A few weeks later, a different developer came in, recognized that the code was only able to handle twelve users, and rewrote the code to use an array of arbitrary size. The rewrite allowed the array to be created when the report needed to be made. The array would be created with whatever length was necessary, then the contents of the array would be filled-in, the array would be sorted, and the report would be generated. Note that the bubble-sort method was retained.

After that, a developer rewrote the code yet again ... this developer apparently felt that fixed-length arrays were evil, so a doubly-linked list was used. Bubble-sort was retained, but the comparison function was re-written so that more fields of the user record were examined.

After this rewrite, the code seemed to scale arbitrarily -- the problem, as you can guess, was that the bubble-sort caused the performance to lag noticeably.

This lag prompted the next set of changes -- somebody noticed that the system was taking too long to produce the report. The front-end webpage which displayed the report was not getting its results fast enough from the back-end code we're discussing here, so the front-end webpage code was timing out and displaying an empty report.

The next fix was to lengthen the timeout: wait five seconds, instead of one second, for this particular report to be generated.

And this is where I stepped in, to fix the whole mess.

First, I noticed that the web front-end applied its own sort method to the result set, so there was no need to do back-end sorting -- the code which generated the report could just traverse the user-list, and pass an unsorted array of user records to the web front end. Problem solved.

Well, almost solved. I went back to each developer who had touched the code, and gave them a thorough description of how poorly the entire project had gone -- and reinforced the need to really understand the problem before writing code.

And thus I was able to save the project, just by removing a whole bunch of un-necessary code.

MSProject is Evil

2006-06-01T10:53:00.000-07:00

OK, here's my Microsoft Project rant. But before I get into the profanity, let's look at the problem that MSProject tries to solve ...

Software scheduling is hard. It is provably harder than software development itself. How can I say that? Well, if software development was a fully solved problem, it would be predictable, quantifiable, and most importantly, schedulable. Software development, especially the innovative software engineering that drives most high-tech companies today, is inherently unpredictable. You don't know how long or how difficult the development process will be until you are finished, and then you can look backward and see how long it took.

One way to do predictable software development is to exhaustively analyze the problem before the development process begins. A team can study, debate, and document the project before the first line of code is written. This team identifies all of the hazard areas, proposes ways to address those hazards, and writes a full specification of the project and the development process.

How long will this analysis phase take? You don't know. So, we're back to the original problem: until you know the difficulty of the project, you won't know how long the project will take. How long will it take to understand the difficulty? Nobody can predict that either.

Enter Microsoft Project.

MSProject provides a framework for breaking down large problems into smaller problems. Each task can be assigned an expected time duration, start date, and one or more "resources" (i.e. developers) can be assigned to the task.

Once an MSProject schedule is created, you have a picture of the future -- you know how long the project will take, who is involved in the project, and what everyone's role will be for the remainder of the project.

The problem is that this schedule is completely false, because it is based upon faulty assumptions. For example, the schedule shows that Bill will be working on the "Faucet" class from August 7th to August 10th. It is now June 1st. If everything works correctly and everyone does everything according to the schedule, then indeed Bill will start working on "Faucet" on August 7th.

But Bill will never get to do that. Guaranteed. Bill will have to fix a bug that comes up on August 6th, or Sally won't finish the "Handle" class which is a predecessor to "Faucet", or Jim will decide, in July, that a new set of requirements needs to be fulfilled and the entire schedule has to be rewritten.

The fundamental problem with MSProject is false accuracy. MSProject allows you to believe that the Faucet class will take four days. What is that based upon? Guesswork. Hunches. Not science. The net effect is that MSProject attempts to apply metrics to what is inherently un-quantifiable.

What is the alternative to MSProject? Agile development, plus a healthy dose of maturity and restraint, at least for the sort of cutting-edge projects I've worked on. Life at the cutting edge means unpredictability. Don't try to squeeze open-ended projects into a fixed schedule, unless you've allowed for the unpredictability via wide error bands.

MSProject can be effectively applied to "cookie-cutter" projects, where the development effort involves small modifications to existing code, or replication of work previously done. In those cases, you can quantify the effort needed to do the work, and you can do it with some accuracy.

MSProject is a way of automating GANTT charts, and GANTT charts came from the construction industry. The construction industry has been using practices, materials, and methods that have been refined for hundreds of years and typically involve replication of known practices, not cutting-edge exploratory methods.

So leave MSProject for those environments, and don't try to apply precision and accuracy to unquantifiable activities.

End of rant.

Brooks' Law and debugging

2006-04-09T16:24:00.000-07:00

Fred Brooks' book "The Mythical Man-Month" has many gems, the brightest of which is the aphorism "Adding manpower to a late software project makes it later."

And, there is what Eric Scott Raymond calls "Linus' Law": "Given enough eyeballs, all bugs are shallow."

Is it possible to weave these two principles into a method to expedite software development? Sure -- employ a mob of debuggers. Debugging (and, to an equal extend, QA testing) benefits from scale. Once primary development of a software project is complete, it is possible and profitable to have a large number of folks banging away at the application and reporting bugs. You can also have a squadron of engineers peering through the code looking for the flaws that lie behind the bugs. And, these folks can write up their discoveries for review by the original developers.

In a future column, I'll talk about the debugging process in detail, but the above process does work. Often, the squadron examining the code can be junior developers who are learning the structure of the code. Or, you can have your wizened geniuses participating: either way (or in mixture), you're subjecting the code to a peer review which may not be possible or convenient during the original development cycle.

If you're Microsoft, just call it a "beta test", and give your product to tens of thousands of motivated users and developers. Or, call it a "technology preview". Just don't call it an "eternal beta".

Crystal Clarity

2006-03-18T14:03:00.000-08:00

In most American high-end department stores, there is a housewares department with a section that sells fine china and crystal. The display of crystal drinkware, vases, and other glassware is often the most radiant location in the store. The exquisitely-cut goblets and candlesticks are carefully arranged under high-intensity halogen spotlights, with a matte black backdrop providing a dramatic contrast.

The image is of precise, almost mathematical purity. The eye is dazzled by the reflected and refracted light. You want to transplant these sparkling objects to your own house, and let their light shine in your everyday surroundings.

I feel the same way about code: textbooks show algorithms implemented in clean, clear fashion and express a purity of thought that I love to transport into my commercial and hobby programs. Further, I want to clean, polish, and refine my own code so that it expresses the same clarity as the textbook examples.

That's why I've enjoyed applying Martin Fowler's techniques from "Refactoring" into my own code. Intentionally or not, he shows how code can be refined, buffed, and polished so that it shows the same simplicity and brilliance as a vase by Waterford, Swarovski, or Steuben.

The Peter Principle

2006-03-09T15:08:00.000-08:00

"The Mythical Man Month" is one of the Ancient Tomes of software engineering, revered for its wisdom. Another book which should be canonized is "The Peter Principle" by Laurence J. Peter.

Peter's book describes the inevitable consequence of modern organizational dynamics: "In a hierarchy every employee tends to rise to his level of incompetence."

How can we apply this maxim to software engineering? Peter explains the inefficiencies and irrationalities of bureaucracy with this principle, but we see similar effects in any mature body of software. Without ongoing scrutiny and diligence, software-maintenance efforts create results that are bloated, unclear, and poorly-organized. This isn't the fault of the software itself, but rather how it is designed, written, and maintained.

Often, the first version of a software program is written in haste, to solve a particular problem. Over time, the problem-definition changes and broadens. As that happens, the software expands to solve the larger problem. If this is done carefully, the result matches the problem definition and retains clarity. But, so often the result is exactly the opposite: ongoing casual bugfixing, hasty extensions, and inattention cause the program to become riddled with inefficiencies, redundant code and unused code. The result is software that is performing far below its potential.

Organizationally, the Peter Principle ensures that a collection of humans will function in a less-than-optimum manner. In software, the Peter Principle ensures that code will contain bugs, is inefficient, and is difficult to understand.

How to escape the Peter Principle? A much-less popular book, "The Peter Prescription", outlines some remedies. In the realm of software engineering, the current vogue is extreme programming, which via oversight, test cases, and a mood of continual monitoring and improvement attempts to keep the code clean.

Antibodies

2006-02-27T14:19:00.000-08:00

The old saw that "bureaucracies grow to fill the need of a growing bureaucracy" means that eventually your organization will acquire a few (or more) antibodies.

What's an antibody? I first heard the term at Sun Microsystems, referring (in a derogatory fashion) to chair-warming bureaucrats who are resistant to change. Change, to an antibody, represents danger, rather than opportunity. Hence, the antibodies swarm around the new idea and attempt to neutralize it. With the new idea succcessfully destroyed, the antibody can return to its relaxed, inactive state.

At Adobe Systems, we used a different term: the "yeahbuts". A yeahbut, upon hearing a new idea, would respond with "Yeah, but ..." and then a set of canned reasons why innovation wasn't practical, prudent, or necessary at this particular time.

The problem with antibodies is that they're hard to get out of the system, short of simply firing them. So often, the antibody will cloak their resistive actions in a set of plausible justifications:

"We haven't completed our analysis of the product requirements, so it is really premature to commit to this course of action."

"My staff is fully occupied with maintenance on the most-recent release, and we haven't had time to review your proposal."

or, even more insidious, the trap of false comparatives:

"Your plan to put out mouse traps sounds very similar to Pete's proposal to fumigate the flooded toolshed, and we simply can't undertake the poison-abatement process with our current staff. Therefore, I won't approve your request for new mouse-traps."

Antibodies find strength in numbers, and it isn't hard to find them clustered around the drinking fountain, discussing tactics for resisting the latest "change agent".

This year's must-attend conference ...

2006-02-07T11:00:00.000-08:00

Let's all go to Waterfall 2006, and use the processes of yesterday to build the products of tomorrow!

Using the Wrong Tool for the Job

2006-01-06T13:47:00.000-08:00

I've collected a few amusing examples of good intentions gone bad. In these cases, folks are using the wrong tool for the job ..

Misuse of Microsoft Excel: Would you use Excel as an outline processor? There's no good way to represent a hierarchy using this grid-based tool. Even with variable-height and variable-width cells, it just looks wrong. Use a proper outline processor, people!

Misuse of PowerPoint: I've seen the line- and box-drawing tools used to prepare elaborate organization charts, network-flow diagrams, and other two-dimensional graphics illustrations. No dice: get a copy of Visio or Adobe Illustrator!

Excel as a Meeting Agenda Organizer: Rows of text. Meeting agendas. Sound like the same thing, right? Well, when you see somebody painfully adding/removing/copying/moving elements of a meeting agenda, especially during the meeting while everyone is waiting impatiently, you'll realize why tools were developed just for that purpose! If you don't want to get a custom tool, at least use an outline processor!

Misuse of HTML: Folks are still editing HTML documents in a raw text editor (such as Emacs, VI, or Notepad). This is the twenty-first century! HTML should be a generated format, not a source format

Using Excel (or Word, or other document-creation tool) to track software bugs, feature-requests, help-desk issues, etc. There are low-cost (even "free", if you count Bugzilla) products that solve this problem much more elegantly.

Corollary to the above: using a bug-tracking tool to manage development tasks or other activities with a high degree of inter-dependence. Foo! Use a tool built for the purpose. Microsoft sells something called "MSProject", which is almost adequate (it lacks groupware capabilities, the topic of another rant)

Considering the realities of human nature, I expect to add more to the above list as time goes on.

Finally, one must take a look at Peter Norvig's rendition of the Gettysburg Address in PowerPoint.

Quick sip of common sense ...

2005-12-24T07:38:00.000-08:00

Here's a reminder of some fundamental principles of computer programming.

Fully-developed bureaucracies ...

2005-12-20T20:46:00.000-08:00

A bureaucracy grows because one of the functions of a bureaucracy is to divide up responsibility. If one person is the expert in a field, or the sole source of knowledge in an organization, the organization is in danger. What happens if that person were to leave or suffer from an extended illness?

So another person is hired to fill the same role, creating a deliberate redundancy. Then, you have two people of perhaps similar skills and interests. These two people need to have a communications system, so that they are aware of each other's thoughts, actions, and beliefs. Paperwork is created. Then, either of these two people have to communicate with others in the organizations, hence meetings. Meetings expand to fill the available time and the available set of topics. One person can't attend all of the meetings. So, more people are hired in order to attend the meetings and represent the interests of the original person (who is otherwise occupied with her preparations for another meeting).

In time, the worker is either attending a meeting or preparing for the next meeting. Decisions are made in meetings, as that's the only place where consensus can be struck. Nobody can, or should, make an independent decision, as that would possibly result in an action that wasn't approved by all "stakeholders".

In a fully developed bureaucracy, every individual is redundant. But everyone is essential, as decisions must be approved by everyone. When one person isn't available, progress must stop. The geometrical expansion of a bureaucracy is usually limited by the salary money available.

I've found fully-developed bureaucracies in moderately-successful companies, but never in un-successful or wildly successful companies. Why is that? I believe that the unsuccessful companies never develop enough financial resources to hire enough individuals to build a bureaucracy to the crucial self-sustaining level. And extremely successful companies are usually staffed with a few very clever (or accidentally successful) people. Clever people won't put up with bureaucratic nonsense. Accidentally successful people aren't secure enough to share the success, so they won't hire people who will spend "their" money.

Age and Talent

2005-12-13T17:30:00.000-08:00

I recently listened to an NPR program "Science Friday". Ira Flatow was interviewing Alan Lightman, author of "The Discoveries: Great Breakthroughs in 20th-century Sciednce, Including the Original Papers".

A listener called in with a question, asking the relationship between age and talent in science. The common perception is that the great discoveries are made by people in their twenties and thirties. Lightman's response was interesting: he pointed out that writers and artists generally benefit from life experiences such as families, financial ups and downs, decline and death of parents. Such experiences enrich and add nuance to the works of novelists, painters, poets, and playwrights. But the work of scientists doesn't benefit from these events.

OK, maybe that's true. But I have a different perspective: experienced engineers and scientists may not be more capable of doing "the right thing", but I assert that they have a greater ability to avoid doing "the wrong thing".

Anecdotally, I've noticed this over and over throughout my career: trust the aged guru to point out the hazards of the path. She might not have the solution, but she'll warn you of all the mis-steps along the way to the solution.

Footnote: Richard Hamming talks a bit about age and creativity here.

Roadtrip software engineering

2005-12-12T17:36:00.000-08:00

When I was a teenager in the mid-Seventies, I had a copy of David Ahl's "101 BASIC Computer Games". Since I didn't actually own a computer, I'd read through the program listings and figure out how the programs work. Then, I'd mentally think about the code that I'd write, if there were a real computer for me to work with. A few years later, I was able to buy a Commodore PET microcomputer, and then all of those programs could be actually written and run.

Last weekend, I was driving my family through rural Ohio on a family vacation, and found myself mentally composing code again. This time, there was a laptop computer in the car, but my wife had banned its use. Instead of writing and debugging code on the laptop, I was writing it in my head.

Now that we've returned from the vacation, I can translate all of these mental notes into some code and try them out. Thirty years ago, it was BASIC on a microcomputer with 8 kilobytes of RAM. Now, it is Java on a gigaherz Apple laptop with a gigabyte of RAM. Funny, my typing speed is about the same.

How do you build a pyramid?

2005-12-05T20:14:00.000-08:00

I was talking "management science" with a friend today, and discussing company growth patterns.

His pithy comment was "You don't build a pyramid from the top down", referring to the need for a strong foundation for growth.

Hmmm ... obviously, you need somebody in charge of the growth, but I also believe that great engineering organizations have bottoms-up organizational structures, and many poorly-run engineering organizations are run in a top-down fashion.

But growth-wise? How about "from the inside out"? Meaning that you add more people at the top, when the foundation can support it, and you continue to add to the base as requirements (and the pool of qualified candidates) allows.

Organizations which are too strong at the top are typically too tightly controlled and generate unimaginitive products. Organizations which are too strong at the bottom can thrash around rudderless. Balance is paramount.

Confidence factor on estimates?

2005-12-01T21:57:00.000-08:00

According to the schedule, one of our projects is supposed to take another six months. Recently, somebody asked me how certain I was about that number.

"Oh, I'm about ninety-percent sure that it will happen", I replied, casually.

"What's your confidence factor?", asked my questioner.

I thought a minute, somewhat dumbfounded. A confidence factor on an estimate? Uhh, should I reply with a standard deviation? Some sort of error band?

"Oh, I'm about eighty-five percent sure that I'm ninety-percent sure that it will take six months", I answered gamely.

"Thanks ... I'll check back in a few weeks and let's see how it is going".

I was confused. What had I just said? A few minutes later, I realized that while my questioner was satisfied with the answer, I would have been more satisfied had I replied, "Oh, I think it will take five to seven months. Dead sure."

Just another surreal moment in software development.

Bug Synchronicity ...

2005-11-29T10:05:00.000-08:00

Have you ever noticed that a bug can lay dormant in a product for months (maybe YEARS), then all of a sudden it becomes manifest, everywhere, instantly, simultaneously.

I've often puzzled over this ... how can it happen that nobody, then everybody, sees a bug?

Recently, this happened again in a product that I'm managing. This time, I wanted to find out how it had remained unobserved for so long. We found that the bug had been present for about two years, but hadn't been revealed until a new bit of code had subtly altered the program flow. At that point, the original problem (a data corruption) started to really affect the system in a negative way.

I was disassapointed. I had hoped to find evidence of UFOs, gremlins, or astral convergence. In the end, it was a simple buffer overflow (scourge of C programmers everywhere).

What type of engineers do you hire?

2005-11-27T14:27:00.000-08:00

Recently, I wrote about the type of managers I like.

Let's discuss the type of engineers I hire. The best engineers don't just write code, they obsess over it. They live it.

How can you tell if you're interviewing a developer who is obsessed with code? Ask her about hobby projects -- the little bits of code written out of curiousity, as a hobby, or to solve a particular interesting problem. Ask her to send you a code sample, so you can take a look at coding style, organization, and clarity.

Over the years, I've seen some interesting hobby projects: code for digital control of a model railroad, code to organize mp3 files, and most recently, a particularly elegant solution to sudoku puzzles.

People who are motivated to write great software for themselves will typically be motivated to write great software for you.

I look for other aspects in a candidate, but el numero uno is an obsession with code.

Software Debugging vs. Product Maintenance

2005-11-26T12:51:00.000-08:00

What's the difference between debugging and maintenance?

You debug software, then you ship it. Then, you maintain it.

In most high-tech development environments, the "ship" decision is made almost arbitrarily: it may be a date on the calendar, it may be when the customer is ready to cancel the project, or it may be when the last high-priority bug is fixed. I even remember a release was made because the project-manager said she'd quit if we didn't ship the code -- so, the release was determined by PM pique.

Once the code has been delivered to an external customer, many development organizations hand off responsibility to a "maintenance" team. The maintenance team needs the same skills, the same levels of insight, creativity, and imagination as the original development team. The maintenance team is continuing the implementation: molding the software to fit into unexpected environments, shaping it to handle unanticipated situations, and generally tracking down the problems that the original team was too rushed to consider.

Ideally, the maintenance team is the development team. Often, that isn't the case, as the original development team is off trying to design and implement a new raft of features. But if you don't staff your maintenance team with people equally skilled in software engineering, your product will suffer. Just as submarines, nuclear reactors, and MRI machines aren't run by new college hires, don't assume that your multi-million dollar software product can be maintained by junior programmers. I've always kept my senior developers involved in the debugging process. Many times, the most experienced developer is the most effective debugger.

What type of management do you have?

2005-11-16T22:31:00.000-08:00

The best development houses have management structure similar to successful law firms and good big-city newspapers.

When I mention these parallels, folks usually scratch their heads in wonder. Maybe you are doing so now, so I'll explain.

In good law firms, you usually have a few senior partners who are rainmakers and try important cases. They are lawyers, not bureaucrats. They will sweat over briefs, dig up interesting facts for presentation to the jury, and show up at 8:30 AM to face the judge. Sure, they delegate, and sometimes overwork the junior partners, associates, clerks and secretaries. But, they're also in court on a regular and frequent basis.

In well-run newspapers, the senior staff are the editors: they oversee the daily operations of reporting, story selection, and editing. They run around making sure that the reporters are diligent with sources, double-verify all accounts, and behave in an honorable and principled fashion. When the time is right, they duck back into the office (or maybe to a local bar) and write a column or editorial.

Just like the senior law partner, the senior editor produces.

And, well-run tech companies have senior management staff that produce. The zenith of Silicon Valley companies (IMHO) was the "old HP", when Hewlett and Packard were in charge. They were a familiar sight in the lab, used their own products, and had a real sense of product design and customer support. When the professional managers took over at HP, the company's decline began. But Hewlett and Packard continued to invent, even in retirement. For another example, walk down Stevens' Creek Boulevard to Apple. Ask about the Sculley/Spindler years. Find out what happened when professional managers took over Apple. Then, ask about the renaissance when Jobs returned.

Moral: Hire an engineer to run an engineering company. Hire a professional manager to run a soap factory, a soft-drink company, or an advertising agency. Let the editors run the newspapers, and let the senior partners sit in a courtroom from time to time.

Rules are made to be broken

2005-11-10T06:52:00.000-08:00

We recently came up with a new "engineering process" document, listing all of the steps and milestones along the path from product conception to release.

Since this was a committee effort, the resulting document was huge. Everyone got to put in their favorite little pet idea, and there was very little effort to simplify, streamline, and clarify the result. After all, committees just want to get the job done: there wasn't much desire to offend somebody by killing off any personal proposals.

Some committee-members were very intent upon reining-in the unmanaged developers. So, the process document requires specification reviews, design reviews, code inspections, verification passes, unit-tests, QA inspection of unit-tests, test-plans written by both the developer and tester, test-plan reviews, spot-checks by QA management, feature-content verification by program management, signoff by the project manager and an extended beta period for shake-out at a customer's site.

Actually writing the code was a very small portion of the resulting development schedule. The assembled committee wants most of the developer's time to be spent writing, reviewing, discussing, editing, integrating, inspecting, reviewing, and commenting.

Which, of course, is no way to run a railroad. Our company makes money by selling software, not processes or documents. So, I made sure to insert my own pet idea: a codicil that allows a development manager to waive certain requirements if deemed necessary.

OK, I'm now fine with the document. The junior engineers can spend all of their time in review and edit sessions, so they can learn the wisdom of careful, methodical code development. But the senior folks, the wizards who end up writing 95% of the code anyway, are exempt from most or all of the process foolishness.

The process princes are happy (lots of review meetings), the truly great engineers are happy (they get to work unfettered by the committee's rules), and the negligent or junior engineers are watched carefully via the various review items. This is perfection.

Single-author documents ...

2005-11-09T06:45:00.000-08:00

Recently, I was discussing project-management tools with some engineering managers. We discussed the various pros and cons of Microsoft Project, and decided that we'd go with a simple Excel spreadsheet, with employee names on one axis and project names on the other axis. Data cells would contain the percentage of time that an engineer would be spending on the project. A new copy of the spreadsheet would be generated every calendar quarter.

Every project manager would have to update the spreadsheet with information about their group. We use CVS for document control, and we'd have to make sure that two people weren't editing the file at the same time. So, we'd need to agree, via email, who was editing the file at any given time.

This is wrong in so many ways. First, the single-author restriction is crippling. Any time you want to edit the document, you have to check with everyone else to see if the latest copy is in the repository. Then, you have to make sure you have the latest copy, and then edit it and promptly check it back in.

CVS excels with text documents: there is a automatic merge system that can reconcile the efforts of multiple concurrent users. (the "C" in CVS stands for "concurrent"). But with binary files from Microsoft Word, Microsoft Project, etc the automatic merge function won't work. Text is easy, binary is almost impossible.

Most of my documents are written in plain ASCII text for just that reason. I can read them everywhere, I can edit them almost everywhere, and I can email them with impunity. CVS will maintain an edit history, so I can view each change if necessary. Text documents are small, efficient, and I write in a terse, concise style, without having to worry about font choice, text sizes, etc.

Second reason why the single-author form is lame: it restricts the number of people who can contribute information. My staff members can't update the file, because that creates even more opportunities for edit collisions. The whole paradigm of a single document with a single author is broken. The approach does not scale up as teams grow and more managers are added.

What we really need is a technology that inverts the model: one that allows incremental updates by a wide variety of users. The Web, with its client-server approach is ideal. A friend showed me two products that are hitting this target: CopperProject and TeamInteractions. I'm going to check them out ...

Virtual Consumerism

2005-10-30T13:31:00.000-08:00

I recently took a bunch of phonograph albums down to Amoeba Music. The Amoeba used-record buyer gave me about thirty dollars in store credit, which I used to puchase some CDs.

When I got home, I ripped the CDs into my Macintosh and loaded the tunes to my iPod.

My iPod mostly contains ripped CDs, however there are about fifty songs that I've purchased from the iTunes music store.

The CDs I can sell - on eBay, via Craigslist, or at Amoeba. The iTunes songs -- no sale. They are licensed & locked to my particular iTunes account, and transferring them to another user is possible but is a real headache.

Thus, the resale value of the iTunes songs is nil. In the near future, when I'm buying videos, video games, audiobooks, and (someday) immersive holographic participatory adventures, the resale value will also be nil.

The record companies (and in the future, movie producers etc) love this. They don't have to worry about a secondary market. Amoeba Records will be out of business. Furniture stores won't sell CD racks or bookshelves for DVDs. Phonograph albums are obsolete. CDs and DVDs will soon be obsolete. I guess this is progress of some sort.

Seed corn ...

2005-10-30T08:40:00.000-08:00

Bob Cringeley's article Changing the Guard discusses what will happen when the baby-boomer generation of engineers and developers goes into retirement. Bob proposes that open-source projects will benefit from the retiree's experience and wisdom.

There's another question that isn't asked by Cringeley: who will replace this generation of skilled engineers?

I visit many Silicon Valley tech companies as part of my current job, and I see an increasing reliance upon outsourced development. Junior programmers aren't being hired in the Valley. Instead, companies are outsourcing software maintenance and bug-fixing to contractors in India, China, Russia, and Eastern Europe. Cost-savings are typically cited as the reason.

The pros and cons of outsourcing have been fully discussed in Forbes, Fortune, and other business magazines, and the topic was an issue of (some) debate in the 2004 U.S. Presidential election. We won't rehash those topics here, today.

Getting back to Cringeley, I think that companies aren't investing in the next generation of U.S. programmers. Instead, the mentorship, training, and teaching is going to a new crop of engineers in Hyderabad, Taipei, and Kuala Lumpur. As the knowledge flows overseas, and isn't replaced locally, U.S. companies will be faced with a talent deficit that is overwhelming.

Now, outsourcing isn't universally practiced, and even the companies who are currently outsourcing haven't stopped hiring locally. But the trend is accelerating: we are now training team leads and mid-level managers in the remote development site. The rate of local growth is slowing, the rate of remote growth is accelerating. The trend is clear.

Is this a problem? It is, if you think that one of the advantages of the U.S. is our world leadership in technology. Largely inspired by Sputnik and the Space Race, we've developed a huge technical advantage which has had wonderful economic benefit to the United States. Can we afford to give that advantage away? Only if we're ready to replace that advantage with something even more miraculous: gene therapy is certainly one tantalizing candidate.

We're giving away our advantage in software engineering, simply because we're not replacing the skillsets accumulated by Cringeley's coder generation.

Tom Friedman's book

The World Is Flat: A Brief History of the Twenty-first Century covers these topics far more eloquently. Recommended reading.

Grid computing misses the point ...

2005-10-27T10:59:00.000-07:00

I've viewed the Sun Grid announcement with some skepticism. Here is the original BBC article describing the service.

Having dealt with the need for massive computing resources (verilog, etc) before, I can understand the appeal of their product -- I don't have to purchase & maintain dozens or hundreds of servers which will be obsolete from a technical and financial perspective in three short years.

But Sun doesn't realize that the blocker is data. I don't want to send my Verilog off to a remote location. Security is paramount, thank you very much. I don't want my circuit designs to leave the building.

And Wall Street firms probably don't want their market-analysis algorithms out of the building. Pixar doesn't want to ship their gigabytes (or terabytes) of rendering data to a remote data center.

So, I'll continue to buy blade servers, multi-core CPUs, or whatever is the best ratio of mips/dollar/btu.

Internet-connected change machines?

2005-08-26T16:23:00.000-07:00

No treatise on software engineering today ... here's something in a lighter vein.

Yesterday, I took a bucket full of spare change down to the local Albertson's grocery store. I wanted to use the CoinStar machine there to take the coins and give me a Starbuck's gift card, as promised by the advertising billboards I see every day.

The CoinStar machine gave me many more payout options, including a credit certificat for Amazon.com. Since I spend more money at Amazon than at Starbuck's, that's the button I pressed after all the coins went jingling down into the belly of the machine.

A minute later, I was walking out the door, with a printed receipt including a special credit code. Ten minutes later, I typed that number into an Amazon webpage using my home computer. The amount was then credited to my Amazon account.

Boggle. The CoinStar machine must have been connected to the Internet. In days past, credit transactions such as this would mean filling out a form, mailing some paperwork, waiting days or weeks for a credit statement, then including that credit statement in the next transaction with the merchant.

So, I was at the Albertson's grocery store again today. Yes, there appears to be a network cable going into the CoinStar machine. And, the Coke machine seems to have a similar connection ... does the soft-drink distributor fetch an inventory report from the machine daily? Is that how they schedule their refill deliveries? Once there was an Internet-connected Coke machine at CMU. Now they are everywhere.

P.S. The credit code was over twenty alphanumeric digits. I recognize that it may have been an encrypted encoding of the credit amount, along with encrypted authentication information, but that doesn't make such a good story, does it?

There is a little bit of NASA in every tech company

2005-08-23T01:39:00.000-07:00

The Columbia disaster, like the Challenger explosion before, put the organizational inefficiencies of NASA into embarrassingly public view. Precautions overruled, advice ignored, decisions made without full consideration of risk or cost.

And, as of August 2005, there is no real date for when the Shuttle will fly again. NASA is saying that it won't be until spring of 2006.

The Shuttle is like every large software project -- it is more complex than its creators or current maintainers can comprehend. Unforseen consequences abound, subsystems interact in unforseen (and perhaps unpredictable) ways.

In the Eighties, chaos theory arose to classify the unexpected consequences arising in complex systems, typically as the result of feedback loops that magnify a fundamental uncertainty in the system.

Large software projects develop a similar sort of chaos: engineers pile quick bugfixes atop hurried modifications or casual redesign create, at first, sloppy code and eventually a system that behaves unpredictably and often surprisingly.

Organizational imperatives: market pressure, overly-optimistic schedules, and aggressive promises drive haste and hurry, resulting in poorly-designed or poorly-implemented software systems. At NASA, "go fever" was the term that described the decision-making process that allowed mid-level and senior managers to over-rule and minimize the precautions urged by the designers and technicians.

The same "go fever" exists in every high-tech company. And while the results aren't as spectacularly fatal as those arising from NASA's bad decisions, they arise from exactly the same human faults. We can't fault NASA for the tendencies that we follow every day.

Oh, and the Challenger wasn't the first time. Read Andrew Chaiken's "A Man On The Moon" for a description of how NASA's similar tendencies led to the death of three astronauts in the Apollo 1 fire.

The law of unintended consequences is inescapable.

Little Victories

2005-08-14T09:31:00.000-07:00

So it was Friday afternoon, and I was finishing up a stressful week. Too many meetings, too many bugs, and not enough sleep. Since it was only 2:00, I couldn't leave -- it wouldn't look good to the boss and troops.

Time for a Little Victory! An LV is a small bug that can be fixed with maybe 30% of the brain. A bug with enough challenge so that it is interesting, but not so complex that the fix is risky or likely to take more than a few hours.

Scanning the buglist, I found one: a bug that could be knocked out quickly with no chance that I'd break something else. With the door closed, iPod rocking, and one last cup of espresso, I got to work.

When fixing bugs, I spend most of the time reading existing code -- looking for weaknesses, trying to understand the mindset of whoever wrote the code, and figuring out the best way to fix the problem. This case was no different: for two hours I sat there, Peter Gabriel on the headphones, reading about a thousand lines of code.

At 5:15, I was writing new code and simplifying old code. At 5:45, I was verifying with unit tests, and by 6:00 the code was running in a "real" test environment. At 6:30, the new code got submitted to the repository.

Five minutes later, I was grinning. All the tension of the week drained away, and Friday now looked much better than it had several hours ago. I updated the bug-report status: Fixed

Hopped in the car, and drove home. On Fridays, the best way to head into the weekend is with a Little Victory.

Copy-Paste Coding is Evil

2005-08-06T07:36:00.000-07:00

The developer needed to create code for a new webpage: the new page is specific to a certain type of customer. The desired result is some different verbiage, a few color changes in the text, and some new illustrations.

The developer made copies of the source files that create the current page(s), did some simple edits to the text, color tags, and image paths, tested the results, and was done. Quick operation, completed before lunch.

Wrong.

The problem is that the developer also copied all of the business logic, plus all of the style-sheet and other layout specifications. Why is that a problem? Because there are two (almost) identical code paths now in the system, each containing over six thousand lines of code, and the only difference is a few lines which specify some visual attributes. The developer has unwittingly created a maintenance nightmare.

Suppose, one day later, a bug is found in the business logic of the new page. The developer will, in good faith, go to the code which specifies the actions of the new page, and correct the bug. Did the developer also look in the code for the original page and fix the corresponding page there? Maybe so, if they are very thorough. In the hustle-bustle world that I live in, probably not. Or, the developer will say to herself, "I'll fix that one later", and later never comes.

The better approach would be to examine the original webpage logic, figure out the exact set of changes needed, then create a new set of code that correctly and efficiently presented either the new or old presentation of the page. Sure, it is a lot more work: perhaps the deveoper has to work on this project after lunch, and probably the next day. In doing so, the developer saves herself (or some hapless other person trying to fix the code later) from confusion.

Sure, copy-paste might create a good foundation if a major rewrite is necessary. But if you're going to do a major rewrite, why start with the old code? Allow yourself the luxury of a blank sheet of paper: study the original implementation, but try to improve the system by taking a fresh look at the problem.

An Ultimatum Gets You Out

2005-07-29T23:56:00.000-07:00

Recently, I had to fire a brilliant but temperamental engineer. His work? Flawless. His enthusiasm? Unbounded. His temper? Fatal flaw.

This fellow had been arguing with a colleague over a certain aspect of software design. The discussion had been going on, intermittently, for about a month. I let it continue, because the debate was healthy and all parties were learning new perspectives.

But one morning, Mr. Genius came in and said: "It's final -- I can't work this way. I can't get things done under this constant criticism ..."

I asked, "What do you mean? I thought you guys were working it out. Seems to me you were close to an agreement, and what's going on now?"

"Well, I can't work with this guy. I've tried reason, persuasion, even pleading, and he won't accept my ideas. So, you've got to fire him -- or maybe I'll quit"

I was stunned, but in the back of my mind I realized that something like this would probably happen. The guy's a hothead. And if I indulge him this time, it will be trouble forever.

"No, he's not fired. You've got to work with him, or YOU are fired!"

Now it was Mr. Genius' turn to be stunned. "What? You can't fire me -- you know that would kill the project and probably the company!"

And on it went. Finally, I told him -- think about it over the weekend, and if you aren't ready to work things out with this other engineer, you've got two weeks notice. And I made it stick, and he resigned.

Was I sorry? You bet. Was it the right thing to do? In this case, certainly. Because Mr. Genius, as good as he was, had created an unbearable environment at work. Despite his excellent work, nobody wanted to collaborate with him. And, giving in to one unreasonable demand would only lead to more.

Only give an ultimatum if you're ready to accept either outcome.

(The alert reader will notice that I gave Mr. Genius an ultimatum in response to his ultimatum. Yes, I did. Deliberately. And I was prepared & willing to accept either outcome)

Beware the Bungee Debugger

2005-07-24T15:28:00.000-07:00

Careful developers, thoroughly testing their code, finding all of their bugs -- what's not to like? If we had careful developers in every stage of the development process, we would not need to test code and QA departments would be redundant. While there are isolated examples of coders who don't write bugs, the typical development project is staffed with developers who are in a rush to meet a rapidly-approaching deadline and are responsible for too many areas of functionality.

Repairing that situation is an essay in itself, and one that will appear in these pages soon. But let's first consider what generally happens in the throes of development and test. The usual Grade-A programmer has found most of the obvious bugs, and has fixed them all before the code gets near the QA team. But, the QA team is also composed of experienced engineers, and of course they find a significant number of bugs.

In many cases, the developer who wrote the code is unavailable: maybe she's on vacation, maybe on another project. Enter the Bungee Debugger. Typically a seasoned engineer, but sometimes a junior trainee, this person is pressed into service to find and fix the bug, preferably before the looming ship date.

A skilled Bungee Debugger will find and fix the bug, with minimal impact. Further, the Bungee Debugger will write up extensive notes, pointing out how the bugfix works and identifying risk factors associated with the fix. In addition, the experienced Bungee Debugger will make some recommendations for the original developer, so that the fix can be re-considered once the heat of the release is past, and a more substantial or careful repair can be made.

Just like you might slap on the spare tire if you get a flat near Barstow on your way to Las Vegas, but once you get to Vegas, you're going to the tire shop to get a new tire (or full set).

All too often, the junior engineer takes a quick look and throws together a few modifications that appear to fix the bug but in fact just address a visible symptom of an underlying problem. No review is conducted, and the bug fix "disappears" into the code -- the next person working with the code can't discriminate between the bugfix and the original code. This is a problem when the Bungee Developer didn't fully understand the original code. Now, we have two semantic models present in the implementation: the Original Intent of the developer, and a Misguided Hack that happens to fix a bug.

Once you get three or four instances of Bungee Bugfixes in a certain section of the code, you're in real trouble. The code is messy with patches and fixes, and the original design clarity is muddied by contradictory contributions made in the haste of bugfixing. Now, a careful developer has to go through the code, reviewing all of the bugs and all of the putative fixes, and try to decide how the design and implementation should be fixed. At this time we're not fixing bugs: instead, we're unwriting all of the misguided repairs and trying to re-fix the original bugs in a more careful and consistent way.

And nobody likes that kind of work: the original developer will probably do it, because she is the one who best understands the original intent of the code and is (hopefully) the expert in the problem space. If a Bungee Bugfixer tries to do the rework, your project risks additional distortion and deviation.

The best way to manage a Bungee Bugfixer is to review their work carefully, at the time that the bug-fix is attempted, and not later. And, when the original developer returns, make sure that the bug-fix receives yet another review.

Software in Cement Shoes

2005-07-23T08:07:00.000-07:00

The classic refactoring books (Beck, Fowler, et al) recommend continual refactoring as the code development process continues. Refactoring brings many benefits, but in many commercial software environments, well-intentioned business practices get in the way.

Consider the case of a moderately large codebase under revision control. Developers have been adding new features, fixing bugs, and refactoring awkward code. At some point, the release deadline nears, and this code goes into a "lockdown" state. Typically, the version control system allows a "branch" to be made, so that the software-development effort can continue, but the code destined for the release is kept stable. Now, developers must make bug-fix modifications in the release branch, and matching changes must be made in the ongoing development branch.

Once the release branch is made, you don't want to be making refactoring (i.e. gratuitous) changes in the development branch. Why? If you adjust interfaces, clean up variable naming, extract methods, or any of the other beneficial peturbations that we call "refactorings", you've introduced textual differences between the release branch and the development branch. Such changes make it more difficult for the engineer who is making bug-fixes to do his work: if there is a significant variant, the engineer may have to restart his bug-fix effort from scratch.

Ah, but release branches are typically short-lived, and soon the release is made and non-bug-fix modifications can be made in the development branch without impacting the bug-fix effort, which has ceased. So refactoring can begin again. At least that's what is supposed to happen ...

What I've found is that the business (remember, this is commercial software development -- open-source doesn't generally have this problem) will need to bow to customer or partner pressure and "keep the release branch open" -- that is, to continue making bug fixes or minor feature additions for a significant period of time.

That's when the cement starts to harden around the software. Since the company is already keeping one release branch open, why not two or three? The business managers will argue that we've also got to keep an even older release alive, since we are doing so well keeping one alive. Then, the poor developer is making bug-fix modifications to three, four, or more branches of the code.

It is ugly, and it happens. At this point, we can't afford to burden the developer with the challenges of trying to understand four widely-diverging codebases: we have to keep them as similar as possible. Even the main development branch. Refactoring stops.

I've actually worked for a company that required periodic "re-opening" of previously released versions of the code -- not for the purpose of bugfixes, but to add new features! Sometimes these new features were back-ports of features from the newest or development branch, sometimes it was a "custom" feature just for one client. After all, why not? That particular old release was stable, well-debugged, and is an excellent platform for releasing a new feature!

This is when good software companies start to go downhill. Tracking bug-fixes, feature-content, and other miscellaneous modifications becomes burdensome. The talented engineers don't want to spend their entire days just doing revision management, so the task typically falls either on a junior engineer or a project manager who has enthusiasm but often not much technical acumen. Finally, bug-tracking software systems are often unprepared for simultaneously tracking a bug-fix across multiple divergent software releases.

Fast forward six months or a year -- the software has become a complete mess, as multiple branches are open and nobody is effectively tracking what modifications are going where. Bugs are getting fixed in some branches, not all. Developers are hastily patching code which has diverged -- often creating new bugs in the process. The QA department is going nuts, because each release branch has unique testing requirements and a distinct set of bugs. Design-wise, the source code is a tangle, because therapeutic refactorings ceased long ago.

At this point, senior management starts to get a hint that something is wrong. External consultants are called in, and they might recommend a thorough overhaul of the development and release process. But senior management, addicted to the idea of rapid releases to a divergent set of customers, refuses. Re-imposing the discipline would hurt the business, so it doesn't happen.

Now, the only way to clean things up is to refactor the internal human processes. Slowly, methodically, and carefully optimize the entire organization: trim releases, tighten up the bug-fix process, and clarify the goals. It isn't hopeless, it just looks bad. But getting back on track will take at least as long as it took to "mess things up" in the first place. Once the process is refactored, the code can be refactored.

Nothing Ventured, Nothing Gained

2005-07-22T00:03:00.000-07:00

I recently had my first round with the Sand Hill venture capitalists, and learned a few valuable lessons.

Everyone will listen, everyone has questions. You will never hear a "no", but rarely will you hear "yes".

You will get to present to four or six partners, and they will all have questions. Some will wonder where you got your tie. Others might ask about the intricacies of XML parsing. Everything will be very friendly. You will know when the pitch is done when you are told: "Very interesting ... let's keep talking".

But you won't see that checkbook come out. A venture capitalist won't buy in on the strength of the presentation alone. It takes something else. Maybe you know somebody who knows somebody else. Maybe you got some angel funding from a retired entrepreneur who sits on the same charity board as one of the partners. It is the inside game.

What annoyed me the most about the VC game is that nobody ever says "no". You will get the "soft no" -- your calls will be answered by a cheerful and sympathetic office assistant, who will say "Oh, he just stepped out", or "She's in a meeting right now, but I'll put it at the top of her callback list."

The VC doesn't want to say no, because if your proposal goes anywhere (say, to a competing VC who does decide to help fund you), they can jump back into the action. Or, if you've managed to scrape together some operations cash from a rich uncle, the VC might call back, three months later, just to "check in". This happened to a friend of mine: he had been able to get an engineering prototype installed in another startup funded by the VC, and when the startup's CEO mentioned this at a board meeting, the call came in.

How can you get a "yes"? It depends: in the example I mentioned in the previous paragraph, the founder of the startup still had to go through some pretty tough due-diligence before the first round was closed. In his case, it almost seemed like dumb luck. It helps if you have made the acquaintance of various Silicon Valley luminaries. And I can't tell you how to do that, because they'd shoot me :-)