2013-07-03

Here's a short experience from a project I was on long before joining
Relevance. It might be interesting or educational.

It was October in Minnesota. Fall is a time for endings, and my consulting partnership had recently decided to separate. Amid the lengthening shadows, I found myself in need of a gig. Something relaxing, where I could put in my hours, pull myself back together, and restore my drained spirit. I joined a project that had a January launch date, right after the holiday freeze lifted. The team was already in "crunch time," and had been for six months when I joined. We were working from before dawn until after dark in a
converted warehouse in Minnesota. To maximize our productivity,
management had lunch brought in every day from a rotating pool of
local restaurants. This wasn't as nice as it sounds. There's simply no
way to make lunch for 300 people palatable. Turkey tacos every Tuesday? Yikes.

Crunch time is standard practice in our industry. Everyone knows it
exacts a toll on the team, but (almost) everyone does it anyway. It's
that tantalizing burst of speed at the end of the project. Somehow, one extra day
of work at the end seems much more valuable than it was back in the beginning. I've been through more than a few crunches in my
career, and I have to confess that the big push to the finish line can
be exhilarating. As perverse as it sounds, humans bond through
adversity more than through success. So a hard push for the last
couple of weeks of a long project can actually bring people together
and create an even stronger sense of success when the project
launches.

Six months is a long time to be crunching, though. And as Fall turned into Winter, it became clear that the January launch date was a complete fiction, despite all the green lights on the project managers' dashboards. Winter became Spring, and we continued in crunch time.

As you might imagine, morale suffered a bit as the release date kept
receding into the distance. It seemed to slip one day for every day
that we worked. The management team tried a bunch of different things to bolster
our spirits. For one all-hands meeting, each subteam nominated a theme
song to be the project anthem. All the songs were compiled into a CD
that got pressed and distributed. You can get an idea of our
collective state of mind from the tracks we chose: "A Million Miles
Away," "I Wanna Be Sedated," "If I Had A Million Dollars," and so
on. Someone later realized that this could be construed as a violation
of a dozen copyrights, with a total liability greater than the
project's budget, so nearly all the discs were recovered. Only a few
survive. They are desirable collector's items.

In another amusing effort to boost morale, the management team brought
in a movie-style popcorn machine. Who can be grumpy when the air is
suffused with the delectable scent of coconut oil and popped corn?

The team needed a sense of progress, not novelty CDs or a popcorn
machine. In fact, the whole thing kind of backfired, because we ended up cleaning the popcorn machine ourselves, on a daily
rotation among the subteams. (Except, if I recall correctly, the business analysts, project managers, and architects were exempt.) Giving us dishes to wash certainly let every designer, programmer, and admin know what management thought of their value.

Progress came late that spring, when we finally passed through load
testing, security testing, and functional testing. The critical bugs
were fixed. The "final" release was cut, and we started the migration
and cutover process. I'm sure you won't be surprised to learn that
there were many emergency builds between the "final" release and the
one that went live. Eventually, thanks to a team that lived
in a co-opted conference room for three weeks, we reached the day that
so many of us thought would never come: launch!

It was a disaster.

The system was totally unprepared for contact with the real world. It
had architectural problems nobody had discovered. It would crash
whenever Google tried to index it. There were critical problems in the
code, configuration files, and the user interface. It needed immediate
surgery, but there was nobody left to operate. The developers were
gone. They were all contractors, eager to roll off this hellish
project. Most of the project managers had been released back to the
project management office (the dreaded PMO) to be assigned elsewhere. There was an application maintenance and
support group assigned, but few of them had any knowledge of the
system internals.

Almost everyone who could fix things headed for the hills the first
chance they got. The patient was bleeding out on the table, and there
wasn't a surgeon in the building.

The Dangers of Project Thinking

The first of the many ills on that project was that it was defined as a project
at all! We were building an array of systems meant to be the future of
this business. The end of the project is the true birth of the
system. Like any newborn, a software-based system needs people to keep
it alive and growing.

During the build phase of a product, people accumulate knowledge about
how it works, how it's built, and most importantly why things are
the way they are. Everyone makes thousands upon thousands of
decisions. All of those decisions have to be made real, and the place
they are made real is in the code. There are side effects, though: As you make decisions, you also learn how
the product is built and why.

If your product team is staffed like most companies', you are part of
a group of people attached to a line of business. The business defines
a need, supplies a budget, and decides what is in scope for the
project. The build team gets drawn from pools of people, mostly in
IT. You'll need a project manager, some developers, some UI designers,
and one or more database administrators. If you're lucky, some of them
will have met each other before this project starts. If you're
unlucky, each of them will still be supporting their last few
projects. (They won't have any budget or time allocated, of course. Those projects are finished and their charge codes are closed out. It comes out of the time they bill to your project. But don't worry, you'll get that time back from the next project, after yours is already finished.) One developer joked
that he was supporting ten years' worth of old projects. He said the only
way to get rid of them would be to quit, so he did.

In a large company, each of these groups of people reports to someone
different: project managers report to the corporate Project Management
Office, developers report to a manager of application
development, database administrators report to a DBA manager,
and so on. At some point, the project ends, the budget is used, and
the team goes back to their respective pools.

You might also be using contractors for extra capacity. Our project
had hundreds. In that case, you know they will be
temporary, either due to cost or policy concerns. Many companies have
policies that limit how long a contractor can be engaged. (To avoid
lawsuits about "perma-temps" and benefits.) One company I know had
a limit of 18 months. They had a project that ran long and crossed
that deadline, so they booted out the entire development team. In the
last few months before launch, they brought in an entirely new crop of
contractors who had no knowledge or context about the existing work!
You can imagine how well that worked.

Once the original builders are gone, no support team in the world can
tell which decisions were carefully thought out and which were quick,
"good-enough" hacks. So the hacks will hang around for a long
time. Worse yet, within the code, hacks tend to breed other hacks near
them. The original team would know to beware of those hacks, but the code
doesn't come with any markers to warn their successors where the
quicksand is. The net result is what we affectionately call "bit
rot." Each successive project will take longer and cost more, until
you decide (or your development team tells you) it's time for a
rewrite.

Limited Vision of a Project

Project thinking is limited-term thinking. Projects optimize for a
target delivery date and budget. That induces them to make choices
that favor the date, but might not help your product after the project
is done. A project team may decide to skip data purging to save a
developer-week of effort. That's great for the project but incurs
an ongoing operations cost, because someone has to purge data
manually for the life of your product. Now, the development cost may
turn out to be more or less than the total of future operations
costs. Either way, you should make that decision based on how it
affects your product's lifetime profitability. It shouldn't be a
project manager's decision based on a delivery date.

This problem happens with business processes too. I can't tell you
how many times I've seen people switching between multiple internal
tools to do their jobs, usually with a spreadsheet open to keep track
of the IDs or codes that the tools don't share. In every case, it's
because the individual tools were built one at a time without regard
to the overall workflow. None of the projects could accept the extra
scope of integrating the tools because it would blow the
schedule. The result: saving some development cost once, but paying
the price in efficiency forever.

A Better Way

Instead of project thinking, exercise "product thinking." Design the
organization together with the software so you can keep it alive and
growing after 1.0 goes out the door.

As much as possible, try to create a self-sufficient product
team. That means the team should be able to handle routine daily
activities internally: analyzing customer metrics, planning new
features, implementing them, releasing them, and measuring their
effect on key performance indicators. The team should also handle
those ugly non-routine daily activities, too: fixing bugs, restoring
service when things crash... all that stuff we lump into "operations."

This dedicated product team gets a bunch of advantages. First, they
will communicate better. Second, they will be more motivated to do
good work. Temporary team members assigned to a project may be loyal
to the project, but team members permanently attached to the product
will be loyal to the product. These two goals are often in conflict.

The third big advantage of the self-sufficient team is dramatically
reduced cycle time. With a temporary team, everything becomes a
project. Want a set of bugs fixed? Make a list, define the scope, get
an estimate, and charter a project. Meanwhile, your customers either
learn to work around the bugs or they give up on you and go somewhere
else. With a dedicated team, you can deliver continuously and fix a
bug every day. All you need to do is put up a page with the "hit
list" of top bugs and give your team some slack, and they'll fix the
bugs out of their own sense of pride.

I won't claim that product organization would have cured all the ills
of my "Turkey Tacos" project. I'm quite sure, however, that people
would have done better work if they knew they would live with that
system for the next few years. They would have built a living
ecosystem instead of a bunch of dead artifacts.

Show more