I dramatically increased my notoriety a few weeks ago by telling modern game developers, you’re kidding yourself if you think you can manage yourself out of crunching to ship games. I got a lot of lectures about how I’m not in touch with “modern project management” which is really funny considering that I participated in pioneering many of the project management techniques in widespread use for agile content production today and continue to practice and refine them. In the early 1990’s the concept of “Agile” software development was so new that very few had ever heard of it. It wasn’t adopted widely anywhere in the industry, especially at Microsoft. In order to ship the first five generations of DirectX with no time and no budget, the early DirectX team was forced to resort to a project management style that was unheard of in the industry and at Microsoft at the time.
Today it is called “Agile” but we had never heard of that term at the time. We just knew that it would be impossible to deliver as ambitious a new technology initiative as DirectX by any traditional means. The early DirectX API’s were made by several small, self organized teams scattered around Microsoft coordinated by myself and the other early DirectX creators Eric Engstrom and Craig Eisler. We had no official resources to deliver the product and a fixed deadline in which to ship it. As a consequence the product definition of DirectX 1.0 had to be… whatever gaming technology we can get into shipping condition in time for the launch of Windows 95. Since several API’s (DirectDraw, DirectSound, DirectInput, DirectPlay and DirectVideo) were considered essential components of a shipping game OS, we had to ensure that none of those API’s developed by different teams missed their launch window. To accomplish that we had to focus on getting the barest most essential functionality to each API in place and working immediately and then iterate on them as fast as possible to add features and robustness as the deadline for shipping Windows 95 approached. As the deadline got closer we would push out features that we didn’t think could be refined enough to hit the launch Windows to the DirectX 2.0 release. The result was that we were able to ship five generations of the DirectX API’s in the space of time that it took Microsoft to develop Windows 98.
The speed and efficiency of our approach to iterative software development was phenomenal by Microsoft standards at the time and I would credit it as being one of the principal reasons that the DirectX family of API’s ultimately completely dominated the Windows media architecture. The game industry in that era also became early adopters of agile project methodology, as it was a very natural fit for the life cycle of game development. By the time I had founded WildTangent in 1998 the term “Agile” development had been coined and shortly thereafter the Agile Manifesto was spreading fast among developers everywhere.
“We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value:
Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan
That is, while there is value in the items on the right, we value the items on the left more.”
WildTangent was one of the very first online game publishers in that era. We aspired to produce console quality games written in Java and streamed to players via the browser over modem connections. At first I had attempted to hire professional console game producers but that was a disaster. The pace and cost at which they were accustomed to operating would not keep up with Internet time. Instead I hired very junior engineers and web designers and trained them to develop games on a common engine developed by our engine development team. Each game was assigned an engineer and artist and a producer. Ideally, they came to me with the game idea themselves. They had six months to develop their game and I reviewed their projects on each milestone. If a project was missing a milestone badly, we killed it quickly. That discipline ensured that almost every game that got completed would be profitable if not a hit. Over the years those individual downloadable games became persistent worlds that needed to be continuously updated and developed. Ironically modern online and mobile game developers have actually regressed in their methodology since those years through no fault of their own. In the early days of the Internet we had direct relationships with our customers and could collect detailed information from them about what they liked, how they played and what caused them to make a purchasing decision. Modern Apple and Facebook controlled markets work hard to hide vital customer information from their developers so that it has become very hard for modern developers to “own” their audiences the way we did in the early days of online gaming.
That inability to own customer relationships has made the mobile game market far more volatile and risky for game developers because they live or die by each hit title they produce with little ability to cross promote their players. The result is INCREASED pressure to crunch out new games rather than REDUCED pressure. So for you millennial game developers who are sourly reading this post… consider for a moment the possibility that an old veteran of online publishing might have something to teach you that can save you some pain.
The power of Agile development to hit a target works best when you throw it like a spear instead of like a pillow.
When I hear statements like this from allegedly professional game developers, I think; “Pillow throwers!”
“Long crunches are the result of bad management”
“If they need me to work more than 40hr/wk they should hire more people or pay me overtime!”
“I’m not at my most productive or creative best after 40hrs/wk”
Crunching is an essential tool to game development, when you reject it wholesale, you’re immediately in serious danger of failure. If there is any poor management involved in these projects it is due to HIRING people who don’t understand when and where crunch should be used to deliver a better game. For the purposes of brevity and clarity I’m going to describe a highly simplified model of game development process with the full recognition and acknowledgement that the real-world is actually much bigger, more complex, and fraught with more nuances. I also concede that iteratively maintaining an existing game is much easier to do with a 40hr/wk team than creating new original games from scratch. Hiring more people or paying unplanned overtime consumes your budget faster without increasing productivity. You can’t bribe lazy people to work harder and you can’t add new people to a struggling project productively… these would be BAD management ideas.
In our simplified world, let’s assume that we have to deliver a new original mobile game from scratch in six months on a fixed budget and on a fixed deadline. How do you do it? Actually that was a trick question… you can’t. You can deliver a dozen new original mobile games from scratch in an average of six months on an aggregate budget with good management but there is no magic management technique that always works on a sample of one project. Making games is inherently very risky. Let’s talk about the impossible to control risks that often derail the best managed game projects.
The Four Horsemen of project failure dominate the risk equation in making games which is why there is no magical management solution for utterly avoiding crunch periods.
The four horsemen of failed game projects
Team member failure: Somebody critical always gets hit by an asteroid
Acts of God: There’s always a flood/power outage/traffic accident/hardware failure that you couldn’t predict
Channel failure: Apple, Facebook, Microsoft, Nintendo, etc. decide to have a surprise change of policy that blows your plans
Design failure: Despite our best creative efforts,the core game mechanic just isn’t fun for some reason nobody could have predicted. (You can’t do original without risking it)
The sum of “supernatural” forces that conspire against shipping a great original game on time on budget are greater than any human manager and greater than any reasonable single budget can anticipate by a range of many multiples. If you think you can control for these forces, you’re doomed to disappointment. In this respect games are like quantum particles, you can’t really identify their individual potential for success on time on budget WHILE they are being made. You only get this insight after the fact. The first thing you have to do to succeed at managing game projects is humble yourself to accept that you’re not in complete control… now make a plan.
Any individual game project on a fixed schedule and a fixed budget is very hard to risk manage to deliver on time. The only variable you really control is the ability to reduce the games scope. This observation turns out to be a “feature” when embraced as such. Agile management practices teach us to build products from the outside in. In other words, you start with a “complete” product and iteratively improve it. So what is the smallest unit possible of a complete unit of game? Core game mechanics working as intended with programmer art. Once those elements are in place you iteratively increase the scope in content and features until you run out of time or budget. As a game designer when you design games this way, it’s best to strip the game concept down to its bare bones and then build the game up from its foundation. Design the game as you see it ideally, then formulate it in a way that you can deliver a “complete” nearly shipable version of the game during each sprint with a few more features and a little more content.
There are several huge benefits to this approach.
A bad game mechanic reveals itself early before you’ve spent a lot of time and money on it
You always have a game to ship which greatly reduces your project cancellation risk by management
The discipline to produce a constantly “complete” game exposes the game to a lot of iterative feedback and refinement that greatly improves its polish
The game you ultimately deliver is generally much more polished and refined. The absence of features you wanted but didn’t actually need improves the overall game experience.
In this context when and where is crunching appropriate, and when and where is adding resources appropriate?
Keep the team as small as humanly possible until you have a complete core game mechanic that rocks. It (generally) only really takes one engineer and one artist to prototype any given mobile or casual game concept. The cancellation cost is minimal during early stages, you want the flexibility to be able to throw everything away and try another approach early on.
Crunch early, crunch often. Games that are not designed from the outside in almost always run out of time and/or budget, they just can’t come together from parts and get the polish they need at the end of development. Once you have a game project that is being developed this way, the way to minimize project risk is to crunch early and crunch often. When you crunch early in a game development project it’s because you have the project under control and are pushing hard to deliver the maximum content experience you can with the limited resources available to you. Push for long hours to deliver compelling feature sets in early project sprints. The reward to your team for completing a sprint successfully (A complete unit of improved game) is a few days off, post sprint. Sprints should generally be 2-3 weeks, 6 days/wk, 12 hours/day. The Agile guys don’t say this because the view obviously isn’t widely popular, but it’s true none-the-less. Sprints work better when they are intense. Teams function better, resolve communication problems faster and get things done better under pressure.
The problem with the idea of a 40hr/wk in game production is that people also generally mean the 40hrs of their choice. Engineers and creative types work best at very different times under very different circumstances, so trying to force them all to be at their best and in contact with one another from 9-5 doesn’t work. What works is setting a marker in the day when everyone on the team is expected to be present at the same time. I usually try to synchronize the day around a late morning stand-up meeting. Following stand-up all the various planning and communication meetings take place. Basically mornings until lunch time are for meetings, communicating, etc. After lunch you want people to settle down and work. In my experience a lot of creative folks and engineers work best at night. The problem is that even with the best agile practices people need to collaborate and if they’re in the office on a scattered basis it’s very hard to maintain efficient timely communication with other team members… and this is really the crux of team based software development. Individual productivity needs to be a distant second priority to team productivity. If the work of one or more people depends on information from a team member who is missing, productivity can quickly bottleneck. The larger the team is the more bottlenecks come to dominate overall productivity. Again, assuming you’re doing all the right agile stuff to ensure that tasks people accept have no dependencies and are fully defined, bottlenecks will still tend to limit overall project velocity. It’s better if everyone is always around all the time even if some folks are playing LOL in their cubes for a break.
When everything works exactly according to plan (Which it rarely does, in-spite of everyone’s best efforts) the result of crunching early and often in a project is that the game “glides” to completion. It’s not necessarily that the long hours abate entirely, but that the crunch ends as a diminishing set of the team are required to polish the game and ready it for deployment.
When is the time to scale up?
When things are running smoothly and sprints are getting achieved without major bottlenecks, you can add any number of people to the project and expect that the game will most likely benefit from their participation. When things are running poorly and sprints are being missed, adding more people will only waste time and money faster. Generally a well-run game project should hit it’s peak headcount 50%-65% of the way through the game and glide down to completion. You can often afford to let a game find it’s “soul” and it’s development rhythm if the resources dedicated to it are kept small until it has momentum. If a game project doesn’t find it’s “soul” or a development rhythm, cancel it, don’t invest in it. I recite a phrase I learned from my Poker buddies on this point, “fold early, fold often”. It saves you money to invest in the big hands that are really going to pay off.
One of the things I hear a lot from people who… obviously… don’t actually know much about project management is that when they are feeling pressured at work the management should just add more people. The problem with doing this is that the communication/bottleneck cost of adding NEW people to an active project is exponential compared to the linear benefit of having extra hands available. It slows things down a lot before it speeds anything up.
Teams that crunch early and crunch often are far more likely to deliver a quality game on schedule and within budget than teams that try to mail in a 40hr/wk and then discover deep into the project that they are going to be far from complete on time. The time to operate like your hair is on fire is at the BEGINNING of game development, it’s too late by the end. If you want “balance” as a game developer, take a few days off after delivering an intense sprint and a nice long vacation after completing major products. Personally I think weekends are the enemy of Agile development practices. I would much rather crunch through 2-3 weeks sprints with a floating off day for everybody on non-essential planning days and then have the whole team take a few days off together after the sprint is completed successfully.
Failure!
That’s all great sunshine idealism but in reality one or more of the four horsemen of the apocalypse listed above always rears its head during the project. When it happens you’re screwed! There’s nothing anybody can really do to salvage the situation. There aren’t enough electrons on the internet to describe all the ways I’ve seen projects go awry. The most annoying ones are unprofessional people related but that’s a whole other subject. In practice a lean agile organization can’t afford to keep a lot of spare people or resources around… just-in-case something goes to hell.
Now despite your best efforts at great project management, you have to scramble like hell to recover. Fortunately, you already have a team that is used to crunching when called upon. Your team does it regularly and consistently under well controlled conditions already. Because you did it EARLY in the project, you might actually be ahead of schedule which means you can absorb some pain. Remember that the probability that a four horsemen event occurs to your project is a function of how long the project takes and how many people are working on it, so keeping projects tight and lean is the best way to avoid unmanageable disasters. The best way to avoid a four horsemen event and survive them when they hit is by running lean and fast. Think of it this way… imagine you are riding a motorcycle on a dark freeway at night to a destination you must arrive at on time and you know that a random deer will cross the road at some random point every 10 seconds. What can you do to minimize your chances of a fatal collision with a deer?
Drive slowly and be wary
Drive as fast as you possibly can
#2 is the approach that will minimize your chances of hitting a deer because it minimizes the amount of time you will be on the road. That’s why I advocate throwing spears instead of pillows when it comes to developing games. Crunch is an essential tool to success, not to be casually discarded.
Now the 40hr/wk crowd likes to claim that “studies prove” that people are less productive/creative/whatever when they work more than 40hr/wk. These studies are written by academics who have all chosen careers of low accountability and high job security, not by managers accustomed to systematically producing software with large development teams. Games are 5% creativity, 95% drudgery, don’t kid yourself folks, do the creative work when you are fresh. Relative to the long hours of work associated with production the creative energy that goes into games is small. Secondly as I attempted to illustrate earlier, individual productivity is a distant second priority to team productivity. Having the entire team on hand to communicate with during the wide spread of mixed hours when people are most productive is actually most productive because preventing production bottlenecks due to communication issues is exponentially more important than every individual team member being at their absolute best all the time.
Generally when I hire people I tell them; “We have flex-time, you can work any 80hr/wk you like” Of course I’m kidding because we do need people to be present during some core hours for meetings and planning. Other than those core meeting times, it’s generally not a problem for people to be in and out of the office periodically during the day or to take random days off during the middle of a sprint because everyone is “present” with enough density that bottlenecks can usually be resolved quickly. After a long sprint, you throw people out of the office together for a few days before doing it again.
Compensation
How do you fairly compensate people for working this intensely over long periods of time? The short answer is you can’t. No amount of money I have ever paid a lazy or unmotivated person has ever made them really work longer hours for any protracted period of time. The only people who can consistently work long hours are people with a genuine passion for what they are doing and a strong sense of dedication to their team, management and/or company. The only way to fairly compensate such people is with equity in the game or company they are working for. I am also a big believer in spot bonuses. When you pay people a higher salary, they quickly take it for granted, and feel entitled to more. When you quickly and generously hand out spot bonuses to individuals and teams for hitting sprint milestones, it’s a very effective incentive. People feel they have earned those bonuses and consider them recognition for a task well done. It’s also much easier to manage the fixed budget of a project when you can pay people directly for delivery of well understood milestones. The same goes for time off, it’s much more rewarding when people get it after having accomplished important milestones. When compensation is closely linked to actual productivity it is far more effective at achieving the desired end result. A great game on time and on budget.
To Summarize:
Throw spears not pillows at your delivery target
There’s no magic management technique to deliver every game on time on budget, take a portfolio view
Keep projects lean in order to keep them fast
Crunch early crunch often
Tie compensation and breaks to sprint aligned milestones not to the regular work week
Cancel early, cancel often, save your resources for the games that find their “soul” and development rhythm early
Scale up when things are running smoothly, cut when they are dysfunctional
Concentrate your teams time and energy
That’s the recipe for making new original products. It’s not a leisurely or relaxed process, so if you can’t get your head around accepting that this is the best way to create original games and you want to be in the game business, you should look at roles that involve maintaining existing content franchises. I suspect that often people with families want to tell themselves that they can have it all, they can work on original game titles AND go home everyday for dinner. If you’ve had the experience of pulling that off, you got very lucky, it’s not consistently possible and it risks entire projects and teams to pretend that original games can be developed at a leisurely pace. Even when it’s possible, it’s still good management to crunch early and often to reduce project risk at the beginning and try to glide into shipping rather than CRASH into it which is what really happens with many game projects.
Here’s an informative video tutorial on how game projects usually progress.
The post Mythical Management appeared first on The Saint.