…we frankly couldn't afford to outbid Le Chiffre… accountants seem to be running MI-6… not that I have anything against accountants. Many of them are lovely people…
It's amazing what you can do with Photoshop these days, isn't it? I think your odds are improving, Mr. Bond.
- Rene Mathis, "Casino Royale"
Betting is a Form of Estimation, isn't it Mr. Bond?
The Fixed Bid Estimate
Fixed bids are here to stay.
Some clients, such as government agencies, are required by statute to solicit fixed bids from vendors. Other organizations require fixed bids as a matter of internal policy. If we want to effectively compete for these projects, we need to adopt a methodology for estimating under uncertainty.
The key question is: how much do we bid?
Vesper: So you're telling me it's a matter of probability and odds. I was worried there was some chance involved.
The Fixed Bid Constraints
It is pretty clear why some clients like fixed bids: cost is fixed.
Looking at it from the client side, as one wag put it, "so long as it's what I want and I get it on time for a fixed price, I don't care what it costs them to build it."
This of course works both ways.
Looking at it from the developer's side: If the vendor can do it for a song, but charge an arm-and-a-leg, the vendor makes a huge profit and the clients say that they don't care. Or do they?
Yes they do. Most client's would be livid if they paid an arm-and-a-leg for a "song." They want their money's worth, and to get the biggest bang for the buck, they mistakenly think that their best bet is the fixed bid route. The client will send out literally scores of solicitations for bids from vendors — sometimes known as a "cattle call."
All things being equal, the lowest bid will be awarded the work.
The Price is Right?
In the game show, The Price is Right, the object of the game is to guess the actual retail price of an item. The winner is the one who comes closest without going over the retail price.
In fixed-bid proposals, the object is to come in with the lowest price.
Sometimes the best bid is "no bid," because the lowest price does not favor the bidder.
Unlike the game show where the losers go home with "lovely parting gifts," in the world of price estimation, everyone including the winner, could lose his shirt.
What Are My Chances, Mother?
We start with a persona/role I will call the Estimator. This can be one individual, a small group of individuals, or the whole team. The process goes something like this: Given what we understand the client wants, and drawing from our own experience with projects of similar size and type, how long did it take to deliver a similar or analogous feature or functionality?
The Estimator, whether an individual or a team, will make an informed estimate of the most likely outcomes.
In our experience, the Monte Carlo risk assessment has proven itself as a valuable tool in project time estimation. Several firms offer software packages to run Monte Carlo analyses. We have adopted the @RISK Decision Tools Suite at our firm, along with Liquid Planner. We will draw on these tools by way of example in this blog post series.
In a throw of a pair of fair dice, the most likely outcome is "7." Throw the dice 100 times, or 1000 times or more, 7 comes up more than any other value. In fact, we can figure out the odds in rolling dice.
FIG 1. Histogram of outcomes of rolling a pair of fair dice.
FIG 1. The 50/50 point, also called the "indifference point," is "7" for a pair of fair dice
Engineers, designers, developers, and specialists describe the project, assessing the parts and time it takes to achieve a solution. Individuals work out their differences quantitatively.
A Hypothetical Project
We start off with a modest-sized hypothetical project. In this simplified example, Project 01, there are only four parts, which our Estimator has broken out. In an actual project there would likely be more parts and each would probably broken out down to the ticket level.
In this case, the Estimator filled out a grid (below) that has three categories,
Least - one chance in 20
50/50 chance
Most - 19 chances out of 20
TABLE 1. Project 01 — a hypothetical project with four parts, each broken into estimates, low/most likely/high, to complete the work.
TABLE 1. Drawing from previous projects, the person estimating the project will come up with a range of possibilities.
Putting these cases into English. There is a one in 20 chance (least) that the part for Architecture can be done in 80 hours; a 50/50 chance it can be done in 100 hours, and a 19 out of 20 chance (Most) it can be done in 120 hours.
In the case at hand, we assume a normal distribution. Other distributions might serve as well, but other distributions, too, will be addressed in a later discussion.
If the Estimator adds all the 50/50 points, the total estimate comes to 450 hours. Most clients are looking for that single value to establish the hours, and hence the cost.
But as an experience Estimator knows, it is statistically unlikely the estimates will hit the 50/50 cases on the nose each and every time any more than a fair dice comes out "7" on each and every toss. For the sake of this example, we will assume that each section of the project is fairly independent of the others in term of the estimated time.
The Estimator's curve for Architecture appears as FIG 2., below.
FIG 2. Probability estimate of Architecture phase for Project 01.
FIG 2.Probability curve of Architecture Hours. The small text-table to the right of the curve is a listing of specific metrics for this curve.
FIG 3. Probability estimate of Front End Development phase for Project 01.
FIG 3. Probability curve of Front End Dev Hours. The small text-table to the right of the curve is a listing of specific metrics for this curve.
Likewise for Front End Development, FIG 3., above. In this case the estimate is greater than for Architecture, 120 hours, and in this case the uncertainty is also greater, ±40 hours, than for Architecture. Therefore Front End development estimate variances will play a greater role than Architecture variances.
Site coding, FIG 4., below, in this example, is the largest component, and also represents the greatest amount of absolute hours of uncertainty, ±60 hours among the fours phases.
FIG 4. Probability estimate of Site Coding for phase for Project 01.
FIG 4. Probability curve of Site Coding Hours. The small text-table to the right of the curve is a listing of specific metrics for this curve.
Finally, the launch related hours, Fig 5., below, are smaller in total than the other categories, and the uncertainty in hours is ±20 hours.
FIG 5. Probability estimate of Launch phase for Project 01.
FIG 5. Probability curve of Launch Hours. The small text-table to the right of the curve is a listing of specific metrics for this curve.
Blended Probabilities
Simply adding of best and worst case estimates rarely produces the true high and low of the entire project. Adding the high and low values of each separate curve will over-emphasizes the actual range of the project.
Jumping ahead, the entire project is 450 hours, ±75 hours. If we added the least and most cases values, it would come out ±140 hours.
FIG 6. Probability estimate of all four phases combined for Project 01.
FIG 6. Monte Carlo probability estimate for all curves combined. The small text-table to the right of the curve is a listing of specific metrics for this curve.
The representation below shows the four individual uncertainties to the left, and the combined uncertainties (black) to the right. Intuitively, we see why simple addition of the four shapes on the left does not yield an accurate result.
So how does it work?
FIG 7. Probability estimate of all four phases separately and combined for Project 01.
FIG 7. Component curves and resultant curve overlaid - Total project estimate. The orange, blue red, and green curves to the left are the probabilities for Launch, Architecture, Front End Development, and Site Coding, respectively, while the black curve to the right is the combined probability for all phases. The probability of the five curves is displayed by the software TO SCALE. The small text-table to the right of the curve is a truncated listing of specific metrics for the five curves.
Note that the green curve to the left, which is the same curve as Site Coding, FIG 4., is also the largest of the four component curves in Fig. 7. The green curve is also somewhat squat due in part to the fact that overall the Site Coding curve is less certain than the other three curves. Now note that the combined estimate's curve shape, black curve in Fig 7., is more like that of Site Coding Fig 4., and less like the other three phase curves.
Put another way, although FIGS 2. thru 5. seem to have identical shapes, they do not have identical values, and when they are superimposed on a grid that takes the actual values into account, the curves reveal their actual shapes. The more area under the curve of any component, the more it influences the melded outcome, FIG 7.
Bidding Implications and Risk Aversion
Estimating the hours (hence cost) of the project solves half of the equation. The second question is: What is the acceptable level risk our firm is willing to take? The converse is: at what point is this project too risky?
Will you take this bet?
We toss a fair coin and you call it.
If you win, you get $10M tax free.
If you lose, you forfeit your life.
Will you take the bet? If no, then what if we increase the amount?
What if we make it $100M? $1B? $10B?
At what point, if ever, will you take this bet?
Though this is an extreme example, there are more common bets with huge payoffs that we will not make.
How might this effect the our actual bid?
Fixed Bid Odds-setting
FIG 8. The probability outcome of Project 01 is ± 2 sigma.
FIG 8. is a graph of the expected outcome of Project 01 under a normal distribution. The question for the bidder is: what should we bid?
An answer that immediately comes to mind is 450 hours, and that's because the number is in the middle; at the 50/50 indifference point as well as the most likely outcome.
Let's go back to our coin toss: We toss a fair coin and you call it. Call it right and you win a dollar. Call it wrong, and you lose a dollar. No big deal if you lose.
Now, if you were to play this game over and over, you'd be no better off than when you started. The net is zero.
If you a business were operated at that level (50/50-style) over time the net would be zero, unless you have a string of bad tosses, in which case you may end up having to close shop permanently.
FIG 9. The probability outcome of Project 01 where the odds are in the 50/50 to 95% bracket.
FIG 9. looks at odds that are in the 50/50 to 95% bracket, which are shown in dark green.
A growth company ought to consider where in the green the bid might should be. In this example, the bid might be 450 to 525 hours.
The closer to 450 in the green curve, the closer the firm is to a net zero outcome.
FIG 10. The probability outcome of Project 01 is in the 75% to 95% bracket.
FIG 10. looks at odds that are in the 75% to 95% bracket, which are shown in orange. At 75% the odds are 3 out of 4 that the hours are sufficient. At 95% the odds are 19 out of 20 that the hours are sufficient.
The entrepreneur who operates in this zone would be pondering a bid of 482 to 527 hours.
Summarizing the Bidding Brackets.
FIG 11. Setting the acceptable risk from ±2 sigma to one in for or better.
In this short blog post, we have looked how a Monte Carlo analysis helps quantify risk. We see that generating a curve does not take us all the way there. It is the power of the analysis that informs us when we take a risk by making a fixed bid.
Summary and Final Thoughts When Using Monte Carlo Methodology in Making a Fixed Bid.
Monte Carlo risk analysis is one way to quantify, meld, and graphically represent multiple uncertainties.
Integrated software packages exist to run Monte Carlo analysis, and you don't have to be a math or business major to implement the methodology.
The curves should be generated from what is believed to be the most accurate representation; no sandbagging; no Pollyanna; no Chicken Little.
Combining multiple probability distributions through simple addition does not reflect the uncertainty of the project. A Monte Carlo simulation will meld the curves.
The narrower and more pointed a probability curve is, the more it tends toward a discreet value; the broader and flatter a probability curve is, the greater the uncertainty.
In a fixed bid, the bidder must has the burden of shouldering the risk, not the client.
Estimating probabilities of sub-steps will sharpen the overall probability of the estimate, while at the same time incurring a cost for the additional estimation work.
Before going taking the time and incurring the cost of an estimate, the bidder should ask:
Has the client provided enough information for the shop to make an informed estimate? If not, consider the amount of time it will take to gather that information and act on it.
Is the project one that plays to the shop's strengths? Interests? Profit margins? If not, is there another project that is more in align with these?
Will the cost of drawing up a proposal be outweighed by the expected value, adjusted for the probability of getting the project? If not, consider a "no bid."
Will the time is takes to do an estimate be better spent with a client who is more forthcoming? If a potential client's willingness to engage the shop and answer questions ahead of time is a fairly good indication of how the project will go once the work is started.
Preview of part two
In part two of this blog, we will look more closely at real-world project estimate curves.
FIG 12. Three Probability Distribution Curves That Are Useful for Estimating.
In part one of this series on estimating projects, we have exclusively used a normal distribution, and indeed some of the time it is the preferred distribution. In part two, we will examine how using other distributions will help further refine the estimate.
The three curves are known as:
Normal
Exponential
Erlang
How these are applied in more detailed cases will be addressed in the next part in this blog series.