Vincent Delaroche's blog post was featured
The Economic Impact of ADM Quality and Productivity Measurement
It is commonly known that analytics-driven management yields superior results. A recent study by IBM and MIT Sloan found that businesses managed with analytics perform 2.2 times better than those without. The same is true for the CIO function. Fact-based metrics allow CIOs to make better and quicker decisions, with analytics to support meaningful discussions about projects, service providers, business, and even budget sessions with the COO and CFO. You can outsource everything but your judgment.The problem with deploying analytics in the enterprise is the business case. How does one quantify the positive impact of measurement on development teams whether these are internal or outsourced? The ROI that supports investment in measurement competence is not that obvious. The objective of this essay is to make it crystal clear, and to propose a path to estimating the payback that the C-level can count on.Measurement in ADM is new, hence the business case has proved especially hard for the CIO, leaving the IT function like the shoemaker’s children while everyone else invests in analytics.CIOs have very limited visibility on structural quality -- the resilience, efficiency, security, and maintainability of the code and architecture perpetually under construction. And no visibility on true software development productivity. As for any new paradigm, the canon of empirical data to support the evidence is still developing, so some parallels with other disciplines are useful:In the 1930s we were introduced to the Observer Effect, where worker productivity at an electrical factory increased when the organization showed interest in the workers, but immediately dropped once the study was over. Researcher, Henry A. Landsberger, who performed a study and analysis of data from numerous factories over eight years, found that workers under observation increased their output by about five percent.A study led by Charles P. Friedman and Jeremy Wyatt (Evaluation Methods in Biomedical Informatics, 2nd Edition, Health Informatics Series), based on diagnostic interviews of hundreds of physicians shows that doctors with knowledge of being measured improved their diagnostic accuracy by a staggering 27%! This is compared to the control group that is not being measured. Human nature is the same regardless of degree of education – the M.D.’s and PhD’s, also improve their performance when they know their work is going to be measured.In many areas, the concept of a watchdog institution has proven popular, especially in finance, although in this field it also increases creativity to circumvent rules. One of the most impressive observed impacts was actually related to the effects of speed enforcement with radar along the roads. European surveys shows that the number of injury accidents declined by 20% across the continent. Most interestingly, the Institute of Transport Economics of Norway, reported the largest percent decline in the number of accidents of almost 90% for highway sections complying with visible warnings.Beyond the empirical data, no one doubts the value executives get from measurement, at least conceptually. What gets measured gets improved … You can’t manage what you don’t measure. Conventional post-Drucker management wisdom. In recent pop management culture, the idea of measurement is introduced as gamification or leaderboards. Measurement drives behavior in modern organizations.The measurement mantra is easily translatable to the software-intensive enterprise. To reduce software risk and improve application health we can institute a software quality model based on non-disputable coding standards, along with a demonstration of follow-through from management to ensure compliance to see a real and positive impact on development teams and suppliers. It’s up to management to ensure that follow-through is recognized. This is not an intrusion on the developer, rather a management technique, using analytics to control something that drives business cost and value, while helping developers reach their maximum potential. This includes:Decision support. Immediate access to important analytics means that critical project-saving decisions can be made sooner rather than later.Fact-based discussions among teams. This clears the air, because there is nothing worse than colleagues’ or client’s judgment based on “feeling and impressions.” And, this has the highest benefit of fairness and recognition for your star technologists.Fact-based negotiations with your ADM suppliers. Discussions with vendors don’t have to rely on guesswork or taking the client’s or vendor’s word for the quality of an asset.Fact-based discussions with the business. Because sometimes they need to understand that “minor changes” were actually not “that simple.”Fact-based reporting to the CEO or CFO office. Because they are increasingly tired and frustrated to allocate budgets to something they can’t really measure themselves, making multi-million dollar decisions in the dark.An immediate and global improvement in development outcomes. Both in terms of productivity and quality, which means lower technical debt creation, less slipshod code, always leading to hidden rework, impacting productivity and worse, time-to-market; higher reuse rate: In case anyone hasn’t noticed, software developers really love to reinvent the wheel; and overall higher structural quality. For more resilient and responsive systems, high performing end-users and happy, loyal customers.I’ll let you put your own value on each of the above points, but we’ve collected real life metrics across many different customers in various industry verticals. What we found is after the implementation of a proper tool and methodology, eventually all customers end up with anything from a 5 percent to 15+ percent net gain, in percentage of ADM spending excluding the business impact (see graph below). The gain variation depends on the size of the IT shop, its geographical distribution, its sourcing balance (in-house vs outsource) and its level of process maturity (a CMM Level 3 organization can usually count on a 10+ percentage gain). Personally, I’ve seen a lot of evidence to each of the above value categories. Some examples:A Fortune 100 in North America -- fully outsourced $110M ADM services per year went down to $100M in less than 9 months, for the exact same amount of function points delivered to the business. Want to hear the icing on the cake? It was all at higher structural quality.A very large European system integrator cut their costly troubled projects by 3 percent and, in one of these projects I met an IT executive who saved her career by proving she was making progress on a complex billing system.A CIO of a large investment bank in NYC told me that his suppliers suddenly became more collaborative, just one week after the announcement of their introduction of an ADM measurement system. A CEO of one of the top-5 GSIs stating they had monetized an ADM productivity improvement of “a few points”, directly derived from better work productivity of their 25,000 developers within multiple software factories.A large telecom operator, the rate of newly introduced “critical violations” (i.e., software flaws affecting system stability) going down from around 500 per release to less than 10 -- all within a 10 month period.And a capital markets CIO in NYC who confessed to me that it was even more valuable for him to use our analytics to look good vis-a-vis his boss in the business than to manage his teams.Lastly, have a look at a real story from a large IT-intensive organization that experienced savings and service level improvements by working measurement into their organization and vendor relationships.In this day in age, your software is your business. And although difficult to quantify, better software means more satisfied customers, better time to market and superior business performance. This isn’t rocket science -- it simply requires a measurement process that identifies vulnerabilities to developers, informs managers that procedures are adhered to and proves to executives that the organization is moving in the right direction.Vincent Delaroche founded CAST in 1990 with the vision that software analysis would change the face of software development.See More