On behalf of my co-authors, George Spafford and Kevin Behr, thank you for all your kind words and support for The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win.
One of the most requested items has been a list of recommended reading and further resources to learn more about the philosophies, tools and techniques that were used in the book.
This will be the first in a series of blog posts describing the “body of knowledge” that underpins the novel. In future blog posts, my goal will be to describe the actual planning and construction of the novel, how I became trained as a Theory of Constraints Jonah, a Kanban practitioner, an acolyte in the Toyota Kata, how Visible Ops is woven into the book, as well as more concrete practice aids to replicate the transformation that Bill pulled off with Erik’s help at Parts Unlimited.
After posting this, I realized that I’ve forgotten several key concepts:
The GAIT and GAIT-R methodologies (Institute of Internal Auditors) and the Risk-Adjusted Value Management (Gartner): these are what explain John’s seemingly miraculous transformation as a CISO.
Two books on kanbans, which completely changed the way I visualize and manage work
I’ll write a Part 2 to this post very soon.
Dr. Eliyahu Goldratt’s Theory of Constraints
Dr. Eliyahu Goldratt wrote his seminal book, “The Goal: A Process Of Ongoing Improvement” in 1984. It’s a Socratic novel about Alex Rogo, a plant manager who must fix his cost and due date issues in ninety days, or his plant will be shut down. This book has been incorporated into many MBA curriculums and has influenced multiple generations of business leaders, and has sold over 6 million copies to date.
My co-authors and I studied this book for nearly a decade, getting ready to write “The Phoenix Project.” In many ways, I view our book as an homage to “The Goal.” We attempted to mirror most of the book structure and plot elements, while making it contemporary, relevant, and hopefully more dramatic. (I’d like to think that “The Phoenix Project” is what Dr. Goldratt would have written if he wrote “The Goal” today, and had Tarantino or Scorsese as a novel coach :)
In “The Goal,” Dr. Goldratt starts to describes the steps in the Theory of Constraints (TOC) methodology, which I’ll write more about in a future blog post. Briefly, the five orginial TOC steps are:
Identify the constraint
Exploit the constraint
Subordinate all other activities to the constraint
Elevate the constraint to new levels
Find the next constraint
In “The Goal,” the constraints were initially the famous NCX-10 robot, then the heat treat ovens, and then the constraint becomes market demand. In “The Phoenix Project,” the constraint was initially Brent because he was always having to deal with unplanned work, then the application deployment process, and then the constraint moved outside the organization because the needed MRP application support was outsourced.
In Dr. Goldratt’s following book, “It’s Not Luck,” he describes what he called the Thinking Processes, which is a fantastic (but somewhat inaccessible and often very slow) methodology of identifying core, chronic conflicts, methods of capturing the current reality, describing the desired future reality, and various planning techniques to increase the likelihood of getting there successfully.
By far, the very best overview of of the entire TOC, the Thinking Processes, and Goldratt’s body of knowledge is an audiobook called “Beyond The Goal”. It includes all of his recorded lectures from 2005, and is a breathtaking tour of Dr. Goldratt’s life journey, describing his contributions, tools, and case studies.
Eight years ago, it was well-known that the Thinking Processes were the tools and techniques he used to construct “The Goal.” That’s why my co-authors and I tried to learn more about it, but couldn’t find any training or books beyond Dr. Goldratt’s writings that were feasible for us to attend or buy. Nor were there any non-trivial (or correct) examples that we could find scouring the Internet. Much has changed since then.
For anyone interested in becoming expert in TOC and the Thinking Processes, I wholehearted recommend Dr. James Holt’s EM526 Constraints Management and EM530 Applications Of Constraints Management courses, offered online through Washington State University. They both prepare students to become trained as “a Jonah,” the Yoda-like character in “The Goal,” who was obviously the embodiment of Dr. Goldratt. (“Jonah trained” is actually an outdated term. Official certification programs now exist through TOCICO, the Theory of Constraints International Certification Organization.)
George Spafford and I both took Dr. Holt’s courses and loved them. You not only gain an understanding of TOC and the Thinking Processes, but it also is a fantastic primer on plant floor planning and operations. For those of you who are interested in this course, EM526 is currently being taught as a Spring 2013 class. Maybe it’s not too late for you to register and attend by emailing him — Dr. Holt’s contact information is at the top of the course description page!
(If you can’t take Dr. Holt’s course and really want to learn more about the Thinking Processes, I’d recommend Dr. Dettmer’s textbook, “The Logical Thinking Process”, although it is not light reading. And many shortcuts have been developed that haven’t been reflected in Dr. Dettmer’s book, so danger lies in this path.)
If you’re interested in Dr. Goldratt’s work, I’d first recommend reading “The Goal,” and then listening to “Beyond The Goal.” And if you are still interested in learning more, take Dr. Holt’s fantastic courses. I hope you enjoy them as much as I did.
5 Dysfunctions Of A Team
The technique that Steve, the Parts Unlimited CEO, uses in Chapter 18 in Part 2 is modeled after Patrick Lencioni’s methodology described in “The Five Dysfunctions of a Team: A Leadership Fable.” He posits that one of the core contributors to a team’s inability to achieve goals is due to lack of trust. In his model, the five dysfunctions are described as:
Absence of trust—unwilling to be vulnerable within the group
Fear of conflict—seeking artificial harmony over constructive passionate debate
Lack of commitment—feigning buy-in for group decisions creates ambiguity throughout the organization
Avoidance of accountability—ducking the responsibility to call peers on counterproductive behaviour which sets low standards
Inattention to results—focusing on personal success, status and ego before team success
When I think about the long, bitter intertribal warfare that has existed between Development and IT Operations, as well as between IT and “the business,” I suspect that we will very much need the lessons of Mr. Lencioni to achieve the DevOps ideal.
Often, the first step in using Mr. Lencioni’s methodology is for leaders to enable themselves to become vulnerable (or at the very least, start by modeling vulnerable behaviors). In “The Phoenix Project,” Steve has already internalized these practices for decades, and leads what is called a personal history exercise.
I was fortunate enough to have personally observed and benefited from his techniques being used. When my old boss, Jim B. Johnson, used this technique when he first joined as CEO of Tripwire, Inc., frankly it blew my mind. He shared his own personal story, which was so personal and touching, it left the rest of us on the executive team emotionally raw, with tears in (almost) everyone’s eyes.
In turn, we all had to share some elements of our own personal story, showing vulnerability to each other, and enabling the next step, which is to stop fearing conflict. Jim set the tone of the honesty and candor he demanded from everyone, and trust me, it changed how we behaved, executed and we started acting more like a team.
This was probably one of the most important lessons in my life. It is now my aspiration in every domain of my life to never fear conflict, never be afraid to tell the truth, and never be afraid to say what I really think. Of course, I’d be delusional to think I can fully achieve this, but I think it’s still a worthy goal.
I’ve been in situations where I’ve observed leadership teams locked in chronic underperformance and strife, because of the utter inability for the team members to trust each other. And when leaders don’t trust each other, then almost certainly, their respective teams won’t trust each other.
From my professional experience, the cost and true consequence of not being able to have candid discussions about problems that everyone knows about, but is unwilling to confront is incredibly high. Tackling this problem requires overcoming some of our most ingrained and learned behaviors, but the rewards are worth it.
Toyota Kata
Another one of the books that deeply influenced us is “Toyota Kata: Managing People for Improvement, Adaptiveness and Superior Results” by the Shingo Prize winner Mike Rother. (The Shingo Prize is often called the “Nobel Prize For Manufacturing. Mr. Rother has been awarded this prize three times.)
I’ve had the pleasure of meeting him, as well as taking his three day course, “Improvement Kata and Coaching Kata,” offered through University of Michigan. It includes two days of fieldwork in a real manufacturing plant — I’ll be posting my review of this course sometime soon.
If I could take the liberty of describing Mr. Rother’s journey in my own words, it began over twenty years ago as he was visiting Toyota plants with a team of researchers and American car manufacturing executives. He describes that stage of his journey as capturing and codifying the observed Toyota practices that led to their extraordinary and market-leading performance.
However, when he looks back at that phase of his career, he’ll characterize it as merely teaching people how to mimic the behaviors observed at the Toyota plants, but missing the most important parts of the Toyota culture and values that framed their management practices.
Listening to him talk about this, it’s as if he was calling a significant portion of the Lean community that he helped train merely a “cargo cult.” (This term refers to how pre-industrial tribes behaved in New Guinea and Micronesia shortly after World War II. Having observed an influx of material goods brought by American and Japanese soldiers, these tribes were baffled when the war and supply deliveries ended. In an effort to bring back the deliveries, the tribes replicated the observed behaviors of the U.S. Corp of Engineers, building imitation runways and radio equipment. Despite all of this, for obvious reasons, the planes never returned.)
Mr. Rother’s learnings have been codified in the book, “Toyota Kata,” which frames the thought process and culture that must exist to enable the Lean PDCA cycle (Plan, Do, Check, Act). I believe that this is one of the most extraordinary contributions to the world of process improvement.
The most obvious manifestation of the Toyota Kata is the “two week improvement cycle,” in which every work center supervisor must improve something (anything!) every two weeks. To quote Mr. Rother, “The practice of kata is the act of practicing a a pattern so it becomes second nature. In its day-to-day management, Toyota teaches a way of working — a kata — that has helped make ti so successful over the last six decades.”
The notion of the need for daily repetition in order to create habits, in order to change the outcomes is now well-established in the domains of sports training, learning to play a musical instrument, how Special Forces in the military trains, and now in modern manufacturing.
This forms the basis of Erik’s Third Way that is “about creating a culture that fosters two things: continual experimentation, which requires taking risks and learning from success and failure; and understanding that repetition and practice is the prerequisite to mastery.”
In my mind, Patty’s ITIL/ITSM crusade is very much like the Lean practitioners that Mr. Rother describes who were never able to replicate the performance of Toyota. Why? They’d do a Lean Kaizan event once per year, but then get marginalized from daily operations the remainder of the year.
For us to get the performance gains promised by ITIL/ITSM, Lean, or whatever, we must create a culture of relentless improvement described by Mr. Rother.
Kata impacts your organization by:
Providing a systematic, scientific routine that can be applied to any problem or challenge.
Commonizing how the members of an organization develop solutions.
Migrating managers toward a role of coach and mentor, by having them practice coaching cycles.
Framing PDCA in a way that has people taking small steps every day.
These two week improvement cycles put constant pressure into the system, forcing it to improve. Mr. Rother asserts that if a system is not improving, the result is not a steady state. Instead, because of entropy, organizational performance declines.
In one of the most startling case studies, Mr. Rother describes observing how a certain work center that was able to decrease the number of workers from six to four. Over the next six weeks, however, the number of workers gradually grew back to six. Entropy.
In my mind, patterns like the Netflix culture that has created relentless improvement and innovation, ruthless eradication of variance, and injecting faults into the production environment (embodied in tools such as the famous Chaos Monkey) is the perfect day embodiment of the Improvement Kata that Mr. Rother describes.
Continuous Delivery
Erik’s First Way underscores the importance of “the performance of the entire system, as opposed to the performance of a specific silo of work or department — this as can be as large a division (e.g., Development or IT Operations) or as small as an individual contributor (e.g., a developer, system administrator).”
In the IT value stream, this is all about the left to right flow of work from Development into IT Operations. Probably the best embodiment of this work is Jez Humble and David Farley’s seminal book, “Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation.”
They codify many of the techniques required to replicate the famous 2009 Velocity Conference presentation,“10 Deploys A Day” given by John Allspaw and Paul Hammond, as well as the Agile system administration movement, described further here by John Willis.
Continuous delivery is the extension of continuous integration, which are the Development practices that include continuous builds, continuous testing, daily integration of branches back into trunk, testing in a clone of the production environment, etc. Continuous delivery techniques extend these processes all the way into the production environment.
(When I read this book, I almost doubled over in pain, realizing how much pain and poor decisions I could have averted if I had read this book four years prior. I was associated with an unnamed software company where we had a broken build system for over a year. Without automated builds, you can’t have automated testing, and without automated testing, you’re doomed to painful integration of developer branches, which leads to a downward spiral of less frequent and far more painful integrations, which slows down feature delivery and reduces software quality.)
Imagine my delight when Jez Humble told me that the exercise that the value stream map exercise that Bill and team went through in Chapter 31 of the Development to IT Operations production deployment process was almost precisely what he and Farley did in their book. Awesome.
Continuous delivery is the perfect embodiment of the First, Second and Third Ways, as it emphasizes small batch sizes (e.g., check into trunk daily), stopping the line when problems occur (e.g., no new work allowed when builds, tests or deployments fail; elevating the integrity of the system of work over the work itself), and the need to continually build the validation tests necessary to either prevent failures in production, or at the very least, detect and correct them quickly (e.g., the transition from manual process reviews to automated tests, especially in the ITSM release, change and configuration process areas).
Continuous deployment is a prerequisite for the high deploy rates characterized by DevOps, and is therefore a needed skillset for the modern DevOps practitioner. It will also be the salvation for a generation of ITSM practitioners. Read it.
Release It!: Design and Deploy Production-Ready Software
When you watch talks at the Velocity Conference about how organizations like Google, Netflix, Amazon, Twitter, Pinterest and others are engineering code and environments to operate at scale, you’re often hearing many of the ideas that were first promulgated by Mike Nygard in his fantastic book, “Release It!: Design and Deploy Production-Ready Software.”
(In fact, I distinctly remember sitting next to Adrian Cockcroft in one Velocity Conference session, listening to him rattle off the chapter/verse of the Nygard patterns as he recognized them in the talk. The person giving the presentation should have read the book.)
This is a book that helps span the Development and IT Operations divide, by showing developers and architects how to build applications that can be deployed, managed and survive in even the most hostile production environments. When you read this book, you’ll see in his patterns and lessons horror stories from your own past.
IT Operations practitioners need to read this book, too, in order to connect the dots of how specific Development decisions lead to bad production outcomes that they’ve experienced in the past. And more importantly, it will enable them to go to architecture or development meetings with concrete suggestions on how to avoid them in the future.
When Chris and his team deploy Project Unicorn that performs so well in production in ”The Phoenix Project,” they obviously read Mr. Nygard’s book.
A little story: I saw Mr. Nygard give an Ignite presentation at the 2010 DevOpsDays in Mountain View, and it was one of the most amazing talks I’ve ever seen. It was one of the most high-fidelity and intense descriptions of a disastrous application deployment that took weeks to recover from, requiring heroics from IT Operations. It still remains my genuine hope that the scenes of the Phoenix deployment failure are as disturbing as my experience watching Mr. Nygard’s presentation. :)
Visible Ops and ITIL Service Support
When Kevin Behr and I started studying high performing IT organizations in 1999, we found that these were the organizations that were simultaneously achieving the “highest IT service levels (e.g., MTTR, MTBF, change success rates, etc.), the earliest integration of information security into the software/service development lifecycle, the best posture of compliance (e.g., fewest number of repeat audit findings), and amazingly, the best IT efficiencies (e.g., server/sysadmin ratios).”
We had studied 11 high performing IT organizations, which included a bank, a stock exchange, a wireless billing service, a domain name service provider, and two IT service providers.
Kevin and I worked together to understand how these organizations made their “good to great” IT transformations, and codified this transformation in the books “The Visible Ops Handbook: Implementing ITIL in 4 Practical and Auditable Steps” and “Visible Ops Security: Achieving Common Security and IT Operations Objectives in 4 Practical Steps”, written by the same author team as “The Phoenix Project”.
A critical part of this journey was made possible by the “ITIL Service Support Book v2”, which no discussion about IT Operations would be complete. I loved this book when I first read it in 2000, as it catalogued and normalized the key processes that must exist in any high performing IT organization. But, ITIL remains a descriptive framework, and organizations don’t do descriptive frameworks — they do projects.
Our goal with the Visible Ops series of books was to create a prescriptive, ordered set of steps (projects) to replicate the outcomes we observed in high performers. (Incidentally, this is what we’re attempting to replicate in the upcoming “DevOps Cookbook.”
Conclusion
Well, I hope this list of resources was useful! Please let me know what you think, or if you have any comments or questions. Next up will be more about Dr. Goldratt’s Theory of Constraints.
The post Where To Learn More About Concepts In “The Phoenix Project” (Part 1) appeared first on IT Revolution.