DevOps – One small step for a man, one giant leap for….the organization!
More than 40 IT leaders from a leading Aerospace manufacturing company took part in The Phoenix Project Business Simulation, organized by GamingWorks together with CGI. Why? Driven by mounting business pressure the organization needs to explore the possibilities of DevOps. The senior IT Leadership recognized the ‘cultural’ barriers between Dev and Ops and wanted to use the simulation workshop to create a mindset shift, and to gain more commitment for adopting DevOps practices.
The workshop location, on one side of the road, was a sturdy 1960’s building, dating from the glory days of the Apollo space program in which Neil Armstrong stepped onto the Lunar surface and uttered those immortal words ‘That is one small step for a man, and one giant leap for mankind’. The building housed the traditional project driven – plan, build and run IT organization. Across the road in stark contrast was a gleaming, minimalistic glass and metal building housing the business and the new ‘Digital’ team, representing the shiny new future.
Why this article?
This article describes some key learning and considerations captured by this organization. A recent Forrester report stated that only 13% of organizations surveyed have fully implemented DevOps, 27% of organizations are planning to implement, and 9% have no plans in the immediate future, the rest are still experimenting. This means a lot of organizations will be faced with the challenges discovered by the leaders in this workshop.
The Phoenix Project DevOps business simulation
The Phoenix Project business simulation is a dynamic, classroom based, business game. A team of players, representing the end-to-end delivery chain are assigned roles and responsibilities and placed in a realistic environment. In the simulation the team must effectively communicate and collaborate and apply DevOps practices in order to succeed. The simulation is played in a number of rounds. At the end of each round the team must demonstrate performance improvements in terms of ‘Number of deployments’, ‘Successful deployments’, ‘Resolution effectiveness’ and ‘Business Value realized’. Between rounds the team learns to ‘reflect’ and apply ‘continual learning and improvement’
What happened?
During the initial game round the team experienced many situations they faced in their daily work. Confusion of roles, no clear ‘flow’ of work, wastage and rework caused by poor testing, unclear priority of work, lack of insight into what people are working on, unplanned work caused by mistakes and rework, delayed deployments….the business was being put at risk.
Between rounds the team explored elements from the 3 ways of DevOps and made small iterative improvements. The whole team, representing business, development and operations stood around a Visual Management System that they had created to give more insight and aid decision making. In the next round they effectively collaborated end-to-end as opposed to Silo’s and gave feedback whenever the flow was stopping or a defect (or lack of information) was passed downstream. The results were a significantly improved time to deploy through a smoother release and a ‘single piece flow’ approach, reduction in waste and unplanned work, and business value.
What were the key takeaways?
At the end of the session delegates were asked ‘what did you apply in this simulation that you need to take away and start to apply in your organization’?
These were some of their key discoveries:
“There is no doubt, DevOps practices could make a huge difference to our capabilities”
BizDevOpsBiz: ‘Currently we have 147 new business projects, with poor business cases (Value to be realized) and a history of poor measurement downstream (was Value realized). Having business also engaged at the end will help with the second way – ‘Feedback’ learning how to improve business cases, requirements specification and the need for more involvement in testing in subsequent business initiatives, preventing wasted resources, rework and damaged business value.
Value Stream mapping: ‘Now we understand more about the First way of DevOps we need to select some Pilot business ‘products’ and map all the stakeholders and activity flow. Identifying where waste is occurring, where the constraints are in the system, and where the most ‘waiting’ occurs now. This will also help point out to Business product owners their responsibilities and where they need to fit in’.
Feedback -‘STOP’: ‘In terms of the Second way in DevOps we need to call ‘STOP’ whenever we see a defect (can also be incorrect information, poor requirements, assumptions) so that we can give direct feedback and explore ways to stop this happening again. It also helps to start shifting the mindset and shape a new culture ‘never pass a defect downstream’ and ‘build quality in’ – in everything you give downstream’.
Prioritize: ‘We need to agree together with Business Product owners a Priority mechanism for the 147 initiatives, especially when we recognize we have constraints in shared operations resources. Prioritization MUST be balanced around new features, removing technical debt, AND experimenting and improving our DevOps capabilities. Priority criteria must be clear to teams to empower them to make the right decisions and focus on doing the right work’.
Visual Management: ‘The act of building the Visual management System (or KanBan) is a great way to create understanding of the flow, the dependencies and helps create a discussion on WIP limits, limiting work in progress and making tough decisions on Priorities. It also helped us identify when testing needs to be done and the steps we could eventually automate. A key learning in this was to build this with an end-to-end view’.
Test Driven Development: ‘In terms of the second way we need to ‘shift left’ with testing. Build testing in earlier and make it everybody’s responsibility to think ‘test’ to ensure defects are not passed downstream. Testers are too late in the process now and don’t always know what else has changed and what needs testing. They always lack time. Often the business doesn’t get engaged, which also results in rework and waste or features that don’t meet user needs’.
Autonomy& empowerment: ‘We need to give more autonomy to teams to prioritize and make decisions. Managers must provide the ‘Context’ – ensuring priorities are understood, and not micro-manage all decision making. Managers must ensure there are less escalations that cause a ping-pong effect – bouncing ‘up and down’ through management layers (waste and waiting) and ping-ponging back and forward between departments (waste and waiting)’.
Automation: ‘On the one side we need to automate the flow, which is a Lean exercise of automating for inefficiencies and preventing waste, on the other side we need to consider the complete way in which solutions are architected and provisioned. Infrastructure as code is a scary concept when you own your own ‘pets’ (Servers). This needs exploring’.
An Inconvenient fact
An uncomfortable discovery was the fact that there is no ‘DevOps best practice’, the fact that many organizations have their own interpretation of what DevOps is! Where to start? How big to make it? How to organize around It? ‘This is not simply a matter of adopting and implementing a set of best practices like ITIL’ said one manager. ‘This requires Leadership, top level commitment and some tough decisions’.
As one delegate stated ‘Breaking down existing departments, span of control, political and emotional fiefdoms is going to be a challenge. How did other organizations deal with the tough senior level decisions and commitment that will be necessary?’
Those of you reading this who are also asking ‘How to organize around this’ may want to look at DevOps Topologies . This page describes the types of structures organizations need to consider when starting their Journey, stating ‘Clearly there is no magic conformation or team topology which will suit every organization’, adding ‘in reality a combination of more than one pattern, or one pattern transforming into another, will often be the best approach’.
It is a Leap into the unknown for the organization! An experimentation, and a willingness to make mistakes, learn from them and improve.
What did the IT organization decide after the workshop?
It was agreed to select 2 Pilot business products, to do a Value Stream Mapping exercise. To use this exercise to significantly reduce waste and speed up deployment, to demonstrate an ability to deliver business value and at the same time ‘experiment and learn’ from applying ‘DevOps type’ practices. It was also agreed to arrange site visits to see how other organizations starting their DevOps journeys structured the teams and organization and how they architected the technology….
Just in case you think this might be a good idea here are 2 interesting blogs on Value Stream mapping.
Lean Value Stream Mapping for DevOps – which provides a handy little template.
Case Study using Value Stream Mapping by my good friend Daniel Breston which reveals what might happen when you start to do this.
….As we left the workshop back into the glaring Sunlight reflecting off the glass faced building across the street, I reflected. The building may only be a stone’s throw away, but there is a huge gulf between the current IT organization, with its structure and ways of working, and what is happening across the street. The business owners are already embarking upon a ‘Digital Transformation strategy’, Digital is owned by the Business. It is spawning a lot of Shadow-IT investments, and the traditional IT organization is not part of this strategic initiative. The digital initiatives driven by the business stem from frustrations caused by IT responsiveness and lack of agility in responding to business needs.
As one delegate observed ‘DevOps may be one small step for the engineers and the teams, but is represents one giant leap for the organization. It represents a leap into the unknown. Are we willing to take that leap and accept that risk’?
But as a wall street journal article warned ‘Enterprises are not ready for DevOps but will not survive without it’ – you could replace word ‘Enterprises’ with ‘Traditional IT organizations’.
Final thought
If you haven’t already embarked upon your DevOps journey you may want to ask yourselves ‘Do you still have the trust and credibility to be seen as a partner for the business or are you in danger of becoming irrelevant?’ The choice may no longer even be in your hands.
You may want to fill in this ‘Digital Readiness Quiz’ from the Institute for Digital Transformation and see how you compare to Industry results.
I am curious to see how you score on point 4. Perhaps a reason to engage you end-to-end decision makers and teams in a Simulation exercise!