Transforming Quality Culture
My thoughts on some current challenges relating to software delivery and quality culture
So many organisations delivering software are trying do it faster, but there has often been the view that to go fast, you must sacrifice quality.
In Accelerate, the findings of Forsgren, Humble and Kim show that deployment frequency is a driver of software delivery performance. In my experience, there has been a fair amount of cynicism directed towards these findings. I suspect that's because the headline claims are taken not only without digging into the data that support them, but without the experience of working in a way that is described in the book in terms of the key capabilities.
Strictly speaking, my job title is "Test Manager". Whether or not that's the only/actual role I perform these days is another discussion altogether, but because I have done a fair bit of software testing over the years, I was always particularly interested in this specific finding:
"…having automated tests primarily created and maintained either by QA or an outsourced party is not correlated with IT performance" (Forsgren, Humble and Kim, 2018)
There are debates (even in my own head) about whether this is related to external test teams, i.e. those who, as is described in Team Topologies, are not in a stream aligned team, or those whose role as test analyst/test automation engineer/etc is embedded within the stream aligned team.
Regardless, the research tells us that developers owning test automation is good.
And in my organisation, this feels like it makes lots of sense. We have some great testers and some great test automation people, but not enough to do it all. And even going out to the market to recruit, we've struggled to find enough good people with the skills required to come in as dedicated test automation engineers.
Two big questions come from proposing that developers now own automated testing:
What do we do with our testers?
Is this yet another thing for developers to have to deal with?
Let's start with question 2.
In my organisation, we are going through a technical transformation everywhere. Moving away from waterfall delivery, WinForms, and the Oracle monolith to more of a service-oriented architecture, trying to be more agile, deliver more frequently, implement CI/CD etc. All the things that everyone is trying to do.
Feedback from our developers has been that there is a lot going on at one time, it’s hard to keep up with the pace of changes and some are experiencing extraneous cognitive overload.
The question, or problem statement, we have appears to be along the lines of:
How, with limited technical testing resource, can an organisation implement a shift left automated testing approach without overloading developers who are already experiencing elevated levels of extraneous cognitive load?
I would argue that the question is wrong.
Testing is an activity that is as germane to software development as writing application code. Fast feedback on the functional correctness of an application against the requirements is fundamental to achieving the acceleration of shippable quality to our business users.
I do not believe that software testing, of any kind, should be categorised as extraneous cognitive load in the same way that writing YAML files, or configuring Roadie, or understanding Helm charts may be.
Who is responsible for the quality of this software?
Particularly with trunk-based development and CI/CD, as a developer, if you commit a code change to main, it can/should go to production behind a feature flag. So, who is responsible for the quality of the software?
I believe that a lot of (not all!) developers don't want to write automated acceptance tests, not out of a place of laziness, but rather they see it as someone else's job. And why wouldn't they when we have people in the team whose title is test automation engineer? Their behaviour reflects the environment around them. Developers don't manage the project backlog, that's what a project manager or product owner does. So why would a developer write automated tests? That's what the test automation engineer does.
Having developers own all automated testing is a big cultural change, so how can we change the quality culture in an organisation?
In an MIT Sloan Management Review paper, John Shook describes his experiences in transforming the culture of the NUMMI (New United Motor Manufacturing Incorporated) car manufacturing plant in California that became the genesis of the Lean manufacturing movement in the US:
"What my NUMMI experience taught me that was so powerful was that the way to change culture is not to first change how people think, but instead to start by changing how people behave - what they do". (Shook, 2010)
There's a fascinating radio show where they talk about this transformation here: 561: NUMMI (2015) - This American Life
If we apply this example here, it amounts to a top down (even visually depicted on the diagram above) instruction on how people will behave. “You are a developer, and you now do testing”.
This clearly worked on the production line, but what impact would strongly issuing a mandate around developer behaviour have on morale and productivity?
Players or pawns?
"As the strategy guru Gary Hamel has observed, management is a technology… Its central ethic remains control; its chief tools remain extrinsic motivators. That leaves it largely out of sync with the nonroutine, right-brain abilities on which many of the world's economies now depend" (Pink, 2018)
Edward Deci and Richard Ryan, two American Psychologists who developed the Self Determination Theory, categorise behaviour in two ways: controlled and autonomous.
"Autonomous motivation involves behaving with a sense of volition and choice… whereas controlled motivation involves behaving with the experience of pressure and demand toward specific outcomes that comes from forces perceived to be external to the self" (Pink, 2018)
Pink argues that controlling people's behaviour extrinsically can damage motivation and productivity.
We want to increase motivation and improve productivity, so how can we change the culture in teams without stifling people's autonomous motivation?
Decisions…
If we agree with John Shook, that to change culture you must change how people behave, and we also agree with Dan Pink, that controlled motivation is damaging to morale and productivity, we need to find a way to change behaviour without telling people what to do…
I've pondered over this a lot, and unfortunately for some of the testers we had at my organisation, we recently reduced the contractor headcount and so we have had to adapt to less testers and behaviour had to change. We had no choice. I do not think this is a terrible thing. We're in a strange middle ground place just now though where some teams have test automation engineers embedded in a stream aligned team, but others do not have any testers at all.
Truthfully, if I had my druthers, I'd reduce the number of testers in project teams, such that it is unreasonable for them to be expected to write and implement all of the test automation. This isn't about getting rid of testers and forcing developers to do testing. It's about creating the environment where it makes sense for the behaviours to change in a way that the data tells us will drive software delivery performance. This brings us back to question number 1, way back towards the start of the (way longer than I intended it to be) article: what do we do with our testers?
We have testers (albeit fewer) in project teams to help the team test better. I often hear developers who are sceptical about taking on more testing talk about the "tester mindset", and how testers approach things from a distinct perspective which adds value in a way developers cannot. Whether you agree with this or not, fine. We can have the "tester mindset" in the team, but they don't implement the test automation. They can go to 3 amigos and bring their "tester mindset" with them, help drive out ambiguity in requirements, ask questions that others may not ask and help to define the test scenarios. But the developers implement them, as part of development. The developers are responsible for the functional correctness of the system. I hope that over time developers can enjoy doing this more independently.
"Inspection does not improve the quality, nor guarantee quality. Inspection is too late. The quality, good or bad, is already in the product. As Harold F. Dodge said, “You cannot inspect quality into a product.”" (W Edwards Deming, 1986)
In my organisation we have a team working like this now, developers own all test automation, and a tester reviews their scenarios with their trusty mindset. The early developer feedback has been positive. They feel motivated, they can feel work moving across the board quicker and there is more of a focus on delivery than getting through tasks. The quantitative metrics dashboard that we use to measure deployment frequency, product delivery lead time, call frequency and mean time to restore is also showing favourable readings. It's too early to draw concrete conclusions from these numbers at this stage though.
A welcome side effect of this approach is that we can place less emphasis on testers' technical skills. In my organisation we have loads of testers with deep domain knowledge who don't necessarily have the technical skills to independently implement automated tests. Rather than penalising them for not being able to write C# or shoehorning them into a role that does not suit their skills, we can use the technical skills of the developers and the domain knowledge of testers to maximise the strengths of the team. We see real value from team members having strong business knowledge in our investments and trading domain.
It's maybe not considered best practice, I'm not sure, but I believe it's a positive step in moving towards continuous testing and faster quality delivery.
To (finally) conclude
I believe that if we remove the safety net of dedicated testing specialists, who are there to do testing (NB: we still have testers, they are important, but their job is to help the team test better, not to do the testing), we can change the behaviour, and therefore the quality culture of teams without directly issuing a mandate for a specific behaviour in an environment where it doesn't make sense, potentially damaging motivation and productivity. It can have the opposite effect.
Like all things, it’s important to approach this with an open mind and with empathy. Change is hard, and in the end, we’re all on the same team.