Tag Archives: KPIs

Agile Transformation Metrics

TL;DR

When asked to provide metrics to assess “how well” an Agile transformation is going, re-frame the discussion around measuring changes in the impact the IT organization is having (or not) on it’s Business environment, and define a small set of “fitness for purpose” metrics.

The Inevitable Question about Agile Transformation Metrics

Sooner or later, as an IT organization embarks on a transformation towards Agile mindset and practices, someone will be asked to provide “hard evidence” that the effort is paying off, and the conclusion will be that metrics is the vehicle to satisfy that request. What are your Agile transformation metrics?

It’s been my experience that this request usually leads to a discussion about measuring the specific Agile initiatives the IT organization has launched. In organizations where the emphasis has been around engineering disciplines, such metrics might be things like unit test code coverage, or integration build times. If the focus  was around teams and process, then counting number of teams “converted” to Scrum, or people sent to Scrum Master training may appear as the choice.

While those measurement might be useful indicators in some context, they have two problems. First, they are akin to measuring the performance of the car engine without looking outside the window; the engine might be performing well, but if the car doesn’t have the wheels attached, we’re going nowhere. More importantly, though, these figures are usually meaningless for Business stakeholders, who are the ones usually asking for them in the first place.  Agile transformation metrics need to be meaningful to the Business.

Re-framing the Agile Transformation Metrics Question

Agile transformation efforts can be very costly exercises, therefore it is legitimate to ask about the results of such endeavour. The important thing to realize, though, is that this question is really equivalent to another question: “is the IT organization improving its impact on its Business environment.” Another way to put it is, borrowing from the terminology used by the Kanban community: “is the IT organization becoming more and more fit for purpose?” Answering this question, of course, requires a clear understanding of what is that the Business expects from its interactions with IT.

The IT organization can be seen as providing various services to customers. Arguably, if IT has decided to “transform” in some way (perhaps by moving towards an Agile mindset), it’s doing so to improve its impact on those customers, so this is what needs to be measured to know “how the transformation” is going.

Some of those customers are different areas of the organization (like Finance, or HR.) But it doesn’t stop there, because the Business’ engagement with IT doesn’t have value for its own sake. Ultimately, the Business is using IT as a way to optimize its operations so that it can provide external customers with more effective products and services. Moreover, IT is these days the direct channel through which those products and services are delivered to external customers (for example, through web sites and mobile applications.) Therefore, the concept of “fitness for purpose” of the IT organization can be extended to consider the fitness for purpose of the Business respect the external customers it intends to serve.

Defining the “Agile” Transformation Metrics

Measuring “agile transformation success” really means measuring the success of the exchanges between IT and the Business, and between the Business and its external customers.  Measuring the internal processes and practices that IT puts in place as part of that “transformation” is beside the point. This implies starting with a careful definition of the boundaries that delineate the exchanges to be measured. There might be more to external customer fitness for purpose than IT operations, for example, and that needs to be considered when defining Agile transformation metrics, especially if we’re later going to be drawing causation conclusions.

Defining Agile transformation metrics will be, of course,  a highly contextual exercise because every business organization is different.  But we can, however, draw again from the Kanban community for some general guidelines on what to look for. Their thought leaders talk about classifying metrics into 3 categories: fitness for purpose metrics, health indicators and improvement drivers.  Using this framework, when talking about “agile transformation metrics” we are referring mainly to the first category, and perhaps a bit to the second. Based on those, improvement initiatives can be put in place, and perhaps driven with metrics belonging to the third category.

A fitness for purpose metric (also known as KPI) is an indicator of something a customer will care about. This is a key distinction: if the metric is not easily recognizable and meaningful for the customer, then it’s not a KPI. Another key characteristic is that a minimum threshold for its value can be defined: if the metric goes below the threshold, the Business is putting the relation with its customers at risk (perhaps they will walk away, initiate legal actions, etc.). In other words, the Business is no longer “fit for purpose”. We can then measure the effectiveness of the “agile transformation” by analyzing how KPI values over time compare to their respective thresholds. A typical KPI is delivery time, measured from the moment a customer request is accepted and committed to, until the moment it’s delivered to production.  This is usually a good Agile transformation metric.

Health indicators are metrics that are inwards facing. Customers don’t really care about them (or even understand), but they indicate how a given aspect of the system is operating. The key characteristic is that they are not directly actionable; they only provide information that needs to be analyzed and put in context. As the value of a health indicator changes, we can draw some conclusions about how the system works, or explain why something is happening (or not), but it doesn’t necessarily leads to concrete action. Defect count is an example of this. Customers will certainly care about quality of the product, and we can make inferences about that quality by looking at how many defects we have, but the absolute number of defects will not necessarily make the product more or less fit for purpose. It may happen that customers consider the current quality to be “good enough”, irrespective of the number of defects.

Finally, improvement driver metrics are metrics put in place to influence behaviour towards a particular change. Their key characteristic is that they are temporary: we set a target on them and once the target is achieved, the metric is no longer necessary. The reason for this is related to the unintended behaviours that a metric might encourage in people, which may lead to locally optimizing the metric at the expense of other aspects, leading to global sub-optimization of the system. An example is unit testing code coverage: if we have determined that a given service is not fit for purpose and the cause is related to poor unit test coverage, then establishing a target for minimum coverage may influence developers to work on adding tests to reverse the situation.


Affiliated Promotions:

Try our automated online Scrum coach: Scrum Insight - free scores and basic advice, upgrade to get in-depth insight for your team. It takes between 8 and 11 minutes for each team member to fill in the survey, and your results are available immediately. Try it in your next retrospective.

Please share!
Facebooktwittergoogle_plusredditpinterestlinkedinmailFacebooktwittergoogle_plusredditpinterestlinkedinmail
Berteig
Upcoming Courses
View Full Course Schedule
Certified ScrumMaster® (CSM) [Virtual Learning]
Online
C$1595.00
Jan 26
2021
Details
Seven Design Thinking Tips for Product Owners - Lunch Session
Online
C$0.00
Feb 2
2021
Details
Certified ScrumMaster® (CSM) [Virtual Learning]
Online
C$1595.00
Feb 8
2021
Details
Team Kanban Practitioner® (TKP) [Virtual Learning]
Online
C$1195.00
Feb 10
2021
Details
Certified Scrum Product Owner® (CSPO) [Virtual Learning]
Online
C$1795.00
Feb 11
2021
Details
Banking on Kanban - Kanban in the Financial Sector
Online - Instructor Led
C$149.00
Feb 11
2021
Details
Kanban System Design® (KMP I) [Virtual Learning]
Online
C$1795.00
Feb 17
2021
Details
Advanced Certified ScrumMaster® (A-CSM) [Virtual Learning]
Online
C$1695.00
Feb 17
2021
Details
Certified ScrumMaster® (CSM) [Virtual Learning]
Online
C$1355.75
Feb 23
2021
Details
Dealing with Slackers, Idiots and Subversives in Agile Teams - Lunch Session
Online
C$0.00
Feb 23
2021
Details
Deadlines Are Dead To Me - Dealing with Deadlines in Agile Teams
Online - Instructor Led
C$149.00
Feb 23
2021
Details
Kanban Systems Improvement® (KMP II) [Virtual Learning]
Online
C$1525.75
Feb 25
2021
Details
Master Kanban for Scrum - Kanban for Scrum Masters
Online - Instructor Led
C$149.00
Feb 25
2021
Details
Stop the BS Metrics! Performance Reviews in Agile Teams - Lunch Session
Online
C$0.00
Mar 2
2021
Details
Kanban Your Projects - Kanban for Project Managers
Online - Instructor Led
C$149.00
Mar 4
2021
Details
Advanced Certified Scrum Product Owner® (A-CSPO) [Virtual Learning}
Online
C$1440.75
Mar 9
2021
Details
Leave Us Alone! - Dealing with Urgent Interruptions in Agile Teams
Online - Instructor Led
C$149.00
Mar 9
2021
Details
Team Kanban Practitioner® (TKP) [Virtual Learning]
Online
C$1015.75
Mar 10
2021
Details
Banking on Kanban - Kanban in the Financial Sector
Online - Instructor Led
C$149.00
Mar 10
2021
Details
Deadlines Are Dead To Me - Dealing with Deadlines in Agile Teams
Online - Instructor Led
C$149.00
Mar 18
2021
Details
Certified ScrumMaster® (CSM) [Virtual Learning]
Online
C$1355.75
Mar 22
2021
Details
Certified Scrum Product Owner® (CSPO) [Virtual Learning]
Online
C$1525.75
Mar 25
2021
Details
What the Heck is Kanban? - Introduction to the Kanban Method
Online - Instructor Led
C$149.00
Mar 25
2021
Details
Kanban System Design® (KMP I) [Virtual Learning]
Online
C$1525.75
Mar 30
2021
Details
Advanced Certified ScrumMaster® (A-CSM) [Virtual Learning]
Online
C$1440.75
Mar 30
2021
Details
Kanban Your Projects - Kanban for Project Managers
Online - Instructor Led
C$149.00
Mar 31
2021
Details
Master Kanban for Scrum - Kanban for Scrum Masters
Online - Instructor Led
C$149.00
Apr 1
2021
Details
Certified ScrumMaster® (CSM) [Virtual Learning]
Online
C$1355.75
Apr 6
2021
Details
Banking on Kanban - Kanban in the Financial Sector
Online - Instructor Led
C$149.00
Apr 6
2021
Details
Leave Us Alone! - Dealing with Urgent Interruptions in Agile Teams
Online - Instructor Led
C$149.00
Apr 6
2021
Details
Four Strategic Tips on using Technology and Digital Media Effectively - Lunch Session
Online
C$0.00
Apr 6
2021
Details
How to Effectively Achieve Work/Life Balance in your Environment - Lunch Session
Online
C$0.00
Apr 7
2021
Details
Team Kanban Practitioner® (TKP) [Virtual Learning]
Online
C$1015.75
Apr 8
2021
Details
Should we just Ban Facebook (and other Social Media) in the Workplace? - Lunch Session
Online
C$0.00
Apr 8
2021
Details
Critical Steps on how to be an Effective Colleague/Manager in the Digital Age - Lunch Session
Online
C$0.00
Apr 9
2021
Details
Certified Scrum Product Owner® (CSPO) [Virtual Learning]
Online
C$1525.75
Apr 13
2021
Details
What the Heck is Kanban? - Introduction to the Kanban Method
Online - Instructor Led
C$149.00
Apr 15
2021
Details
Certified ScrumMaster® (CSM) [Virtual Learning]
Online
C$1355.75
Apr 21
2021
Details
Master Kanban for Scrum - Kanban for Scrum Masters
Online - Instructor Led
C$149.00
Apr 22
2021
Details
Deadlines Are Dead To Me - Dealing with Deadlines in Agile Teams
Online - Instructor Led
C$149.00
Apr 22
2021
Details
Advanced Certified Scrum Product Owner® (A-CSPO) [Virtual Learning]
Online
C$1440.75
Apr 27
2021
Details
Kanban Your Projects - Kanban for Project Managers
Online - Instructor Led
C$149.00
Apr 27
2021
Details
Advanced Certified ScrumMaster® (A-CSM) [Virtual Learning]
Online
C$1440.75
Apr 29
2021
Details
Certified ScrumMaster® (CSM) [Virtual Training]
Online
C$1355.75
May 3
2021
Details
Team Kanban Practitioner® (TKP) [Virtual Learning]
Online
C$1015.75
May 5
2021
Details
What the Heck is Kanban? - Introduction to the Kanban Method
Online - Instructor Led
C$149.00
May 6
2021
Details
Leave Us Alone! - Dealing with Urgent Interruptions in Agile Teams
Online - Instructor Led
C$149.00
May 6
2021
Details
Certified Scrum Product Owner® (CSPO) [Virtual Learning]
Online
C$1525.75
May 6
2021
Details
Certified ScrumMaster® (CSM) [Virtual Training]
Online
C$1355.75
May 18
2021
Details
Kanban System Design® (KMP I) [Virtual Learning]
Online
C$1525.75
May 18
2021
Details
Certified Scrum Product Owner® (CSPO) [Virtual Learning]
Online
C$1525.75
May 25
2021
Details
Advanced Certified ScrumMaster® (A-CSM) [Virtual Learning]
Online
C$1440.75
May 27
2021
Details
Kanban Systems Improvement® (KMP II) [Virtual Learning]
Online
C$1525.75
Jun 1
2021
Details