How to measure R&D effectiveness

As a CTO or an organization manager, how can you understand and measure the R&D effectiveness of your team?

As a PMO or an effectiveness manager, what dimensions should you use to answer questions about R&D effectiveness?

With these two questions in mind, we will explore the performance analysis scenario to explain how it helps organization managers make R&D performance levels and trends transparent, analyze the root causes of performance issues, guide improvement actions, and measure the effects of improvements.

You can measure and analyze a team's R&D effectiveness across three dimensions: capacity, efficiency, and quality. During the analysis, focus on the following:

• Delivery rate: The number of requirements a team can deliver per unit of time. This is the requirement delivery throughput.

• Responsiveness: The time from when a requirement is proposed to when it is delivered online. This is the requirement delivery cycle.

• Delivery quality: The timeliness of discovering and fixing bugs during delivery. This also includes the total number of bugs.

Check the delivery rate

In the reports for the data analytics scenario, the Requirement Delivery Rate metric card lets you:

• View the number of requirements delivered per unit of time and the average for the selected period.

• View the delivery rate trend. You can use recent delivery volumes to plan the team's future delivery pace and external commitments.

Data on the metric card:

1. The horizontal axis represents time, divided into weeks.

2. The vertical axis represents the number of requirements. The height of the bar shows the number of requirements delivered in a week. The color of the bar shows the length of the delivery cycle.

On the Requirement Delivery Rate metric card, you can perform a deeper analysis:

1. Evaluate team delivery capability based on the delivery rate

You can use the team's recent delivery rate to predict its future rate. This helps you better plan the team's future workload. For example, if the number of requirements delivered per week for the last six weeks is 10, 12, 15, 13, 11, and 17, the average is 13. You can predict that the team can deliver about 13 requirements per week. This data helps you better schedule the delivery pace and make external commitments.

2. Promote continuous delivery by observing the release frequency

If there is a bar for every week, it means there is a weekly release. If there is a bar every two weeks, it means there is a release every two weeks, and so on.

Check responsiveness

In the reports for the data analytics scenario, you can check responsiveness using the Requirement Delivery Distribution and Requirement Cumulative Flow Diagram metric cards.

Requirement delivery distribution

First, look at the Requirement Delivery Distribution card. It lets you:

• View the distribution of go-live times for each requirement. This shows the team's release frequency.

• View the trend of the requirement delivery cycle. This shows the team's responsiveness and how it changes over time.

• Analyze historical data to predict future responsiveness.

Data on the metric card:

The requirement delivery distribution, also known as a requirement control chart, shows time on the horizontal axis and the requirement delivery cycle (in days) on the vertical axis. On the chart:

• A dot represents a delivered requirement. Its position on the horizontal axis is the delivery time. Its position on the vertical axis is the delivery duration for that requirement.

• The polyline represents the rolling mean of the requirement delivery cycle. It is the average of the current point and the 1, 3, 5, 7, or 9 points before and after it. The number of points used in the calculation varies with the number of items in the range.

• The blue shaded area represents the rolling standard deviation. This is the amount of drift between the actual data and the rolling mean.

• The horizontal line represents the average value of the requirement delivery cycle within the selected time range.

When you view the data on the Requirement Delivery Distribution card, you can analyze it from five perspectives:

1. Vertically, the lower the dots for delivered requirements, the better. This indicates a shorter cycle time, faster responsiveness, and better predictability.

2. Horizontally, the denser the dots, the better. This indicates that requirements are delivered frequently, meaning a higher release frequency.

3. Horizontally, the more uniform the distribution of dots, the better. This indicates that requirements are delivered continuously and stably, which is closer to continuous delivery. If the dots are intermittent but clustered, it indicates that requirements are delivered in batches.

4. The delivery cycle line represents a baseline level for the delivery cycle within the selected time period. The lower this level, the better.

5. The moving average polyline shows the trend of the requirement delivery cycle. A downward trend is ideal. It indicates that the team's responsiveness is continuously improving.

The Requirement Delivery Distribution chart shows whether a team has the ability to deliver requirements continuously and quickly. It helps the team review and analyze its effectiveness and set effectiveness goals based on historical performance. For business personnel, it lets them check the delivery team's effectiveness at any time and predict the go-live time for requirements.

The Requirement Delivery Distribution chart measures the results of delivery. To further analyze the entire delivery process, you can focus on the Requirement Cumulative Flow Diagram. This diagram provides a comprehensive view of metrics such as lead time (delivery cycle), work-in-progress (WIP) count, and delivery rate. It also shows the team's patterns for collaboration, planning, and requirement delivery. It is often used to identify opportunities for systematic improvement. The following section describes this diagram in more detail.

Requirement cumulative flow diagram

The Requirement Cumulative Flow Diagram metric card lets you:

• Check the average delivery cycle: The sum of the time a requirement spends in each stage. This is the time elapsed from start to finish before the requirement is delivered.

• Check the work-in-progress (WIP) count: The number of requirements in each stage. This can show the batch size and degree of parallelism for processing requirements.

• Check the delivery rate: The overall slope of the curve in the release stage. This can show the team's requirement delivery rate.

Data on the metric card:

1. The horizontal axis represents the date.

2. The vertical axis represents the cumulative number of completed requirements in each stage. The curve connects the cumulative values of each stage to show the completion trend of requirements over time.

The Requirement Cumulative Flow Diagram is both holistic and dynamic. It shows the team's overall collaboration pattern and the end-to-end dynamic delivery process. It also shows trends in the delivery pattern and capability. You can use the cumulative flow diagram to analyze the team's collaboration and delivery patterns and identify opportunities for improvement. You can analyze it from the following three aspects:

1. Team's planning pattern

Focus on the number and frequency of requirements entering the development stage. For example, in a project, if large batches of requirements enter the development stage infrequently (for example, once a month), this is a large-batch input model. This can easily lead to many requirements being processed in parallel, which lengthens the delivery cycle. Conversely, small-batch, high-frequency inputs keep the WIP count low and shorten the requirement delivery cycle.

2. Requirement transfer-to-test pattern

Transferring large batches of requirements to testing causes problems. Developed requirements must wait a long time before testing begins. This leads to more WIP and a longer requirement delivery cycle.

3. Requirement release pattern

The requirement release curve may look like a staircase. A wider step indicates a lower release frequency, meaning a longer interval between releases. You can also see that the longer the release interval, the more requirements are included in each release. The difficulty of a release increases with the number of requirements.

Check delivery quality

In the reports for the data analytics scenario, the Bug Trend and Bug Fix Distribution metric cards let you:

• View the trend of discovering and fixing bugs. This shows the team's delivery pattern.

• View the trend of the bug backlog. You can see if the discovery and fix distribution is reasonable. This shows the project's quality status.

• View the trend of the bug fix cycle. This shows the team's ability to fix bugs promptly.

Bug trend

First, look at the Bug Trend chart, as shown in the following figure:

Data on the metric card:

1. The horizontal axis represents the date. The red bars above the axis show the number of new bugs for the day. The green bars below the axis show the number of resolved bugs for the day.

2. The vertical axis represents the number of bugs. The orange curve shows the daily bug backlog (cumulative unresolved bugs).

On the Bug Trend metric card, you can analyze the following:

1. Analyze the team's delivery pattern

If no bugs are found for a long time, and then many new bugs appear in a concentrated period, this indicates a waterfall delivery model. If bugs are continuously discovered and resolved, and the bug backlog remains at a low level, this is closer to a continuous delivery model.

2. Judge delivery quality based on the size of the bug backlog

Before a requirement goes live, the number of bugs usually needs to be reduced to zero. If the project's bug backlog consistently remains at a low level, it indicates high delivery quality.

The following figure shows an example of a team transitioning from a mini-waterfall model to a continuous delivery model:

Bug fix distribution

Next, look at the Bug Fix Distribution chart:

Data on the metric card:

The bug fix distribution, also known as a bug control chart, shows time on the horizontal axis and the bug fix cycle (in days) on the vertical axis. On the chart:

• A dot represents a fixed bug. Its position on the horizontal axis is the fix time. Its position on the vertical axis is the fix duration for that bug.

• The polyline represents the rolling mean of the bug fix cycle. It is the average of the current point and the 1, 3, 5, 7, or 9 points before and after it. The number of points used in the calculation varies with the number of items in the range.

• The red shaded area represents the rolling standard deviation. This is the amount of drift between the actual data and the rolling mean.

• The horizontal line represents the average value of the bug fix cycle within the selected time range.

When you view the data on the Bug Fix Distribution chart, you can analyze it from four perspectives:

1. Vertically, the lower the dots for fixed bugs, the better. This indicates a shorter fix cycle and a faster fix capability.

2. Horizontally, the fewer the dots for fixed bugs, the better. Fewer dots mean fewer bugs, which indicates higher quality of the code submitted for testing.

3. The average fix duration line represents a baseline level for the team's bug fix cycle. The lower, the better. Many teams set bug fix goals, such as clearing bugs daily, which means bugs must be fixed within 24 hours of discovery.

4. The rolling mean polyline shows the trend of the bug fix cycle. A downward trend is ideal. It indicates that the team is fixing bugs faster and faster.

For a team, the bug fix distribution chart can be used in review meetings to analyze past quality performance. It can also be used to set bug fix goals based on historical data.

Overall review

You can observe a team's current R&D effectiveness across the three dimensions of capacity, efficiency, and quality. You can perform a targeted analysis by focusing on five charts:

• Requirement Delivery Rate: Shows the team's historical requirement delivery throughput. You can use it to predict future delivery capacity.

• Requirement Delivery Distribution: Shows the team's historical responsiveness. You can use it to predict future requirement delivery speed.

• Requirement Cumulative Flow Diagram: Shows the team's overall collaboration pattern. You can use it to analyze the team's delivery pattern and capability.

• Bug Trend Chart: Shows the team's historical process quality. You can use it to analyze the team's delivery pattern and quality status.

• Bug Fix Distribution: Shows the team's historical bug fix speed. You can use it to predict the team's future bug fix speed.

For more data analysis, you can also refer to charts such as Requirement Delivery Duration by Stage, Requirement Cumulative Flow Diagram, Bug Backlog by Member, and Bug Backlog Percentage.

Both within Alibaba and among most of our customers, the common goal is to shorten the requirement delivery cycle. In the 2-1-1 goal proposed by Alibaba, the first '2' stands for shortening the requirement delivery cycle to two weeks.

To try the reports for the data analytics scenario, you can use the 'Effectiveness Analysis and Measurement' report template in Apsara DevOps Insight. Go to Apsara DevOps Insight →