data: exploration of review times

[cpcloud]

This PR should not be merged.

This is a notebook that explores the P95 and P99 values of a few review metrics.

The purpose of putting it in a PR is to allow folks to explore the data
themselves, hopefully driving discussions on how we can improve.

How to use the notebook

cd cuda_core
pixi run -e rev-stats jupyter lab rev-stats.ipynb

then run all the cells.

Metrics

• Time to First Review: Duration between PR creation and first review
• Time to Merge: Duration between PR creation and time to merge
• Time to Close: Duration between PR creation and either closing or merging it
• Time from Final Review to Close: Duration between the final review comment and merge

Places where we are doing well

1. Time from Final Review to Close: This is in a solid place, we're not
   waiting too long to click the merge button after approval.
   P95: 2 days, P99: 14 days (this isn't ideal, but not concerning).

Places we can improve

1. Time to First Review: P95 here is 9.7 days, which means most PRs get
   a review within that time. P99 is 45 days, which is something we should
   address.
2. Time to Close: This metric includes merges along with PRs that are closed but not merged. P95 is 28D, P99 is 76D.
   While these are distributions with long tails, I think we can greatly improve the P95 here.
3. Time to Merge: This is a subset of Time to Close, and it's a bit
   better (but not much). Given that a PR is going to get merged, it tends to
   be merged faster than one that isn't. We of course don't know a priori whether a PR is guaranteed to be merged.

Would love to see what others think!

Perhaps there are other interesting metrics to calculate that would help us
determine how to improve our PR turnaround times.

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[mdboom]

Perhaps there are other interesting metrics to calculate that would help us
determine how to improve our PR turnaround times.

Just brainstorming:

• In an ideal world, the reviews would be spread evenly across all potential reviewers. In practice, most PRs are probably bottlenecked on the most experienced reviewers. It might be interesting to track this overtime and make sure we are "growing more experts".

• I don't think you could glean this from the data set, but it would be interesting to know why reviews are delayed -- is it because the reviewer doesn't feel like they have enough context or experience in the code base? Maybe encouraging a culture of self-reporting "I don't feel like I can give this a good review", and then bonus points for "I will spend the time reading / poking / experimenting enough so that I feel confident". That last part is expensive, of course.