Release scorecard definitions

Release Output

Metric
How to Measure
Ratio of story-to-bug tickets
Total of feature tickets over total of bug tickets. Include current quality release and previous feature release.
Mean time from P1 bug report to delivery
With a new or existing Jira filter, with:
  1. 1.
    Project = Mattermost
  2. 2.
    Fix Versions = Latest released version
  3. 3.
    Issue Type = Bug
  4. 4.
    Label = P1
  5. 5.
    Created Date = 17th of the previous month
  6. 6.
    Release Date = 16th of the current month
Copy a list of Jira tickets with the above information and paste them into a spreadsheet. Calculate the average by using a formula “=Release Day-Created”.
Mean time from P2 bug report to delivery
With a new or existing Jira filter, with:
  1. 1.
    Project = Mattermost
  2. 2.
    Fix Versions = Latest released version
  3. 3.
    Issue Type = Bug
  4. 4.
    Label = P2
  5. 5.
    Created Date = 17th of the previous month
  6. 6.
    Release Date = 16th of the current month
Copy a list of Jira tickets with the above information and paste them into a spreadsheet. Calculate the average by using a formula “=Release Day-Created”.
Mean time from P3 bug report to delivery
With a new or existing Jira filter, with:
  1. 1.
    Project = Mattermost
  2. 2.
    Fix Versions = Latest released version
  3. 3.
    Issue Type = Bug
  4. 4.
    Label = P3
  5. 5.
    Created Date = 17th of the previous month
  6. 6.
    Release Date = 16th of the current month
Copy a list of Jira tickets with the above information and paste them into a spreadsheet. Calculate the average by using a formula “=Release Day-Created”.
Number of P1 bugs reported in production
Count of major issues reported in product, documentation or marketing by customers, CSMs, Support, or Product (e.g. dot releases from customer reports, undocumented breaking changes).
Number of P2 bugs reported in production
Count of medium level issues reported in product, documentation or marketing by customers, CSMs, Support, or Product (e.g. dot releases from customer reports, undocumented breaking changes).
Number of P3 bugs reported in production
Count of small issues reported in product, documentation or marketing by customers, CSMs, Support, or Product (e.g. dot releases from customer reports, undocumented breaking changes).

Release Management

Metric
How to Measure
Number of misses in each release cycle
Count of issues encountered during the release cycle (e.g. no process for an early advance notification for upcoming deprecated ESR).
Number of setbacks to users/staff resulting in a negative reaction
Manual count of negative reactions reported by CSMs, Support, or Product (e.g. features that got pushed).
Number of measurable actions from each Release Retrospective driven from root causes and completed and reported to R&D
Count of completed and reported items from Release Retrospective document.
Number of deadlines missed in Release Checklist
Manual count of deadlines not met from items posted to Release Checklist.

Release Hearbeat

Metric
How to Measure
Time updated to RC1 (PST)
Check Release Discussion channel history for date/time RC1 was cut.
How many RCs cut
Check Release Discussion channel history for how many RCs were cut for that release.
Number of days between when final RC is cut and the release date
Check Release Discussion channel for post with official release build. Oxygen = 16th - Day Final RC is cut
Community + customer bugs reported during release timeframe (17th to 16th)
With a new or existing Jira filter, with:
  1. 1.
    Project = Mattermost
  2. 2.
    Fix Versions = Latest released version
  3. 3.
    Issue Type = Bug
  4. 4.
    Label = Customer-bug and Community-bug
  5. 5.
    Created Date = 17th of the previous month
  6. 6.
    Release Date = 16th of the current month
Number of customer bugs fixed during release
With a new or existing Jira filter, with:
  1. 1.
    Project = Mattermost
  2. 2.
    Fix Versions = Latest released version
  3. 3.
    Issue Type = Bug
  4. 4.
    Status = Closed and Resolved
  5. 5.
    Label = Customer-bug
Total valid bugs in fix version
After closing current release:
project = Mattermost AND issuetype = Bug AND resolution not in (Duplicate, "Cannot Reproduce", "Won't Fix") AND fixVersion = latestReleasedVersion()
Total valid bugs in fix version found by test automation
Check "Se", "Selenium-found, "Rainforest-found" Jira labels.
Total valid bugs found after RC1 is cut
After closing current release, adjust dates as per above, and use this Jira query:
  1. 1.
    Check Jira timezone + Pre-release timezone and make sure times match
  2. 2.
    Replace START with date (yyyy-MM-dd HH:mm) RC1 was cut
  3. 3.
    Replace END with date (yyyy-MM-dd HH:mm) test servers returned to master
project = Mattermost AND issuetype = Bug AND resolution not in (Duplicate, "Cannot Reproduce", "Won't fix") AND created > "START" AND created < "END"
Valid bugs found after RC1 fixed in release
After closing current release, adjust dates as per above, and use this Jira query:
project = Mattermost AND issuetype = Bug AND resolution not in (Duplicate, "Cannot Reproduce", "Won't Fix") AND created > "START" AND created < "END" AND fixVersion = latestReleasedVersion()
Valid bugs found after RC1 pushed to next release
After closing current release, adjust dates as per above, and use this Jira query:
project = Mattermost AND issuetype = Bug AND resolution not in (Duplicate, "Cannot Reproduce", "Won't Fix") AND created > "START" AND created < "END" AND fixVersion = earliestUnreleasedVersion()
Valid bugs found after RC1 fix version = other (eg unscheduled, not set)
After closing current release, adjust dates as per above, and use this Jira query:
project = Mattermost AND issuetype = Bug AND created > "START" AND created < "END" AND resolution not in (Duplicate, "Cannot Reproduce", "Won't Fix") AND (fixVersion not in (latestReleasedVersion(), earliestUnreleasedVersion()) OR fixVersion is EMPTY)
(Non-security) Bugs requiring patch release
After any patch release goes out (after the normal release date): Check Changelog for total number of non-security patch releases.
Total features/improvements in fix version
With a new or existing Jira filter, with:
  1. 1.
    Project = Mattermost
  2. 2.
    Fix Versions = Latest released version
  3. 3.
    Issue Type = Story
  4. 4.
    Status = Closed and Resolved
Critical security issues found during release timeframe
With a new or existing Jira filter, check for Security Vulnerability tickets:
  1. 1.
    Project = Mattermost
  2. 2.
    Fix Versions = Latest released version
  3. 3.
    Impact = High
Moderate security issues found during release timeframe
With a new or existing Jira filter, check for Security Vulnerability tickets:
  1. 1.
    Project = Mattermost
  2. 2.
    Fix Versions = Latest released version
  3. 3.
    Impact = Medium
Minor security issues found during release timeframe
With a new or existing Jira filter, check for Security Vulnerability tickets:
  1. 1.
    Project = Mattermost
  2. 2.
    Fix Versions = Latest released version
  3. 3.
    Impact = Low
Export as PDF
Copy link
Edit on GitHub