Release Scorecard Definitions

For https://docs.google.com/spreadsheets/d/1Aoj4OTaWoyrKIcQNiHH1MVoRG51T20Y_0w2tg5oVw-M/edit#gid=825551144

Release Output

Metric

How to Measure

Ratio of story-to-bug tickets

Total of feature tickets over total of bug tickets. Include current quality release and previous feature release.

Mean time from P1 bug report to delivery

With a new or existing Jira filter, with:

  1. Project = Mattermost

  2. Fix Versions = Latest released version

  3. Issue Type = Bug

  4. Label = P1

  5. Created Date = 17th of the previous month

  6. Release Date = 16th of the current month

Copy a list of Jira tickets with the above information and paste them into a spreadsheet. Calculate the average by using a formula “=Release Day-Created”.

Mean time from P2 bug report to delivery

With a new or existing Jira filter, with:

  1. Project = Mattermost

  2. Fix Versions = Latest released version

  3. Issue Type = Bug

  4. Label = P2

  5. Created Date = 17th of the previous month

  6. Release Date = 16th of the current month

Copy a list of Jira tickets with the above information and paste them into a spreadsheet. Calculate the average by using a formula “=Release Day-Created”.

Mean time from P3 bug report to delivery

With a new or existing Jira filter, with:

  1. Project = Mattermost

  2. Fix Versions = Latest released version

  3. Issue Type = Bug

  4. Label = P3

  5. Created Date = 17th of the previous month

  6. Release Date = 16th of the current month

Copy a list of Jira tickets with the above information and paste them into a spreadsheet. Calculate the average by using a formula “=Release Day-Created”.

Number of P1 bugs reported in production

Count of major issues reported in product, documentation or marketing by customers, CSMs, Support or Product. E.g. dot releases from customer reports, undocumented breaking changes.

Number of P2 bugs reported in production

Count of medium level issues reported in product, documentation or marketing by customers, CSMs, Support or Product. E.g. dot releases from customer reports, undocumented breaking changes.

Number of P3 bugs reported in production

Count of small issues reported in product, documentation or marketing by customers, CSMs, Support or Product. E.g. dot releases from customer reports, undocumented breaking changes.

Release Management

Metric

How to Measure

Number of misses in each release cycle

Count of issues encountered during the release cycle. E.g. no process for an early advance notification for upcoming deprecated ESR.

Number of setbacks to users/staff resulting in a negative reaction

Manual count of negative reactions reported by CSMs, Support or Product. E.g. features that got pushed.

Number of measurable actions from each release retrospective driven from root causes & completed and reported to R&D

Count of completed and reported items from Release Retrospective document.

Number of deadlines missed in release checklist

Manual count of deadlines not met from items posted to Release Checklist.

Release Hearbeat

Metric

How to Measure

Time updated to RC1 (PST)

Check Release Discussion channel history for date/time RC1 was cut.

How many RCs cut

Check Release Discussion channel history for how many RCs were cut for that release.

Number of days between when final RC is cut and the release date

Check Release Discussion channel for post with official release build. Oxygen = 16th - Day Final RC is cut

Community + customer bugs reported during release timeframe (17th to 16th)

With a new or existing Jira filter, with:

  1. Project = Mattermost

  2. Fix Versions = Latest released version

  3. Issue Type = Bug

  4. Label = Customer-bug and Community-bug

  5. Created Date = 17th of the previous month

  6. Release Date = 16th of the current month

Number of customer bugs fixed during release

With a new or existing Jira filter, with:

  1. Project = Mattermost

  2. Fix Versions = Latest released version

  3. Issue Type = Bug

  4. Status = Closed and Resolved

  5. Label = Customer-bug

Total valid bugs in fix version

After closing current release:

project = Mattermost AND issuetype = Bug AND resolution not in (Duplicate, "Cannot Reproduce", "Won't Fix") AND fixVersion = latestReleasedVersion()

Total valid bugs in fix version found by test automation

Check "Se", "Selenium-found, "Rainforest-found" Jira labels.

Total valid bugs found after RC1 is cut

After closing current release, adjust dates as per above, and use this Jira query:

  1. Check Jira timezone + Pre-release timezone and make sure times match

  2. Replace START with date (yyyy-MM-dd HH:mm) RC1 was cut

  3. Replace END with date (yyyy-MM-dd HH:mm) test servers returned to master

project = Mattermost AND issuetype = Bug AND resolution not in (Duplicate, "Cannot Reproduce", "Won't fix") AND created > "START" AND created < "END"

Valid bugs found after RC1 fixed in release

After closing current release, adjust dates as per above, and use this Jira query:

project = Mattermost AND issuetype = Bug AND resolution not in (Duplicate, "Cannot Reproduce", "Won't Fix") AND created > "START" AND created < "END" AND fixVersion = latestReleasedVersion()

Valid bugs found after RC1 pushed to next release

After closing current release, adjust dates as per above, and use this Jira query:

project = Mattermost AND issuetype = Bug AND resolution not in (Duplicate, "Cannot Reproduce", "Won't Fix") AND created > "START" AND created < "END" AND fixVersion = earliestUnreleasedVersion()

Valid bugs found after RC1 fix version = other (eg unscheduled, not set)

After closing current release, adjust dates as per above, and use this Jira query:

project = Mattermost AND issuetype = Bug AND created > "START" AND created < "END" AND resolution not in (Duplicate, "Cannot Reproduce", "Won't Fix") AND (fixVersion not in (latestReleasedVersion(), earliestUnreleasedVersion()) OR fixVersion is EMPTY)

(Non-security) Bugs requiring patch release

After any patch release goes out (after the normal release date): Check Changelog for total number of non-security patch releases.

Total features/improvements in fix version

With a new or existing Jira filter, with:

  1. Project = Mattermost

  2. Fix Versions = Latest released version

  3. Issue Type = Story

  4. Status = Closed and Resolved

Critical security issues found during release timeframe

With a new or existing Jira filter, check for Security Vulnerability tickets:

  1. Project = Mattermost

  2. Fix Versions = Latest released version

  3. Impact = High

Moderate security issues found during release timeframe

With a new or existing Jira filter, check for Security Vulnerability tickets:

  1. Project = Mattermost

  2. Fix Versions = Latest released version

  3. Impact = Medium

Minor security issues found during release timeframe

With a new or existing Jira filter, check for Security Vulnerability tickets:

  1. Project = Mattermost

  2. Fix Versions = Latest released version

  3. Impact = Low