Release scorecard definitions

Release Hearbeat

MetricHow to Measure

Date/time updated to RC1 (PST)

Check Release Discussion channel history for date/time RC1 was cut.

How many RCs cut

Check Release Discussion channel history for how many RCs were cut for that release.

Number of days between when QA approval is posted and the release date

Check Release Discussion channel for post with official release build. Oxygen = 16th - Day Final RC is cut

Community + customer bugs reported during release timeframe (17th to 16th)

With a new or existing Jira filter, with:

  1. Project = Mattermost

  2. Fix Versions = Latest released version

  3. Issue Type = Bug

  4. Label = Customer-bug and Community-bug

  5. Created Date = 17th of the previous month

  6. Release Date = 16th of the current month

Number of bugs reported within a week after the release

With a new or existing Jira filter, with:

  1. Project = Mattermost

  2. Issue Type = Bug

  3. Status = Open

  4. Label = Customer-bug and Community-bug

  5. Created Date = Between the 16th and 23rd of the month

Number of customer bugs fixed during release

With a new or existing Jira filter, with:

  1. Project = Mattermost

  2. Fix Versions = Latest released version

  3. Issue Type = Bug

  4. Status = Closed and Resolved

  5. Label = Customer-bug

Total valid bug fixes in fix version

After closing current release:

project = Mattermost AND issuetype = Bug AND resolution not in (Duplicate, "Cannot Reproduce", "Won't Fix") AND fixVersion = latestReleasedVersion()

Total valid regression fixes in fix version

After closing current release:

project = Mattermost AND issuetype = Bug AND resolution not in (Duplicate, "Cannot Reproduce", "Won't Fix") AND fixVersion = latestReleasedVersion() AND label = rc2, rc3

Valid bugs found after RC1 is pushed to next release

After closing current release, adjust dates as per above, and use this Jira query:

project = Mattermost AND issuetype = Bug AND resolution not in (Duplicate, "Cannot Reproduce", "Won't Fix") AND created > "START" AND created < "END" AND fixVersion = earliestUnreleasedVersion()

Valid bugs found after RC1 fix version = other (eg unscheduled, not set)

After closing current release, adjust dates as per above, and use this Jira query:

project = Mattermost AND issuetype = Bug AND created > "START" AND created < "END" AND resolution not in (Duplicate, "Cannot Reproduce", "Won't Fix") AND (fixVersion not in (latestReleasedVersion(), earliestUnreleasedVersion()) OR fixVersion is EMPTY)

Total valid bugs found after RC1 is cut

After closing current release, adjust dates as per above, and use this Jira query:

  1. Check Jira timezone + Community server timezone and make sure times match

  2. Replace START with date (yyyy-MM-dd HH:mm) RC1 was cut

  3. Replace END with date (yyyy-MM-dd HH:mm) release was published

project = Mattermost AND issuetype = Bug AND resolution not in (Duplicate, "Cannot Reproduce", "Won't fix") AND created > "START" AND created < "END"

(Non-security) Bugs requiring patch release

After any patch release goes out (after the normal release date): Check Changelog for total number of non-security patch releases.

Total features/improvements in fix version

With a new or existing Jira filter, with:

  1. Project = Mattermost

  2. Fix Versions = Latest released version

  3. Issue Type = Story

  4. Status = Closed and Resolved

Critical security issues found during release timeframe

With a new or existing Jira filter, check for Security Vulnerability tickets:

  1. Project = Mattermost

  2. Fix Versions = Latest released version

  3. Impact = High

Moderate security issues found during release timeframe

With a new or existing Jira filter, check for Security Vulnerability tickets:

  1. Project = Mattermost

  2. Fix Versions = Latest released version

  3. Impact = Medium

Minor security issues found during release timeframe

With a new or existing Jira filter, check for Security Vulnerability tickets:

  1. Project = Mattermost

  2. Fix Versions = Latest released version

  3. Impact = Low

Average lead time per fix version

Average number of days for all bugs with Resolution equal to Done.

  1. Project = Mattermost

  2. Issue type = Bug

  3. Resolution = Done

  4. Fix Version = Any

  5. Component = Cloud or null (on-prem)

Last updated