Release scorecard definitions

​Google spreadsheet​
Release Hearbeat
Metric
How to Measure
Time updated to RC1 (PST)
Check Release Discussion channel history for date/time RC1 was cut.
How many RCs cut
Check Release Discussion channel history for how many RCs were cut for that release.
Number of days between when final RC is cut and the release date
Check Release Discussion channel for post with official release build. Oxygen = 16th - Day Final RC is cut
Community + customer bugs reported during release timeframe (17th to 16th)
With a new or existing Jira filter, with:
1.Project = Mattermost
2.Fix Versions = Latest released version
3.Issue Type = Bug
4.Label = Customer-bug and Community-bug
5.Created Date = 17th of the previous month
6.Release Date = 16th of the current month
Number of bugs reported within a week after the release
With a new or existing Jira filter, with:
1.Project = Mattermost
2.Issue Type = Bug
3.Status = Open
4.Label = Customer-bug and Community-bug
5.Created Date = Between the 16th and 23rd of the month
Number of customer bugs fixed during release
With a new or existing Jira filter, with:
1.Project = Mattermost
2.Fix Versions = Latest released version
3.Issue Type = Bug
4.Status = Closed and Resolved
5.Label = Customer-bug
Total valid bugs in fix version
After closing current release:
project = Mattermost AND issuetype = Bug AND resolution not in (Duplicate, "Cannot Reproduce", "Won't Fix") AND fixVersion = latestReleasedVersion()
Total valid regressions in fix version
After closing current release:
project = Mattermost AND issuetype = Bug AND resolution not in (Duplicate, "Cannot Reproduce", "Won't Fix") AND fixVersion = latestReleasedVersion() AND label = rc2, rc3
Total valid bugs found after RC1 is cut
After closing current release, adjust dates as per above, and use this Jira query:
1.Check Jira timezone + Pre-release timezone and make sure times match
2.Replace START with date (yyyy-MM-dd HH:mm) RC1 was cut
3.Replace END with date (yyyy-MM-dd HH:mm) test servers returned to master
project = Mattermost AND issuetype = Bug AND resolution not in (Duplicate, "Cannot Reproduce", "Won't fix") AND created > "START" AND created < "END"
Valid bugs found after RC1 pushed to next release
After closing current release, adjust dates as per above, and use this Jira query:
project = Mattermost AND issuetype = Bug AND resolution not in (Duplicate, "Cannot Reproduce", "Won't Fix") AND created > "START" AND created < "END" AND fixVersion = earliestUnreleasedVersion()
Valid bugs found after RC1 fix version = other (eg unscheduled, not set)
After closing current release, adjust dates as per above, and use this Jira query:
project = Mattermost AND issuetype = Bug AND created > "START" AND created < "END" AND resolution not in (Duplicate, "Cannot Reproduce", "Won't Fix") AND (fixVersion not in (latestReleasedVersion(), earliestUnreleasedVersion()) OR fixVersion is EMPTY)
(Non-security) Bugs requiring patch release
After any patch release goes out (after the normal release date): Check Changelog for total number of non-security patch releases.
Total features/improvements in fix version
With a new or existing Jira filter, with:
1.Project = Mattermost
2.Fix Versions = Latest released version
3.Issue Type = Story
4.Status = Closed and Resolved
Critical security issues found during release timeframe
With a new or existing Jira filter, check for Security Vulnerability tickets:
1.Project = Mattermost
2.Fix Versions = Latest released version
3.Impact = High
Moderate security issues found during release timeframe
With a new or existing Jira filter, check for Security Vulnerability tickets:
1.Project = Mattermost
2.Fix Versions = Latest released version
3.Impact = Medium
Minor security issues found during release timeframe
With a new or existing Jira filter, check for Security Vulnerability tickets:
1.Project = Mattermost
2.Fix Versions = Latest released version
3.Impact = Low

Metric	How to Measure
Time updated to RC1 (PST)	Check Release Discussion channel history for date/time RC1 was cut.
How many RCs cut	Check Release Discussion channel history for how many RCs were cut for that release.
Number of days between when final RC is cut and the release date	Check Release Discussion channel for post with official release build. Oxygen = 16th - Day Final RC is cut
Community + customer bugs reported during release timeframe (17th to 16th)	With a new or existing Jira filter, with: 1. Project = Mattermost 2. Fix Versions = Latest released version 3. Issue Type = Bug 4. Label = Customer-bug and Community-bug 5. Created Date = 17th of the previous month 6. Release Date = 16th of the current month
Number of bugs reported within a week after the release	With a new or existing Jira filter, with: 1. Project = Mattermost 2. Issue Type = Bug 3. Status = Open 4. Label = Customer-bug and Community-bug 5. Created Date = Between the 16th and 23rd of the month
Number of customer bugs fixed during release	With a new or existing Jira filter, with: 1. Project = Mattermost 2. Fix Versions = Latest released version 3. Issue Type = Bug 4. Status = Closed and Resolved 5. Label = Customer-bug
Total valid bugs in fix version	After closing current release: project = Mattermost AND issuetype = Bug AND resolution not in (Duplicate, "Cannot Reproduce", "Won't Fix") AND fixVersion = latestReleasedVersion()
Total valid regressions in fix version	After closing current release: project = Mattermost AND issuetype = Bug AND resolution not in (Duplicate, "Cannot Reproduce", "Won't Fix") AND fixVersion = latestReleasedVersion() AND label = rc2, rc3
Total valid bugs found after RC1 is cut	After closing current release, adjust dates as per above, and use this Jira query: 1. Check Jira timezone + Pre-release timezone and make sure times match 2. Replace START with date (yyyy-MM-dd HH:mm) RC1 was cut 3. Replace END with date (yyyy-MM-dd HH:mm) test servers returned to master project = Mattermost AND issuetype = Bug AND resolution not in (Duplicate, "Cannot Reproduce", "Won't fix") AND created > "START" AND created < "END"
Valid bugs found after RC1 pushed to next release	After closing current release, adjust dates as per above, and use this Jira query: project = Mattermost AND issuetype = Bug AND resolution not in (Duplicate, "Cannot Reproduce", "Won't Fix") AND created > "START" AND created < "END" AND fixVersion = earliestUnreleasedVersion()
Valid bugs found after RC1 fix version = other (eg unscheduled, not set)	After closing current release, adjust dates as per above, and use this Jira query: project = Mattermost AND issuetype = Bug AND created > "START" AND created < "END" AND resolution not in (Duplicate, "Cannot Reproduce", "Won't Fix") AND (fixVersion not in (latestReleasedVersion(), earliestUnreleasedVersion()) OR fixVersion is EMPTY)
(Non-security) Bugs requiring patch release	After any patch release goes out (after the normal release date): Check Changelog for total number of non-security patch releases.
Total features/improvements in fix version	With a new or existing Jira filter, with: 1. Project = Mattermost 2. Fix Versions = Latest released version 3. Issue Type = Story 4. Status = Closed and Resolved
Critical security issues found during release timeframe	With a new or existing Jira filter, check for Security Vulnerability tickets: 1. Project = Mattermost 2. Fix Versions = Latest released version 3. Impact = High
Moderate security issues found during release timeframe	With a new or existing Jira filter, check for Security Vulnerability tickets: 1. Project = Mattermost 2. Fix Versions = Latest released version 3. Impact = Medium
Minor security issues found during release timeframe	With a new or existing Jira filter, check for Security Vulnerability tickets: 1. Project = Mattermost 2. Fix Versions = Latest released version 3. Impact = Low

Release tips

How-to guides for Product

Last modified 6mo ago