Release scorecard definitions

• Looker dashboard

• Google spreadsheet

Release Hearbeat

Metric	How to Measure
Date/time updated to RC1 (PST)	Check Release Self-Hosted/Cloud channel history for date/time RC1 was cut.
How many RCs cut	Check Release Self-Hosted/Cloud channel history for how many RCs were cut for that release.
Number of days between when QA approval is posted and the release date	Check Release Self-Hosted/Cloud channel for post with official release build. Oxygen = 16th - Day Final RC is cut
Community + customer bugs reported during release timeframe (17th to 16th)	With a new or existing Jira filter, with: Project = Mattermost Fix Versions = Latest released version Issue Type = Bug Label = Customer-bug and Community-bug Created Date = 17th of the previous month Release Date = 16th of the current month
Number of bugs reported within a week after the release	With a new or existing Jira filter, with: Project = Mattermost Issue Type = Bug Status = Open Label = Customer-bug and Community-bug Created Date = Between the 16th and 23rd of the month
Number of customer bugs fixed during release	With a new or existing Jira filter, with: Project = Mattermost Fix Versions = Latest released version Issue Type = Bug Status = Closed and Resolved Label = Customer-bug
Total valid bug fixes in fix version	After closing current release: project = Mattermost AND issuetype = Bug AND resolution not in (Duplicate, "Cannot Reproduce", "Won't Fix") AND fixVersion = latestReleasedVersion()
Total valid regression fixes in fix version	After closing current release: project = Mattermost AND issuetype = Bug AND resolution not in (Duplicate, "Cannot Reproduce", "Won't Fix") AND fixVersion = latestReleasedVersion() AND label = rc2, rc3
Valid bugs found after RC1 is pushed to next release	After closing current release, adjust dates as per above, and use this Jira query: project = Mattermost AND issuetype = Bug AND resolution not in (Duplicate, "Cannot Reproduce", "Won't Fix") AND created > "START" AND created < "END" AND fixVersion = earliestUnreleasedVersion()
Valid bugs found after RC1 fix version = other (eg unscheduled, not set)	After closing current release, adjust dates as per above, and use this Jira query: project = Mattermost AND issuetype = Bug AND created > "START" AND created < "END" AND resolution not in (Duplicate, "Cannot Reproduce", "Won't Fix") AND (fixVersion not in (latestReleasedVersion(), earliestUnreleasedVersion()) OR fixVersion is EMPTY)
Total valid bugs found after RC1 is cut	After closing current release, adjust dates as per above, and use this Jira query: Check Jira timezone + Community server timezone and make sure times match Replace START with date (yyyy-MM-dd HH:mm) RC1 was cut Replace END with date (yyyy-MM-dd HH:mm) release was published project = Mattermost AND issuetype = Bug AND resolution not in (Duplicate, "Cannot Reproduce", "Won't fix") AND created > "START" AND created < "END"
(Non-security) Bugs requiring patch release	After any patch release goes out (after the normal release date): Check Changelog for total number of non-security patch releases.
Total features/improvements in fix version	With a new or existing Jira filter, with: Project = Mattermost Fix Versions = Latest released version Issue Type = Story Status = Closed and Resolved
Critical security issues found during release timeframe	With a new or existing Jira filter, check for Security Vulnerability tickets: Project = Mattermost Fix Versions = Latest released version Impact = High
Moderate security issues found during release timeframe	With a new or existing Jira filter, check for Security Vulnerability tickets: Project = Mattermost Fix Versions = Latest released version Impact = Medium
Minor security issues found during release timeframe	With a new or existing Jira filter, check for Security Vulnerability tickets: Project = Mattermost Fix Versions = Latest released version Impact = Low
Average lead time per fix version	Average number of days for all bugs with Resolution equal to Done. Project = Mattermost Issue type = Bug Resolution = Done Fix Version = Any Component = Cloud or null (on-prem)