Release scorecard definitions
Last updated
Last updated
Metric | How to Measure |
---|
Date/time updated to RC1 (PST) | Check Release Self-Hosted/Cloud channel history for date/time RC1 was cut. |
How many RCs cut | Check Release Self-Hosted/Cloud channel history for how many RCs were cut for that release. |
Number of days between when QA approval is posted and the release date | Check Release Self-Hosted/Cloud channel for post with official release build. Oxygen = 16th - Day Final RC is cut |
Community + customer bugs reported during release timeframe (17th to 16th) | With a new or existing Jira filter, with:
|
Number of bugs reported within a week after the release | With a new or existing Jira filter, with:
|
Number of customer bugs fixed during release | With a new or existing Jira filter, with:
|
Total valid bug fixes in fix version | After closing current release: project = Mattermost AND issuetype = Bug AND resolution not in (Duplicate, "Cannot Reproduce", "Won't Fix") AND fixVersion = latestReleasedVersion() |
Total valid regression fixes in fix version | After closing current release: project = Mattermost AND issuetype = Bug AND resolution not in (Duplicate, "Cannot Reproduce", "Won't Fix") AND fixVersion = latestReleasedVersion() AND label = rc2, rc3 |
Valid bugs found after RC1 is pushed to next release | After closing current release, adjust dates as per above, and use this Jira query: project = Mattermost AND issuetype = Bug AND resolution not in (Duplicate, "Cannot Reproduce", "Won't Fix") AND created > "START" AND created < "END" AND fixVersion = earliestUnreleasedVersion() |
Valid bugs found after RC1 fix version = other (eg unscheduled, not set) | After closing current release, adjust dates as per above, and use this Jira query: project = Mattermost AND issuetype = Bug AND created > "START" AND created < "END" AND resolution not in (Duplicate, "Cannot Reproduce", "Won't Fix") AND (fixVersion not in (latestReleasedVersion(), earliestUnreleasedVersion()) OR fixVersion is EMPTY) |
Total valid bugs found after RC1 is cut | After closing current release, adjust dates as per above, and use this Jira query:
project = Mattermost AND issuetype = Bug AND resolution not in (Duplicate, "Cannot Reproduce", "Won't fix") AND created > "START" AND created < "END" |
(Non-security) Bugs requiring patch release | After any patch release goes out (after the normal release date): Check Changelog for total number of non-security patch releases. |
Total features/improvements in fix version | With a new or existing Jira filter, with:
|
Critical security issues found during release timeframe | With a new or existing Jira filter, check for Security Vulnerability tickets:
|
Moderate security issues found during release timeframe | With a new or existing Jira filter, check for Security Vulnerability tickets:
|
Minor security issues found during release timeframe | With a new or existing Jira filter, check for Security Vulnerability tickets:
|
Average lead time per fix version | Average number of days for all bugs with Resolution equal to Done.
|