Feature Release Process

Mattermost core team works on a monthly release process, with a new version shipping on the 16th of each month in binary form.
This document outlines the development process for the Mattermost core team, which draws from what we find works best for us from Agile, Scrum, and Software Development Lifecycle approaches.
Recommended pre-reading:
​Mattermost Software Development Process training materials​
​Mattermost Security Practices training (particularly NIST standards)
Release Timeline
Notes:
All cut-off dates are based on 10am (San Francisco Time) on the day stated.
T-minus counts are measured in "working days" (weekdays, Monday through Friday, excluding the listed statutory holidays) prior to release day.
A. (Code complete date of previous release) Beginning of Release
Pre-work for the current release begins at the code complete date of the previous release. See "Code Complete" section below for details.
B. (T-minus 30 working days) Judgment Day & Release Branch Cut
Day when Leads and PMs decide which major features are included in the release, and which are postponed. Stabilization period begins when all features for release have been committed. During this period, only bugs can be committed to the release branch. Non-bug pull requests are tagged for the next version. Exceptions can be made by the Release Manager during triage. Review the Release Features & Bugs Quality Gate Guidelines​
(Team) Judgment Day Meeting (10:00am San Francisco time):
Begin daily triage of tickets with the team
Also triage tickets in the backlog
Release Manager:
Post this checklist in Release Checklist channel
Prior to the cut to Self-Managed, the Release manager reviews new feature flags and provides a report to the PM/QA team for their review. Essentially a feature will be included in a Self-Managed release once the feature flag has been removed.
Discuss turning off feature flags if severe bugs found
Confirm with PMs that each Enterprise feature is in the correct pricing SKU​
Review any features that are currently in beta and confirm with PMs if there are any to be promoted
Review the Jira tickets remaining in the current release fix version and push those that won't make it to the next fix version
After mobile release branch is cut, ask dev to cut an RN build
Queue a list of MVP candidates in alphabetical order to the MVP Discussion channel See example​
Draft changelog in a WIP PR with updates for highlights, feature additions, known issues, compatibility updates for deprecated features, config.json, database changes, API changes, WebSocket event changes, and Go runtime version; see example​
Specify the release type with a link to the Release Definitions doc​
Review that software requirements are up-to-date based on these step-by-step guidelines​
Ask PMs and Dev Leads if there are any notable breaking changes or deprecated features in the release
Update Upgrade Guide with any special notes for upgrading to the new version
If there are any breaking compatibility changes in the release, open an issue in the GitLab Omnibus to make sure GitLab is aware. Post a link to the issue in the Release: Self-Managed channel
Start posting a daily list of open and submitted bugs and PRs
Submit NOTICE.txt PRs for any new libraries added from dev, and ensure they are cherry-picked to feature release branch
Confirm that NOTICE.txt PRs, Translations PRs, database upgrade, and any needed pre-packaged plugins, backports, and recent regression fixes are included in the release
Create meta issue for release in GitHub (see example)
Post a reminder in the Localization channel about the due date for translations, similar to this example​
Follow that translations are prioritized at https://translate.mattermost.com/projects/mattermost/​
Prepare bullet points for release announcement and share for PMs to work on. Refer to the process here​
Ask PMs and Tech Writer to complete release documentation by T-9 deadline. Example request​
Schedule a meeting with PMs and QAs to discuss upcoming features in the next feature release
Dev:
Cut release branches for server and mobile
Merge database upgrade before cutting the branch
Update model/version.go​
Point translation server and CI servers to the release branch after cutting
Update https://prev.test.mattermost.com to the previous release version
Prioritize reviewing, updating, and merging of pull requests for current release until there are no more tickets in the pull request queue marked for the current release
After the cut-off, any PRs that include significant code changes require approval of the Release Manager before merging
QA:
Prioritize testing PRs and resolved tickets for this release
Ensure that new features are also properly tested on mobile apps
Prioritize updating release tests in test management and automated tests
Identify any new teammates who will be joining release testing, send them an intro to the testing process and timeframe, send them the hardware/software survey​
Set up DM/GM channels in preparation for testing auto-closing after 7 days
C. (T-minus 9 working days) RC1 Cut
Release Manager:
Post this checklist in Release Checklist channel
Verify all items in the last posted release checklist are complete
Post in the Release: Self-Managed channel the rough timing when the release candidate will be cut
Cut a Release Candidate using this process and check CI servers running on release branch
Generate a list of contributors for changelog
Confirm doc submission
Post list of tickets to be fixed to the Release: Self-Managed channel (see example)
Update the GitHub meta issue:
Include a link to the changelog on the documentation branch
Update download links and testing server links to the latest RCs
After build is cut, tweet announcement that RC1 is ready (see example)
Update Open Source Components and Release Lifecycle docs
Logistics @hanna.park:
Mail out contributor and security researcher mugs
Space out the ordering of mugs over the next three weeks to prevent mistakes being made by the supplier (e.g., If there are 12 contributors to order mugs for, place an order every 2nd or 3rd day over the next three weeks)
Update Team page with new contributors
QA:
Confirm up to date with testing PRs and resolved tickets
Confirm up to date with test updates and known issues in release test management and automated tests
Assign release testing areas to team members
After RC1 is cut: Update test server invite links in Release Testing instructions
After RC1 is cut: Run automated regression tests
Build:
Review all TODO notes, including one for uncommenting upgrade code
Confirm all PRs in /enterprise repo have been merged
Update Redux before each RC and Final build
Master is tagged and branched and Release Candidate 1 is cut (e.g. 3.5.0-RC1) according to the Release Candidate Checklist in mattermost/process
Update version for each Mattermost Helm chart​
Make PRs for bug fixes to the release branch
Review PRs made from release branch and merge changes into the release branch as required
Run daily automated upgrade tests to avoid catching upgrade bugs late
D. (T-minus 8 to T-minus 7 working days) Release Candidate Testing
(Team) Daily Release Update Meeting
Triage Jira tickets
Decide when to cut next RC
Release Manager:
Post this checklist in Release Checklist channel
Verify all items in the last posted release checklist are complete
Confirm changelog reflects any changes since it was merged (including known issues and contributors from all repositories)
Confirm translators have been added
Confirm Open Source Components changes have been added
Find www-gitlab-com merge request for latest GitLab release blog post and make request for adding GitLab Mattermost update (see example request, example update). Post to Release: Self-Managed channel with link to request
Confirm the required URLs for the blog/release announcement were prepared
QA:
Update Release: Self-Managed header with links to RC instances and testing spreadsheet (template)
Post release testing instructions to Release: Self-Managed channel (template)
Post reminders in Announcements channel (template) and Customer Support channel (template)
DM reminders to team members who are not QA or devs, or who are new to release testing
Midday: Post reminders about testing, at-mentioning team members whose tests are not yet complete
At end of day, post reminders about release testing in Release: Self-Managed and Announcements channels, DM any team members who have zero test cells marked Done
Find QA or other teammates to help finish unfinished tests if needed
End of day or next morning: Verify all release tests are finished, bring any concerns to Triage/Release meeting
Go through all tabs of testing spreadsheet and verify all comments and questions have been filed in Jira as needed
Logistics @hanna.park:
Generate an E20 5000 seat test licence and email to Lindy for release testing
PM:
Finish draft of blog post for mattermost.com and all art work (screenshots, GIFs, and Twitter banners) used for the blog post
E. (T-minus 5 working days) Code Freeze
Day after which only S1 regressions should be fixed (crashes, security vulnerabilities) and no other code changes should be committed. Exceptions can be made by the Release Manager during triage on a case-by-case basis. Review the Bug Severity Guidelines.
(Team) Daily Release Update Meeting
Continue to triage Jira tickets
If no blocking issues are found, PM, Dev, and QA sign off on the release and begin final smoke tests
Release Manager:
Post this checklist in Release Checklist channel
Verify all items in the last posted release checklist are complete
Verify that the final translations PR for the release is submitted
Update Known Issues section with any significant issues that were found and not fixed for the final release
QA:
Verify all Jira tickets other than newly-filed bugs have been tested, verified, and closed
As bug fixes are merged and RCs are cut, verify fixes on new RCs, and post in Release Channel after testing
After all tickets are verified and closed, run smoke tests on webapp/server, desktop app, and RN apps as appropriate 
Docs:
Submit Changelog PR for team review
Submit any remaining documentation PRs for product updates in the release
Check that a redirect page has been set up in mattermost.com for any links added to the System Console
Submit documentation for API changes and WebSocket event changes to API documentation
Confirm changes to config.json in compatibility section of changelog are written back to settings documentation​
Confirm all new diagnostics are documented in the telemetry docs (https://docs.mattermost.com/administration/telemetry.html)
F. (T-minus 2 working days) Release Build Cut
The final release is cut - RC cuts and bug fixes should be completed by this date. Only urgent and critical issues are considered for fixing.
Release Manager:
Post this checklist in Release Checklist channel
Verify all items in the last posted release checklist are complete
Work with a developer to submit GitLab MR following this process and test the upgrade once the GitLab MR is merged and included in their RC
Close GitHub meta ticket for the release
Add the download links, SHA-256 hash, and GPG signature to the version archive​
Merge changelog PR after review is complete
If there is a security fix, confirm the changelog recommends upgrade, with a note mentioning the security level and thanking the security researcher (security process doc for example)
Update the Mattermost server download page with the links to the Enterprise Edition and Team Edition sections
Test the download links before and after updating the page
Check security issues and confirm disclosure text
Update the security mugs spreadsheet​
Check the security researcher was added to the Responsible Disclosure Policy page
Update deprecated feature list in mattermost.com with new and scheduled deprecations
Draft Mattermost Security Updates, but do not post until 30 days after official release 
Add a placeholder text saying "Details on the security update will be posted here on X date, as per our Responsible Disclosure Policy"
QA:
Verify all PRs and tickets for the release have been tested/closed
Verify Selenium server was put on final RC and build passed
Verify smoke tests on webapp/server, desktop app, and RN apps all passed
Post QA approval in Release: Self-Managed channel
Build:
Update Redux before each RC and Final build
Tags a new release (e.g. 1.1.0) and runs an official build which should be essentially identical to the last RC
Posts SHA key, md5 sum, and GPG signatures of the final build to Release: Self-Managed channel
Post in Release: Self-Managed with links to the Enterprise Edition and Team Edition sections
Dev:
Verify the hashes (SHA-1, SHA-256, and md5) and GPG signatures are correct for both Enterprise Edition and Team Edition
Test upgrade from previous version to current version, following the Upgrade Guide with database upgrades on both MySQL and Postgres
Test upgrade from Team Edition to Enterprise Edition based on the Upgrade Guide​
Test fresh install of current version, following the Install Guide​
Review any changes made to install guides, and test if necessary
Ensure Security Policies page has been updated
Update dependancies after release branch is cut in mattermost-server, mattermost-webapp, desktop, mattermost-mobile, and mattermost-redux
Logistics @hanna.park:
Order the release MVP winner's coaster
Docs:
Remind PMs and Tech Writer to finalize docs
Merge the docs release branch to master and verify all changes on docs.mattermost.com once the build is up
Submit a correction PR for any incorrect formatting or other errors missed during the initial review
Marketing:
Receive sign off on final version of MailChimp email blast and Twitter announcement and schedule for 08:00 PST on the date of marketing announcement
Note: If the release contains a security update, also draft a Mailchimp email blast for the Security Bulletin mailing list​
Finalize blog post for mattermost.com, test on mobile view, and set timer for 08:00 PST on the day of release
Add links to Admin guide in the release blog post where needed
G. (T-minus 0 working days) Release Day
Release Manager:
Post this checklist in Release Checklist channel
Verify all items in the last posted release checklist are complete
Update the server upgrade in-product notice​
Schedule a release retrospective meeting, to be held within five days of the release
Prepare and post release metrics​
Finalize release summary slide deck to prepare for posting for CSMs
Review and update company roadmap with which major features made it into the release
Review with Head of Product prior to updating
Also update feature lists on https://mattermost.com/pricing/ and https://mattermost.com/product/ with relevant new features where needed
Update the Mattermost Wikipedia page with the latest version
Post the MVP winner announcement in the Contributors channel​
Update MVP page with the most valued professional of the release
Add new release fix versions in Jira for the next few releases
Post key dates for the next release in the Release: Self-Managed channel and remove links to RC candidates and testing spreadsheet from the header
Make sure that statutory holidays for Canada and US are accounted for in the release dates
Check for any UserVoice feature suggestions that were completed in the current release
Find the release tweet and insert a link to the tweet next to the feature that shipped with the release
Close the release in Jira both for webapp and mobile (releases page)
For the next release, create the following team meetings. If they conflict with existing meetings, check with meeting owner to reschedule or reschedule the release meeting
Bug Bash Meeting on the week after Code Complete at 10:00am San Francisco time
Release Triage and Update Meeting each weekday starting at T-20 and ending at T-0 at 9:30am San Francisco time for PM, QA, and release dev
Prepare tickets for the next release, with a corresponding vX.X prefix, and put the tickets in the appropriate sprints as follows:
The week RC is cut:
​RC Build Testing for core team​
The week RC is cut:
​Loadtest x.x release candidate compared to x.x release​
The week RC is cut (for GitLab dev owner):
Test RC1 with the latest GitLab build during release testing cycle
Release week (for dependencies owner)
Upgrade dependencies for webapp, server, and Redux
Week after release (for GitLab dev owner)
​Submit GitLab Omnibus RC install of Mattermost​
The week before code complete (one for Apps and for Mobile):
If an existing ESR is going out of support next month, update the in-app prompts (mobile and desktop) to start detecting for the new minimum supported ESR version
If a new ESR is released next month, update the prompts to recommend upgrading to that version instead of the older ESR
Confirm that mattermost-docker has been updated to the latest version (contact the maintainer via direct message on community server if necessary)
Contact owners of community installers or submit PRs to update install version number
For Yunohost, open a new pull request to update the version. See example​
Turn on CrazyEgg for blog post page
Confirm marketing has been posted (screenshots, mail announcement, tweets, blog posts)
Post in the Announcements channel
Update @mattermost Twitter profile with the next release date
Prepare retweet of GitLab release tweet (see example here)
Docs:
Create a new branch on docs for the next release - vX.X-documentation
Submit a PR for changelog against the vX.X-documentation branch and add a Work in Progress label for it
Submit a PR to change version number in docs/source/conf.py against the vX.X-documentation branch
Build:
Ensure https://community.mattermost.com is on the most recently released version and that https://prev.test.mattermost.com is on the previous release version
Verify that the release tags were added to GitHub
QA:
Merge any updates made to Selenium tests during release testing
Update RN server URLs to Rainforest
H. (T-plus 30 days) Update Mattermost Security Page
Release Manager:
Post Mattermost Security Updates after reviewing with security lead
If a dot release is shipping with security fixes, do not post new details until T-plus 10 working days from the dot release ship date
Release Overview
Cloud Release Process
Last updated 2 weeks ago