Feature Release Process

Mattermost core team works on a monthly release process, with a new version shipping on the 16th of each month in binary form.

This document outlines the development process for the Mattermost core team, which draws from what we find works best for us from Agile, Scrum, and Software Development Lifecycle approaches.

Recommended pre-reading:

Release Timeline

Notes:

A. (Code complete date of previous release) Beginning of Release

Pre-work for the current release begins at the code complete date of the previous release. See "Code Complete" section below for details.

B. (T-minus 30 working days) Judgment Day & Release Branch Cut

Day when Leads and PMs decide which major features are included in the release, and which are postponed. Stabilization period begins when all features for release have been committed. During this period, only bugs can be committed to the release branch. Non-bug pull requests are tagged for the next version. Exceptions can be made by the Release Manager during triage. Review the Release Features & Bugs Quality Gate Guidelines

  1. (Team) Judgment Day Meeting (10:00am San Francisco time):

    • Begin daily triage of tickets with the team

      • Also triage tickets in the backlog

  2. Release Manager:

    • Post this checklist in Release Checklist channel

    • Prior to the cut to Self-Managed, the Release manager reviews new feature flags and provides a report to the PM/QA team for their review. Essentially a feature will be included in a Self-Managed release once the feature flag has been removed.

      • Discuss turning off feature flags if severe bugs found

    • Confirm with PMs that each Enterprise feature is in the correct pricing SKU

    • Review any features that are currently in beta and confirm with PMs if there are any to be promoted

    • Review the Jira tickets remaining in the current release fix version and push those that won't make it to the next fix version

    • After mobile release branch is cut, ask dev to cut an RN build

    • Queue a list of MVP candidates in alphabetical order to the MVP Discussion channel See example

    • Draft changelog in a WIP PR with updates for highlights, feature additions, known issues, compatibility updates for deprecated features, config.json, database changes, API changes, WebSocket event changes, and Go runtime version; see example

    • Review that software requirements are up-to-date based on these step-by-step guidelines

    • Ask PMs and Dev Leads if there are any notable breaking changes or deprecated features in the release

      • Update Upgrade Guide with any special notes for upgrading to the new version

      • If there are any breaking compatibility changes in the release, open an issue in the GitLab Omnibus to make sure GitLab is aware. Post a link to the issue in the Release: Self-Managed channel

    • Start posting a daily list of open and submitted bugs and PRs

    • Submit NOTICE.txt PRs for any new libraries added from dev, and ensure they are cherry-picked to feature release branch

    • Confirm that NOTICE.txt PRs, Translations PRs, database upgrade, and any needed pre-packaged plugins, backports, and recent regression fixes are included in the release

    • Create meta issue for release in GitHub (see example)

    • Post a reminder in the Localization channel about the due date for translations, similar to this example

    • Prepare bullet points for release announcement and share for PMs to work on. Refer to the process here

    • Ask PMs and Tech Writer to complete release documentation by T-9 deadline. Example request

    • Schedule a meeting with PMs and QAs to discuss upcoming features in the next feature release

  3. Dev:

    • Cut release branches for server and mobile

      • Merge database upgrade before cutting the branch

      • Point translation server and CI servers to the release branch after cutting

      • Update https://prev.test.mattermost.com to the previous release version

        • Prioritize reviewing, updating, and merging of pull requests for current release until there are no more tickets in the pull request queue marked for the current release

        • After the cut-off, any PRs that include significant code changes require approval of the Release Manager before merging

  4. QA:

    • Prioritize testing PRs and resolved tickets for this release

    • Ensure that new features are also properly tested on mobile apps

    • Prioritize updating release tests in test management and automated tests

    • Identify any new teammates who will be joining release testing, send them an intro to the testing process and timeframe, send them the hardware/software survey

    • Set up DM/GM channels in preparation for testing auto-closing after 7 days

C. (T-minus 9 working days) RC1 Cut

  1. Release Manager:

    • Post this checklist in Release Checklist channel

    • Verify all items in the last posted release checklist are complete

    • Post in the Release: Self-Managed channel the rough timing when the release candidate will be cut

    • Cut a Release Candidate using this process and check CI servers running on release branch

    • Generate a list of contributors for changelog

    • Confirm doc submission

    • Post list of tickets to be fixed to the Release: Self-Managed channel (see example)

    • Update the GitHub meta issue:

      • Include a link to the changelog on the documentation branch

      • Update download links and testing server links to the latest RCs

    • After build is cut, tweet announcement that RC1 is ready (see example)

  2. Logistics @hanna.park:

    • Mail out contributor and security researcher mugs

      • Space out the ordering of mugs over the next three weeks to prevent mistakes being made by the supplier (e.g., If there are 12 contributors to order mugs for, place an order every 2nd or 3rd day over the next three weeks)

    • Update Team page with new contributors

  3. QA:

    • Confirm up to date with testing PRs and resolved tickets

    • Confirm up to date with test updates and known issues in release test management and automated tests

    • Assign release testing areas to team members

    • After RC1 is cut: Update test server invite links in Release Testing instructions

    • After RC1 is cut: Run automated regression tests

  4. Build:

    • Review all TODO notes, including one for uncommenting upgrade code

    • Confirm all PRs in /enterprise repo have been merged

    • Update Redux before each RC and Final build

    • Master is tagged and branched and Release Candidate 1 is cut (e.g. 3.5.0-RC1) according to the Release Candidate Checklist in mattermost/process

    • Update version for each Mattermost Helm chart

    • Make PRs for bug fixes to the release branch

    • Review PRs made from release branch and merge changes into the release branch as required

    • Run daily automated upgrade tests to avoid catching upgrade bugs late

D. (T-minus 8 to T-minus 7 working days) Release Candidate Testing

  1. (Team) Daily Release Update Meeting

    • Triage Jira tickets

    • Decide when to cut next RC

  2. Release Manager:

    • Post this checklist in Release Checklist channel

    • Verify all items in the last posted release checklist are complete

    • Confirm changelog reflects any changes since it was merged (including known issues and contributors from all repositories)

      • Confirm translators have been added

      • Confirm Open Source Components changes have been added

    • Find www-gitlab-com merge request for latest GitLab release blog post and make request for adding GitLab Mattermost update (see example request, example update). Post to Release: Self-Managed channel with link to request

    • Confirm the required URLs for the blog/release announcement were prepared

  3. QA:

    • Update Release: Self-Managed header with links to RC instances and testing spreadsheet (template)

    • Post release testing instructions to Release: Self-Managed channel (template)

    • Post reminders in Announcements channel (template) and Customer Support channel (template)

    • DM reminders to team members who are not QA or devs, or who are new to release testing

    • Midday: Post reminders about testing, at-mentioning team members whose tests are not yet complete

    • At end of day, post reminders about release testing in Release: Self-Managed and Announcements channels, DM any team members who have zero test cells marked Done

    • Find QA or other teammates to help finish unfinished tests if needed

    • End of day or next morning: Verify all release tests are finished, bring any concerns to Triage/Release meeting

    • Go through all tabs of testing spreadsheet and verify all comments and questions have been filed in Jira as needed

  4. Logistics @hanna.park:

    • Generate an E20 5000 seat test licence and email to Lindy for release testing

  5. PM:

    • Finish draft of blog post for mattermost.com and all art work (screenshots, GIFs, and Twitter banners) used for the blog post

E. (T-minus 5 working days) Code Freeze

Day after which only S1 regressions should be fixed (crashes, security vulnerabilities) and no other code changes should be committed. Exceptions can be made by the Release Manager during triage on a case-by-case basis. Review the Bug Severity Guidelines.

  1. (Team) Daily Release Update Meeting

    • Continue to triage Jira tickets

    • If no blocking issues are found, PM, Dev, and QA sign off on the release and begin final smoke tests

  2. Release Manager:

    • Post this checklist in Release Checklist channel

    • Verify all items in the last posted release checklist are complete

    • Verify that the final translations PR for the release is submitted

    • Update Known Issues section with any significant issues that were found and not fixed for the final release

  3. QA:

    • Verify all Jira tickets other than newly-filed bugs have been tested, verified, and closed

    • As bug fixes are merged and RCs are cut, verify fixes on new RCs, and post in Release Channel after testing

    • After all tickets are verified and closed, run smoke tests on webapp/server, desktop app, and RN apps as appropriate

  4. Docs:

F. (T-minus 2 working days) Release Build Cut

The final release is cut - RC cuts and bug fixes should be completed by this date. Only urgent and critical issues are considered for fixing.

  1. Release Manager:

    • Post this checklist in Release Checklist channel

    • Verify all items in the last posted release checklist are complete

    • Work with a developer to submit GitLab MR following this process and test the upgrade once the GitLab MR is merged and included in their RC

    • Close GitHub meta ticket for the release

    • Add the download links, SHA-256 hash, and GPG signature to the version archive

    • Merge changelog PR after review is complete

      • If there is a security fix, confirm the changelog recommends upgrade, with a note mentioning the security level and thanking the security researcher (security process doc for example)

    • Update the Mattermost server download page with the links to the Enterprise Edition and Team Edition sections

      • Test the download links before and after updating the page

    • Check security issues and confirm disclosure text

    • Check the security researcher was added to the Responsible Disclosure Policy page

    • Update deprecated feature list in mattermost.com with new and scheduled deprecations

    • Draft Mattermost Security Updates, but do not post until 30 days after official release

      • Add a placeholder text saying "Details on the security update will be posted here on X date, as per our Responsible Disclosure Policy"

  2. QA:

    • Verify all PRs and tickets for the release have been tested/closed

    • Verify Selenium server was put on final RC and build passed

    • Verify smoke tests on webapp/server, desktop app, and RN apps all passed

    • Post QA approval in Release: Self-Managed channel

  3. Build:

    • Update Redux before each RC and Final build

    • Tags a new release (e.g. 1.1.0) and runs an official build which should be essentially identical to the last RC

    • Posts SHA key, md5 sum, and GPG signatures of the final build to Release: Self-Managed channel

    • Post in Release: Self-Managed with links to the Enterprise Edition and Team Edition sections

  4. Dev:

    • Verify the hashes (SHA-1, SHA-256, and md5) and GPG signatures are correct for both Enterprise Edition and Team Edition

    • Test upgrade from previous version to current version, following the Upgrade Guide with database upgrades on both MySQL and Postgres

    • Test upgrade from Team Edition to Enterprise Edition based on the Upgrade Guide

    • Test fresh install of current version, following the Install Guide

    • Review any changes made to install guides, and test if necessary

    • Ensure Security Policies page has been updated

    • Update dependancies after release branch is cut in mattermost-server, mattermost-webapp, desktop, mattermost-mobile, and mattermost-redux

  5. Logistics @hanna.park:

    • Order the release MVP winner's coaster

  6. Docs:

    • Remind PMs and Tech Writer to finalize docs

    • Merge the docs release branch to master and verify all changes on docs.mattermost.com once the build is up

    • Submit a correction PR for any incorrect formatting or other errors missed during the initial review

  7. Marketing:

    • Receive sign off on final version of MailChimp email blast and Twitter announcement and schedule for 08:00 PST on the date of marketing announcement

    • Finalize blog post for mattermost.com, test on mobile view, and set timer for 08:00 PST on the day of release

    • Add links to Admin guide in the release blog post where needed

G. (T-minus 0 working days) Release Day

  1. Release Manager:

    • Post this checklist in Release Checklist channel

    • Verify all items in the last posted release checklist are complete

    • Update the server upgrade in-product notice

    • Schedule a release retrospective meeting, to be held within five days of the release

    • Prepare and post release metrics

    • Finalize release summary slide deck to prepare for posting for CSMs

    • Review and update company roadmap with which major features made it into the release

    • Update the Mattermost Wikipedia page with the latest version

    • Post the MVP winner announcement in the Contributors channel

      • Update MVP page with the most valued professional of the release

    • Add new release fix versions in Jira for the next few releases

    • Post key dates for the next release in the Release: Self-Managed channel and remove links to RC candidates and testing spreadsheet from the header

      • Make sure that statutory holidays for Canada and US are accounted for in the release dates

    • Check for any UserVoice feature suggestions that were completed in the current release

      • Find the release tweet and insert a link to the tweet next to the feature that shipped with the release

    • Close the release in Jira both for webapp and mobile (releases page)

    • For the next release, create the following team meetings. If they conflict with existing meetings, check with meeting owner to reschedule or reschedule the release meeting

      • Bug Bash Meeting on the week after Code Complete at 10:00am San Francisco time

      • Release Triage and Update Meeting each weekday starting at T-20 and ending at T-0 at 9:30am San Francisco time for PM, QA, and release dev

    • Prepare tickets for the next release, with a corresponding vX.X prefix, and put the tickets in the appropriate sprints as follows:

      • The week RC is cut:

      • The week RC is cut (for GitLab dev owner):

        • Test RC1 with the latest GitLab build during release testing cycle

      • Release week (for dependencies owner)

        • Upgrade dependencies for webapp, server, and Redux

      • Week after release (for GitLab dev owner)

      • The week before code complete (one for Apps and for Mobile):

        • If an existing ESR is going out of support next month, update the in-app prompts (mobile and desktop) to start detecting for the new minimum supported ESR version

        • If a new ESR is released next month, update the prompts to recommend upgrading to that version instead of the older ESR

    • Confirm that mattermost-docker has been updated to the latest version (contact the maintainer via direct message on community server if necessary)

    • Contact owners of community installers or submit PRs to update install version number

      • For Yunohost, open a new pull request to update the version. See example

    • Turn on CrazyEgg for blog post page

    • Confirm marketing has been posted (screenshots, mail announcement, tweets, blog posts)

      • Post in the Announcements channel

    • Update @mattermost Twitter profile with the next release date

    • Prepare retweet of GitLab release tweet (see example here)

  2. Docs:

    • Create a new branch on docs for the next release - vX.X-documentation

      • Submit a PR for changelog against the vX.X-documentation branch and add a Work in Progress label for it

      • Submit a PR to change version number in docs/source/conf.py against the vX.X-documentation branch

  3. Build:

  4. QA:

    • Merge any updates made to Selenium tests during release testing

    • Update RN server URLs to Rainforest

H. (T-plus 30 days) Update Mattermost Security Page

  1. Release Manager:

    • Post Mattermost Security Updates after reviewing with security lead

      • If a dot release is shipping with security fixes, do not post new details until T-plus 10 working days from the dot release ship date