Mattermost Handbook
Mattermost Handbook
Mattermost Handbook
Company
About Mattermost
"How to" guides for staff
Operations
Company Operations
Research and Development
Product
Product Planning
Release Process
Release Overview
Feature Release Process
Quality Release Process
Dot Release Process
Security Release Process
Mobile App Release Process
Release Tips
Release Scorecard Definitions
Development Process
How to Guides for Product
Product Management Team Handbook
Technical Writing Team Handbook
Engineering
Quality Assurance
Messaging and Math
Sales
Customer Success
IT
Business Operations
Finance
Workplace
Contributors
Contributors
Join Us
Powered by GitBook

Feature Release Process

Mattermost core team works on a monthly release process, with a new version shipping on the 16th of each month in binary form.

This document outlines the development process for the Mattermost core team, which draws from what we find works best for us from Agile, Scrum, and Software Development Lifecycle approaches.

Recommended pre-reading:

Release Timeline

Notes:

A. (Code complete date of previous release) Beginning of Release

Pre-work for the current release begins at the code complete date of the previous release. See "Code Complete" section below for details.

B. (T-minus 23 working days) Cut-off for Submitting Features

No pull requests for major features should be submitted to the current release after this date. In special cases, exceptions can be made by the Release Manager.

  1. Release Manager:

    • Post this checklist in Release Checklist channel

    • Follow that feature PR reviews are prioritized and post a list of outstanding feature PRs in the Release Discussion channel

    • Confirm with PMs that each Enterprise feature is in the correct pricing SKU​

    • Review any features that are currently in beta and confirm with PMs if there are any to be promoted

    • Confirm with PMs that all features are behind a feature flag and that all config settings and new features have diagnostics

C. (T-minus 20 working days) Feature Complete

No pull requests for major features should be merged to the current release after this date. In special cases, exceptions can be made by the Release Manager.

  1. (Team) Major Feature Complete Meeting (10:00am San Francisco time):

    • Review status of remaining feature PRs to be merged

    • Begin daily triage of tickets

      • Also start to triage tickets in the backlog

  2. Release Manager:

  3. Dev:

    • Cut release branch both for server and mobile

      • Merge database upgrade before cutting the branch

      • Point translation server and CI servers to the release branch after cutting

      • Update https://prev.test.mattermost.com to the previous release version

    • Ensure community-release is on the feature branch

    • Prioritize reviewing, updating, and merging of pull requests for current release until there are no more tickets in the pull request queue marked for the current release

      • After the cut-off, any PRs that include significant code changes require approval of the Release Manager before merging

D. (T-minus 19 working days) Feature Testing

  1. QA:

    • Prioritize testing PRs and resolved tickets for this release

    • Ensure that new features are also properly tested on mobile apps

    • Prioritize updating release tests in test management and automated tests

E. (T-minus 15 working days) Judgment Day

Day when Leads and PMs decide which major features are included in the release, and which are postponed.

  1. (Team) Judgment Day Meeting (10:00am San Francisco time):

    • Finalize which major features will be in or out for the release

      • Discuss reverting feature(s) if five or more bugs found

  2. Release Manager:

    • Post this checklist in Release Checklist channel

    • Verify all items in the last posted release checklist are complete

    • Update Changelog PR based on what's in/out of the release

    • Create meta issue for release in GitHub (see example)

    • Confirm date of marketing announcement for the release date

      • If release day falls on a Friday, the blog post goes out on the Friday and the emailed newsletter goes out the following Tuesday

    • Post a reminder to devs in the Release Discussion channel of the the code complete date with the list of open and submitted bugs and PRs see example​

    • Review the Jira tickets remaining in the current release fix version and push those that won't make it to the next fix version

    • Ask PMs and Tech Writer to complete release documentation by T-9 deadline. Example request​

    • Finalize release summary slide deck to prepare for posting for MTL and CSMs

  3. PM:

    • Finalize roadmap for next release, and identify planned marketing bullet points

F. (T-minus 14 working days) Code Complete

Stabilization period begins when all features for release have been committed. During this period, only bugs can be committed to the release branch. Non-bug pull requests are tagged for next version. Exceptions can be made by the Release Manager during triage. Review the Release Features & Bugs Quality Gate Guidelines​

  1. (Team) Code Complete Meeting (10:00am San Francisco time):

    • Last check of tickets that need to be merged before RC1

  2. Release Manager:

    • Post this checklist in Release Checklist channel

    • Verify all items in the last posted release checklist are complete

    • Submit NOTICE.txt PRs for any new libraries added from dev, and ensure they are cherry-picked to feature release branch

    • If there are any breaking compatibility changes in the release, open an issue in the GitLab Omnibus to make sure GitLab is aware. Post a link to the issue in the Release Discussion channel

  3. Dev:

    • Prioritize reviewing, updating, and merging of pull requests for current release until there are no more tickets in the pull request queue marked for the current release

  4. QA:

    • Identify any new teammates who will be joining release testing, send them an intro to the testing process and timeframe, send them the hardware/software survey​

    • Set up DM/GM channels in preparation for testing auto-closing after 7 days

G. (T-minus 9 working days) Release Candidate Cut

  1. Release Manager:

    • Post this checklist in Release Checklist channel

    • Verify all items in the last posted release checklist are complete

    • Cut a Release Candidate using this process and check CI servers running on release branch

    • Generate a list of contributors for Changelog

    • Confirm doc submission

  2. Logistics @hanna.park:

    • Mail out contributor and security researcher mugs

      • Space out the ordering of mugs over the next three weeks to prevent mistakes being made by the supplier (e.g., If there are 12 contributors to order mugs for, place an order every 2nd or 3rd day over the next three weeks)

    • Update Team page with new contributors

  3. QA:

    • Confirm up to date with testing PRs and resolved tickets

    • Confirm up to date with test updates and known issues in release test management and automated tests

    • Assign release testing areas to team members

    • After RC1 is cut: Update test server invite links in Release Testing instructions

    • After RC1 is cut: Run automated regression tests

  4. Build:

    • Review all TODO notes, including one for uncommenting upgrade code

    • Confirm all PRs in /enterprise repo have been merged

    • Update Redux before each RC and Final build

    • Update package version in Mattermost DockerFile​

    • Master is tagged and branched and Release Candidate 1 is cut (e.g. 3.5.0-RC1) according to the Release Candidate Checklist in mattermost/process

    • After branching, the database version in sql_upgrade.go on master is set to the next scheduled release version (e.g., 3.6.0)

    • Update version for each Mattermost Helm chart​

H. (T-minus 8 working days) Release Candidate Testing

  1. (Team) Daily Release Update Meeting

    • Triage Jira tickets

    • Decide when to cut next RC

  2. Release Manager:

    • Post this checklist in Release Checklist channel

    • Verify all items in the last posted release checklist are complete

    • Post list of tickets to be fixed to the Release Discussion channel (see example)

    • Update the GitHub meta issue:

      • Include a link to the changelog on the documentation branch

      • Update download links and testing server links to the latest RCs

    • After build is cut, tweet announcement that RC1 is ready (see example)

    • Update https://docs.mattermost.com/administration/open-source-components.html​

    • Ask martin.kraft to provide a list of Database changes, API changes, WebSocket event changes, and config.json setting changes for Changelog. See example request​

  3. QA:

    • Update Release Discussion header with links to RC instances and testing spreadsheet (template)

    • Post release testing instructions to Release Discussion channel (template)

    • Post reminders in Announcements channel (template) and Customer Support channel (template)

    • DM reminders to team members who are not QA or devs, or who are new to release testing

    • At end of day, post reminders about release testing in Release Discussion and Announcements channels, DM any team members who have zero test cells marked Done

  4. Logistics @hanna.park:

    • Generate an E20 5000 seat test licence and email to Lindy for release testing

  5. Dev:

    • Make PRs for bug fixes to the release branch

    • Review PRs made from release branch and merge changes into the release branch as required

    • Run daily automated upgrade tests to avoid catching upgrade bugs late

  6. PM:

    • Finish draft of blog post for mattermost.com and all art work (screenshots, GIFs, and Twitter banners) used for the blog post

      • Upgrade should be recommended if there are security fixes in this version, with a note thanking the security researcher

I. (T-minus 7 working days) Release Candidate Testing Finished

  1. (Team) Daily Release Update Meeting

    • Confirm testing assigned in the release testing spreadsheet is complete

    • Continue to triage Jira tickets

  2. Release Manager:

    • Post this checklist in Release Checklist channel

    • Verify all items in the last posted release checklist are complete

    • Confirm Changelog reflects any changes since it was merged (including known issues and contributors from all repositories)

      • Confirm translators have been added

      • Confirm Open Source Components changes have been added

    • Find www-gitlab-com merge request for latest GitLab release blog post and make request for adding GitLab Mattermost update (see example request, example update). Post to Release Discussion channel with link to request

    • Confirm the required URLs for the blog/release announcement where prepared

  3. QA:

    • Midday: Post reminders about testing, at-mentioning team members whose tests are not yet complete

    • Find QA or other teammates to help finish unfinished tests if needed

    • End of day or next morning: Verify all release tests are finished, bring any concerns to Triage/Release meeting

    • Go through all tabs of testing spreadsheet and verify all comments and questions have been filed in Jira as needed

J. (T-minus 5 working days) Code Freeze

Day after which only S1 regressions should be fixed (crashes, security vulnerabilities) and no other code changes should be committed. Exceptions can be made by the Release Manager during triage on a case-by-case basis. Review the Bug Severity Guidelines.

  1. (Team) Daily Release Update Meeting

    • Continue to triage Jira tickets

    • If no blocking issues are found, PM, Dev, and QA sign off on the release and begin final smoke tests

  2. Release Manager:

    • Post this checklist in Release Checklist channel

    • Verify all items in the last posted release checklist are complete

    • Verify that the final translations PR for the release is submitted

    • Update Known Issues section with any significant issues that were found and not fixed for the final release

  3. QA:

    • Verify all Jira tickets other than newly-filed bugs have been tested, verified, and closed

    • As bug fixes are merged and RCs are cut, verify fixes on new RCs, and post in Release Channel after testing

    • After all tickets are verified and closed, run smoke tests on webapp/server, desktop app, and RN apps as appropriate

  4. Docs:

K. (T-minus 2 working days) Release Build Cut

The final release is cut - RC cuts and bug fixes should be completed by this date. Only urgent and critical issues are considered for fixing.

  1. Release Manager:

    • Post this checklist in Release Checklist channel

    • Verify all items in the last posted release checklist are complete

    • Work with a developer to submit GitLab MR following this process and test the upgrade once the GitLab MR is merged and included in their RC

    • Close GitHub meta ticket for the release

    • Add the download links, SHA-256 hash, and GPG signature to the version archive​

    • Merge Changelog PR after review is complete

      • If there is a security fix, confirm the Changelog recommends upgrade, with a note mentioning the security level and thanking the security researcher (security process doc for example)

    • Update the Mattermost server download page with the links to the Enterprise Edition and Team Edition sections

      • Test the download links before and after updating the page

    • Check security issues and confirm disclosure text

    • Update the security mugs spreadsheet​

    • Check the security researcher was added to the Responsible Disclosure Policy page

    • Confirm link to security updates appears in blog post if there are security updates in this release, with a note thanking the security researcher

    • Update deprecated feature list in mattermost.com with new and scheduled deprecations

    • Draft Mattermost Security Updates, but do not post until 30 days after official release

      • Add a placeholder text saying "Details on the security update will be posted here on X date, as per our Responsible Disclosure Policy"

  2. QA:

    • Verify all PRs and tickets for the release have been tested/closed

    • Verify Selenium server was put on final RC and build passed

    • Verify smoke tests on webapp/server, desktop app, and RN apps all passed

    • Post QA approval in Release Discussion channel

  3. Build:

    • Update Redux before each RC and Final build

    • Tags a new release (e.g. 1.1.0) and runs an official build which should be essentially identical to the last RC

    • Posts SHA key, md5 sum, and GPG signatures of the final build to Release Discussion channel

    • Post in Release Discussion with links to the Enterprise Edition and Team Edition sections

  4. Dev:

    • Verify the hashes (SHA-1, SHA-256, and md5) and GPG signatures are correct for both Enterprise Edition and Team Edition

    • Test upgrade from previous version to current version, following the Upgrade Guide with database upgrades on both MySQL and Postgres

    • Test upgrade from Team Edition to Enterprise Edition based on the Upgrade Guide​

    • Test fresh install of current version, following the Install Guide​

    • Review any changes made to install guides, and test if necessary

    • Ensure Security Policies page has been updated

    • Update dependancies after release branch is cut in mattermost-server, mattermost-webapp, desktop, mattermost-mobile, and mattermost-redux

  5. Logistics @hanna.park:

    • Order the release MVP winner's coaster

  6. Docs:

    • Finalize docs

      • If reviews are not complete, hold a 30-minute doc review meeting with PMs and anyone else who has changed or reviewed docs this release and wants to join

      • Merge the docs release branch to master and verify all changes on docs.mattermost.com once the build is up

      • Submit a correction PR for any incorrect formatting or other errors missed during the initial review

  7. Marketing:

    • Receive sign off on final version of MailChimp email blast and Twitter announcement and schedule for 08:00 PST on the date of marketing announcement

    • Finalize blog post for mattermost.com, test on mobile view, and set timer for 08:00 PST on the day of release

    • Add links to Admin guide in the release blog post where needed

L. (T-minus 0 working days) Release Day

  1. Release Manager:

    • Post this checklist in Release Checklist channel

    • Verify all items in the last posted release checklist are complete

    • Schedule a release retrospective meeting, to be held within five days from the release

    • Update the server upgrade in-product notice​

    • Prepare and post release metrics​

    • Review and update company roadmap with which major features made it into the release

    • Update Integrations Directory on the Mattermost Apps and Integrations page

    • Post the MVP winner announcement in the Contributors channel​

      • Update MVP page with the most valued professional of the release

    • Add new release fix versions in Jira for the next few releases

    • Post key dates for the next release in the Release Discussion channel and remove links to RC candidates and testing spreadsheet from the header

      • Make sure that statutory holidays for Canada and US are accounted for in the release dates

    • Check for any UserVoice feature suggestions that were completed in the current release

      • Find the release tweet and insert a link to the tweet next to the feature that shipped with the release

    • Close the release in Jira both for webapp and mobile (releases page)

    • For the next release, create the following team meetings. If they conflict with existing meetings, check with meeting owner to reschedule or reschedule the release meeting

      • Bug Bash Meeting on the week after Code Complete at 10:00am San Francisco time

      • Release Triage and Update Meeting each weekday starting at T-15 and ending at T-2 at 9:30am San Francisco time for PM, QA and release dev

    • Prepare tickets for the next release, with a corresponding vX.X prefix, and put the tickets in the appropriate sprints as follows:

    • Confirm that mattermost-docker has been updated to the latest version (contact the maintainer via direct message on community server if necessary)

    • Contact owners of community installers or submit PRs to update install version number

      • For Chef Cookbook, open a new issue to announce the new release. See example​

      • For Yunohost, open a new pull request to update the version. See example​

    • Turn on CrazyEgg for blog post page

    • Confirm marketing has been posted (animated GIFs, screenshots, mail announcement, tweets, blog posts)

    • Update @mattermosthq Twitter profile with the next release date

    • Prepare retweet of GitLab release tweet (see example here)

  2. Docs:

    • Create a new branch on docs for the next release - vX.X-documentation

      • Submit a PR for Changelog against the vX.X-documentation branch and add a Work in Progress label for it

      • Submit a PR to change version number in docs/source/conf.py against the vX.X-documentation branch

  3. Build:

    • Cut release branch for Bug Fix release

      • Ensure community-release is on the quality release branch

    • Put CI servers and translation server back onto master, and post in Private Developers channel once done

    • Ensure https://community.mattermost.com is on the most recently released version and that https://prev.test.mattermost.com is on the most recently released bug fix release

  4. QA:

    • Merge any updates made to Selenium tests during release testing

    • Update RN server URLs to Rainforest

M. (T-plus 30 days) Update Mattermost Security Page

  1. Release Manager:

    • Post Mattermost Security Updates after reviewing with security lead

      • If a dot release is shipping with security fixes, do not post new details until T-plus 10 working days from the dot release ship date

Previous
Release Overview
Next
Quality Release Process
Last updated 2 months ago
Edit on GitHub
Contents
Release Timeline
A. (Code complete date of previous release) Beginning of Release
B. (T-minus 23 working days) Cut-off for Submitting Features
C. (T-minus 20 working days) Feature Complete
D. (T-minus 19 working days) Feature Testing
E. (T-minus 15 working days) Judgment Day
F. (T-minus 14 working days) Code Complete
G. (T-minus 9 working days) Release Candidate Cut
H. (T-minus 8 working days) Release Candidate Testing
I. (T-minus 7 working days) Release Candidate Testing Finished
J. (T-minus 5 working days) Code Freeze
K. (T-minus 2 working days) Release Build Cut
L. (T-minus 0 working days) Release Day
M. (T-plus 30 days) Update Mattermost Security Page