Overview

This document aims to lay down some guidelines on how to work with security-sensitive changes in public Git repositories.

Goals

Establish and document a way of working that:

1. Minimizes the risk of vulnerability information leaking out before a fix is made available.

2. Minimizes the friction caused for developers, QA, and security engineers actively working on a security issue.

3. Is easy to understand and adopt, given the existing PR process.

4. Is easy to remember and follow.

Scope

The guidelines documented here only apply to open source products maintained by Mattermost. Closed-source work does not have the same considerations and does not need to follow the guidelines laid out here.

Background reading

Similar documentation by Mozilla used as the inspiration for this document: https://firefox-source-docs.mozilla.org/bug-mgmt/processes/fixing-security-bugs.html

Guidelines by engineering area

The guidelines below are split by work area; start by reading the section, then proceed to whichever follow-up section is relevant to you.

Common guidelines

When working on security-sensitive changes, pay attention to what is public and what is not. Be mindful about entering information into systems such as Mattermost Channels and Boards, GitHub Issues and Pull Requests, and Jira.

When discussing related work in public channels, such as a CI pipeline failure blocking the security release or feature work that had to be postponed because of the incident, pay attention to the ways you reference the private information. Avoid sharing ticket IDs or mentioning key words like "security": every time you reference the issue in an identifiable way, you add contextual information, and that information can eventually add up to something that's useful to an attacker. Specific words and phrases to be extra careful with:

• Obvious key words: "security", "vulnerability", "exploit", "insecure", "attack".

• Anything that identifies a vulnerability type: "XSS", "authentication bypass", "authorization bug", "SQL injection", "remote code execution", etc.

• Security-specific tools and terminology: "HackerOne", "Bugcrowd", "CVE", "CVSS", "advisory", "pentest".

Guidelines for developers

When developing a fix for a security issue, bear in mind that the code, comments, unit tests, and commit messages become public information as soon as they are pushed to the remote repository. Apply accordingly.

When commenting on your code, keep references to security issues to a minimum. For example: If it's necessary to highlight that a non-obvious validation check is necessary for security, avoid the following:

Instead refer to the ticket ID or some other opaque identifier only:

With the latter, any other staff member or your future self can still easily find the context, but you avoid pointing out to would-be attackers that the change is critical for security.

When developing unit/e2e tests, do not use the Proof-of-Concept (PoC) exploits received from the security team or documented on the vulnerability ticket as-is: the purpose of these PoCs is to clearly demonstrate the security issue and its implications, which often means they are very easy to "weaponize". Instead, test the intended functionality using more benign payloads. For example in the case of a Cross-Site Scripting (XSS) issue resulting from missing HTML special character escaping, test the fix using sets of individual characters or garbage input instead of a full XSS payload. If you are unsure about your approach to testing, ask the Security team!

When submitting PRs, pay attention to the PR title and description. Remember the Common Guidelines and do not use the title of the Jira ticket as-is. The title of the PR is one of the most visible and obvious indicators of a security issue being worked on! It appears on the PR itself, the pull requests index page—both of which attackers are likely to monitor—and eventually ends up in Git history when the PR is squashed and merged. The guidelines also apply to individual commit messages and the name of the branch!

• If you notice you've made a mistake in the code, unit tests, comments, or commit messages and already pushed it to GitHub, fix it as soon as possible by rewriting the Git history and force-pushing the new, sanitized changes.

• If you notice you've made a mistake in the PR title or description, edit it out and alert the Security team: Edits will still be visible in the GitHub UI and if the information revealed is sensitive enough, Security may choose to fully delete the PR in question.

Guidelines for QA

When QA testing a security fix, keep in mind that any test steps documented on the Jira ticket are most likely sensitive and should not be publicly disclosed. When a test step fails or the instructions are unclear, feel free to request changes on the Pull Request if applicable, but try to keep the details private: use Jira or Mattermost Channels.

If unsure about how to thoroughly QA test a security fix or how to set up the environment required, feel free to leave QA testing to the security team. For automated testing, also see the section.

Guidelines for security engineering

As security engineers coordinate an incident with both engineering staff and external contributors, the same Common Guidelines apply. The security engineer currently responsible for coordinating an incident is also in the end responsible for making sure other engineering staff is aware of the guidelines and follows them to the best of their ability.

Security engineers should monitor any public activity around an incident, such as information unintentionally leaking, outside parties commenting on Pull Requests, or the reporter of the issue actively leaking information on social media.

The responsibility of maintaining this document also rests with security engineering: If a specific guideline cannot be reasonably applied in a large number of cases, the guideline should be updated. Similarly, if information appears to be consistently leaking despite following these guidelines, improvements should be planned and documented.

This document provides guidelines to secure software development at Mattermost. The document is at the moment a 50% draft and not conclusive.

Document scope

This document covers the secure software development practices on all source repositories used to create software products offered by Mattermost.

The product security team is overall in charge of secure software development practices by providing help and guidance and performing various verification activities, including code reviews and testing. Development teams are encouraged to adopt the secure software development practices described here. Third-party plugin developers are also welcome to use this document for guidance.

Recommended reading

• , a curated and updated list on most common security risks in web applications.

Code reviews

Manual code reviews

The product security team performs reviews on new features and plugins when necessary. Here's some of the things included in the checks:

• Manual diff review

• Static application security testing (SAST) using multiple tools

• Software composition analysis (SCA)

• Dynamic application security testing (DAST)

Detailed review checklists are documented in the , accessible to team members only.

Automated code reviews

Mattermost production repositories have automated code security analysis tools set up. If this is not the case for a specific repository, please reach out to the product security team for assistance.

The automated tools in each repository depend on the technologies used in the repository. In practice, we use as a Github Action to automatically analyze all pull requests. Additionally, the use of the is strongly encouraged. This document describes some of the best practices checked by the Scorecards action.

Dependency pinning

Dependency pinning hardens software development workflows against . The specific threat being mitigated is upstream dependencies being taken over by attackers and replaced with malicious content. Attacks are more likely to be successful against projects that refer to their dependencies without explicitly specifying a computed hash value for the dependency: In case of a dependency takeover, pulling the latest version triggers the attack. The likelihood of this risk materializing is low, but the impact can be bad.

In practice, defining build-time dependencies with an explicit hash is the preferred solution. When specifying hashes to existing dependencies that did not use hash-based references before, picking the hash of the currently used dependency is a good starting point as it is often not practical to do a full audit of all dependencies. Picking a hash and sticking to it doesn't ensure that the hash represents a safe version, but it does ensures that a possibly malicious later version is not introduced in the build process afterwards.

Many dependency managers support automatic dependency pinning so that a developer doesn't need to explicitly find and set dependency hashes. For example both Go and npm do, as described below.

Npm dependencies

With npm, always use npm ci instead of npm install in build scripts and automated pipelines. This ensures that only packages matching the existing package-lock.json file are installed. See for more details.

When adding new dependencies, prefer over the default semver range.

Docker

Docker image dependencies exist in Dockerfiles and elsewhere, such as in CircleCI config files. The tool can be used to pin Docker base images to their currently latest versions. To pin a dockerfile, run dockpin docker pin -f Dockerfile. To pin Docker base image dependencies elsewhere, run dockpin docker resolve [base-image]; for example dockpin docker resolve [email protected].

Here's a handy script to do this on CircleCI yaml files, replacing the image reference with one with a hash, and additionally commenting with the timestamp the hash was obtained:

Go

When adding a Go dependency using go get, the dependency can be installed specifying either a specific version (go get example.com/[email protected]), the latest available version (go get example.com/theirmodule@latest), or a commit hash (go get example.com/theirmodule@4cf76c2) or branchname (@branchname). In all cases, the installed dependencies are added to the automatically generated go.mod file and the content hashes of the dependencies are written to the go.sum file. Make sure to commit both the go.mod and the go.sum file to your repository. This ensures that subsequent builds download exactly the same dependencies.

Github Actions

GitHub Actions are specified in .yml files in the .github/workflows directory inside a GitHub repository. Check out by that can harden the workflow spec. Check at least the "Restrict permissions..." and "Pin actions..." boxes, then paste in the workflow .yml file and click "Secure workflow". Review the changes and paste the result back into your workflow .yml file.

A few other things

(most from the -- consult with product security team)

• Check that the git repository doesn't contain any secrets anywhere in its history

• Make sure the SECURITY.md file exists and describes the security policy

Overview

This document aims to provide guidance on the handling of product vulnerabilities. It describes the process and provides background information and definitions for the daily usage of all relevant stakeholders.

Goals

Establish and document a way of working that ensures:

• Vulnerabilities are triaged efficiently and quickly

• All parties are aligned around ownership and expectations

• Patch releases are available within the agreed SLAs

• Minimize the risk for Mattermost Cloud, on-premises customers, and Mattermost, Inc.

Scope

This process covers any vulnerabilities found in officially published Mattermost products or services that were either found internally, reported to us by a customer, or come in through the bug bounty program or via another third party.

Infrastructure vulnerabilities are explicitly out of scope for this process. Vulnerability management, as it relates to infrastructure, is owned by the Security Operations Team.

Process Definitions

Responsible Parties

• Product Security Team: Process Ownership; Vulnerability Owner

• Development Team: Vulnerability Remediation

• Release Manager: Release Planning

• Release Team:

Vulnerability Sources

Vulnerabilities are reported from a variety of sources that are explained below.

Internal Finding

The vulnerability has been discovered by the internal security, development, or QA team.

Security Automation

The vulnerability has been discovered by internal security automation tools such as Dependency Track or CodeQL

Responsible Disclosure

The vulnerability has been reported through our .

Bug Bounty Program

The vulnerability has been reported through our .

Customer Report

The vulnerability has been reported from customers as part of an internal audit or on-going operations.

Public Leak

The vulnerability has not been officially reported, but released to the public without any notice. It is possible that it can contain reproduction steps or not.

Vulnerability Severity

Vulnerabilities must be scored according to the Common Vulnerability Scoring System Version 3.1 (CVSS 3.1). The CVSS is an industry-wide accepted standard to determine the impact of vulnerabilities in software products and takes impacts such as attack vector, complexity, user interaction, scope, and various other components into consideration. The Product Security Engineer may overwrite the CVSS severity if we think the risk to our customers is lower or higher than the calculated score. A justification must be provided in the vulnerability ticket.

FIRST provides a CVSS calculator at the following URL: https://www.first.org/cvss/calculator/3.1

The CVSS 3.1 provides a textural representation of the severity score in Section 5 of the Specification Document. The mapping is included below:

Remediation SLAs

Refer to this .

Issue Types

Vulnerabilities must be assigned one of the following vulnerability types:

Backport Policy

For Mattermost server, the following backport policy applies:

Vulnerabilities with Low Score:

• Upcoming Release

• Extended Support Release

Vulnerabilities with Medium to Critical Score:

• Upcoming Release

• Extended Support Release

• Last 3 Releases

Notes:

• The Mattermost server policy also applies to vulnerabilities in bundled plugins within the respective version

• There is no backport policy for Desktop, Mobile and standalone plugins.

• Bundled plugins are the plugins that are pre-installed in the Mattermost server. The list of bundled plugins can be found in Mattermost server's . Any other plugin is considered standalone.

Process Steps

Report

Vulnerability reports are received from internal or external parties via one of the methods outlined above. The Product Security Team must acknowledge to the reporter that it received the report within 48 hours on business days and will investigate whether it’s a valid issue.

After the report was received, a “Security Vulnerability” ticket must be created in the Jira ticket that contains the following information:

• Title

• Description

• Category [If possible]

• Reporter

The security engineer that acknowledges the initial report must create a Mattermost Playbook Run using the "Security Vulnerability Playbook" and follow the process steps documented to continue with the Triage step. The playbook links to the Jira ticket created above, ensures the right process steps are completed, and informs relevant parties.

If the vulnerability was published on a news site/blog/etc. and is considered a public leak the Head of Marketing and VP Legal should be notified and invited to the playbook run.

Triage

After a vulnerability report was acknowledged it must be confirmed whether it is a valid finding within 96 hours of receiving it on business days. The security engineer should determine:

• Is the report a duplicate of a previous or current report from another reporter?

• Does the report have a security impact?

• Can the report be reproduced?

• If neither of the above hold true, does the report have an impact on any production infrastructure or systems and thus should be reviewed nevertheless?

After confirming that the report is valid the Product Security Team member must move the report to the next stage, called “Remediation” and must fill in the following information in the ticket:

• Vulnerability Category

• Steps to Reproduce

• Recommended Mitigation

• Security Update Text

Additional auto-populated fields

• A Mattermost-specific vulnerability ID is automatically assigned when a report is moved from “Triage” to “Fix Development”.

• A Due Date field is automatically populated when when a report is moved from “Triage” to “Fix Development” based on the Final Severity and the SLAs.

If the vulnerability is not confirmed valid or considered a duplicate, the security engineer must notify the reporter about the decision. If we don’t consider it a vulnerability, the justification for our decision should be included both in the response to the reporter and as a comment in the Product Vulnerability ticket. The ticket should afterwards be closed either as duplicate or as invalid.

If the report is deemed a security improvement (but not vulnerability), a development ticket should be created and the security vulnerability ticket closed with the development ticket added as a reference. A report is considered a security improvement when it is not exploitable without the presence of any additional vulnerabilities or only increases the general security posture.

Remediation

The Product Security Engineer must create a development ticket and link it to the vulnerability ticket. The Lead Engineer and the Product Manager of the feature team responsible for the vulnerability fix and the Release Manager should be invited in the vulnerability’s playbook run channel and be informed about the vulnerability and its severity. The Lead Engineer must acknowledge the report and assign the development ticket to an engineer. The Lead Engineer should work together with the Product Manager on the prioritization of the reported security vulnerability.

The development team is responsible for the development of the fix. The assigned Product Security Engineer is responsible to notify the developer teams if an agreed SLAs is at risk for the fix development and document once the SLA is broken. Developers should work together with the assigned Product Security Engineer during the vulnerability remediation to clarify questions and mitigation approaches. Once the development team finishes the development of the fix for the current version, the assigned Product Security Engineer should review that the fix properly mitigates the vulnerability. The fix must then be backported to previous releases. This is not the case if:

• The affected feature did not exist in the version in question

• The vulnerability has a low severity

If in doubt, refer to the Backport Policy section above. After the fix development has finished and has been reviewed by peers, security and QA, a new dot release must be prepared.

In parallel with the development of the fix by the Engineering team, the Product Security Engineer must invite in the playbook run channel the Security Operations team, if the vulnerability is of “High” or “Critical” severity or if cloud installations are at risk of exploitation. If cloud installations are at risk of exploitation, the Security Operations team must look at possible no-code mitigation steps to prevent the vulnerability from being exploited in the cloud until a fix is deployed. If the vulnerability is of “High” or “Critical” severity, the Security Operations team must investigate for any exploitation attempts with the guidance of Product Security. If exploitation is confirmed, a separate process will be triggered which is outside of the scope of this document. The Product Security Engineer may invite the Security Operations team to look for exploitation attempts for “Medium” severity vulnerabilities on a case by case manner.

Release Preparation

After reviews have been completed, the developer must notify the Cloud SRE team and the Release Manager that a new cloud release version can be cut. Cloud rings soak times for patches with vulnerability fixes can be found in this . Once the cloud release patch with the fix has been rolled out in the cloud, the Release Manager must cut a new dot release for all versions that received the security fix and release it. In case there are any problems during the release, the Delivery team should be notified to fix potential issues.

After the new versions are released, the Release Manager must update the based on the Reporter field from the Jira ticket.

Post Mortem

The Product Security Engineer must conduct a post mortem using the provided template for any vulnerabilities with “High” or “Critical” severity. The Product Security Engineer and the Lead Engineer of the feature team responsible for the vulnerability fix should meet and discuss:

• Why did this vulnerability occur?

• How can we verify that the vulnerability doesn’t exist in other parts of the product?

• What actions do we need to take to prevent it from happening again?

The outcome of the post mortem must be action items based on the answers to the questions above. The Product Security Engineer must then fill the retrospective of the playbook run based on the answers in the above questions and create backlog tickets with the agreed action items.

The Release Manager must create a draft for the email newsletter. The Marketing team and Product Security Engineering Lead must review the draft and send it out to all subscribers of the Security Updates mailing list.

Responsible Disclosure

This section is applicable if the vulnerability affects publicly released software artifacts.

Once the versions with the vulnerability fix are released, the Product Security Engineer must verify that all required information is included in the initial vulnerability ticket. The Release Manager must publish the security update notes for the vulnerabilities that have been fixed and released in the page. No details must be added in the “Issue Details”. Instead the following message must be posted: “Details on the security update will be posted here on DATE, as per our Responsible Disclosure Policy.” The DATE refers to 30 days after the release day.

The Product Security Engineer must prepare for a CVE disclosure. Specifically the Product Security Engineer must create a JSON file with all the needed CVE information. A tool like can be used for this purpose. A CVE ID can be reserved using the library.

30 days after the release with the vulnerability fix, the Product Security Engineer must publish the CVE details using the library. At the same time, the release manager must update the security update notes for vulnerabilities released in the page, this time properly populating the content of the “Issue Details” column.

Responsibilities Matrix