Product Vulnerability Process

Overview

This document aims to provide guidance on the handling of product vulnerabilities. It describes the process and provides background information and definitions for the daily usage of all relevant stakeholders.

Goals

Establish and document a way of working that ensures:

• Vulnerabilities are triaged efficiently and quickly

• All parties are aligned around ownership and expectations

• Patch releases are available within the agreed SLAs

• Minimize the risk for Mattermost Cloud, on-premises customers, and Mattermost, Inc.

Scope

This process covers any vulnerabilities found in officially published Mattermost products or services that were either found internally, reported to us by a customer, or come in through the bug bounty program or via another third party.

Infrastructure vulnerabilities are explicitly out of scope for this process. Vulnerability management, as it relates to infrastructure, is owned by the Security Operations Team.

Process Definitions

Responsible Parties

• Product Security Team: Process Ownership; Vulnerability Owner

• Development Team: Vulnerability Remediation

• Release Manager: Release Planning

• Release Team: Build Process

• Marketing Team: Security Updates Email Newsletter Distribution

• Security Operations: Vulnerability mitigation for Mattermost Cloud, verification whether it’s been exploited in Mattermost Cloud.

Vulnerability Sources

Vulnerabilities are reported from a variety of sources that are explained below.

Internal Finding

The vulnerability has been discovered by the internal security, development, or QA team.

Security Automation

The vulnerability has been discovered by internal security automation tools such as Dependency Track or CodeQL

Responsible Disclosure

The vulnerability has been reported through our responsible disclosure program.

Bug Bounty Program

The vulnerability has been reported through our Bug Bounty program running on Bugcrowd.

Customer Report

The vulnerability has been reported from customers as part of an internal audit or on-going operations.

Public Leak

The vulnerability has not been officially reported, but released to the public without any notice. It is possible that it can contain reproduction steps or not.

Vulnerability Severity

Vulnerabilities must be scored according to the Common Vulnerability Scoring System Version 3.1 (CVSS 3.1). The CVSS is an industry-wide accepted standard to determine the impact of vulnerabilities in software products and takes impacts such as attack vector, complexity, user interaction, scope, and various other components into consideration. The Product Security Engineer may overwrite the CVSS severity if we think the risk to our customers is lower or higher than the calculated score. A justification must be provided in the vulnerability ticket.

FIRST provides a CVSS calculator at the following URL: https://www.first.org/cvss/calculator/3.1

The CVSS 3.1 provides a textural representation of the severity score in Section 5 of the Specification Document. The mapping is included below:

Rating	CVSS Score
None	0.0
Low	0.1 - 3.9
Medium	4.0 - 6.9
High	7.0 - 8.9
Critical	9.0 - 10.0

Remediation SLAs

Refer to this internal matrix.

Issue Types

Vulnerabilities must be assigned one of the following vulnerability types:

• CWE-79: Improper Neutralization of Input During Web Page Generation ('Cross-site Scripting')
• CWE-352: Cross-Site Request Forgery (CSRF)
• CWE-601: URL Redirection to Untrusted Site ('Open Redirect')
• CWE-287: Improper Authentication
• CWE-284: Improper Access Control
• CWE-400: Uncontrolled Resource Consumption
• CWE-200: Exposure of Sensitive Information to an Unauthorized Actor
• CWE-1395: Dependency on Vulnerable Third-Party Component
• CWE-74: Improper Neutralization of Special Elements in Output Used by a Downstream Component ('Injection')
• CWE-922: Insecure Storage of Sensitive Information
• CWE-319: Cleartext Transmission of Sensitive Information
• CWE-327: Use of a Broken or Risky Cryptographic Algorithm
• CWE-918: Server-Side Request Forgery (SSRF)

Backport Policy

For Mattermost server, the following backport policy applies:

Vulnerabilities with Low Score:

• Upcoming Release

• Extended Support Release

Vulnerabilities with Medium to Critical Score:

• Upcoming Release

• Extended Support Release

• Last 3 Releases

Notes:

• The Mattermost server policy also applies to vulnerabilities in bundled plugins within the respective version

• There is no backport policy for Desktop, Mobile and standalone plugins.

• Bundled plugins are the plugins that are pre-installed in the Mattermost server. The list of bundled plugins can be found in Mattermost server's Makefile. Any other plugin is considered standalone.

Process Steps

Report

Vulnerability reports are received from internal or external parties via one of the methods outlined above. The Product Security Team must acknowledge to the reporter that it received the report within 48 hours on business days and will investigate whether it’s a valid issue.

After the report was received, a “Security Vulnerability” ticket must be created in the Jira ticket that contains the following information:

• Title

• Description

• Category [If possible]

• Reporter

• Source

• Component

The security engineer that acknowledges the initial report must create a Mattermost Playbook Run using the "Security Vulnerability Playbook" and follow the process steps documented to continue with the Triage step. The playbook links to the Jira ticket created above, ensures the right process steps are completed, and informs relevant parties.

If the vulnerability was published on a news site/blog/etc. and is considered a public leak the Head of Marketing and VP Legal should be notified and invited to the playbook run.

Triage

After a vulnerability report was acknowledged it must be confirmed whether it is a valid finding within 96 hours of receiving it on business days. The security engineer should determine:

• Is the report a duplicate of a previous or current report from another reporter?

• Does the report have a security impact?

• Can the report be reproduced?

• If neither of the above hold true, does the report have an impact on any production infrastructure or systems and thus should be reviewed nevertheless?

After confirming that the report is valid the Product Security Team member must move the report to the next stage, called “Remediation” and must fill in the following information in the ticket:

• Vulnerability Category

• Steps to Reproduce

• Recommended Mitigation

• Security Update Text

• CVSS Score

• CVSS Severity

• Final Severity

• Mattermost Team

• Development Team

• Playbook Run link

• Bug Bounty Link

• Affected On-Premises Versions [Text]

• Affects Cloud [Yes/No]

Additional auto-populated fields

• A Mattermost-specific vulnerability ID is automatically assigned when a report is moved from “Triage” to “Fix Development”.

• A Due Date field is automatically populated when when a report is moved from “Triage” to “Fix Development” based on the Final Severity and the SLAs.

If the vulnerability is not confirmed valid or considered a duplicate, the security engineer must notify the reporter about the decision. If we don’t consider it a vulnerability, the justification for our decision should be included both in the response to the reporter and as a comment in the Product Vulnerability ticket. The ticket should afterwards be closed either as duplicate or as invalid.

If the report is deemed a security improvement (but not vulnerability), a development ticket should be created and the security vulnerability ticket closed with the development ticket added as a reference. A report is considered a security improvement when it is not exploitable without the presence of any additional vulnerabilities or only increases the general security posture.

Remediation

The Product Security Engineer must create a development ticket and link it to the vulnerability ticket. The Lead Engineer and the Product Manager of the feature team responsible for the vulnerability fix and the Release Manager should be invited in the vulnerability’s playbook run channel and be informed about the vulnerability and its severity. The Lead Engineer must acknowledge the report and assign the development ticket to an engineer. The Lead Engineer should work together with the Product Manager on the prioritization of the reported security vulnerability.

The development team is responsible for the development of the fix. The assigned Product Security Engineer is responsible to notify the developer teams if an agreed SLAs is at risk for the fix development and document once the SLA is broken. Developers should work together with the assigned Product Security Engineer during the vulnerability remediation to clarify questions and mitigation approaches. Once the development team finishes the development of the fix for the current version, the assigned Product Security Engineer should review that the fix properly mitigates the vulnerability. The fix must then be backported to previous releases. This is not the case if:

• The affected feature did not exist in the version in question

• The vulnerability has a low severity

If in doubt, refer to the Backport Policy section above. After the fix development has finished and has been reviewed by peers, security and QA, a new dot release must be prepared.

In parallel with the development of the fix by the Engineering team, the Product Security Engineer must invite in the playbook run channel the Security Operations team, if the vulnerability is of “High” or “Critical” severity or if cloud installations are at risk of exploitation. If cloud installations are at risk of exploitation, the Security Operations team must look at possible no-code mitigation steps to prevent the vulnerability from being exploited in the cloud until a fix is deployed. If the vulnerability is of “High” or “Critical” severity, the Security Operations team must investigate for any exploitation attempts with the guidance of Product Security. If exploitation is confirmed, a separate process will be triggered which is outside of the scope of this document. The Product Security Engineer may invite the Security Operations team to look for exploitation attempts for “Medium” severity vulnerabilities on a case by case manner.

Release Preparation

After reviews have been completed, the developer must notify the Cloud SRE team and the Release Manager that a new cloud release version can be cut. Cloud rings soak times for patches with vulnerability fixes can be found in this internal matrix. Once the cloud release patch with the fix has been rolled out in the cloud, the Release Manager must cut a new dot release for all versions that received the security fix and release it. In case there are any problems during the release, the Delivery team should be notified to fix potential issues.

After the new versions are released, the Release Manager must update the hall of fame list based on the Reporter field from the Jira ticket.

Post Mortem

The Product Security Engineer must conduct a post mortem using the provided template for any vulnerabilities with “High” or “Critical” severity. The Product Security Engineer and the Lead Engineer of the feature team responsible for the vulnerability fix should meet and discuss:

• Why did this vulnerability occur?

• How can we verify that the vulnerability doesn’t exist in other parts of the product?

• What actions do we need to take to prevent it from happening again?

The outcome of the post mortem must be action items based on the answers to the questions above. The Product Security Engineer must then fill the retrospective of the playbook run based on the answers in the above questions and create backlog tickets with the agreed action items.

The Release Manager must create a draft for the email newsletter. The Marketing team and Product Security Engineering Lead must review the draft and send it out to all subscribers of the Security Updates mailing list.

Responsible Disclosure

This section is applicable if the vulnerability affects publicly released software artifacts.

Once the versions with the vulnerability fix are released, the Product Security Engineer must verify that all required information is included in the initial vulnerability ticket. The Release Manager must publish the security update notes for the vulnerabilities that have been fixed and released in the Mattermost Security Updates page. No details must be added in the “Issue Details”. Instead the following message must be posted: “Details on the security update will be posted here on DATE, as per our Responsible Disclosure Policy.” The DATE refers to 30 days after the release day.

The Product Security Engineer must prepare for a CVE disclosure. Specifically the Product Security Engineer must create a JSON file with all the needed CVE information. A tool like vulnogram can be used for this purpose. A CVE ID can be reserved using the cvelib library.

30 days after the release with the vulnerability fix, the Product Security Engineer must publish the CVE details using the cvelib library. At the same time, the release manager must update the security update notes for vulnerabilities released in the Mattermost Security Updates page, this time properly populating the content of the “Issue Details” column.

Responsibilities Matrix

	Product Security	Development Team	Release Manager	Security Operations	Marketing	SRE
Triage Vulnerability	x
Product Vulnerability & Development fix Jira tickets creation	x
Product Vulnerability Jira status progression & fields completion	x
Security Vulnerability Playbook run creation & administration	x
Vulnerability fix development		x
Discovery of exploitation attempts (High/Critical only)				x
Security Review of the fix	x
Notify the Release Manager for a new release version cut		x
Cloud Release Preparation			x			x
Self Hosted Release Preparation			x
Post Mortem discussion (High/Critical only)	x	x
Post Mortem documentation & action items Jira tickets creation (High/Critical only)	x
Email newsletter draft			x
Update the hall of fame list			x
Review newsletter draft & send it to the Security Updates mailing list					x
Release of the security update notes			x
Prepare and publish CVE	x