Updated Notice Regarding 2016 Security Incident

Hi everyone,

We recently completed our review into the security incident we announced in October, and wanted to share some additional information with you.

As an initial matter, we note that we did not discover any affected customer information other than what we announced could have been impacted in October. During our review, we discovered that a small number of AWS keys used by Zendesk to access our AWS environment, were compromised after having been provided to a third party vendor. These keys were then used to access customer service data across our production environment.

At this time we still have no evidence that customer information was actually misused. Nevertheless, as a precaution, we implemented password rotations for all active agents in Support and Chat, and all end users in Support created prior to November 1, 2016.

In line with our previous communications, we recommend that customers take the following precautions, if you have not done so already:

• If you installed a Zendesk Marketplace or private app prior to November 1, 2016 that saved authentication credentials such as API keys or passwords during installation, rotate all credentials for the respective app.

• In addition, if you uploaded a TLS certificate to Zendesk prior to November 1, 2016 which is still valid, upload a new certificate, and revoke the old one

• While we have no indication at this time that other authentication credentials were accessed, customers may want to consider rotating authentication credentials used in Zendesk products prior to November 1, 2016. API Tokens in Chat do not need to be rotated.

We also have taken additional actions since the incident occurred in order to further enhance our security protocols. For example, we have rolled out several changes since 2016, including:

• Expanded Single Sign On and Multi-Factor Authentication across our environment. This roll-out took place during 2016 and 2017. The services affected by this incident are now protected by these technologies;
• Increased security monitoring and logging. We have upgraded our security monitoring stack, with increased monitoring of logs;
• Increased security scanning, both at an application and infrastructure level: We increased the level and frequency of security scanning we perform, both at an application and infrastructure level. With our migration to AWS completed in June of 2019, we have improved our ability to patch and remediate security vulnerabilities.
• Infrastructure as code: Most of our infrastructure is now managed through code, which is frequently validated for accuracy.

We also understand that security is an ongoing process; thus, we are continuing to evaluate ways to further improve our security posture. Key steps we are taking immediately include:

• In addition to our existing annual and quarterly third party security testing, we are conducting additional third party testing.

• Effective November 4, we have made the key security features Configurable Password Policy, Single sign-on, IP restrictions and Custom Session Expiration available at all plan levels. Customers who used SSO and MFA were less affected by this incident, and we encourage all customers to deploy these security features on their accounts.

Maarten Van Horenbeeck
Chief Information Security Officer

————

Updated Notice Regarding 2016 Security Incident

Relevant to Zendesk Customers with Accounts Activated Prior to November 1, 2016

We recently were alerted by a third party regarding a security matter that may have affected the Zendesk Support and Chat products and customer accounts of those products activated prior to November of 2016. While our investigation is still ongoing, on September 24, 2019, we determined that information belonging to a small percentage of customers was accessed prior to November of 2016.

We deeply regret that this incident occurred. The safety and security of our customers and their data is of paramount importance to us. Our goal is to communicate this information as quickly as possible with transparency and guidance on how to address. We will be updating and sharing more information in this blog post and our help center as it becomes available.

For further information, please go to our frequently asked questions/FAQ page.

What happened and what does this mean to you?

Once we became aware of this security matter, the Zendesk Security teams launched an investigation into the incident, including:

• Engaging a team of outside forensic experts to validate the claims of the third party and to determine the exact data and information that was exposed

• Activating our internal data security response team and protocol. This team continues to investigate with full resources dedicated to determining how this exposure occurred

• Informing law enforcement and the appropriate global regulatory agencies

On September 24, we identified approximately 15,000 Zendesk Support and Chat accounts, including expired trial accounts and accounts that are no longer active, whose account information was accessed without authorization prior to November of 2016. Information accessed included some personally identifiable information (PII) and other Service Data. We have found no evidence that ticket data was accessed in connection with this incident.

For impacted customers, the information accessed from these databases includes the following data:

• Email addresses, names and phone numbers of agents and end-users of certain Zendesk products, potentially up to November 2016

• Agent and end user passwords that were hashed and salted – a security technique used to make them difficult to decipher, potentially up to November 2016. We have found no evidence that these passwords were used to access any Zendesk services in connection with this incident.

UPDATE: We have also determined that certain authentication information was accessed for approximately 7,000 customer accounts, including expired trial accounts and accounts that are no longer active. Upon further analysis, we also found an error and identified a group of customers who had a small number of TLS certificates accessed, almost all of which are currently expired.

Here is the information impacted:

• Transport Layer Security (TLS) encryption keys provided to Zendesk by customers

• Configuration settings of apps installed from the Zendesk app marketplace or private apps. This may include integration keys used by those apps to authenticate against third party services.

What has been done to remedy the situation?

We are taking specific steps to ensure that all potentially impacted customers are protected. These steps include the following actions:

• We are informing all impacted customers directly and sharing the steps we are taking to safeguard their accounts and data and additional actions they can take themselves.

• As a precautionary measure, in the next 24 hours, we are starting to implement password rotations for all active agents in Support and Chat, and all end users in Support created prior to November 1, 2016. This password rotation will impact all other products which share authentication with Support, including Guide, Talk and Explore. Upon their next login, each of these users will be required to create a new password. You will not be impacted by this if we have been able to identify that you have updated your password since November 1, 2016 or have implemented Single Sign-on in connection with your account.

• We are continuing our investigation including working with outside forensics experts and law enforcement.

As a Zendesk customer, what do I need to do?

If you have received an email from us saying that you had an account prior to November 1, 2016, we recommend that you take the following steps:

• If you installed a Zendesk Marketplace or private app prior to November 1, 2016 that saved authentication credentials such as API keys or passwords during installation, we recommend that you rotate all credentials for the respective app.

• In addition, if you uploaded a TLS certificate to Zendesk prior to November 1, 2016 which is still valid, we recommend you upload a new certificate, and revoke the old one

• While we have no indication at this time that other authentication credentials were accessed, customers may want to consider rotating authentication credentials used in Zendesk products prior to November 1, 2016. API Tokens in Chat do not need to be rotated.