SUMMARY

On JUN 14, 2023, between 18:55 and 19:50 UTC, Atlassian customers using Confluence may not have been able to access their Confluence site. The event was triggered by a name change made to an infrastructural service which Confluence uses for monitoring, which impacted customers in US regions (prod-west2 and prod-east). The incident was detected within 10 minutes by monitoring and mitigated by scaling up the Confluence API gateway, as well as reverting the renaming of the infrastructural service. This put Atlassian systems into a known good state. The total time to resolution was about 55 minutes.

IMPACT

Between June 14, 2023 18:55 and June 14, 2023 19:50 UTC, Confluence customers in the US East and West regions may have not been able to access their Confluence sites. This impact did not extend to all Confluence sites or users on a given site, as the nature of the issue meant that some users requests to their site may have succeeded, while others failed.

ROOT CAUSE

This incident was caused by a name change to an infrastructural service that Confluence uses for monitoring purposes. This service is normally invoked by Confluence's gateway on every web request which is received. However when the infrastructural service’s name was updated, a new failure mode was induced in the Confluence gateway service, which in turn began a memory leak. Under high load, Confluence’s gateway hosts experienced an out of memory condition which caused the service degradation encountered during this incident.

REMEDIAL ACTIONS PLAN & NEXT STEPS

We know that outages impact your productivity. While we have a number of testing and preventative processes in place, this specific issue wasn’t identified because:

• There were naming convention challenges and internal team miscommunication during the rollout of the name change to the infrastructural service.
• The infrastructural service which was renamed does not directly provide customer functionality, and is only used internally by engineers for monitoring. As such, while tests do exist for this code, they do not check for name changes such as what occurred in this outage.
• The fault wasn’t detected in our staging environment because when the load was low, the memory leak was very slow, and when the load was higher, active load testing masked node failures.
• The Confluence gateway code which invokes the infrastructural service was not resilient to a potential naming change, and was not using environmental variables.

We are prioritizing the following improvement actions to avoid repeating this type of incident:

• The Confluence gateway service will adopt the recommended practice to use environmental variables when invoking infrastructural services.
• The Confluence gateway service will make the relevant code more robust to ensure the system can bypass this non-critical path infrastructure service if it is not available generally.
• We will implement checks when the Confluence gateway service is deployed to fail at deploy time if a key dependency cannot be invoked (by a specific name or otherwise).
• Our infrastructure team will perform additional verification checks when decommissioning production infrastructure to ensure it isn't still being actively used by critical workloads.

Furthermore, to minimize the impact of breaking changes to our environments, we will implement additional detection measures such as:

• We will improve monitoring and alerts in our staging environment for Confluence's gateway service. While we do have alerts in production which fired properly, we will enable them for staging and reduce lag time between the event firing and alerts being generated in staging.

We apologize to customers whose services were impacted during this incident; we are taking immediate steps to improve the platform’s performance and availability.

Thanks,

Atlassian Customer Support