SUMMARY

Between March 18th, 2022 12:00AM and March 21st, 2022 9:29AM UTC, some Atlassian customers may have experienced varying levels of degradation across our different services and products, including Atlassian partner apps. The full list of impacted products is listed below in the impact section. This event was triggered as a result of a fault in Atlassian’s underlying feature flagging tool. The incident was detected through internal monitoring and support requests from customers and partners. The issue in the underlying service has been resolved, returning impacted Atlassian systems into a known good state. The total time to resolution was 3 Days 9 Hours 29 Minutes.

IMPACT

The overall impact was between March 18th, 2022 12:00AM and March 21st, 2022 9:29AM UTC. The impacted products were:

Jira

• During this incident Jira partners would have had a reduction to the number of web hooks they could create

Confluence (only impacted between March 18th, 2022 12:00AM and March 21st 2022 04:20am UTC)

• During this incident Confluence customers and consumers of their API would have been unable to:

  • Search for user name via the user profile macro
  • Create templates in the new editor
  • Correctly consume the user search API
  • View the people menu

• During this incident customers would have received emailed from bounce@spemail-us.atlassian.net instead of confluence@mail-eu.atlassian.net

Atlassian Account

• During this incident customers would have been unable to login/consent with OpenID or use OpenID to link their Atlassian Account

Customer Support

• During this incident customers would have been unable to perform some actions on their tickets including

  • Escalating their issue on https://getsupport.atlassian.com/
  • Using the bug template on https://jira.atlassian.com/

ROOT CAUSE

The issue was caused by an incident in Atlassian’s underlying feature flag tooling which exposed some errors in Atlassian’s consumption of feature flags. This resulted in some feature flags no longer evaluating as expected, resulting in Atlassian customers losing access to features or being shown older versions of features.

REMEDIAL ACTIONS PLAN & NEXT STEPS

Atlassian acknowledges that outages impact your productivity and business. While we have a number of testing and preventative processes in place, this specific issue wasn’t identified due to minimal monitoring for the specific fault which occurred in our underlying feature flag tooling.

We are prioritizing the following improvement actions to avoid repeating this type of incident in the future:

• Improvements to our consumption of feature flags in the impacted products and services to provide sensible fallbacks in the event this occurs again
• Adding additional monitoring and alerting around our underlying feature flag tooling and its integration points within our products and services reduce our time to detection in the event this occurs again

We apologize to customers whose services were impacted during this incident; we are taking immediate steps to improve the platform’s performance and availability.

Thanks,

Atlassian Customer Support