SUMMARY

On NOV 7, 2022 between 8:10 AM and 7:26 PM UTC, Atlassian customers using Confluence experienced degraded performance. The event was triggered by database migrations which impacted customers in a single database instance in the European region. The incident was detected within 28 minutes by Atlassian’s internal automated monitoring system system and mitigated by scaling the database instance size which put Atlassian systems into a known good state. The total time to resolution was about 11 hours and 16 minutes.

IMPACT

Some customer database instances were over provisioned and experienced intermittent slow performance, request timeouts, and complete downtime.

ROOT CAUSE

The issue was caused by database migrations performed for maintenance operation. The migration resulted in over provisioning and caused database CPU to spike to 100%. The CPU spike resulted in degraded performance for Confluence.

REMEDIAL ACTIONS PLAN & NEXT STEPS

We know that outages impact your productivity. While we have a number of testing and preventative processes in place, this specific issue wasn’t identified earlier because modeling customer usage is nuanced and even slight error in modeling will result in over provisioning or under provisioning.

We are prioritizing the following improvement actions to avoid repeating this type of incident:

• Adding more metrics used in customer usage modeling.
• Improving automated tenant rebalancing to avoid over provisioning or under provisioning.

Thanks,

Atlassian Customer Support