Confluence Bad gateway error

Incident Report for Confluence

Postmortem

SUMMARY

On March 26, 2024, between 13:22 and 16:33 UTC, Atlassian customers using Confluence Cloud experienced longer than expected loading times or failed page loads. The event was triggered by a code change that added a dependency on a missing configuration which impacted prod-east customers only. The incident was detected within minutes by automated monitoring and mitigated by enabling a redundancy to remove the impacted service, which put Atlassian systems into a known good state. The total time to resolution was about three hours and 10 minutes, with most customers recovering within one hour.

IMPACT

The overall impact was between March 26, 2024, between 13:22 and 16:33 UTC UTC on Confluence Cloud, with most customers recovering within one hour.

Confluence customers may have experienced reduced functionality, slower response times, and limited access when loading Confluence pages, space overview pages, and the home page.

ROOT CAUSE

The issue was caused by a code change depending on a missing configuration. After identifying the problem, we quickly reverted the change to mitigate impact to customers.

REMEDIAL ACTIONS PLAN & NEXT STEPS

We know that outages impact your productivity. We are prioritizing the following improvement actions to avoid repeating this type of incident:

Introducing improved testing capabilities to prevent similar issues from reaching production environments.
Adding enhanced redundancy within the impacted service to improve availability.
Adding additional safeguards to prevent missing configurations.

‌

We apologize to customers whose services were impacted during this incident; we are taking immediate steps to improve the platform’s performance and availability.

Thanks,

Atlassian Customer Support

Posted Apr 02, 2024 - 00:39 UTC

Resolved

Between 13:15 UTC to 15:15 UTC, Customers experienced issues accessing Confluence in the US East regions and were getting 502 Bad gateway errors. The issue has been resolved and the service is operating normally

Posted Mar 26, 2024 - 16:37 UTC

Monitoring

As a priority we ensured to address the 502 Bad Gateway errors impacting our Confluence customers which has now been resolved and you should no longer see these errors. We have monitored and observed recovery in the impacted regions and multiple customers have also confirmed recovery. We apologize for the inconvenience caused.

Posted Mar 26, 2024 - 16:09 UTC

Update

We do not have a root cause for the 502 Bad Gateway error impacting Confluence in US east regions. We are investigating and will post updates within the next hour.

Posted Mar 26, 2024 - 15:06 UTC

Update

We are continuing to investigate this issue.

Posted Mar 26, 2024 - 14:07 UTC

Investigating

We are investigating an issue with Confluence where we are getting 502 Bad gateway errors that is impacting some Confluence Cloud customers. We will provide more details within the next hour.

Posted Mar 26, 2024 - 14:02 UTC

This incident affected: View Content, Create and Edit, Comments, Authentication and User Management, Search, Administration, Notifications, Marketplace Apps, Purchasing & Licensing, Signup, Confluence Automations, Cloud to Cloud Migrations - Copy Product Data, Server to Cloud Migrations - Copy Product Data and Mobile (iOS App, Android App).