Pages not loading for customers using specific 3rd party app

Summary

On January 28, 2025, between 13:35 and 15:10 UTC, Atlassian customers using Confluence Cloud experienced difficulties viewing, editing and creating pages. This disruption was caused by a faulty version of a third-party app which caused a grey overlay that obscured fully loaded Confluence pages, affecting customers who had the app installed.

Customers reported the issue, and the Atlassian team began their investigation within 10 minutes after receiving the support tickets.

Timeline of events

• The app vendor released version (1.3.39-AC) of the app on January 28, 2025, 13:35 UTC.
• Atlassian team got notified of the incident on January 28, 2025, 14:02 UTC, tried to reproduce the issue and involved the on-call teams.
• The app vendor released version (1.3.41-AC) of the app on January 28, 2025 14:35 UTC .
• Atlassian team notified customers to update the app to the latest version (1.3.41-AC) through our Statuspage on January 28, 2025, 15:06 UTC.
• The app vendor released a hot-fix for the existing faulty version (1.3.39-AC) of the app on January 28, 2025, 15:10 UTC. This helped mitigate the incident in one hour and 35 minutes.

Furthermore, the Atlassian team initiated a mandatory update to the latest version (1.3.41-AC) for all customers of the app on January 28, 2025, 16:37 UTC.

IMPACT

The overall impact was between January 28, 2025, 13: 35 UTC and 15:10 PM UTC on Confluence Cloud. The incident caused service disruption to customers having the third-party app installed, where they were unable to view, edit, or create pages on Confluence Cloud.

ROOT CAUSE

The issue arose from a problematic version release of a third-party app.

How it started?

The app vendor uploaded a new version of their app (1.3.39-AC) to the marketplace, making it available for automatic updates. Customer sites check for new minor app versions and update automatically within a rolling 24-hour period. However, an issue arose when the vendor deployed new frontend code that relied on a specific component available only in the latest app version. This created a dependency mismatch, leading to an error.

The app’s frontend code is delivered through the vendor's CDN, and during testing, no issues were detected. However, when the updated JavaScript code was deployed to the CDN, it assumed the presence of the new component. Unfortunately, not all customers were updated to the latest app version because of our progressive updates, resulting in some using older app descriptors that lacked the required component.

This mismatch triggered Confluence to display a full-screen dialog on every page load. The Confluence frontend rendered the dialog, which included an iframe to load the app’s code. Since the app had not been updated on some customer sites, the required component’s data could not be loaded, causing a grey overlay with error message to appear on the users’ screens.

The Confluence frontend provided functionality to close the dialog using the escape key. However, because the app failed to load the necessary data, the dialog displayed a generic error message overlaid with a grey opaque screen. Customers were unaware of the escape key workaround, leaving them unable to dismiss the dialog easily.

How was it mitigated?

• The app vendor deployed a new version (1.3.41-AC) of the descriptor.
• Atlassian team notified customers to update to the (1.3.41-AC) on StatusPage
• The app vendor released a hotfix for (1.3.39-AC) which removed the reference to the new component.

The issue was resolved in one hour and 35 minutes.

REMEDIAL ACTIONS PLAN & NEXT STEPS

We know that outages impact your productivity. While we have a number of testing and preventative processes in place, this specific issue wasn’t identified because the third-party app created an overlay on our loaded pages and that was not picked up by our monitoring systems.

We are working to improve the monitoring and detection of such app triggered failures. To prevent the recurrence of this type of incident, and detect such incidents faster, we are prioritizing the following improvement actions:

• We are developing a solution in our UX components to be resilient to app errors.
• We are also developing a solution where we can have clear messaging to exit from overlays that are taking the whole screen.
• We are working on improving our monitoring for this specific scenario.

We are collaborating with app vendors to share guidance on dependency component versioning. Therefore, the app vendors need to make changes in a more backward compatible manner.

We apologize to customers whose services were impacted during this incident; we are taking immediate steps to improve the platform’s performance and availability.

Thanks,

Atlassian Customer Support