Summary

On February 29, 2024, between 05:11 and 08:46 UTC, customers of Jira Software, Jira Service Management, Jira Work Management, Jira Product Discovery, and Confluence experienced an incorrect data update of Automation Rules. This was due to an incomplete feature flag rollout and bugs in the data upgrade code. 99.99% of affected Automation Rules were remediated by 17:00 UTC on February 29, 2024. The remaining 0.01% of Rule Components were edited by users during the impacted time window and required a confirmation to ensure changes weren’t overridden. These customers were contacted proactively.

IMPACT

Some Rules containing Issue Edit, Issue Create, or Issue Clone actions in Jira products had the Advanced Fields section removed. These Rules continued to run as if the Advanced Fields section were empty. In addition, for affected Issue Edit actions, Jira notifications were not sent.

Some Rules using the Send Email action in Jira and Confluence had the “From name” field removed. Emails continued to be sent but fell back to the Automation default for the “From name” field.

ROOT CAUSE

We were in the process of rolling out a new feature for Automation, which required us to perform data upgrades, which could be risky as the current system would run the upgrades automatically upon Rule activity. In an attempt to minimize such risk, we used feature flags to better control when the upgrade occurred.

However, there was an error in the feature flag configuration which resulted in the upgrades immediately kicking in. Additionally, there were bugs in these upgrades that resulted in overriding customers' saved values with default values. This incorrect configuration was used for Rule runs until recovery.

REMEDIAL ACTION PLAN & NEXT STEPS

We are prioritizing the following improvement actions to avoid repeating this type of incident:

• We are changing the approach for upgrading Rule configurations to allow better testing and prevent accidental upgrades.
• Improving feature flag rollout and verification processes to avoid such incorrect configurations.
• Increasing the frequency of backups of the Automation Rule data.

We apologize to customers who were impacted during this incident; we are taking immediate steps to improve the reliability of Automation.

Thanks,

Atlassian Customer Support