AU Jobs degraded processing

Incident Report for Solver

Postmortem

On March 18th, 2024, our Australian data center encountered a partial outage affecting a limited subset of data integration tasks. The issue stemmed from a single node within a cluster responsible for load balancing these tasks, which became unresponsive due to its inability to write new logs to disk. Functionality was restored to the affected node, and services resumed normal operation.

We are actively exploring strategies to enhance our monitoring capabilities and proactively maintain node health to mitigate any future instances of data integration failures.

Posted Apr 11, 2024 - 08:50 PDT

Resolved

This incident has been resolved.

Posted Mar 17, 2024 - 21:13 PDT

Update

We are continuing to monitor for any further issues processing integrations. If the job did not complete, it will be cancelled. And will run at the next scheduled time. Jobs can be re-run manually as well. We have identified the root cause and we will provide more details within the next several days.

Posted Mar 17, 2024 - 20:38 PDT

Monitoring

A fix has been implemented and we are monitoring the results.

Posted Mar 17, 2024 - 20:24 PDT

Identified

The issue has been identified and a fix is being implemented.

Posted Mar 17, 2024 - 20:02 PDT

Investigating

We are currently investigating this issue.

Posted Mar 17, 2024 - 19:53 PDT

This incident affected: AU Cloud (Data Services).