Impact/Scope: Customers experienced significant slowness in CEM (application-wide latency degradation).

Investigation Findings

Recent shifts in application pattern usage introduced additional traffic to several legacy endpoints. These endpoints were expected to participate in the system's standard asynchronous backend processing model, but investigation revealed that they had inadvertently retained synchronous components from earlier implementations. Under normal load, this discrepancy went unnoticed, since request volume to these endpoints remained low enough that synchronous processing did not create a bottleneck.

As traffic to these paths increased, connections were held longer than expected, and available connections across the affected systems were exhausted. This connection starvation caused blocking behavior that cascaded into broader application slowness affecting CEM.

A temporary workaround has been applied to relieve pressure on the affected endpoints and restore normal operation. The permanent fix of converting the retained synchronous components to the standard asynchronous model is in progress.

Plans for Improvement

• Convert affected endpoints to fully asynchronous processing — implement the permanent fix to remove the remaining synchronous components identified during this incident.
• Audit other legacy endpoints for similar synchronous/async inconsistencies — proactively identify any other endpoints that may have drifted from the async standard, before they become load-driven incidents.

This is related to events on July 1st, 9th, 10th and 15th. The intermittent and short occurrence duration of this issue resulted in an unusually long diagnostic and repair cycle.