A UI generation component of our application entered a crash/restart cycle due to an increase in load. This reduced the capacity of the cluster as a whole resulting in other instances of the UI generation component crashing as well. Autoscaling typically ensures sufficient capacity, however, this component was recently separated out and autoscaling was not yet enabled during behavior observation.
The UI generation component was manually scaled resolving the issue.
To prevent similar issues moving forward, we will be enabling autoscaling *up* of components during initial observation periods while still preserving manual scaling *down*. This will prevent resource starvation while still allowing us to ensure correct scale tuning.
Posted Jan 09, 2024 - 12:02 MST
Monitoring
A fix has been deployed and is being monitored
Posted Jan 09, 2024 - 09:44 MST
Investigating
We are investigating an availability issue with the CEM.
Posted Jan 09, 2024 - 08:49 MST
This incident affected: Candidate Experience Manager (User Experience).