Postmortem: My Light
School Outage
Issue Summary
·
Duration of Outage:
September 22, 2024, from 10:30 AM to 12:00 PM (UTC+1)
Impact: My Light School management system experienced slow load times
and intermittent downtime. Around 40% of users, primarily administrative staff,
were affected. The student dasboards were sluggish, with significant delays in
loading pages. Some student users were unable to log in and check their
results..
Root Cause: A sudden spike in traffic combined with a misconfigured
database connection pool led to the overload of the system's database, causing
delays and unavailability for some users.
Timeline
- 10:30 AM:
Issue detected through a monitoring alert indicating high latency on
database queries.
- 10:32 AM:
Automated system checks triggered, confirming slow responses from the
student and dashboards.
- 10:40 AM:
The engineer on call noticed significant CPU usage on the database server.
- 10:50 AM:
Initial assumption was high traffic caused by end-of-term printing of
results by students, so actions taken were to investigate the server load.
- 11:00 AM:
The team attempted to scale the application servers, but performance did
not improve.
- 11:10 AM:
Misleading assumption that the issue was related to server-side caching
led to adjustments that did not resolve the issue.
- 11:30 AM:
Investigation escalated to the database team, who identified the
misconfigured connection pool.
- 11:45 AM:
Database connection pool size was adjusted, and traffic was rerouted to
reduce load.
- 12:00 PM:
Services fully restored, and all users were able to access the system
normally.
Root Cause and Resolution
The core issue stemmed
from the My Light School system's inability to handle a sudden traffic spike.
The database connection pool had been set to a low maximum of connections,
which was sufficient under normal conditions but insufficient during the surge
of requests. As more users accessed the system, the connection pool was quickly
exhausted, resulting in long wait times for new database connections.
The resolution involved
increasing the database connection pool size to accommodate more concurrent
connections. Additionally, the traffic was temporarily rerouted using a load
balancer to relieve pressure on the overloaded database server. Once the connection
pool settings were adjusted, the system gradually returned to normal operation.
Corrective and
Preventative Measures
Areas for improvement:
- Increase the capacity of the database
connection pool based on user load estimates.
- Implement more comprehensive
monitoring to detect spikes earlier and provide better alerts.
- Review server auto-scaling
configurations to handle high traffic events more efficiently.
TODO List:
- Increase the database connection pool
size to accommodate peak traffic.
- Add memory and CPU usage monitoring
for the database server with automated alerts for high usage.
- Optimize load balancer configuration
for dynamic traffic rerouting during surges.
- Conduct load testing to simulate
high-traffic scenarios and ensure system stability.
- Update auto-scaling policies for
application servers to ensure better response during spikes.
Comments
Post a Comment