Postmortem: My Light School Outage

Issue Summary

·         Duration of Outage: September 22, 2024, from 10:30 AM to 12:00 PM (UTC+1)
Impact: My Light School management system experienced slow load times and intermittent downtime. Around 40% of users, primarily administrative staff, were affected. The student dasboards were sluggish, with significant delays in loading pages. Some student users were unable to log in and check their results..
Root Cause: A sudden spike in traffic combined with a misconfigured database connection pool led to the overload of the system's database, causing delays and unavailability for some users.

Timeline

  • 10:30 AM: Issue detected through a monitoring alert indicating high latency on database queries.
  • 10:32 AM: Automated system checks triggered, confirming slow responses from the student and dashboards.
  • 10:40 AM: The engineer on call noticed significant CPU usage on the database server.
  • 10:50 AM: Initial assumption was high traffic caused by end-of-term printing of results by students, so actions taken were to investigate the server load.
  • 11:00 AM: The team attempted to scale the application servers, but performance did not improve.
  • 11:10 AM: Misleading assumption that the issue was related to server-side caching led to adjustments that did not resolve the issue.
  • 11:30 AM: Investigation escalated to the database team, who identified the misconfigured connection pool.
  • 11:45 AM: Database connection pool size was adjusted, and traffic was rerouted to reduce load.
  • 12:00 PM: Services fully restored, and all users were able to access the system normally.

Root Cause and Resolution

The core issue stemmed from the My Light School system's inability to handle a sudden traffic spike. The database connection pool had been set to a low maximum of connections, which was sufficient under normal conditions but insufficient during the surge of requests. As more users accessed the system, the connection pool was quickly exhausted, resulting in long wait times for new database connections.

The resolution involved increasing the database connection pool size to accommodate more concurrent connections. Additionally, the traffic was temporarily rerouted using a load balancer to relieve pressure on the overloaded database server. Once the connection pool settings were adjusted, the system gradually returned to normal operation.

Corrective and Preventative Measures

Areas for improvement:

  • Increase the capacity of the database connection pool based on user load estimates.
  • Implement more comprehensive monitoring to detect spikes earlier and provide better alerts.
  • Review server auto-scaling configurations to handle high traffic events more efficiently.

TODO List:

  1. Increase the database connection pool size to accommodate peak traffic.
  2. Add memory and CPU usage monitoring for the database server with automated alerts for high usage.
  3. Optimize load balancer configuration for dynamic traffic rerouting during surges.
  4. Conduct load testing to simulate high-traffic scenarios and ensure system stability.
  5. Update auto-scaling policies for application servers to ensure better response during spikes.

 

Comments