Operational Excellence Best Practices

In the summer of 2023, my team entered into a code yellow to stabilize the health of the service we own. This service powers the visualization on the dashboard product. The decision was made following high-severity incidents impacting the availability of the service.  

For context, the service provides aggregation data to dashboard visualizations by gathering aggregations through the data pipeline. This is a critical path service for dashboard rendering. Any impact on the availability of this service manifests itself as dashboard viewers experiencing delays in rendering visualizations and rendering failures in some cases.  

Exit Criteria for Code Yellow

•  Fix the scaling issues on the service using the below mechanisms:
  • Enhance observability into service metrics
  • Implement service protection mechanisms
• Investigate and implement asynchronous execution of long-running requests
• Mitigation mechanisms to recover service below 10 minutes

Execution Plan

1. Enhance observability into service metrics

• Enhanced service request body logging
• Replaying traffic to observe patterns after the incident. 
  • Since this service was read-only, and the underlying data was not being modified hence we could rely on the replays.

2. Service protection mechanisms

• An auto restart of the child thread on request hangs.
• Request throttling to protect against overwhelming

3. Investigate and implement asynchronous execution of long-running requests

• Replace deprecated packages (Request with Axios)
•  Optimizing slow-running operations 

Key Takeaways

• Enhanced tooling helped to isolate problematic requests and requests with poor time complexity blocking the node event loop. 
• We set up traffic capture and replay of the traffic on demand. 
• Also, CPU profiling and distributed tracing as observability improvements.
• Optimizations on the critical path: Efforts to optimize operations on the critical path have yielded ~40% improvement in the average latencies across DCs. These efforts include (but are not limited to) package upgrades such as the replacement of Request with Axios (Promise-based HTTP client), caching optimizations — unintentional cache misses, and identification of caching opportunities.
• Scale testing and continuous load testing framework are set up to monitor the service’s scale needs. 
• Mitigation mechanisms rolled out. This included node clustering mode with an auto restart of the thread when an event loop gets blocked. 
• Request throttling is implemented to protect the service in case of bad requests
• Better alert configuration bringing down the time to detect anomalies below 5 minutes in most recent incidents
• Clear definition of Sev-1/Sev-2 Criteria: We now have clear sev-1/sev-2 criteria defined. This is to help on-calls quickly assess if the system is in a degraded state and whether or not they need to pull the sev-2 trigger to get help. 

Next Steps

• To further the momentum of the operation excellence, the plan is to perform quarterly resiliency game days to find the system’s weaknesses and respond gracefully in the event of failures.
• Re-evaluate the Northstar architecture of the service to meet the scaling needs of the future.

At this point, I feel more confident in our overall operational posture and better equipped to deal with potential incidents in the future. At the same time, I recognize that operational improvements are a continuous process, and we will continue to build on top of the work done as a part of Code Yellow.

All the opinions are mine and are not affiliated with any product/company.