Datadog Bits Investigations.

Metrics grow in several places

During the practice incident, the delay increased after release. At the same time, the database is slower to respond. Both events may be part of the same cause, but the order of the signals still needs to be clarified. Bits Investigations collects relevant data from Datadog. Open the original graphs and traces to see the time interval, environment and services that the hypothesis is based on.

If one source is not connected, this should be noted in the analysis. The absence of a signal in the summary does not mean that it did not exist.

Checking the timeline

In the practice scenario, record when latency increased, mark the release, and select a few normal and slow requests. Check which span takes longer. If the change appears in only one application version, inspect the differences. If it affects both, look at shared dependencies and load. Record which data supports each step, which events only coincide in time, and what else you need to collect.

Next up is the duty officer.

Agree with the owner of the service and the current rules of response. For comparison, save the time window, request terms and a link to the graph before changing. After work, check whether the delay has decreased for the desired group, and not only the average value for all requests. Such a journal helps to understand the situation, even if the first hypothesis turned out to be wrong.

Suitable tasks

• Compare API delay with release changes
• Check if different services are affected by one cause