How to survive a horror movie (if that movie is your prod environment)

"PagerDuty screams: your checkout service is failing. You pull up a distributed trace, expecting a clear, end-to-end map of the request, from your frontend service, through product-catalog, recommendation, and so on, through to payment service (maybe). Instead, it looks like someone's taken an axe and chopped it up. There are gaping holes in between calls, and as you reach the bottom of the trace waterfall, there they are, hauntingly out of place. The orphaned spans."

"The ghastly truth (and the antidotes) When spans are missing or have invalid parent span IDs, their children spans become what are referred to as orphaned spans, causing traces to become fragmented. In New Relic, orphaned spans appear at the bottom of the trace, but won't have lines that connect them to the rest of the trace. This can happen for a number of reasons, but you can save future spans from the span orphanage!"

Orphaned spans occur when spans lack valid parent span IDs or are dropped, causing fragmented distributed traces that hide the true sequence of calls. Fragmented traces can make it unclear whether a service is the cause of a failure or merely a victim. Orphaned spans often appear disconnected at the bottom of trace visualizations and lack connecting lines. High throughput and agent or API collection limits can contribute to missing spans. Detecting and addressing parent ID issues, adjusting collection limits, and improving instrumentation can prevent span orphaning and restore end-to-end trace visibility.

Read at New Relic