Scala Profiling Under Fire

"While the codebase is fresh and grows fast under the umbrella of the local environment, we tend to rely on debugging tools, which were created specifically for that purpose. The app is half-baked, and the code is split open. We observe it through the lens of our IDE and with the speed of our brain. Everything is possible; we may pause execution for minutes, and the whole system is a white box - an open book for us."

"Tools such as Lightrun allow developers to dynamically instrument logs and even add basic metrics on the fly. But as load scales, I have found these tools to become prohibitively expensive. Above 100,000 requests per second, agent overhead becomes double-digit percentages of CPU time, and snapshot creation becomes impractical. There are also security constraints, since proprietary information could be easily compromised."

"As services move to production, scale becomes larger, and developers often turn to exposing the application's state to the outer world through metrics. At DV, we define metrics for all our SLAs and incorporate key performance indicators, aiming to cover as much business logic and system-relevant parameters as possible. Grafana dashboards provide us with great tools to analyze system behavior, either at the current moment or over days, weeks, or even months."

Application monitoring strategies vary significantly across development stages. During local development, debugging tools provide complete visibility into application state through IDEs. As applications move to staging and trial environments, traditional debugging becomes impractical, requiring structured logging instead. Dynamic instrumentation tools like Lightrun offer on-the-fly metric and log additions but become prohibitively expensive at scale, consuming double-digit CPU percentages above 100,000 requests per second while creating security risks. Production environments require static metrics defined at compile or deploy time, exposed through systems like Grafana dashboards. These metrics align with service-level agreements and key performance indicators, enabling analysis of system behavior across various timeframes from real-time to months of historical data.

Read at Medium