LinkedIn has transitioned from using Apache Kafka for managing data infrastructure due to unprecedented growth from 90 million to 1.2 billion users. The company now processes 32 trillion records daily across 17 petabytes of data, which has introduced considerable operational complexity. The development of Northguard and Xinfra represents a significant evolution in LinkedIn's handling of event streaming, moving beyond the limitations of their original system to address the challenges posed by such extensive data scaling.
When LinkedIn's engineers published their announcement about Northguard & Xinfra earlier this year, it sent ripples through the event streaming community. Here was the company that created Apache Kafka - the backbone of modern data infrastructure - essentially saying they'd outgrown their own creation.
This isn't just another 'we built a better Kafka' story. This is about what happens when you scale from 90 million to 1.2 billion users, when your system processes 32 trillion records daily across 17 petabytes of data.
In 2010, when Kafka was first developed, LinkedIn had 90 million members. Today, they serve over 1.2 billion users. That's not just a linear scaling problem - it's an exponential complexity challenge that touches every aspect of distributed systems design.
Collection
[
|
...
]