Netflix migrates Ads Event Publisher to Apache Flink for live streaming
Key Takeaway
Netflix rebuilt its ad telemetry enrichment pipeline using stateful stream processing in Apache Flink to handle massive traffic surges during live events.
Summary
Netflix has evolved its Ads Event Publisher to support live streaming events like NFL games and major fights. By replacing a database-driven metadata lookup with real-time streaming joins in Apache Flink, the company can now handle the extreme viewership surges characteristic of live broadcasts. This transition allows for better scaling and removes critical path dependencies during high-traffic periods.
Transition from Key-Value lookups to Flink
- Replaced a globally replicated Key-Value store (Ads Metadata Registry) lookup with stateful stream processing.
- Implemented real-time streaming joins using Apache Flink to enrich ad telemetry.
- Designed to handle massive concurrency spikes, such as those seen during events with over 100 million viewers.
- Decoupled metadata enrichment from the critical serving path to improve scalability for live content.
Why it matters
This demonstrates a large-scale production use case for Apache Flink in managing highly volatile, massive-scale event telemetry.
Original reference: netflixtechblog.com →