Netflix implements tiered storage for TimeSeries using Kafka, Flink, and Cassandra
Netflix reduced storage costs for petabyte-scale temporal data by migrating older datasets from hot storage to S3 using streaming and native Cassandra methods.
Summary
Netflix has detailed its evolution of the TimeSeries Abstraction platform to manage multi-petabyte datasets more cost-effectively. The company moved from a streaming approach using Apache Kafka and Apache Flink to write Parquet files to S3, to a more integrated solution leveraging Cassandra backups and pluggable Memtables for direct S3 cold reads. This tiered storage strategy allows recent data to remain on low-latency hardware while historical data resides on cheaper object storage.
Evolution of Netflix's tiered storage
- Solution 1 used Apache Kafka and Apache Flink to stream data into S3 as Parquet files, with a daily compaction job to merge small files into optimized daily files.
- Solution 2 utilizes Cassandra backups in S3 and pluggable Memtables to enable direct reads from object storage.
- The system uses Time Slices to partition datasets into immutable windows for optimization.
- Data is hash-partitioned into buckets to allow dedicated Flink tasks to batch writes and reduce S3 throughput pressure.
Why it matters
The implementation demonstrates how large-scale operators can combine multiple open source tools like Kafka, Flink, and Cassandra to manage the economic challenges of massive data retention.
Original reference: netflixtechblog.com →