InfoQ HomepageNewsNetflix Moves Toward Open
Architecture & Design
Beyond the PR: The New Control Plane for Agentic Software Delivery (Webinar Sept 17th)
Netflix Moves Toward Open
Sep 07, 2026
2
min read
Netflix is moving toward the open-across multiple AWS regions, after finding that its cluster-level approach was less effective for complex, stateful pipelines with operators that have different processing requirements. Netflix reports that one team reduced annualized Flink compute expenditure by 58%, saving approximately $1.1 million annually
Comparison of Netflix’s two Flink Autoscalers
Netflix has runApache Flink since 2017 and built its first autoscaler around 2019. The system ran on Mantis and consumed cluster-level telemetry fromAtlas, including CPU, network utilization, Kafka lag, input rate, and consume rate. It adjusted the total number of TaskManagers and reduced re
The limitation was the scaling unit. Because the original autoscaler reasoned about the cluster rather than individual operators, all operators in a job effectively shared the same scaling decision. That became less suitable for stateful pipelines containing branches, joins, and terabytes of state, where different parts of the dataflow can have different processing requirements.
TheApache Flink Autoscaler instead uses metrics exposed by the running job to estimate each operator’s true processing rate from throughput and busy time. It then walks the job graph and calculates required parallelism for individual vertices. The approach is described inFLIP-271, which addresses autoscaling for heterogeneous streaming jobs and the cost of rescaling stateful applications.
The technique also builds on research from the DS2 project. Vasiliki Kalavri, a systems researcher involved in the work, said the project initially explored critical path analysis before adopting True Processing Rate as a simpler baseline. It turned out that this very simple idea worked really well. The approach subsequently became part of Flink’s autoscaling work.
Netflix integrated the autoscaler with its internal control plane rather than deploying it directly through the Flink Kubernetes Operator. A Spring Boot service usesTemporal workflows to isolate autoscaling decisions for individual jobs. Netflix also modified JobManager metric collection to support jobs with up to 3,000 subtasks, added server-side metric filtering, preserved forward connected subgraphs during scaling, and added handling for sink backpressure.
The OSS-based Flink Autoscaler architecture with Temporal workflows
The forward connection issue has also appeared inApache Flink developer discussions. Changing parallelism across a FORWARD connection can require redistribution, while Netflix’s implementation keeps forward connected operators together. An openFLINK-38538 issue separately highlights cases where busy operators can be affected by output ratio-based scaling decisions.
Flink Job DAG Showing Current and Desired Parallelism Per Vertex
Unlike generic event-driven autoscalers such asKEDA, which scale workloads from external metrics or events, Flink’s autoscaler reasons about the internal dataflow graph and operator capacity. Netflix currently uses a 0.45 utilization target, below Flink’s 0.7 community default, to reduce aggressive rescaling of large stateful jobs. The company plans to migrate its remaining internal autoscaling use cases to the open-source implementation, while investigating Flink 2’s disaggregated state architecture to address the cost of state recovery during rescaling.
