Evolving Dataflow to process massive datasets for machine learning

Google created MapReduce more than 20 years ago to solve the scaling problems in data processing that the then young company was running into. The AI era that we are in now demands efficient, large-scale data processing for everything from training frontier models like Gemini by Google DeepMind to powering fully autonomous vehicles like Waymo. Many aspects of machine learning, including data ingestion, transformation, and feature extraction, rely heavily on processing massive datasets.

To meet this astronomical scale required by efforts across Google, we evolved our data platform, Flume, the successor to the original MapReduce, with innovations focused on scalability , efficiency , and a better developer experience . And many of those innovations are available as part of Dataflow , our fully managed batch and streaming platform built on the same core technology Google uses to power its most demanding internal workloads. In this blog, we provide an overview of the many innovations in the Flume platform, and a glimpse into how Google Cloud customers are putting those features into action with Dataflow.

Addressing massive scalability The scale of data processing at Google has exploded over the last 20 years and continues to drive innovation. To tackle the challenges of immense scale, we introduced several features within Google's data processing platform, which are also available in Dataflow:: Liquid sharding dynamically splits work units (shards) during execution for on-the-fly rebalancing. This helps pipelines with uneven data distribution and stragglers to maximize worker efficiency as data grows.

Global compute enables enormous scaling by dynamically scheduling workloads across Google's global infrastructure. The system automatically determines the optimal location based on factors like data locality and resource availability. Automatic pipeline optimization fuses consecutive operations into a single stage.

This reduces I/O and stage-transition overhead, allowing large-scale execution to scale more gracefully. Rate-limiting external API calls manages load on external services. This is essential for modern ML pipelines that frequently call external APIs for tasks like model evaluation, preventing high data volumes from overloading systems.

Tandem pools facilitate serverless remote inference. This feature helps overcome scalability limitations often found in remote inference systems by efficiently hosting, sharing, managing, and autoscaling external model servers. Boosting efficiency with accelerators Doing more with less isn't just a constraint; it fuels our progress.

By finding ways to run more efficiently, we create the space and capacity needed for rapid innovation. This is particularly evident for teams that use accelerators like TPUs for their workloads. To improve utilization and cost efficiency, our engineers devised several novel features for our platform, now part of Dataflow: Heterogeneous worker pools allow developers to specify custom resource requirements for different pipeline stages.

For example, TPU-intensive work runs on TPU-equipped workers, while other stages use standard CPU workers. This ensures optimal resource allocation. TPU-aware autoscaling prevents excessive initial assignment of TPU workers and improves efficiency during subsequent autoscaling events.

Duty-cycle policy enforcement automatically scales down TPU workloads when the accelerator's duty cycle (the fraction of time it is active) is low, scaling back up only when utilization improves. TPU fungibility : By working with other infrastructure teams, we developed optimizations to encourage scheduling jobs to the most suitable TPU version and cell location based on quota and resource availability. Enhancing the developer experience Considering the wide mix of backgrounds and tools across Google, rapid prototyping, iteration, and reliable production operations are extremely important.