What's new for Managed Service for Apache Spark clusters

Overview

Google Cloud detailed updates to its Managed Service for Apache Spark, the rebranded Dataproc service, focused on the managed clusters deployment mode. The improvements center on three pillars: making Spark faster, easier to run, and smarter with embedded AI. Announced at Google Cloud Next '26, the headline addition is the Lightning Engine native execution engine, alongside Flexible VMs and FinOps features for cost control.

Key Takeaways

Dataproc is now Managed Service for Apache Spark, offered in serverless and managed cluster deployment modes.
The Lightning Engine native execution engine speeds up Spark DataFrame, Dataset, and SQL workloads.
Lightning Engine requires no code changes to existing Spark applications.
Flexible VMs improve resource obtainability by letting users rank up to ten machine types.
Zero-scale clusters and scheduled stops give better cost control over persistent environments.

Stats & Key Facts

#Lightning Engine delivers up to 4.9x faster performance than standard open-source Spark
#Up to 2x the price-performance over the leading high-speed Spark alternative
#Flexible VMs allow up to ten ranked machine types per node role

What's new for Managed Service for Apache Spark clusters

Two deployment modes

The service supports different architectural needs.

›Serverless mode completely abstracts infrastructure management for ephemeral or ad-hoc jobs.
›Managed clusters mode suits teams needing fine-grained customization, persistent environments, or long-running stateful processing.
›Managed clusters support native integration with custom Compute Engine hardware configurations.

The service reflects Google Cloud's integration with the Agentic Data Cloud, and this update focuses on the managed clusters deployment mode announced at Google Cloud Next '26.

Faster with Lightning Engine

Lightning Engine is described as the biggest update for managed clusters.

›It introduces performance gains for Spark DataFrame and Dataset APIs and heavy Spark SQL queries.
›It uses a native C++ vectorized execution engine built on Velox and Gluten.
›It bypasses JVM execution bottlenecks by compiling query plans into native instructions optimized for SIMD vectorization.

Lightning Engine delivers up to 4.9x faster performance than standard open-source Spark and up to 2x the price-performance over the leading high-speed Spark alternative, without requiring code changes. Because jobs complete faster, customers reduce aggregate Compute Engine runtime hours and overall spend. Users enable it by specifying the Lightning Engine option when creating a cluster.

Easier with Flexible VMs

Flexible VMs improve resilience against capacity constraints.

›Temporary localized shortages of a machine type can stall cluster creation or interrupt autoscaling.
›Flexible VMs for Managed Spark clusters are now generally available.
›Users can define up to ten ranked machine types for master, primary, and secondary worker nodes.

The service pairs this preference with automated regional zone placement, scanning the region to fulfill capacity requests using the best available hardware layout, which reduces resource availability errors and helps capture cost-effective Spot VM capacity during peak demand.

FinOps cost controls

New features give better fiscal control over persistent environments.

›Zero-scale clusters and cluster scheduled stops are generally available.
›Zero-scale clusters let you provision environments that use exclusively secondary capacity.
›Both features were highly requested for development and persistent environments.

Smarter with embedded AI

The third pillar embeds AI into the Spark lifecycle.

›The update embeds AI directly into the development and operational lifecycle.
›Gemini-powered extensions are part of the managed clusters improvements.
›Smarter scaling policies accompany the native execution engine and Gemini extensions.

Frequently Asked Questions

What was Dataproc renamed to?

Dataproc is now called Managed Service for Apache Spark, reflecting its integration with the Agentic Data Cloud.

What is the Lightning Engine?

Lightning Engine is a native C++ vectorized execution engine built on Velox and Gluten that speeds up Spark DataFrame, Dataset, and SQL workloads by compiling query plans into native instructions optimized for SIMD vectorization.

How much faster is Lightning Engine?

It delivers up to 4.9x faster performance than standard open-source Spark and up to 2x the price-performance over the leading high-speed Spark alternative, with no code changes required.

What do Flexible VMs do?

Flexible VMs let users define up to ten ranked machine types per node role and pair that with automated regional zone placement to reduce resource availability errors and capture cost-effective Spot VM capacity.

What cost-control features were added?

Zero-scale clusters and cluster scheduled stops are now generally available, giving better fiscal control over persistent and development environments.

Google Cloud is positioning Managed Service for Apache Spark around faster execution, better resource resilience, and embedded AI.