What's new for Managed Service for Apache Spark clusters

At Google Cloud, our goal is to let you run large-scale analytical and data science workloads with maximum efficiency so you can process big data pipelines, machine learning, and ETL tasks. We recently announced that the Dataproc service is now Managed Service for Apache Spark , reflecting our deep integration with the Agentic Data Cloud . To support the diverse architectural needs of today's modern data teams, we offer the service in two distinct deployment modes: serverless and managed clusters.

The serverless deployment mode completely abstracts infrastructure management for ephemeral or ad-hoc jobs, while the managed clusters deployment mode is designed for teams that require fine-grained infrastructure customization, persistent environments, long-running stateful processing, or native integration with custom Compute Engine hardware configurations. When it comes to managed cluster deployments, we've re-imagined the experience from the ground up, focusing on three core pillars: making Spark faster by supercharging execution speeds, easier to run by maximizing resource obtainability and reducing operational overhead, and smarter by embedding AI directly into the development and operational lifecycle. This blog post focuses specifically on what we announced at Google Cloud Next '26 for the Managed Spark clusters deployment mode: providing enhanced flexibility to fine-tune performance and cost through native execution engine, smarter scaling policies, and Gemini-powered extensions.

For the latest of the serverless deployment mode, check out this blog . Faster, with the Lightning Engine native execution engine Arguably the biggest update for Managed Spark clusters is Lightning Engine , which introduces massive performance gains for Spark DataFrame/Dataset APIs and heavy Spark SQL queries. Powered by a native, C++ vectorized execution engine built on Velox and Gluten, with specialized internal enhancements, Lightning Engine bypasses JVM execution bottlenecks by compiling query plans into native instructions optimized for SIMD (Single Instruction, Multiple Data) vectorization.

This native execution engine delivers: Up to 4.9x faster performance than standard open-source Spark up to 2x the price-performance over the leading high-speed Spark alternative Crucially, taking advantage of these performance gains doesn't require any code changes to your existing Spark applications. Because your jobs complete faster, you directly reduce your aggregate Compute Engine runtime hours and overall spend. To enable Lightning Engine on your managed clusters, simply specify the Lightning Engine option when you're creating a cluster.

The new way to use Spark: Intelligent, automated, and lightning fast Learn technical details and hear Lowe's experience with Lightning Engine Easier: Maximize resource obtainability via Flexible VMs Temporary localized shortages of a specific machine type can stall cluster creation or interrupt autoscaling. To dramatically improve cluster resilience against capacity constraints, Flexible VMs for Managed Spark clusters are now generally available. Flexible VMs allow you to define up to ten ranked machine types for your master, primary, and secondary worker nodes.

Managed Service for Apache Spark pairs this preference with automated regional zone placement, dynamically scanning the entire region to fulfill your capacity requests using the best available hardware layout. This helps ensure your pipelines spin up predictably, drastically reducing resource availability errors, and maximizing your ability to capture cost-effective Spot VM capacity during periods of peak demand. Easier: Zero-scale clusters and scheduled stops To give you better fiscal control over persistent and developmental environments, we recently announced the general availability of two highly requested FinOps features: zero-scale clusters and cluster scheduled stops .