How Trustpilot built a real-time architecture for data enrichment using Gemma

Processing millions of user reviews in real-time, under strict latency and cost constraints, is no easy task. Trustpilot has been doing exactly that with custom machine learning since long before large language models (LLMs) were cool. Now, as the company transitions its core stack to generative AI, here is a look at how we teamed up to build a high-volume streaming pipeline using fine-tuned Gemma models.

Powering deep review intelligence at scale Trustpilot's core business relies on delivering deep, actionable review intelligence. As a platform championing transparency and genuine feedback, it must safeguard data integrity and maximize value. This means extracting every drop of metadata from incoming reviews - making LLMs the perfect tool for the job.

These models excel at parsing messy, human-written text to run named entity recognition (NER), categorize business domains, score sentiment, and pinpoint customer intent. But while prompting an LLM for a few reviews is easy, processing millions in real-time without blowing up costs is a massive engineering hurdle. When pursuing such a big task, why isn't just plugging into a powerful, off-the-shelf, frontier model like Gemini the right approach?

For a pipeline this critical to the core business, closed models are rarely the best option. Instead, by fine-tuning open-weight models like Gemma, Trustpilot takes full ownership of their AI strategy. Here's how: Total model independence: By owning its models, Trustpilot ensures it controls the retraining lifecycle, completely freeing it from a third-party vendor's update schedule or sudden API changes.

Predictable economics: Shifting from a variable per-token pricing model to fixed infrastructure costs makes running millions of predictions financially viable and optimizable. Expanding MLOps capabilities: Building these models in-house enables Trustpilot to bake in the "secret sauce" of its review intelligence while building competencies on open-weight models. Architectural continuity: Standardizing on an open-weight lineage preserves the company's ability to leverage the future iterations of the base model.

This enables performance gains with minimal engineering overhead. Rather than deploying one massive model, Trustpilot built a suite of highly specialized models using the lightweight google/gemma-2-9b as a base. To get heavy-weight performance from a small footprint, the company employed a consensus annotation over a stratified sample of the Trustpilot review corpus, using a selection of teacher models from the Gemini 2.0/2.5 Pro/Flash family.

This process generated high quality training datasets for specialized tasks like topic classification, NER, and sentiment extraction. The datasets were subsequently used to fine-tune a targeted lineup of custom models that considerably outperformed the legacy solution and delivered accuracy just a couple percentage points lower than the teacher models' consensus. System architecture This architecture was built on top of Dataflow and Gemini Enterprise Agent Platform Endpoints, which play together very nicely because of the out-of-the-box VertexAIModelHandlerJSON .

We decoupled business logic and raw LLM inference by creating two separate endpoints: The classifier: a FastAPI-based endpoint that handles the messy stuff, pre/post-processing, prompt templating, and chaining. The LLM: A separate Agent Platform endpoint dedicated strictly to serving the Gemma model via vLLM. This approach keeps the Dataflow job clean and ensures the LLM endpoint sticks to what it does best: generating text.