Quick Overview
In this build-along project video, creator Matt Wolfe attempts to engineer an automated detection tool to identify AI-generated short-form video content on social media. The experiment follows his process of architecting, coding, and testing the application using automated AI development tools and computer vision APIs.
Key Points
- 1.Matt Wolfe attempted to build a web application called AI Slop Detector to automatically identify AI-generated short-form videos from social media links.
- 2.Initial automated detection pipelines built with OpenAI models and Google Gemini video analysis failed to reliably detect obvious synthetic video content.
- 3.Integrating the third-party Sightengine computer vision API improved detection accuracy across test clips from Instagram and X.
- 4.Disagreements between detection engines, such as Sightengine flagging AI markers while Gemini reported no visual indicators, resulted in inconclusive verdicts.
- 5.Hosting a public detection tool proved financially unfeasible due to high per-scan API consumption costs on platforms like Sightengine.
- 6.The complete source code for the local detection app was released on GitHub for users to run with their own API keys.
Summary
Matt Wolfe addresses the pervasive issue of AI-generated video content circulating across platforms like Instagram, TikTok, YouTube, and X. To help users determine whether a shared video is authentic or synthetic, he sets out to create a web tool called the AI Slop Detector. He starts by outlining his project requirements in ChatGPT, planning a workflow that inspects video frames using Google Gemini agentic video understanding models, reviews comments for community claims, and checks digital provenance credentials.
Wolfe exports the chat architecture to OpenAI Codex and generates the initial codebase. When testing the initial build with an Instagram reel featuring a clearly synthetic inflatable water jetpack called FartZooka, the detector fails completely. The app categorizes the video as authentic live-action footage with traditional visual effects, assigning a ninety-four percent confidence rating to a ten percent AI likelihood score.
To solve the detection issues, Wolfe prompts GPT-6 Astra Ultra to autonomously debug the codebase and refine the detection algorithms. The model runs for over eight hours, executing automated tests across sets of real and synthetic video clips. The evaluation reveals persistent recognition gaps, prompting the pipeline to incorporate Sightengine, a specialized third-party computer vision API designed for content moderation and synthetic media identification.
After configuring the Sightengine API within the local environment variables, the system begins scanning video frames at half-second intervals. When retesting the FartZooka clip, Sightengine detects repeated synthetic indicators across nineteen sampled frames. However, because Gemini finds no visual anomalies in the same footage, the application produces an inconclusive result due to provider disagreement.
Wolfe requests a minimalist redesign of the user interface and tests additional clips, including an authentic video from creator Tiffany Janzen and an AI-generated clip of an airplane embedded in a lawn. The refined app correctly identifies the authentic video with zero flags and flags the synthetic airplane video. Despite the working prototype, testing only a few videos consumes over twelve thousand Sightengine API operations. Wolfe concludes that hosting the web app publicly would be financially unfeasible, opting instead to publish the source code on GitHub for local execution.
Conceptualizing the AI Slop Detector
Matt Wolfe begins by addressing the proliferation of synthetic, AI-generated video clips on social media and the difficulty many viewers face in identifying them. He plans a web app that accepts video URLs from TikTok, Instagram, YouTube, and X, evaluating whether the footage is authentic or generated by artificial intelligence. Using ChatGPT and Codex, he drafts the architecture using Google Gemini agentic video understanding models.
Testing Early Prototypes and AI Failures
After generating the initial application code, Wolfe tests an obvious AI video featuring an inflatable jetpack suit named FartZooka. The prototype incorrectly rates the video as authentic live-action footage with traditional visual effects, giving it a high confidence score of ten percent likelihood of being AI. Wolfe tasks OpenAI models with autonomous debugging sessions to refine the scoring logic and detection criteria.
Integrating the Sightengine API
Because general multimodal models struggle to recognize synthetic video artifacts, the system switches to integrating Sightengine, a specialized computer vision and content moderation API. Sightengine successfully samples frames and detects synthetic patterns, but conflicts between Sightengine and Gemini assessments often cause the application to return inconclusive verdicts on test clips.
Redesign, API Costs, and Public Release
Wolfe redesigns the user interface into a clean minimalist layout and tests both synthetic clips and genuine talking-head footage. While the Sightengine integration accurately flags synthetic footage and clears authentic clips, running just a handful of tests consumes over twelve thousand API operations. Because open public hosting would be cost-prohibitive, Wolfe uploads the code to GitHub for local use.
The Bottom Line
The video establishes that current frontier multimodal language models struggle to reliably detect AI-generated video, requiring dedicated computer vision tools like Sightengine to spot synthetic patterns. While a functional hybrid detection interface can be built, significant disagreements between evaluation models frequently yield inconclusive results. The project ultimately resolves as an open-source local application, leaving open public deployment unsolved due to prohibitive API operating costs.
FAQ
What is AI Slop Detector and what does it do?
AI Slop Detector is a web application created by Matt Wolfe designed to scan short-form video URLs from social media platforms and determine whether the footage is AI-generated.
Why did the initial Google Gemini video analysis fail on synthetic clips?
Google Gemini failed to identify clear visual indicators of generative AI in obvious synthetic clips, frequently misinterpreting AI artifacts as conventional visual effects or authentic camera footage.
What role does the Sightengine API play in the detection pipeline?
Sightengine acts as a specialist computer vision detector that samples video frames at regular intervals to flag visual patterns characteristic of AI image generation.
Why did the detector return inconclusive results on certain AI videos?
The application returned inconclusive verdicts when Sightengine detected synthetic indicators across multiple frames but Google Gemini reported no clear visual anomalies in the same clip.
Why was the AI Slop Detector app not released as a public website?
A public release was deemed financially unfeasible because each video scan consumed hundreds of operations on the paid Sightengine API, quickly depleting monthly quotas.
Worth watching for
Developers and tech enthusiasts interested in AI content moderation, video forensics, and the limitations of current multimodal AI models in detecting synthetic media.
- ai-detection
- sightengine
- gemini
- synthetic-video
- codex
- deepfakes