Quick Overview
This product demonstration features Nick Baumann from Developer Experience at OpenAI introducing computer control features in ChatGPT Work. The video outlines how ChatGPT can interact with local desktop applications and web browsers alongside traditional plugins. It shows practical examples of automations run within the ChatGPT desktop app on macOS.
Key Points
- 1.Computer Use and Chrome Use allow ChatGPT to interact directly with desktop applications and websites inside the Work tab of the ChatGPT desktop app.
- 2.Computer Use enables ChatGPT to click, type, and navigate between desktop applications in the background on macOS.
- 3.Chrome Use lets ChatGPT operate within the Chrome browser to perform tasks across websites where the user is already logged in.
- 4.Pressing Command twice captures an App Shot, which acts as a context-aware screenshot so ChatGPT can see the active screen without detailed manual explanations.
- 5.ChatGPT pauses before finalizing critical actions, such as submitting a paid time off request or creating calendar events, allowing users to review and approve drafts first.
Summary
Nick Baumann from Developer Experience at OpenAI introduces how ChatGPT Work interacts with tools that lack native plugin integrations. While plugins offer direct API connections to various services, many essential workflows reside inside desktop software or websites requiring active user logins. Computer Use and Chrome Use allow ChatGPT to operate directly within these interfaces. These capabilities are hosted exclusively in the Work tab of the ChatGPT desktop application rather than the standard web or mobile interfaces.
Computer Use allows ChatGPT to navigate Mac applications by clicking buttons, typing text, and switching across windows in the background. Similarly, Chrome Use executes identical actions within Google Chrome, leveraging existing authenticated sessions. Before ChatGPT accesses an external application or webpage, it prompts the user for permission, which can be granted once or saved for future sessions. Users can issue instructions via voice dictation or text. To demonstrate Computer Use, Baumann directs ChatGPT to open the Spotify desktop application and start playing his daylist, prompting ChatGPT to launch the app and play the track Reckoner by Radiohead without a dedicated Spotify plugin.
To share visual context quickly, users can press Command twice to generate an App Shot. An App Shot functions as a screenshot accompanied by application context, enabling ChatGPT to inspect the visual interface of an app or website directly. In the demonstration, Baumann uses an App Shot of a webpage titled Work in Progress to instruct ChatGPT to inspect the content and find additional relevant posts without typing extensive descriptive prompts.
Baumann then demonstrates how Chrome Use pairs with native plugins to manage multi-step administrative tasks. He requests a paid time off submission in Workday for September 14 and 15, while also asking ChatGPT to check his calendar for conflicts and draft a corresponding out-of-office event. Because Workday lacks a plugin, ChatGPT employs Chrome Use to open Workday, navigate the interface, and fill in the absence request form. In parallel, it calls the Google Calendar plugin to check for schedule overlaps and draft the calendar event. In both cases, ChatGPT halts before final submission, staging the drafts so the user can verify details before confirming.
Bridging Gaps Beyond Plugins
While ChatGPT can integrate with many tools through dedicated plugins, certain workflows live in standalone desktop applications or authenticated websites without plugin support. Computer Use and Chrome Use address these gaps by giving ChatGPT the ability to click, type, and interact with software interfaces directly. These features are exclusive to the Work tab inside the ChatGPT desktop application and request user permission before accessing any app or site.
Desktop Control and App Shots
Computer Use enables ChatGPT to control Mac desktop applications in the background while the user continues other tasks. Users can monitor progress, intervene, or halt actions at any point. By pressing Command twice, users can capture an App Shot, which provides ChatGPT with an immediate, context-rich visual capture of the active desktop app or webpage without requiring lengthy text descriptions.
Combining Chrome Use with Plugins for Multi-Step Workflows
In a practical demonstration, ChatGPT processes an out-of-office schedule by pairing Chrome Use with standard plugins. Because Workday does not feature a plugin, ChatGPT uses Chrome Use to open the site and fill in a paid time off form, while simultaneously using the Google Calendar plugin to verify scheduling conflicts and stage an out-of-office event. ChatGPT stages both items as drafts and stops before final submission, ensuring human oversight over permanent changes.
The Bottom Line
The video establishes how Computer Use, Chrome Use, and plugins complement one another within ChatGPT Work to handle desktop apps and authenticated web tasks. It demonstrates that ChatGPT can stage complex actions across disparate tools in the background while enforcing human review prior to final submission. The demonstration leaves broader platform availability and specific configuration details outside macOS unaddressed.
FAQ
What is Computer Use and what are Chrome Use and plugins in ChatGPT Work?
Computer Use and Chrome Use are features in ChatGPT Work that allow ChatGPT to interact directly with desktop software and browser windows by clicking and typing. Plugins provide direct integrations with specific tools, while Computer Use controls local desktop apps and Chrome Use interacts with websites where the user is already logged in.
Where can users access Computer Use and Chrome Use in ChatGPT?
Computer Use and Chrome Use are available only in the Work tab within the ChatGPT desktop application, and not in regular ChatGPT on the web or mobile devices.
How does the Command shortcut create an App Shot in the ChatGPT desktop application?
Pressing Command twice captures an App Shot, which acts as a screenshot with context that allows ChatGPT to inspect and understand what is currently displayed on the user's screen.
How does ChatGPT handle security permissions when accessing desktop applications or websites?
Before opening or interacting with an application or website, ChatGPT asks the user for permission, which can be granted for a single instance or remembered for future sessions.
Why does ChatGPT stop before submitting forms or creating calendar events during automated workflows?
ChatGPT stages the actions and stops before submitting requests or creating events so the user has the final decision to review and approve real changes that affect systems, calendars, or team schedules.
Worth watching for
Developers and knowledge workers looking to automate complex desktop and browser-based workflows using the ChatGPT desktop application.
- chatgpt
- chatgpt-work
- computer-use
- chrome-use
- automation
- openai