Join our daily and weekly newsletters for the latest updates and exclusive content on industry-leading AI coverage. Learn More
OpenAI has expanded the number of applications its desktop apps can work with, including allowing Advanced Voice Mode to work with other apps, and is moving closer to ChatGPT using computers.
The desktop app introduced integrations in November with an initial four applications. During Day 11 of its “12 Days of OpenAI” event, OpenAI announced several new integrated development environments (IDEs), terminals and text apps it will support.
ChatGPT now supports BBEdit, MatLab, Nova, Script Editor, TextMate as IDEs, VS Code for VSCode Insiders, VSCodium, Cursor, WindSurf, the JetBrains family of IDEs of Android Studio, AppCode, CLion, DataGrip, GoLand, IntelliJ IDEA, PHPStorm, PyCharm, RubyMine, RustRover and WebStorm. It also added the Warp and Prompt terminal apps as integration. These applications join VS Code, Xcode, Terminal, iTerm 2 and TextEdit as integrated apps.
But coding applications won’t be the only applications that ChatGPT desktop apps can access. OpenAI also added Apple Notes, Notion and Quip to its integrations. Advanced Voice Mode can work with these applications, considering the context of projects in the integrations.
OpenAI emphasized that users must give ChatGPT permission to access these applications.
Letting ChatGPT use your computer for you
App integrations with AI chatbots are, of course, nothing new. In October, GitHub Copilot added coding platform integrations. Connecting applications to ChatGPT or Copilot brings context from those platforms into the chat experience. Developers can prompt ChatGPT for some coding help for a project they have on VS Code, and the chatbot understands what they’ve been working on.
Kevin Weil, chief product officer at OpenAI, said during a live stream that improving the desktop app will help the company get closer to a more agentic user experience for ChatGPT.
“We’ve been putting a lot of effort into our desktop apps,” said Weil. “As our models get increasingly powerful, ChatGPT will more and more become agentic. That means we’ll go beyond just questions and answers; ChatGPT will begin doing things for you.”
He added that the desktop apps “are a big part” of that transformation.
“Being a desktop app you can do much more than you can than just a browser tab,” he said. “That includes things like, with your permission, being able to see what’s on your screen and being able to automate a lot of the work you’re doing on your desktop. We’ll have a lot more to say on that as we go into 2025.”
If OpenAI lets ChatGPT see more of your computer, ChatGPT will get closer to Anthropic’s Claude Computer Use feature, which allows Claude to click around a person’s computer, navigate screens and even type text.
OpenAI already announced a fairly similar feature for the mobile version of ChatGPT, although the chatbot cannot yet access computers or phones the same way. Users can share their screens with the chatbot so it can “see” what they’re reading or looking at. Microsoft and Google also developed comparable features with Copilot Vision and Project Astra.
How to access
On MacOS, users who want to open ChatGPT while using other applications can use “option + space” to pull up ChatGPT and choose the application they need through a button on the chat screen. Another shortcut, “option + shift + 1,” brings up the topmost application used.
From that window, users can also access Advanced Voice Mode in the same way. Voice mode automatically detects context from the application.
Integrations are available to ChatGPT Plus, Pro, Team, Enterprise and Edu users. However, Enterprise and Edu subscribers must ask their IT administrators to turn on the feature.
Source link