UI-TARS-desktop

UI-TARS-desktop is ByteDance's open-source multimodal AI agent stack that sees your screen and operates your computer and browser like a human. It ships two projects: Agent TARS, a general multimodal agent stack with a CLI and Web UI, and UI-TARS Desktop, a native desktop app built on the UI-TARS model. Instead of driving through APIs, it looks at pixels and performs mouse and keyboard actions, clicking buttons, filling forms, dragging windows, and navigating. The repository was created January 19, 2025, and shows about 38.7K stars.

Key Features

• GUI agent by vision: The agent interprets screenshots and performs real mouse and keyboard actions, so it works with apps that have no API.
• Agent TARS CLI: One-command install with headful Web UI and headless server execution for multimodal task completion.
• Remote Computer and Browser Operators: v0.2.0 added Remote Computer Operator and Remote Browser Operator, both free, to control a computer or browser from anywhere.
• UI-TARS-1.5 model support: The desktop app supports the advanced UI-TARS-1.5 model for improved performance and precise control.
• MCP integration: Extends the agent with real-world tools through Model Context Protocol, including a Chrome DevTools MCP for DOM and network inspection.
• State-of-the-art benchmarks: UI-TARS-1.5 reports strong results across 10+ GUI benchmarks including OSWorld and AndroidWorld, above prior Claude and GPT-4o results.

Use Cases

Who Should Use This Tool?

• Automation builders: Automate repetitive form filling, UI testing, and multi-step desktop workflows.
• QA engineers: Drive browser and desktop apps for end-to-end testing with vision-based interaction.
• AI engineers: Embed GUI agent capabilities in products or use the CLI for human-like task completion.
• Researchers: Use the UI-TARS model stack to evaluate multimodal GUI agents.

Problems It Solves

1. Apps without APIs: When no API exists, the agent interacts with the UI directly like a person would.
2. Screen data privacy: Local operation means your screen data does not need to leave your machine.
3. Cross-app workflows: Multi-step workflows across different desktop applications become automatable.

Pricing

UI-TARS-desktop is open source under the Apache-2.0 license. The desktop app and Agent TARS CLI are free; v0.2.0's Remote Computer and Browser Operators are also free. You pay only for the model inference you choose to run, with local and remote model options.

Advantages & Unique Selling Points

1. True computer use: Operates via pixels and input events, not injected code, so it works on anything visible on screen.
2. Two products in one repo: Agent TARS for terminal/Web UI automation and UI-TARS Desktop for native GUI control.
3. Free remote operators: Remote computer and browser operators shipped without a paid tier.

Getting Started

Quick Start Guide

1. Download the latest release from GitHub releases, or build from source with git clone, npm install, npm run build.
2. Run the app and pick a model: run UI-TARS-1.5 locally or connect a remote operator.
3. Give the agent a task in natural language, such as "book the earliest flight from San Jose to New York."
4. Use the Agent TARS CLI or Web UI for headless and terminal-driven automation.
5. Extend with MCP servers for charts, DevTools, or other real-world tools.

Frequently Asked Questions

Does it need an API key?

Not necessarily. You can run a local model or connect a remote operator; the app itself is free and open source.

Is my screen data sent anywhere?

The project supports local operation, so screen data can stay on your machine. Remote operator mode transfers control sessions; review the docs for details.

What models does it support?

It supports the UI-TARS-1.5 model and multimodal LLMs that the stack is designed around, plus MCP tools for extension.

How is this different from browser-use?

browser-use is a Python library focused on web automation via a real browser. UI-TARS-desktop is a native desktop agent covering both computer and browser control with the UI-TARS multimodal stack.

Alternatives

• browser-use: open-source Python library for browser automation for AI agents.
• Playwright: browser automation framework for scripted web testing.
• Steel: browser infrastructure for AI agents and computer-use workloads.

Tips & Best Practices

1. Start on a known task: Verify the agent's pixel-based control on one workflow before trusting it with many.
2. Combine with MCP tools: Chrome DevTools MCP and chart MCP servers make the agent far more useful for inspection and output.
3. Watch remote sessions: Remote operator mode is powerful; limit access and review sessions for sensitive screens.

Conclusion

UI-TARS-desktop makes vision-based computer use practical and open source. If you need an agent that clicks, fills, and navigates real applications and browsers, and you want it local and free, it is one of the most capable stacks available today.