Control Your Computer
Your employees can control your desktop. Move the mouse, type on the keyboard, manage files, and run terminal commands.
TL;DR
Employees can control your desktop to click, type, manage files and run terminal commands. They can automate anything you'd do yourself sitting at the computer. Requires the Desktop Companion app (the same app that powers Browser Control).
One app, two capabilities. Download the Desktop Companion once and your employees get both Computer Control and Browser Control. You can enable or disable each one independently per employee.
How It Works
You describe a desktop task. The employee takes a screenshot to see your screen, plans the actions, and executes them: mouse clicks, keyboard input, file operations, or terminal commands. Results come back as screenshots and text.
| Step | What Happens |
|---|---|
| 1. You instruct | "Open the monthly-report.xlsx file and update the revenue row" |
| 2. Employee screenshots | Takes a screenshot to see your current screen |
| 3. Plans actions | Reasons about what to click, type, or run |
| 4. Executes | Sends mouse/keyboard/file/terminal commands via the companion app |
| 5. Verifies | Takes another screenshot to confirm the result |
What It Can Do
Computer Controller has three action categories:
Desktop Control
| Action | Example |
|---|---|
| Click | Click on buttons, menus, links, anywhere on screen |
| Type | Enter text into any application |
| Scroll | Scroll through pages, documents, feeds |
| Keyboard shortcuts | Cmd+S, Ctrl+C, Alt+Tab, any key combination |
| Take screenshots | Capture what's on screen at any moment |
File Management
| Action | Example |
|---|---|
| List files | "Show me what's in my Downloads folder" |
| Read files | "Read the config.yaml on my desktop" |
| Write files | "Create a new file called notes.txt with these bullet points" |
| Search files | "Find all .csv files in my Documents folder" |
Terminal Commands
| Action | Example |
|---|---|
| Run commands | "Run npm install in the project directory" |
| Check status | "Run docker ps and tell me which containers are running" |
| Process data | "Run this Python script and show me the output" |
How to Set It Up
Computer Control is one of two capabilities that come with the Desktop Companion app. Download it once and you get both Computer Control and Browser Control.
- Download the Desktop Companion app from the employee's Tools tab
- Install and run it on your computer
- Grant permissions when prompted. The app shows two switches: Screen Recording and Keyboard & Mouse. Click the Enable button next to each. macOS opens System Settings so you can flip them on
- Quit and reopen the app once so the freshly granted permissions take effect (macOS caches the previous denial until a full relaunch)
- Click Connect. A browser tab opens, pairs your account, and closes. The app status turns green
- Done. The Tools tab in the webapp shows Ready
Already have the companion app installed for Browser Control? You're all set. Both capabilities share the same connection.
First Task, First Time
The first time you ask an employee to click, type, take a screenshot, or run a command, they check if the companion is ready. If it is not, they point you back here, share a download link, and wait. Once the app shows Connected and the tool row says Ready, ask again and the work starts immediately. There is no need to repeat the instruction in different words, the employee is simply waiting on the app.
Once connected, you can manage the tool per employee:
- Select the employee
- Click the Tools tab
- Find "Computer Controller" in the Actions section
| Action | What it does | How |
|---|---|---|
| Enable / Disable | Controls whether the employee can use this tool | Toggle the switch |
| Tool Rules | Custom instructions that guide how the employee uses this specific tool, e.g. "never run commands outside the project directory" or "always confirm before deleting files" | Expand the tool, then write your rules in the text field |
| Delete | Permanently removes the tool from the employee | Click the delete button |
Tips & Tricks
- Describe what you see. "Click the blue 'Export' button in the top-right corner" helps the employee locate elements faster
- Pre-open applications. If you want the employee to work in Excel, open it first so it's visible on screen
- Use terminal for precision. File operations via terminal commands are more reliable than GUI clicks for batch work
- Check screenshots. The employee takes screenshots to verify actions. Review them if the result looks unexpected
Behind the Scene
| Powered by | Desktop Companion app (native OS integration) |
| How it works | The employee takes screenshots to see your screen, reasons about what to do, then sends mouse/keyboard/file/terminal commands to the companion app |
| Vision-based | The employee "sees" your screen via screenshots and reasons about what to do. Ambiguous interfaces may need clarification |
| Platforms | macOS, Windows, Linux |
Computer Controller vs Browser Controller
| Computer Controller | Browser Controller | |
|---|---|---|
| Scope | Full desktop: any app, files, terminal | Web browser only |
| Input method | Mouse, keyboard, file ops, terminal commands | DOM-aware browser navigation |
| Best for | Native apps, files, terminal, multi-app workflows | Web-specific tasks: forms, navigation, data extraction |
| Precision | Coordinate-based (click at x,y) | Element-based (click the "Submit" button) |
| When to use | Desktop apps, file management, terminal tasks | Websites, web apps, online dashboards |
| Requires | Desktop Companion app | Desktop Companion app |
Both capabilities come from the same Desktop Companion app. One download, one connection. The employee chooses the right one based on the task. Browser Control for web work, Computer Control for everything else. You can enable or disable each one independently per employee.
What It Costs
| Cost | Runtime credits based on processing time |
Is It Safe
Computer Controller has built-in safety restrictions:
| Category | Blocked |
|---|---|
| Terminal | rm -rf /, mkfs, dd, shutdown, reboot, sudo, su, fork bombs |
| Files | Access outside home directory, files > 10 MB, symlink traversal |
| Credentials | Employee will never type passwords, SSH keys, or API tokens |
The employee operates within your home directory only and cannot escalate privileges. Dangerous commands are blocked before execution.
Good to Know
- Desktop Companion required. Computer Control only works when the companion app is installed and running. Without it, the tool is unavailable. The same app also provides Browser Control
- Home directory only. File operations are restricted to your home directory. System files, application internals, and root-level paths are inaccessible
- No privilege escalation.
sudo,su, and similar commands are blocked. The employee operates with your user-level permissions only - Screenshots saved to Drive. Every screenshot is persisted to the employee's Drive tab for your records
- Terminal output limits. Commands that produce more than 100 KB of output are truncated. Long-running commands timeout after 120 seconds
Frequently Asked Questions
Q: Can the employee install software on my computer?
A: They can run terminal commands like npm install or pip install within your user permissions, but cannot use sudo for system-level installs.
Q: Is it safe to let employees control my computer? A: Yes. Dangerous commands are blocked, file access is restricted to your home directory, and the employee never handles passwords or credentials. You can also enable approval mode to require sign-off before each action.
Q: Can the employee work while I'm using the computer? A: Computer Controller takes over mouse and keyboard input, so it works best when you're not actively using the computer. Consider letting it run while you're on a break or working on a different device.
Q: What operating systems are supported? A: macOS, Windows, and Linux, anywhere the Desktop Companion app runs.
Q: Can the employee access my browser through Computer Controller? A: Technically yes (by clicking on the browser window), but Browser Controller is much better for web tasks. It's DOM-aware and more precise. Computer Controller is for everything that isn't a browser.
Q: I granted permissions but the employee still says it can't see my screen. Why? A: macOS caches the previous denial. Quit the companion app fully and reopen it. The first action after reopening picks up the new permission state. This is a one-time thing, after that the permission sticks across restarts.
Q: What permissions does the app actually need? A: Only two, and only when the employee needs to use your computer or browser: Screen Recording so they can see what is on screen, and Keyboard & Mouse (Accessibility on macOS) so they can click and type. Both are grantable from the app's Permissions section with one click each. Nothing else is requested.