About Services Tools Blog Contact us

Desktop Computer-Use Control

Operates a Mac app that has no direct connection: edits its file if it can, then scripting, labelled controls or screen text, and checks each change with screenshots.

  • Browser & Desktop Automation
  • Cross-sector
  • Tool
  • Productised
  • Local/private
  • 2026

The problem

Some vendor tools can only be used by clicking through them. Clicks at fixed screen positions break when a window moves. A second automated helper can also grab the mouse at the same time and spoil the first one's work.

What it does

The run edits the file on disk when that file is the real source. Otherwise it uses the app's own scripting and finds named buttons and menus. If those are missing or ignore automated clicks, it reads the text on screen. A lock stops two helpers using the mouse at once, and each change is checked against a before and after image.

Outcomes

  • A labelled Save button is clicked by name, so a moved window still works.
  • Apps that hide their structure still give a combined list from the controls and the screenshot.
  • Controls that ignore automated clicks get a click that behaves like a person's.
  • Focus returns to the app the person was using.

Simplified mockup

Interactive preview with sample data. Nothing is sent to a server.

Drive Preferences

Cheapest route that can reach Save
1 File 2 Script 3 Accessibility 4 On-screen text

The preferences file is the real source, so the pointer stays still.

DeskWindowHelpCounter desk is waiting
Preferences
Screen lock is free. One agent at a time.
Ready. File route is the cheapest.

Named controls

From the tree
ControlRoleSource
SaveButtonTree
CancelButtonTree
Tab widthPopupTree
CloseButtonTree

The tree has no Continue label and no human-check. Those are only painted on the window.

Screenshot text not merged

Find on the screen

Save your work

Unsaved changes in Preferences.

Point

The crosshair sits on Continue. The offset walks it onto the checkbox, which ignores a synthetic click.

Crosshair on Continue

How it works

  1. Confirm the app is running and note which window is at the front.
  2. Edit the underlying file when the step does not need the window.
  3. Otherwise lock the screen to one helper and keep the display awake.
  4. Use the app's scripting if it has any, or its labelled controls to find a named button.
  5. If those are thin, read the on-screen text and click the label as a person would.
  6. Capture before and after, then bring back the app that was at the front.

Pipeline

flowchart LR
  A[Preflight] --> B[FileEdit]
  B --> C[Script]
  C --> D[AccessTree]
  D --> E[OnScreenText]
  E --> F[Click]
  F --> G[BeforeAfter]
  G --> H[Restore]