About Services Tools Blog Contact us

Phone App Control

A helper follows a written step list on an iPhone, reads the screen between taps and stops if the screen is not the one expected.

  • Browser & Desktop Automation
  • Cross-sector
  • Agent skill
  • Productised
  • Local/private
  • 2026

The problem

Some jobs exist only inside a phone app, with no desktop version and no export. Someone had to tap through them by hand. A wrong screen was easy to miss, and the next tap went ahead anyway.

What it does

The helper is pointed at the phone and given a written list of taps. It reads the screen between steps and stops with a report if the screen does not match the step. So far this has been built and tested, and has not been run as a finished job on a real handset.

Outcomes

  • A phone-only task can be repeated from a written list.
  • A mismatch stops the run before the next tap.
  • The stop report shows what was on the screen.

Simplified mockup

Interactive preview with sample data. Nothing is sent to a server.

Handover on the handset

Handover

Ready

3 collections are waiting. The agent has not tapped yet.

4418 Jordan Hale · UK 2 · Bay 2
4420 Nia Cole · UK 13 · Bay 2
4426 Sam Reed · UK 1 · Bay 4
1. Open collections
Expect: Collections, 3 waiting
2. Open order 4418
Expect: Order 4418, Jordan Hale, UK 2
3. Confirm handover
Expect: Collected, receipt sent
List loaded, not started

Expected screen for each step

Step 1 · Open collections
Title must be Collections. The body lists three waiting orders, including 4418.
Step 2 · Open order 4418
Title must be Order 4418. The name on screen must be Jordan Hale, size UK 2, bay 2. A sign-in wall is a mismatch.
Step 3 · Confirm handover
Title must be Collected. The receipt line names Jordan Hale. This step is not tapped if step 2 failed.

Earlier stop reports

WhenStoppedScreen readingNext tap
Yesterday 16:40 Step 2 No network. Order 4402 did not open. Not tapped
Tuesday 11:05 Step 1 Wrong shop. Bay list was for the other site. Not tapped
Monday 09:20 Finished Collected. Receipt sent for order 4388. All three

Each row is a canned reading from a previous list. The live run is on the Run tab.

How it works

  1. Load the step list and the expected screen for each step.
  2. Read the phone screen before the first tap.
  3. Perform the tap or the key entry.
  4. Read the screen again and compare it with the expected step.
  5. Stop and show what was read if the screen does not match.

Pipeline

flowchart LR
  A[Steps] --> B[ReadScreen]
  B --> C[Tap]
  C --> D[Check]
  D --> E[Next]
  D --> F[StopReport]