Mob.Test (mob v0.7.39)

Copy Markdown View Source

Remote inspection and interaction helpers for connected Mob apps.

All functions accept a node atom and operate on the running screen via Erlang distribution. Connect first with mix mob.connect, then use these from IEx or from an agent via :rpc.call/4.

Quick reference

node = :"my_app_ios@127.0.0.1"

# Inspection
Mob.Test.screen(node)               #=> MyApp.HomeScreen
Mob.Test.assigns(node)              #=> %{count: 3, ...}
Mob.Test.tree(node)                 #=> %{type: :column, ...}
Mob.Test.find(node, "Save")         #=> [{[0, 2], %{...}}]
Mob.Test.inspect(node)              #=> %{screen: ..., assigns: ..., tree: ...}

# Interaction
Mob.Test.tap(node, :increment)      # tap a button by tag
Mob.Test.back(node)                 # system back gesture
Mob.Test.pop(node)                  # pop to previous screen (synchronous)
Mob.Test.navigate(node, MyApp.DetailScreen, %{id: 42})
Mob.Test.pop_to(node, MyApp.HomeScreen)
Mob.Test.pop_to_root(node)

# Lists
Mob.Test.select(node, :my_list, 0)  # select first row

# Visual capture + scroll (in-process, over dist — no adb/xcrun)
{:ok, png} = Mob.Test.screenshot(node)
Mob.Test.scroll_info(node, "feed")            # offset/content/viewport
Mob.Test.scroll_to(node, "feed", :bottom)
Mob.Test.screenshot_tour(node, "feed")        # page top→bottom, capture each

# Element positions without a screenshot (elements need an :id)
Mob.Test.element_frames(node)                 # %{id => {x, y, w, h}}
Mob.Test.frame(node, "save")                  # {x, y, w, h}
Mob.Test.tap_id(node, "save")                 # drive by id at real coords

# What colour did the app actually draw? (samples pixels — the view tree can't)
Mob.Test.sample_color(node, "my-card")        # %{average: 0xFF2196F3, ...}

# Device API simulation
Mob.Test.send_message(node, {:permission, :camera, :granted})
Mob.Test.send_message(node, {:camera, :photo, %{path: "/tmp/photo.jpg", width: 1920, height: 1080}})
Mob.Test.send_message(node, {:location, %{lat: 43.65, lon: -79.38, accuracy: 10.0, altitude: 80.0}})
Mob.Test.send_message(node, {:notification, %{id: "n1", title: "Hi", body: "Hey", data: %{}, source: :push}})

Tap vs send_message

tap/2 sends the same {:tap, tag} message a native tap produces, so it arrives in the screen's handle_info/2 exactly like a real button press. send_message/2 delivers any term to handle_info/2. Use send_message/2 to simulate async results from device APIs (camera, location, notifications, etc.) without having to trigger the actual hardware.

Synchronous vs fire-and-forget

Navigation functions (pop, navigate, pop_to, pop_to_root) are synchronous — they block until the navigation and re-render complete. This makes them safe to follow immediately with screen/1 or assigns/1 to verify the result.

back/1, tap/2 and send_message/2 are fire-and-forget (they send a message to the screen process and return immediately). Use settle/2 as a sync point if you need to wait before reading state:

Mob.Test.send_message(node, {:permission, :camera, :granted})
Mob.Test.settle(node)
Mob.Test.assigns(node)

:sys.get_state/1 on :mob_screen is no longer sufficient on its own: since MOB-110 the tree is handed to Mob.Sender and committed asynchronously, and since MOB-112 :mob_screen is the navigation owner rather than the screen itself. That only matters for the functions that read the native side — view_tree/1, screenshot/2, tap_id/2, element_frames/2. tree/1 and assigns/1 re-render in-process and are unaffected.

Two layers of inspection: render tree vs native UI

Mob.Test exposes two complementary views of what the app is showing:

APISourceWhen to use
tree/1, find/2Mob render tree (logical components)Mob apps you control. Fast, exact, has on_tap tags, no AX activation needed.
view_tree/1, find_view/2Native view hierarchy via NIFNative pixel frames and painted colours; works for any app on iOS UIKit; shallow on SwiftUI/Compose.
ui_tree/1OS accessibility treeWhat sighted users read; works on any app if AX is active (iOS: VoiceOver). Strict superset of view_tree for UIKit; the only path to semantics inside SwiftUI/Compose.

Choose render tree first if your app is Mob-rendered. Reach for view_tree when you want native frames or geometry. Reach for ui_tree when you need to inspect non-Mob content (alerts, system overlays, third-party SDK UI), or to verify the rendered state matches the logical render.

Driving controls beyond plain taps

  • Buttons / nav itemstap/2 (by tag, fastest), or mob_nif:tap/1 (by accessibility label), or tap_xy/3 (by coordinate).
  • Sliders, steppers, pickersadjust_slider/4 and the underlying ax_action/3 / ax_action_at_xy/4 use accessibilityIncrement / accessibilityDecrement. Synthetic drag gestures don't fire SwiftUI's DragGesture reliably; AX actions do.
  • Switches / togglestoggle/2 finds the switch by nearby label and activates it via the AX path (sends accessibilityActivate).
  • Modals / alerts / sheetsdismiss_alert/2 uses accessibilityActivate on the named button; ax_action/3 with :escape sends accessibilityPerformEscape.
  • Scroll viewsax_action/3 with :scroll_up/:scroll_down/ :scroll_left/:scroll_right sends accessibilityScroll:.
  • System backback/1 (Mob screens, framework-level) or — for sidecar mode against arbitrary apps — synthetic edge-pan via swipe/5 from x=0, but iOS owns that gesture above the app process and the synthetic pan won't fire. Use back/1 for Mob, document the limitation for sidecar.

Platform support matrix

HelperiOS simiOS deviceAndroid
screen/1, assigns/1
tap/2 (by tag)
back/1, pop/1, navigate
send_message/2
screen_info/1
view_tree/1✅ (shallow†)✅ (shallow†)❌ not_loaded‡
sample_color/2❌ not_loaded°
find_view/2❌ not_loaded‡
ui_tree/1 (legacy AX)⚠️ AX active§⚠️ AX active§❌ not_loaded
ax_action/3⚠️ AX active§⚠️ AX active§❌ not_supported
ax_action_at_xy/4⚠️ AX active§⚠️ AX active§❌ not_supported
toggle/2⚠️ AX active§⚠️ AX active§❌ ui_tree_unavailable
dismiss_alert/2⚠️ AX active§⚠️ AX active§❌ ui_tree_unavailable
adjust_slider/4⚠️ AX active§⚠️ AX active§❌ ui_tree_unavailable
tap_xy/3⚠️ AX-activatable only¶❌ no_effect¶n/a
swipe/5⚠️ scroll only⚠️ unverified✱n/a
  • SwiftUI doesn't expose its content as separate UIView instances — view_tree reaches the SwiftUI hosting view's container and stops. For semantic content on Mob screens use tree/1 (render tree); for any other SwiftUI-based content use ui_tree/1.
  • Android's ui_view_tree NIF delegates to a MobBridge.uiViewTree() Kotlin method that no shipped template implements, so it returns {:error, :not_loaded}. When it lands it must emit the same keys iOS does (including bg_color/text_color); the contract is documented at the NIF in android/jni/mob_nif.zig. The Mob renderer is Compose, so the View walk would stop at the AndroidComposeView host anyway — the real fix is Modifier.onGloballyPositioned in Mob's components writing to a registry the NIF reads. See issues.md #11.
  • ° sample_region/4 is implemented in ios/mob_nif.m only. Android would need the same crop-in-the-render treatment against the activity window; until then sample_color/2 returns {:error, {:badrpc, _}} there.
  • § "AX active" means an iOS accessibility client is asking for the AX tree so SwiftUI materializes it. Today: VoiceOver toggle. Production: XCAXClient_iOS activation, debug-only — see WireTap stretch goals in future_developments.md.
  • tap_xy/3 now verifies that the tap actually produced an event before returning :ok. On the simulator that limits it to elements SwiftUI exposes an accessibility action for (Button, text fields) — a Box with on_tap: returns {:error, :no_effect}. On a physical device the IOHID-injected touch is accepted but never delivered, so every coordinate returns {:error, :no_effect}. Drive taps with tap/2 (by tag); see tap_xy/3 and decisions/2026-08-09-ios-device-tap-injection-has-no-effect.md.
  • swipe/5 and long_press/4 use the same device injection path as tap_xy/3 and still report :ok on acceptance rather than on effect. Same root cause, not yet converted — treat their :ok as unverified.

Helpers that depend on AX return clear error tuples on Android instead of raising. Callers should match on {:error, :not_supported_on_android} and {:error, :ui_tree_unavailable} and either skip or fall back to send_message/2 for state mutations.

Known limitations affecting AX automation

Even on iOS with AX active, three Mob component defects keep the natural paths from working today. Workarounds in each helper's docstring:

  • SlideraccessibilityIncrement/Decrement are no-ops because Mob's iOS Slider doesn't attach .accessibilityAdjustableAction. See issues.md #7.
  • Toggle — the label: prop doesn't reach the AX tree; toggle/2 can't find the switch by label name. Use ax_action_at_xy/4 with coordinates for now. See issues.md #8.
  • Alert OK buttonaccessibilityActivate on the AX-tree button doesn't fire the underlying UIAlertAction. Use Mob Alert with action: atoms and send_message/2 to dismiss programmatically. See issues.md #9.

System-level gestures iOS owns above the app process (edge-pan back, swipe-up app switcher, pull-down notification center) are out of reach for in-process synthetic touches on physical devices. Use back/1 for Mob screens; for sidecar mode against arbitrary apps, document the limitation rather than promising the gesture.

Summary

Functions

Step a slider toward a target percentage (0.0..1.0) using accessibility increment/decrement actions. Reliable when synthetic-drag won't fire (SwiftUI Slider's DragGesture ignores in-process touches on iOS).

Return the current screen's assigns map, or nil while that screen is being restarted after a crash (MOB-112 — the socket lives in the screen's own process, which is briefly absent).

Invoke an accessibility action on the first AX element matching match.

Invoke an AX action on whatever element occupies the given screen coordinates.

Simulate the system back gesture (Android hardware back / iOS edge-pan).

Clear all text in the focused input (select-all + delete).

Tally of the distinct painted colours in a view tree — the cheap way to assert a styling change actually reached the screen.

Delete one character behind the cursor (backspace).

Dismiss a modal/alert overlay by tapping its first button labelled with button_label (e.g. "OK", "Cancel"). Mirrors what a user does when an alert pops up.

Return the on-screen frame of every rendered element that carries an :id, as %{id => {x, y, w, h}} in logical units (points on iOS, dp on Android).

Find all nodes in the current tree whose text contains substring. Returns a list of {path, node} tuples where path is a list of indices from the root.

Find elements in the native accessibility tree whose label or value contains text.

Find nodes in the view tree whose label or value contains text.

Flatten an already-fetched view tree. Pure function — useful for tests and for inspecting a captured tree without re-fetching.

Frame {x, y, w, h} of the element with id, or nil if it has no tracked position. See element_frames/1.

Return a map with :screen, :assigns, :nav_history, and :tree (the raw render tree from calling render/1 on the current screen).

Press a special key on the focused text input.

Locate an element by visible label text or accessibility ID (tag atom name). Returns the element's screen frame.

Long-press at screen coordinates for duration_ms milliseconds (default 800ms).

Push a new screen onto the navigation stack. Synchronous.

Normalize an Android-shaped (JSON-decoded, string-keyed) view tree into the iOS map shape: atom keys, atom :type, {x, y, w, h} frame tuple.

Census of where colour lives in the native view tree — the diagnostic to reach for when view_tree/1 reports nil colours and you need to know why.

Pop the current screen and return to the previous one. Synchronous.

Pop the stack until dest is at the top. Synchronous.

Pop all screens back to the root of the current stack. Synchronous.

Reduce a raw RGBA buffer to colour statistics. Pure — no device needed.

Replace the current navigation stack with a new root screen. Synchronous.

What colour did the app actually draw in a region? Samples real pixels.

Return the current screen module.

Return screen geometry in logical units (points on iOS, dp on Android).

Capture the running app's own window in-process and return the image bytes.

Walk a scroll view top→bottom, capturing a screenshot at each page. Returns a list of {offset, image_binary} pairs — the agent's "see the whole long screen" path, entirely over dist.

Read a scroll view's current offset and extent, addressed by its :id prop (the same :id you set on a type: :scroll or type: :list node).

Scroll a view (by :id) to a target position. Reads scroll_info/2 first to resolve and clamp the absolute offset, then drives the native scroll view.

Select a row in a :list component by index.

Send an arbitrary message to the screen's handle_info/2. Fire-and-forget.

Block until the app has finished processing and the current frame is on screen.

Swipe from (x1, y1) to (x2, y2). Drives UIScrollView contentOffset on simulator; synthesises a drag gesture on real device.

Switch to a named tab stack. Synchronous.

Send a tap event to the current screen by tag atom.

Tap the element with id at the center of its tracked frame — driving by id without a screenshot or coordinate guess. The element must carry an :id (see element_frames/1).

Locate an element and tap it via the simulator's native UI mechanism.

Tap at screen coordinates on the native app.

Toggle a switch by a label substring. SwiftUI exposes Toggle as a button with an empty accessibility label and value "0" or "1" — so we find the Text element matching label_match, then activate the next button below it.

Return the current rendered tree (calls render/1 on the live assigns).

Type text into the currently focused text field.

Return the live accessibility tree from the running native app.

Return the live UI tree as a nested map, walking native views directly.

Return the view tree flattened to a list of {path, node} tuples.

Wait until predicate returns true when called with the current ui_tree, polling every interval_ms until timeout_ms elapses.

Wait until an element whose label or value contains text appears in the accessibility tree.

Functions

adjust_slider(node, match, target, opts \\ [])

@spec adjust_slider(node(), String.t(), float(), keyword()) ::
  {:ok, float()} | {:error, term()}

Step a slider toward a target percentage (0.0..1.0) using accessibility increment/decrement actions. Reliable when synthetic-drag won't fire (SwiftUI Slider's DragGesture ignores in-process touches on iOS).

match is a substring of the slider's label or value (e.g. "Volume"). target is a fraction 0.0..1.0. max_steps caps the increment loop (default 30) so a wrong match can't spin forever.

Returns {:ok, final_pct} or {:error, reason}.

Mob.Test.adjust_slider(node, "Volume", 0.30)
#=> {:ok, 0.30}

Implementation note: each AX increment/decrement on a SwiftUI slider moves by the slider's .step value (default 0.10 of the range). The function re-reads the slider value after each step to converge.

Known limitation (issues.md #7)

Mob's iOS Slider component does not currently attach .accessibilityAdjustableAction { … }, so accessibilityIncrement and accessibilityDecrement are silently dropped by SwiftUI even though the NIF returns :ok. This helper currently returns {:error, :max_steps_exhausted} against an unfixed slider. Until issue #7 lands, drive sliders via Mob.Test.send_message(node, {:change, :slider_tag, value}).

assigns(node)

@spec assigns(node()) :: map() | nil

Return the current screen's assigns map, or nil while that screen is being restarted after a crash (MOB-112 — the socket lives in the screen's own process, which is briefly absent).

ax_action(node, match, action)

@spec ax_action(node(), String.t(), atom()) :: :ok | {:error, atom()}

Invoke an accessibility action on the first AX element matching match.

Platform support

  • iOS: works once AX is active (today: VoiceOver on; future: XCAXClient_iOS activation, see future_developments.md).
  • Android: returns {:error, :not_supported_on_android}. The Compose semantics walker is queued under WireTap (issues.md #11).

Used for controls where synthetic touches don't reach the gesture recognizer (sliders, scrolls, modal dismissal).

match is a string searched in both label and value. action is one of: :increment, :decrement, :activate, :escape, :scroll_up, :scroll_down, :scroll_left, :scroll_right.

Mob.Test.ax_action(node, "Volume", :decrement)
Mob.Test.ax_action(node, "Cancel", :activate)

ax_action_at_xy(node, x, y, action)

@spec ax_action_at_xy(node(), number(), number(), atom()) :: :ok | {:error, atom()}

Invoke an AX action on whatever element occupies the given screen coordinates.

Useful when label/value substring matching is ambiguous (e.g. multiple sliders that all read "50%", a toggle whose accessibility label is empty). Caller picks coordinates from ui_tree/1 and points at the exact element.

Mob.Test.ax_action_at_xy(node, 187.0, 296.0, :increment)

Platform support

  • iOS: works once AX is active (VoiceOver on, today).
  • Android: returns {:error, :not_supported_on_android} — see ax_action/3.

back(node)

@spec back(node()) :: :ok

Simulate the system back gesture (Android hardware back / iOS edge-pan).

Fire-and-forget — follow with settle/2 before reading the native side. The framework pops the navigation stack; if already at the root, it exits the app. Prefer pop/1 when you need to know that navigation has finished before reading state.

clear_text(node)

@spec clear_text(node()) :: :ok | {:error, atom()}

Clear all text in the focused input (select-all + delete).

color_census(node)

@spec color_census(node() | map()) :: %{background: map(), text: map()}

Tally of the distinct painted colours in a view tree — the cheap way to assert a styling change actually reached the screen.

Pass a node to fetch the tree, or an already-fetched tree to work offline. Returns %{background: %{argb => count}, text: %{argb => count}}, nil colours excluded.

Mob.Test.color_census(node)
#=> %{background: %{0xFF2196F3 => 4, 0xFF1E1E1E => 1}, text: %{0xFFFFFFFF => 9}}

A theme regression that discards backgrounds shows up as an empty (or collapsed) :background map, and two themes that should differ produce different key sets.

delete_backward(node)

@spec delete_backward(node()) :: :ok | {:error, atom()}

Delete one character behind the cursor (backspace).

dismiss_alert(node, button_label)

@spec dismiss_alert(node(), String.t()) :: :ok | {:error, atom()}

Dismiss a modal/alert overlay by tapping its first button labelled with button_label (e.g. "OK", "Cancel"). Mirrors what a user does when an alert pops up.

Mob.Test.dismiss_alert(node, "OK")

Known limitation (issues.md #9)

UIAlertController exposes its buttons twice in the AX tree (visual view + action target). Activating the visual view doesn't fire the action. This helper currently reports :ok while the alert stays on screen. Workaround: define alerts with action: :tag_atom and dismiss via Mob.Test.send_message(node, {:alert, :tag_atom}).

element_frames(node)

@spec element_frames(node()) ::
  %{optional(String.t()) => {float(), float(), float(), float()}}
  | {:error, term()}

Return the on-screen frame of every rendered element that carries an :id, as %{id => {x, y, w, h}} in logical units (points on iOS, dp on Android).

This is the screenshot-free way for an agent to know where things are: give the elements you want to inspect or drive an :id, and their live positions come back as a small structured map — no image bytes, no accessibility activation. The renderer also sets the :id as the element's accessibility identifier, so the same tags are visible to external tools (XCUITest, etc.).

Pairs with tap_id/2 to drive by id at real coordinates.

Mob.Test.element_frames(node)
#=> %{"save" => {24.0, 720.0, 327.0, 48.0}, "row_3" => {0.0, 300.0, 393.0, 56.0}}

What counts as "rendered" — and it differs by platform

On iOS, an element appears here once it has laid out, and is dropped when it leaves the tree or stops being laid out — a lazy-list row scrolled out of range, a tab that isn't the active one, a dismissed sheet's content. Those all stay in the render tree while off screen, so tree membership alone would report them; they're dropped on the platform's own disappear signal instead.

On Android, only the first half holds. Frames are cleared wholesale on a navigation transition and never per-element, so an element that stays in the tree but stops being laid out keeps its last frame until you navigate. A scrolled-away lazy_list row still reports a position there, and tap_id/2 will happily tap it — so on Android, treat a frame for content that may have scrolled or switched tabs as unverified, and confirm against render_tree/1 or a screenshot before acting on it. Bringing Android to parity is MOB-105.

A frame is a last known position, not a synchronous read: it's recorded as the element lays out. After a render that moves an element, there's a brief window before the next layout pass where the previous frame is still what's reported. If you've just triggered a change and are about to act on the result, poll until the frame settles rather than trusting the first read.

On iOS a frame is refreshed when the element appears, when the :id at that position changes, or when the element's own frame value changes — so a rearrangement that moves ids between same-sized slots without moving any slot can briefly report a stale position. For list content where that's a risk, confirm against render_tree/1 or a screenshot before acting on coordinates.

find(node, substring)

@spec find(node(), String.t()) :: [{list(), map()}]

Find all nodes in the current tree whose text contains substring. Returns a list of {path, node} tuples where path is a list of indices from the root.

Mob.Test.find(node, "Device APIs")
#=> [{[0, 1, 8], %{"type" => "button", "props" => %{"text" => "Device APIs →", ...}}}]

find_native(node, text)

@spec find_native(node(), String.t()) :: list()

Find elements in the native accessibility tree whose label or value contains text.

Mob.Test.find_native(node, "Increment")
#=> [{:button, "Increment", "", {164.0, 400.0, 54.0, 54.0}}]

find_view(node, text)

@spec find_view(node(), String.t()) :: [{[non_neg_integer()], map()}]

Find nodes in the view tree whose label or value contains text.

Returns [{path, node}] for each match. Faster and more accurate than find_native/2 (no AX dependency, sees all views).

Mob.Test.find_view(node, "Roll Dice")
#=> [{[0, 0, 0, 4], %{type: :button, label: "Roll Dice", ...}}]

flatten_tree(tree)

@spec flatten_tree(map()) :: [{[non_neg_integer()], map()}]

Flatten an already-fetched view tree. Pure function — useful for tests and for inspecting a captured tree without re-fetching.

tree = Mob.Test.view_tree(node)
flat = Mob.Test.flatten_tree(tree)

frame(node, id)

@spec frame(node(), String.t() | atom()) ::
  {float(), float(), float(), float()} | nil | {:error, term()}

Frame {x, y, w, h} of the element with id, or nil if it has no tracked position. See element_frames/1.

Mob.Test.frame(node, "save")   #=> {24.0, 720.0, 327.0, 48.0}

inspect(node)

@spec inspect(node()) :: map()

Return a map with :screen, :assigns, :nav_history, and :tree (the raw render tree from calling render/1 on the current screen).

key_press(node, key)

@spec key_press(node(), atom()) :: :ok | {:error, atom()}

Press a special key on the focused text input.

Keys: :return | :tab | :escape | :space

Mob.Test.key_press(node, :return)
Mob.Test.key_press(node, :escape)

locate(tag_or_label)

@spec locate(atom() | String.t()) :: {:ok, map()} | {:error, :not_found}

Locate an element by visible label text or accessibility ID (tag atom name). Returns the element's screen frame.

Requires idb (iOS) to be installed.

Mob.Test.locate(:save)
#=> {:ok, %{x: 0.0, y: 412.0, width: 402.0, height: 44.0}}

Mob.Test.locate("Save")
#=> {:ok, %{x: 0.0, y: 412.0, width: 402.0, height: 44.0}}

long_press_xy(node, x, y, duration_ms \\ 800)

@spec long_press_xy(node(), number(), number(), non_neg_integer()) ::
  :ok | {:error, atom()}

Long-press at screen coordinates for duration_ms milliseconds (default 800ms).

Mob.Test.long_press_xy(node, 195.0, 400.0)
Mob.Test.long_press_xy(node, 195.0, 400.0, 1200)

normalize_view_tree(node)

@spec normalize_view_tree(map() | term()) :: map() | term()

Normalize an Android-shaped (JSON-decoded, string-keyed) view tree into the iOS map shape: atom keys, atom :type, {x, y, w, h} frame tuple.

view_tree/1 applies this automatically. It's public so a captured tree can be normalized without a device.

paint_debug(node)

@spec paint_debug(node()) :: map() | {:error, term()}

Census of where colour lives in the native view tree — the diagnostic to reach for when view_tree/1 reports nil colours and you need to know why.

Groups every native view by (view class, layer class, sublayer classes) and reports, per group, how many views set each colour-bearing property:

Mob.Test.paint_debug(node)
#=> %{
#     "total_views" => 443,
#     "groups" => [
#       %{"view" => "SwiftUI.CGDrawingView", "layer" => "SwiftUI.CGDrawingLayer",
#         "sublayers" => ["CAShapeLayer"], "count" => 40,
#         "view_bg" => 0, "layer_bg" => 0, "shape_fill" => 40,
#         "gradient" => 0, "text_layer_fg" => 0, "uikit_text" => 0,
#         "has_contents" => 40},
#       ...
#     ]
#   }

Read a row as: for these 40 views the only colour set is CAShapeLayer.fillColor, so that is the property the extractor has to read. A group where every tally is 0 but has_contents is high is a view that drew itself into a bitmap — its colour is not recoverable without pixel sampling.

iOS only, debug builds only. Android raises :nif_error.

pop(node)

@spec pop(node()) :: :ok

Pop the current screen and return to the previous one. Synchronous.

Returns :ok once the navigation and re-render are complete, so it is safe to call screen/1 or assigns/1 immediately after.

No-op (returns :ok) if already at the root of the stack.

pop_to(node, dest)

@spec pop_to(node(), module() | atom()) :: :ok

Pop the stack until dest is at the top. Synchronous.

dest is a screen module or registered name atom. No-op if not in history.

pop_to_root(node)

@spec pop_to_root(node()) :: :ok

Pop all screens back to the root of the current stack. Synchronous.

reduce_rgba(rgba, width, height)

@spec reduce_rgba(binary(), integer(), integer()) :: {:ok, map()} | {:error, atom()}

Reduce a raw RGBA buffer to colour statistics. Pure — no device needed.

rgba is width * height pixels of 4 bytes each in R, G, B, A order (what :mob_nif.sample_region/4 returns). Colours come back as 0xAARRGGBB integers, alpha first, matching component props (guides/theming.md).

Mob.Test.reduce_rgba(<<0, 0, 255, 255, 0, 0, 255, 255>>, 2, 1)
#=> {:ok, %{average: 0xFF0000FF, dominant: 0xFF0000FF, dominant_share: 1.0,
#           distinct: 1, pixels: 2}}

:average is the per-channel mean (each channel independently, alpha included, rounded to nearest). :dominant is the most frequent exact pixel value, ties broken by the higher 0xAARRGGBB value so the result is deterministic. :dominant_share is its fraction of all pixels and :distinct counts distinct values — together they say whether :dominant describes a flat fill or just the most common pixel of a gradient.

The capture path renders opaque, so alpha is 255 in practice; a buffer with varying alpha is averaged channel-wise and not un-premultiplied.

{:error, :empty_region} for a non-positive dimension, {:error, :size_mismatch} when byte_size(rgba) != width * height * 4.

reset_to(node, dest, params \\ %{}, opts \\ [])

@spec reset_to(node(), module() | atom(), map(),
  transition: atom(),
  scope: :stack | :all
) :: :ok

Replace the current navigation stack with a new root screen. Synchronous.

Use this to simulate auth transitions (e.g. login → home with no back button).

Pass transition: :push or transition: :pop to drive a directional reset, matching Mob.Socket.reset_to/4. Pass scope: :all to discard every parked stack as well as the active one.

sample_color(node, id_or_rect)

@spec sample_color(
  node(),
  String.t() | atom() | {number(), number(), number(), number()}
) ::
  {:ok, map()} | {:error, term()}

What colour did the app actually draw in a region? Samples real pixels.

Address the region either by an element :id (resolved through element_frames/1) or by an explicit {x, y, w, h} rect in logical points:

Mob.Test.sample_color(node, "my-card")
Mob.Test.sample_color(node, {24.0, 416.0, 327.0, 53.5})

Returns {:ok, sample} where sample is the map reduce_rgba/3 produces:

{:ok, %{average: 0xFF2196F3, dominant: 0xFF2196F3, dominant_share: 0.94,
        distinct: 37, pixels: 2400}}

Why pixels and not view_tree/1

view_tree/1's :bg_color is nil for virtually all SwiftUI content — on iOS 26 SwiftUI paints via SDFLayer or rasterises into contents, exposing no readable paint property (measured: colour for 4 of 443 nodes; see decisions/2026-08-09-view-tree-colour-needs-screenshot-sampling.md). Sampling the rendered pixels is the only way to catch a regression like the glass theme that discarded every Box background — under which a background: :primary Box and a :surface_raised Box sample to the same colour, and that difference is what this asserts.

Reading the result

A region is rarely one flat colour — a card has text, a border, antialiased corners — so a bare mean can be misleading. :average is the mean, :dominant is the most common exact pixel value (the background of a mostly-flat region), and :dominant_share says how much to trust it: 0.9 is a flat fill, 0.2 is a gradient or a busy region where only :average means much. Assert on :dominant for solid fills, on :average for anything glassy.

Errors

  • {:error, :not_found} — no element with that :id has a tracked frame (the element needs an :id, and must have laid out at least once)
  • {:error, :empty_frame} — the element's frame has zero width or height
  • {:error, :offscreen} — the rect lies entirely outside the window
  • {:error, :no_window} — app has no visible window (backgrounded)
  • {:error, :size_mismatch} — the buffer didn't match the reported dimensions, so no colour is reported rather than a wrong one
  • {:error, {:badrpc, _}} — no sample_region/4 on this platform; the NIF is iOS-only and debug-build only

A rect that only partly overlaps the window is clamped to the visible part and :pixels reports what was actually sampled.

The payload is w * h * screen_scale^2 * 4 bytes — cropping happens in the native render, so an element-sized region is tens to hundreds of KB, not a framebuffer. Don't hand it a full-screen rect.

screen(node)

@spec screen(node()) :: module()

Return the current screen module.

screen_info(node)

@spec screen_info(node()) :: map()

Return screen geometry in logical units (points on iOS, dp on Android).

Mob.Test.screen_info(node)
#=> %{
#     width: 393.0, height: 852.0, scale: 3.0,
#     safe_area: %{top: 59.0, bottom: 34.0, left: 0.0, right: 0.0}
#   }

:scale is the device-pixel ratio (UIScreen.scale on iOS, displayMetrics.density on Android). All other values are already in logical units; no further conversion needed in the agent.

screenshot(node, opts \\ [])

@spec screenshot(
  node(),
  keyword()
) :: {:ok, binary()} | {:error, term()}

Capture the running app's own window in-process and return the image bytes.

Returns {:ok, binary} (PNG or JPEG) or {:error, reason}. The bytes come back over Erlang distribution — no adb screencap / xcrun simctl io, so it works against a remote device an agent can only reach over dist.

Options:

  • :format:png (default) or :jpeg
  • :quality0..100, JPEG only (default 90)
  • :scale — output scale factor (default 1.0); 0.5 halves resolution

Captures only the app's own surface, not system layers or other processes. Secure text fields (iOS) and FLAG_SECURE windows (Android) render blank by OS policy. A backgrounded app has no live window, so this fails when the app is not foregrounded.

{:ok, png} = Mob.Test.screenshot(node)
File.write!("/tmp/shot.png", png)

{:ok, jpg} = Mob.Test.screenshot(node, format: :jpeg, quality: 60, scale: 0.5)

screenshot_tour(node, id, opts \\ [])

@spec screenshot_tour(node(), String.t() | atom(), keyword()) ::
  [{{float(), float()}, binary()}] | {:error, term()}

Walk a scroll view top→bottom, capturing a screenshot at each page. Returns a list of {offset, image_binary} pairs — the agent's "see the whole long screen" path, entirely over dist.

Options:

  • :format / :quality / :scale — passed through to screenshot/2

  • :overlap0.0..0.9, fraction of a viewport to overlap between pages (default 0.0)

  • :settle_ms — pause after each scroll before capturing (default 150)

    pages = Mob.Test.screenshottour(node, "feed", format: :jpeg, quality: 60) for {{_x, y}, bin} <- pages, do: File.write!("/tmp/page#{trunc(y)}.jpg", bin)

scroll_info(node, id)

@spec scroll_info(node(), String.t() | atom()) :: map() | {:error, term()}

Read a scroll view's current offset and extent, addressed by its :id prop (the same :id you set on a type: :scroll or type: :list node).

Returns a map, or {:error, reason}:

%{
  offset:     {x, y},   # current scroll position
  content:    {w, h},   # full scrollable content size
  viewport:   {w, h},   # visible area
  max_offset: {x, y},   # offset at the bottom/right edge
  kind:       :pixel | :index
}

:kind is :pixel for pixel-precise scroll views (iOS UIScrollView, Android verticalScroll). It is :index for item-indexed lists (Android LazyColumn), where the y components count items, not pixels, and viewport height is the number of visible items. scroll_to/4 and screenshot_tour/3 work in whichever unit :kind reports, so paging stays coherent either way.

Mob.Test.scroll_info(node, "feed")
#=> %{offset: {0.0, 0.0}, content: {393.0, 2400.0}, viewport: {393.0, 756.0},
#     max_offset: {0.0, 1644.0}, kind: :pixel}

scroll_to(node, id, target, opts \\ [])

@spec scroll_to(node(), String.t() | atom(), tuple() | atom(), keyword()) ::
  :ok | {:error, term()}

Scroll a view (by :id) to a target position. Reads scroll_info/2 first to resolve and clamp the absolute offset, then drives the native scroll view.

target:

  • {x, y} — absolute offset (pixels, or item index on an :index list)
  • :top / :bottom — the extremes
  • {:page, n}n viewport-heights down from the top (works on both :pixel and :index views)

Returns :ok or {:error, reason}.

Mob.Test.scroll_to(node, "feed", :bottom)
Mob.Test.scroll_to(node, "feed", {:page, 2})
Mob.Test.scroll_to(node, "feed", {0.0, 500.0})

select(node, list_id, index)

@spec select(node(), atom(), non_neg_integer()) :: :ok

Select a row in a :list component by index.

list_id must match the :id prop on the type: :list node. index is zero-based. Delivers {:select, list_id, index} to handle_info/2.

Fire-and-forget.

Mob.Test.select(node, :my_list, 0)   # first row

send_message(node, message)

@spec send_message(node(), term()) :: :ok

Send an arbitrary message to the screen's handle_info/2. Fire-and-forget.

Use this to simulate results from device APIs without triggering real hardware:

# Permissions
Mob.Test.send_message(node, {:permission, :camera, :granted})
Mob.Test.send_message(node, {:permission, :notifications, :denied})

# Camera
Mob.Test.send_message(node, {:camera, :photo, %{path: "/tmp/photo.jpg", width: 1920, height: 1080}})
Mob.Test.send_message(node, {:camera, :cancelled})

# Location
Mob.Test.send_message(node, {:location, %{lat: 43.6532, lon: -79.3832, accuracy: 10.0, altitude: 80.0}})
Mob.Test.send_message(node, {:location, :error, :denied})

# Photos / Files
Mob.Test.send_message(node, {:photos, :picked, [%{path: "/tmp/photo.jpg", width: 800, height: 600}]})
Mob.Test.send_message(node, {:files, :picked, [%{path: "/tmp/doc.pdf", name: "doc.pdf", size: 4096}]})

# Audio / Motion / Scanner
Mob.Test.send_message(node, {:audio, :recorded, %{path: "/tmp/audio.aac", duration: 12}})
Mob.Test.send_message(node, {:motion, %{ax: 0.1, ay: 9.8, az: 0.0, gx: 0.0, gy: 0.0, gz: 0.0}})
Mob.Test.send_message(node, {:scan, :result, %{type: :qr, value: "https://example.com"}})

# Notifications
Mob.Test.send_message(node, {:notification, %{id: "n1", title: "Hi", body: "Hello", data: %{}, source: :push}})
Mob.Test.send_message(node, {:push_token, :ios, "abc123def456"})

# Biometric
Mob.Test.send_message(node, {:biometric, :success})
Mob.Test.send_message(node, {:biometric, :failure, :user_cancel})

# Custom
Mob.Test.send_message(node, {:my_event, %{key: "value"}})

settle(node, timeout \\ 5000)

@spec settle(node(), timeout()) :: :ok

Block until the app has finished processing and the current frame is on screen.

Drains the navigation owner and the screen process (twice, since an event that navigates hands off to a different screen), then waits for Mob.Sender to commit. All three are needed: the owner forwards the event, the screen builds the tree, and the sender commits it — so a drained owner mailbox alone does not mean the frame has been rendered.

Use after any fire-and-forget call (tap/2, back/1, send_message/2) before reading the native side with view_tree/1, screenshot/2, tap_id/2 or element_frames/2.

Mob.Test.tap(node, :save)
Mob.Test.settle(node)
Mob.Test.view_tree(node)

swipe(node, x1, y1, x2, y2)

@spec swipe(node(), number(), number(), number(), number()) :: :ok | {:error, atom()}

Swipe from (x1, y1) to (x2, y2). Drives UIScrollView contentOffset on simulator; synthesises a drag gesture on real device.

Mob.Test.swipe(node, 195.0, 500.0, 195.0, 100.0)   # scroll down

switch_tab(node, tab, opts \\ [])

@spec switch_tab(node(), atom(), transition: atom(), mount_params: map()) :: :ok

Switch to a named tab stack. Synchronous.

Pass transition: :push, transition: :pop, or transition: :reset to exercise the same directional animation as Mob.Socket.switch_tab/3. mount_params: %{...} is passed to a target root only on its first mount.

tap(node, tag)

@spec tap(node(), atom()) :: :ok

Send a tap event to the current screen by tag atom.

The tag comes from on_tap: {self(), :tag_atom} in the screen's render/1. Check the screen's render function to find available tags.

Fire-and-forget — does not wait for the screen to finish processing. Follow with settle/2 before reading the native side.

Mob.Test.tap(node, :save)
Mob.Test.tap(node, :open_detail)

tap_id(node, id)

@spec tap_id(node(), String.t() | atom()) :: :ok | {:error, term()}

Tap the element with id at the center of its tracked frame — driving by id without a screenshot or coordinate guess. The element must carry an :id (see element_frames/1).

Mob.Test.tap_id(node, "save")

Inherits tap_xy/3's return contract, including its platform limits — read those before treating a non-:ok result as a test failure.

tap_native(tag_or_label)

@spec tap_native(atom() | String.t()) :: :ok | {:error, term()}

Locate an element and tap it via the simulator's native UI mechanism.

Requires idb (iOS) to be installed. Exercises the full native gesture path rather than sending a BEAM message — useful for testing gesture recognizers or verifying that the native layer wired up the tap handler correctly.

Prefer tap/2 for testing Elixir logic; use tap_native/1 when you need the native path.

Mob.Test.tap_native("Save")      # by visible text
Mob.Test.tap_native(:save)       # by accessibility_id (= tag atom name)

tap_xy(node, x, y)

@spec tap_xy(node(), number(), number()) ::
  :ok | {:error, :no_view_at_point | :no_element_at_point | :no_effect | term()}

Tap at screen coordinates on the native app.

Mob.Test.tap_xy(node, 289.7, 518.8)

Return values

:ok means the app reacted — an event reached the BEAM within 300ms of the tap. Every other outcome is an error tuple; there is no "probably worked".

ValueMeaning
:okA tap/focus/change/submit/select event reached the BEAM.
{:error, :no_view_at_point}Hit-test found nothing — the coordinate is outside every visible window.
{:error, :no_element_at_point}iOS simulator only: a view is there but no accessibility element to activate.
{:error, :no_effect}Input was accepted by the OS but no handler ran.
{:error, probe}iOS device only: the private injection API is missing; probe lists which selectors resolved.

Real capability per platform — read before trusting a result

  • iOS simulator — activates the accessibility element under the point. That works for Button, and for text fields (the responder chain is walked to focus them). It does not work for Mob's Box/Row/Column with on_tap:: SwiftUI gives a plain .onTapGesture no accessibility action, so activation is accepted and the handler never runs. A Box with accessibility_role: "button" is a real AX element (.isButton) since #94, but it still has no activate action, so the result is the same. Those taps return {:error, :no_effect}. Use tap/2 (by tag) to drive them.
  • iOS physical device — synthesises an IOHIDEvent. As of iOS 26.5 UIKit accepts the event and delivers no touch, so this returns {:error, :no_effect} for every coordinate. Treat coordinate tapping as not working on device and use tap/2. See decisions/2026-08-09-ios-device-tap-injection-has-no-effect.md.
  • Android — not routed through this function; adb shell input tap works and is what the tooling uses.

:no_effect in sidecar mode

The check is "did an event reach the BEAM", so it only sees handlers Mob owns. Driving a non-Mob app (sidecar mode), a genuinely successful tap still reports {:error, :no_effect} because there is nothing for the NIF to observe. Confirm those with ui_tree/1 or a screenshot instead.

The counter behind the check is process-wide, not per-tap: any Mob event that reaches the BEAM inside the 300ms settle window (a scroll notification, a timer-driven change, another handler) counts as the tap's effect and can turn a miss into a false :ok. The check assumes a serial harness — one synthetic interaction in flight at a time, no concurrent UI activity.

toggle(node, label_match)

@spec toggle(node(), String.t()) :: :ok | {:error, atom()}

Toggle a switch by a label substring. SwiftUI exposes Toggle as a button with an empty accessibility label and value "0" or "1" — so we find the Text element matching label_match, then activate the next button below it.

Mob.Test.toggle(node, "Notifications")

Known limitation (issues.md #8)

Mob's iOS Toggle component does not currently surface its label: prop as a separate :text AX element, so find_label_y/2 returns {:error, :label_not_found}. Workaround: use ax_action_at_xy/4 directly with the toggle's frame from ui_tree/1 (filter for :button with value "0" or "1"). Once issue #8 lands, this helper works as documented.

tree(node)

@spec tree(node()) :: map()

Return the current rendered tree (calls render/1 on the live assigns).

type_text(node, text)

@spec type_text(node(), String.t()) :: :ok | {:error, atom()}

Type text into the currently focused text field.

Tap the field first to give it focus, then call this function.

Mob.Test.tap_xy(node, 195.0, 300.0)
Process.sleep(100)
Mob.Test.type_text(node, "hello@example.com")

ui_tree(node)

@spec ui_tree(node()) :: list()

Return the live accessibility tree from the running native app.

Each element is a tuple: {type, label, value, {x, y, w, h}}

Mob.Test.ui_tree(node)
#=> [{:button, "Increment", "", {164.0, 400.0, 54.0, 54.0}}, ...]

view_tree(node)

@spec view_tree(node()) :: map() | {:error, term()}

Return the live UI tree as a nested map, walking native views directly.

Unlike ui_tree/1 (which uses the accessibility subsystem and requires VoiceOver activation on iOS), this walks UIView/View hierarchies directly: no AX activation needed.

Coverage caveat

  • UIKit apps (sidecar mode): full UIView hierarchy with labels and frames.
  • SwiftUI apps (current Mob): shallow — SwiftUI doesn't expose its content as separate UIView instances under the hosting view. You'll see containers and scroll views but not individual buttons/text. For Mob apps, prefer Mob.Test.tree/1 (the logical render tree, which has all the semantic info) or Mob.Test.ui_tree/1 (AX walk, requires VoiceOver activation).
  • Android (planned): a registry populated via onGloballyPositioned in Mob's Compose components — see future_developments.md "WireTap" section.

Returns a nested map:

%{
  type: :root, class: nil, label: nil, value: nil,
  frame: {0.0, 0.0, 393.0, 852.0},
  bg_color: nil, text_color: nil,
  children: [
    %{type: :window, class: "UIWindow", ..., children: [
      %{type: :scroll, ..., children: [
        %{type: :button, class: "SwiftUI.CGDrawingView", label: "Roll Dice",
          frame: {24.0, 416.0, 327.0, 53.5},
          bg_color: 0xFF2196F3, text_color: 0xFFFFFFFF, children: []}
      ]}
    ]}
  ]
}

:class is the concrete native view class. On SwiftUI it is usually the only thing that identifies a node — :type collapses anything it doesn't recognise to :view — and it's what tells you which renderer drew a node when a colour comes back nil.

Colours

:bg_color and :text_color are the colours the view actually painted, as 0xAARRGGBB integers — the same representation component props use (guides/theming.md). nil means nothing paintable was found, or the colour has no single RGBA value (a multi-stop gradient, a pattern fill).

UIKit puts colour on the view (UIView.backgroundColor, UILabel.textColor). SwiftUI mostly does not.background(Color, in: shape) and .foregroundColor, which is what Mob's renderer uses for every Box and Text, go through SwiftUI's own renderer and land on a CALayer (typically a CAShapeLayer fill) under a structural view whose own backgroundColor stays nil. So each node also harvests from its own layer subtree, excluding layers owned by its subviews so a container never claims a child's paint.

Sources consulted per node, first match wins:

BackgroundText
viewUIView.backgroundColorUILabel/UITextField/UITextView/UIButton
layer subtreeCAShapeLayer.fillColor, single-stop CAGradientLayer, CALayer.backgroundColorCATextLayer.foregroundColor

Fully-transparent colours are treated as no colour, so a Color.clear placeholder doesn't read as "painted black at alpha 0".

Because these are read back off UIView/CALayer rather than echoed from the render tree, they are the way to catch a styling regression where a theme or modifier silently drops a colour Elixir sent. Compare against tree/1 (what Elixir asked for) to see the two diverge.

If colours come back nil across the board, don't guess at the reason — call paint_debug/1, which reports which view/layer classes the renderer produced and which colour properties they actually set. On iOS 26 SwiftUI that is the expected outcome, and sample_color/2 (real pixels) is the way to verify a drawn colour.

On Android, the JSON returned by mob_nif:ui_view_tree/0 is decoded here — but no shipped MobBridge.kt implements uiViewTree(), so today Android returns {:error, :not_loaded}.

view_tree_flat(node)

@spec view_tree_flat(node()) :: [{[non_neg_integer()], map()}]

Return the view tree flattened to a list of {path, node} tuples.

path is the list of child indices from the root — e.g. [0, 2, 1] is "the second child of the third child of the first child of the root."

Useful for filter/find — see find_view/2.

Mob.Test.view_tree_flat(node)
#=> [
#     {[], %{type: :root, ...}},
#     {[0], %{type: :window, ...}},
#     {[0, 0], %{type: :scroll, ...}},
#     ...
#   ]

wait_for(node, predicate, opts \\ [])

@spec wait_for(node(), (list() -> boolean()), keyword()) :: :ok | {:error, :timeout}

Wait until predicate returns true when called with the current ui_tree, polling every interval_ms until timeout_ms elapses.

Mob.Test.wait_for(node, fn tree ->
  Enum.any?(tree, fn {_, label, _, _} -> label == "Success" end)
end)

wait_for_text(node, text, opts \\ [])

@spec wait_for_text(node(), String.t(), keyword()) :: :ok | {:error, :timeout}

Wait until an element whose label or value contains text appears in the accessibility tree.

Mob.Test.wait_for_text(node, "Welcome")
Mob.Test.wait_for_text(node, "Error", timeout_ms: 2000)