Remote inspection and interaction helpers for connected Mob apps.
All functions accept a node atom and operate on the running screen via
Erlang distribution. Connect first with mix mob.connect, then use these
from IEx or from an agent via :rpc.call/4.
Quick reference
node = :"my_app_ios@127.0.0.1"
# Inspection
Mob.Test.screen(node) #=> MyApp.HomeScreen
Mob.Test.assigns(node) #=> %{count: 3, ...}
Mob.Test.tree(node) #=> %{type: :column, ...}
Mob.Test.find(node, "Save") #=> [{[0, 2], %{...}}]
Mob.Test.inspect(node) #=> %{screen: ..., assigns: ..., tree: ...}
# Interaction
Mob.Test.tap(node, :increment) # tap a button by tag
Mob.Test.back(node) # system back gesture
Mob.Test.pop(node) # pop to previous screen (synchronous)
Mob.Test.navigate(node, MyApp.DetailScreen, %{id: 42})
Mob.Test.pop_to(node, MyApp.HomeScreen)
Mob.Test.pop_to_root(node)
# Lists
Mob.Test.select(node, :my_list, 0) # select first row
# Visual capture + scroll (in-process, over dist — no adb/xcrun)
{:ok, png} = Mob.Test.screenshot(node)
Mob.Test.scroll_info(node, "feed") # offset/content/viewport
Mob.Test.scroll_to(node, "feed", :bottom)
Mob.Test.screenshot_tour(node, "feed") # page top→bottom, capture each
# Element positions without a screenshot (elements need an :id)
Mob.Test.element_frames(node) # %{id => {x, y, w, h}}
Mob.Test.frame(node, "save") # {x, y, w, h}
Mob.Test.tap_id(node, "save") # drive by id at real coords
# What colour did the app actually draw? (samples pixels — the view tree can't)
Mob.Test.sample_color(node, "my-card") # %{average: 0xFF2196F3, ...}
# Device API simulation
Mob.Test.send_message(node, {:permission, :camera, :granted})
Mob.Test.send_message(node, {:camera, :photo, %{path: "/tmp/photo.jpg", width: 1920, height: 1080}})
Mob.Test.send_message(node, {:location, %{lat: 43.65, lon: -79.38, accuracy: 10.0, altitude: 80.0}})
Mob.Test.send_message(node, {:notification, %{id: "n1", title: "Hi", body: "Hey", data: %{}, source: :push}})Tap vs send_message
tap/2 sends the same {:tap, tag} message a native tap produces, so it
arrives in the screen's handle_info/2 exactly like a real button press.
send_message/2 delivers any term to handle_info/2.
Use send_message/2 to simulate async results from device APIs (camera, location,
notifications, etc.) without having to trigger the actual hardware.
Synchronous vs fire-and-forget
Navigation functions (pop, navigate, pop_to, pop_to_root) are synchronous —
they block until the navigation and re-render complete. This makes them safe to
follow immediately with screen/1 or assigns/1 to verify the result.
back/1, tap/2 and send_message/2 are fire-and-forget (they send a message
to the screen process and return immediately). Use settle/2 as a sync point
if you need to wait before reading state:
Mob.Test.send_message(node, {:permission, :camera, :granted})
Mob.Test.settle(node)
Mob.Test.assigns(node):sys.get_state/1 on :mob_screen is no longer sufficient on its own: since
MOB-110 the tree is handed to Mob.Sender and committed asynchronously, and
since MOB-112 :mob_screen is the navigation owner rather than the screen
itself. That only matters for the
functions that read the native side — view_tree/1, screenshot/2,
tap_id/2, element_frames/2. tree/1 and assigns/1 re-render in-process
and are unaffected.
Two layers of inspection: render tree vs native UI
Mob.Test exposes two complementary views of what the app is showing:
| API | Source | When to use |
|---|---|---|
tree/1, find/2 | Mob render tree (logical components) | Mob apps you control. Fast, exact, has on_tap tags, no AX activation needed. |
view_tree/1, find_view/2 | Native view hierarchy via NIF | Native pixel frames and painted colours; works for any app on iOS UIKit; shallow on SwiftUI/Compose. |
ui_tree/1 | OS accessibility tree | What sighted users read; works on any app if AX is active (iOS: VoiceOver). Strict superset of view_tree for UIKit; the only path to semantics inside SwiftUI/Compose. |
Choose render tree first if your app is Mob-rendered. Reach for view_tree
when you want native frames or geometry. Reach for ui_tree when you need
to inspect non-Mob content (alerts, system overlays, third-party SDK UI),
or to verify the rendered state matches the logical render.
Driving controls beyond plain taps
- Buttons / nav items —
tap/2(by tag, fastest), ormob_nif:tap/1(by accessibility label), ortap_xy/3(by coordinate). - Sliders, steppers, pickers —
adjust_slider/4and the underlyingax_action/3/ax_action_at_xy/4useaccessibilityIncrement/accessibilityDecrement. Synthetic drag gestures don't fire SwiftUI'sDragGesturereliably; AX actions do. - Switches / toggles —
toggle/2finds the switch by nearby label and activates it via the AX path (sendsaccessibilityActivate). - Modals / alerts / sheets —
dismiss_alert/2usesaccessibilityActivateon the named button;ax_action/3with:escapesendsaccessibilityPerformEscape. - Scroll views —
ax_action/3with:scroll_up/:scroll_down/:scroll_left/:scroll_rightsendsaccessibilityScroll:. - System back —
back/1(Mob screens, framework-level) or — for sidecar mode against arbitrary apps — synthetic edge-pan viaswipe/5fromx=0, but iOS owns that gesture above the app process and the synthetic pan won't fire. Useback/1for Mob, document the limitation for sidecar.
Platform support matrix
| Helper | iOS sim | iOS device | Android |
|---|---|---|---|
screen/1, assigns/1 | ✅ | ✅ | ✅ |
tap/2 (by tag) | ✅ | ✅ | ✅ |
back/1, pop/1, navigate | ✅ | ✅ | ✅ |
send_message/2 | ✅ | ✅ | ✅ |
screen_info/1 | ✅ | ✅ | ✅ |
view_tree/1 | ✅ (shallow†) | ✅ (shallow†) | ❌ not_loaded‡ |
sample_color/2 | ✅ | ✅ | ❌ not_loaded° |
find_view/2 | ✅ | ✅ | ❌ not_loaded‡ |
ui_tree/1 (legacy AX) | ⚠️ AX active§ | ⚠️ AX active§ | ❌ not_loaded |
ax_action/3 | ⚠️ AX active§ | ⚠️ AX active§ | ❌ not_supported |
ax_action_at_xy/4 | ⚠️ AX active§ | ⚠️ AX active§ | ❌ not_supported |
toggle/2 | ⚠️ AX active§ | ⚠️ AX active§ | ❌ ui_tree_unavailable |
dismiss_alert/2 | ⚠️ AX active§ | ⚠️ AX active§ | ❌ ui_tree_unavailable |
adjust_slider/4 | ⚠️ AX active§ | ⚠️ AX active§ | ❌ ui_tree_unavailable |
tap_xy/3 | ⚠️ AX-activatable only¶ | ❌ no_effect¶ | n/a |
swipe/5 | ⚠️ scroll only | ⚠️ unverified✱ | n/a |
- † SwiftUI doesn't expose its content as separate UIView instances —
view_treereaches the SwiftUI hosting view's container and stops. For semantic content on Mob screens usetree/1(render tree); for any other SwiftUI-based content useui_tree/1. - ‡ Android's
ui_view_treeNIF delegates to aMobBridge.uiViewTree()Kotlin method that no shipped template implements, so it returns{:error, :not_loaded}. When it lands it must emit the same keys iOS does (includingbg_color/text_color); the contract is documented at the NIF inandroid/jni/mob_nif.zig. The Mob renderer is Compose, so the View walk would stop at theAndroidComposeViewhost anyway — the real fix isModifier.onGloballyPositionedin Mob's components writing to a registry the NIF reads. Seeissues.md#11. - °
sample_region/4is implemented inios/mob_nif.monly. Android would need the same crop-in-the-render treatment against the activity window; until thensample_color/2returns{:error, {:badrpc, _}}there. - § "AX active" means an iOS accessibility client is asking for the
AX tree so SwiftUI materializes it. Today: VoiceOver toggle. Production:
XCAXClient_iOSactivation, debug-only — see WireTap stretch goals infuture_developments.md. - ¶
tap_xy/3now verifies that the tap actually produced an event before returning:ok. On the simulator that limits it to elements SwiftUI exposes an accessibility action for (Button, text fields) — aBoxwithon_tap:returns{:error, :no_effect}. On a physical device the IOHID-injected touch is accepted but never delivered, so every coordinate returns{:error, :no_effect}. Drive taps withtap/2(by tag); seetap_xy/3anddecisions/2026-08-09-ios-device-tap-injection-has-no-effect.md. - ✱
swipe/5andlong_press/4use the same device injection path astap_xy/3and still report:okon acceptance rather than on effect. Same root cause, not yet converted — treat their:okas unverified.
Helpers that depend on AX return clear error tuples on Android instead of
raising. Callers should match on {:error, :not_supported_on_android} and
{:error, :ui_tree_unavailable} and either skip or fall back to
send_message/2 for state mutations.
Known limitations affecting AX automation
Even on iOS with AX active, three Mob component defects keep the natural paths from working today. Workarounds in each helper's docstring:
- Slider —
accessibilityIncrement/Decrementare no-ops because Mob's iOS Slider doesn't attach.accessibilityAdjustableAction. Seeissues.md#7. - Toggle — the
label:prop doesn't reach the AX tree;toggle/2can't find the switch by label name. Useax_action_at_xy/4with coordinates for now. Seeissues.md#8. - Alert OK button —
accessibilityActivateon the AX-tree button doesn't fire the underlyingUIAlertAction. Use MobAlertwithaction:atoms andsend_message/2to dismiss programmatically. Seeissues.md#9.
System-level gestures iOS owns above the app process (edge-pan back,
swipe-up app switcher, pull-down notification center) are out of reach
for in-process synthetic touches on physical devices. Use back/1 for
Mob screens; for sidecar mode against arbitrary apps, document the
limitation rather than promising the gesture.
Summary
Functions
Step a slider toward a target percentage (0.0..1.0) using accessibility increment/decrement actions. Reliable when synthetic-drag won't fire (SwiftUI Slider's DragGesture ignores in-process touches on iOS).
Return the current screen's assigns map, or nil while that screen is being
restarted after a crash (MOB-112 — the socket lives in the screen's own
process, which is briefly absent).
Invoke an accessibility action on the first AX element matching match.
Invoke an AX action on whatever element occupies the given screen coordinates.
Simulate the system back gesture (Android hardware back / iOS edge-pan).
Clear all text in the focused input (select-all + delete).
Tally of the distinct painted colours in a view tree — the cheap way to assert a styling change actually reached the screen.
Delete one character behind the cursor (backspace).
Dismiss a modal/alert overlay by tapping its first button labelled with
button_label (e.g. "OK", "Cancel"). Mirrors what a user does when an
alert pops up.
Return the on-screen frame of every rendered element that carries an :id,
as %{id => {x, y, w, h}} in logical units (points on iOS, dp on Android).
Find all nodes in the current tree whose text contains substring.
Returns a list of {path, node} tuples where path is a list of
indices from the root.
Find elements in the native accessibility tree whose label or value contains text.
Find nodes in the view tree whose label or value contains text.
Flatten an already-fetched view tree. Pure function — useful for tests and for inspecting a captured tree without re-fetching.
Frame {x, y, w, h} of the element with id, or nil if it has no tracked
position. See element_frames/1.
Return a map with :screen, :assigns, :nav_history, and :tree
(the raw render tree from calling render/1 on the current screen).
Press a special key on the focused text input.
Locate an element by visible label text or accessibility ID (tag atom name). Returns the element's screen frame.
Long-press at screen coordinates for duration_ms milliseconds (default 800ms).
Push a new screen onto the navigation stack. Synchronous.
Normalize an Android-shaped (JSON-decoded, string-keyed) view tree into the
iOS map shape: atom keys, atom :type, {x, y, w, h} frame tuple.
Census of where colour lives in the native view tree — the diagnostic to reach
for when view_tree/1 reports nil colours and you need to know why.
Pop the current screen and return to the previous one. Synchronous.
Pop the stack until dest is at the top. Synchronous.
Pop all screens back to the root of the current stack. Synchronous.
Reduce a raw RGBA buffer to colour statistics. Pure — no device needed.
Replace the current navigation stack with a new root screen. Synchronous.
What colour did the app actually draw in a region? Samples real pixels.
Return the current screen module.
Return screen geometry in logical units (points on iOS, dp on Android).
Capture the running app's own window in-process and return the image bytes.
Walk a scroll view top→bottom, capturing a screenshot at each page. Returns a
list of {offset, image_binary} pairs — the agent's "see the whole long
screen" path, entirely over dist.
Read a scroll view's current offset and extent, addressed by its :id prop
(the same :id you set on a type: :scroll or type: :list node).
Scroll a view (by :id) to a target position. Reads scroll_info/2 first to
resolve and clamp the absolute offset, then drives the native scroll view.
Select a row in a :list component by index.
Send an arbitrary message to the screen's handle_info/2. Fire-and-forget.
Block until the app has finished processing and the current frame is on screen.
Swipe from (x1, y1) to (x2, y2). Drives UIScrollView contentOffset on simulator; synthesises a drag gesture on real device.
Switch to a named tab stack. Synchronous.
Send a tap event to the current screen by tag atom.
Tap the element with id at the center of its tracked frame — driving by id
without a screenshot or coordinate guess. The element must carry an :id
(see element_frames/1).
Locate an element and tap it via the simulator's native UI mechanism.
Tap at screen coordinates on the native app.
Toggle a switch by a label substring. SwiftUI exposes Toggle as a button
with an empty accessibility label and value "0" or "1" — so we find the
Text element matching label_match, then activate the next button below it.
Return the current rendered tree (calls render/1 on the live assigns).
Type text into the currently focused text field.
Return the live accessibility tree from the running native app.
Return the live UI tree as a nested map, walking native views directly.
Return the view tree flattened to a list of {path, node} tuples.
Wait until predicate returns true when called with the current ui_tree,
polling every interval_ms until timeout_ms elapses.
Wait until an element whose label or value contains text appears in the
accessibility tree.
Functions
Step a slider toward a target percentage (0.0..1.0) using accessibility increment/decrement actions. Reliable when synthetic-drag won't fire (SwiftUI Slider's DragGesture ignores in-process touches on iOS).
match is a substring of the slider's label or value (e.g. "Volume").
target is a fraction 0.0..1.0. max_steps caps the increment loop
(default 30) so a wrong match can't spin forever.
Returns {:ok, final_pct} or {:error, reason}.
Mob.Test.adjust_slider(node, "Volume", 0.30)
#=> {:ok, 0.30}Implementation note: each AX increment/decrement on a SwiftUI slider moves
by the slider's .step value (default 0.10 of the range). The function
re-reads the slider value after each step to converge.
Known limitation (issues.md #7)
Mob's iOS Slider component does not currently attach
.accessibilityAdjustableAction { … }, so accessibilityIncrement and
accessibilityDecrement are silently dropped by SwiftUI even though the
NIF returns :ok. This helper currently returns {:error, :max_steps_exhausted} against an unfixed slider. Until issue #7 lands,
drive sliders via Mob.Test.send_message(node, {:change, :slider_tag, value}).
Return the current screen's assigns map, or nil while that screen is being
restarted after a crash (MOB-112 — the socket lives in the screen's own
process, which is briefly absent).
Invoke an accessibility action on the first AX element matching match.
Platform support
- iOS: works once AX is active (today: VoiceOver on; future:
XCAXClient_iOSactivation, seefuture_developments.md). - Android: returns
{:error, :not_supported_on_android}. The Compose semantics walker is queued under WireTap (issues.md #11).
Used for controls where synthetic touches don't reach the gesture recognizer (sliders, scrolls, modal dismissal).
match is a string searched in both label and value. action is one of:
:increment, :decrement, :activate, :escape, :scroll_up,
:scroll_down, :scroll_left, :scroll_right.
Mob.Test.ax_action(node, "Volume", :decrement)
Mob.Test.ax_action(node, "Cancel", :activate)
Invoke an AX action on whatever element occupies the given screen coordinates.
Useful when label/value substring matching is ambiguous (e.g. multiple
sliders that all read "50%", a toggle whose accessibility label is empty).
Caller picks coordinates from ui_tree/1 and points at the exact element.
Mob.Test.ax_action_at_xy(node, 187.0, 296.0, :increment)Platform support
- iOS: works once AX is active (VoiceOver on, today).
- Android: returns
{:error, :not_supported_on_android}— seeax_action/3.
@spec back(node()) :: :ok
Simulate the system back gesture (Android hardware back / iOS edge-pan).
Fire-and-forget — follow with settle/2 before reading the native side. The
framework pops the navigation stack; if already at the root, it exits the app. Prefer pop/1 when you need to know that navigation
has finished before reading state.
Clear all text in the focused input (select-all + delete).
Tally of the distinct painted colours in a view tree — the cheap way to assert a styling change actually reached the screen.
Pass a node to fetch the tree, or an already-fetched tree to work offline.
Returns %{background: %{argb => count}, text: %{argb => count}}, nil
colours excluded.
Mob.Test.color_census(node)
#=> %{background: %{0xFF2196F3 => 4, 0xFF1E1E1E => 1}, text: %{0xFFFFFFFF => 9}}A theme regression that discards backgrounds shows up as an empty (or
collapsed) :background map, and two themes that should differ produce
different key sets.
Delete one character behind the cursor (backspace).
Dismiss a modal/alert overlay by tapping its first button labelled with
button_label (e.g. "OK", "Cancel"). Mirrors what a user does when an
alert pops up.
Mob.Test.dismiss_alert(node, "OK")Known limitation (issues.md #9)
UIAlertController exposes its buttons twice in the AX tree (visual view +
action target). Activating the visual view doesn't fire the action. This
helper currently reports :ok while the alert stays on screen. Workaround:
define alerts with action: :tag_atom and dismiss via
Mob.Test.send_message(node, {:alert, :tag_atom}).
@spec element_frames(node()) :: %{optional(String.t()) => {float(), float(), float(), float()}} | {:error, term()}
Return the on-screen frame of every rendered element that carries an :id,
as %{id => {x, y, w, h}} in logical units (points on iOS, dp on Android).
This is the screenshot-free way for an agent to know where things are: give
the elements you want to inspect or drive an :id, and their live positions
come back as a small structured map — no image bytes, no accessibility
activation. The renderer also sets the :id as the element's accessibility
identifier, so the same tags are visible to external tools (XCUITest, etc.).
Pairs with tap_id/2 to drive by id at real coordinates.
Mob.Test.element_frames(node)
#=> %{"save" => {24.0, 720.0, 327.0, 48.0}, "row_3" => {0.0, 300.0, 393.0, 56.0}}What counts as "rendered" — and it differs by platform
On iOS, an element appears here once it has laid out, and is dropped when it leaves the tree or stops being laid out — a lazy-list row scrolled out of range, a tab that isn't the active one, a dismissed sheet's content. Those all stay in the render tree while off screen, so tree membership alone would report them; they're dropped on the platform's own disappear signal instead.
On Android, only the first half holds. Frames are cleared wholesale on a
navigation transition and never per-element, so an element that stays in the
tree but stops being laid out keeps its last frame until you navigate. A
scrolled-away lazy_list row still reports a position there, and tap_id/2
will happily tap it — so on Android, treat a frame for content that may have
scrolled or switched tabs as unverified, and confirm against render_tree/1
or a screenshot before acting on it. Bringing Android to parity is MOB-105.
A frame is a last known position, not a synchronous read: it's recorded as the element lays out. After a render that moves an element, there's a brief window before the next layout pass where the previous frame is still what's reported. If you've just triggered a change and are about to act on the result, poll until the frame settles rather than trusting the first read.
On iOS a frame is refreshed when the element appears, when the :id at that
position changes, or when the element's own frame value changes — so a
rearrangement that moves ids between same-sized slots without moving any slot
can briefly report a stale position. For list content where that's a risk,
confirm against render_tree/1 or a screenshot before acting on coordinates.
Find all nodes in the current tree whose text contains substring.
Returns a list of {path, node} tuples where path is a list of
indices from the root.
Mob.Test.find(node, "Device APIs")
#=> [{[0, 1, 8], %{"type" => "button", "props" => %{"text" => "Device APIs →", ...}}}]
Find elements in the native accessibility tree whose label or value contains text.
Mob.Test.find_native(node, "Increment")
#=> [{:button, "Increment", "", {164.0, 400.0, 54.0, 54.0}}]
@spec find_view(node(), String.t()) :: [{[non_neg_integer()], map()}]
Find nodes in the view tree whose label or value contains text.
Returns [{path, node}] for each match. Faster and more accurate than
find_native/2 (no AX dependency, sees all views).
Mob.Test.find_view(node, "Roll Dice")
#=> [{[0, 0, 0, 4], %{type: :button, label: "Roll Dice", ...}}]
@spec flatten_tree(map()) :: [{[non_neg_integer()], map()}]
Flatten an already-fetched view tree. Pure function — useful for tests and for inspecting a captured tree without re-fetching.
tree = Mob.Test.view_tree(node)
flat = Mob.Test.flatten_tree(tree)
@spec frame(node(), String.t() | atom()) :: {float(), float(), float(), float()} | nil | {:error, term()}
Frame {x, y, w, h} of the element with id, or nil if it has no tracked
position. See element_frames/1.
Mob.Test.frame(node, "save") #=> {24.0, 720.0, 327.0, 48.0}
Return a map with :screen, :assigns, :nav_history, and :tree
(the raw render tree from calling render/1 on the current screen).
Press a special key on the focused text input.
Keys: :return | :tab | :escape | :space
Mob.Test.key_press(node, :return)
Mob.Test.key_press(node, :escape)
Locate an element by visible label text or accessibility ID (tag atom name). Returns the element's screen frame.
Requires idb (iOS) to be installed.
Mob.Test.locate(:save)
#=> {:ok, %{x: 0.0, y: 412.0, width: 402.0, height: 44.0}}
Mob.Test.locate("Save")
#=> {:ok, %{x: 0.0, y: 412.0, width: 402.0, height: 44.0}}
@spec long_press_xy(node(), number(), number(), non_neg_integer()) :: :ok | {:error, atom()}
Long-press at screen coordinates for duration_ms milliseconds (default 800ms).
Mob.Test.long_press_xy(node, 195.0, 400.0)
Mob.Test.long_press_xy(node, 195.0, 400.0, 1200)
Normalize an Android-shaped (JSON-decoded, string-keyed) view tree into the
iOS map shape: atom keys, atom :type, {x, y, w, h} frame tuple.
view_tree/1 applies this automatically. It's public so a captured tree can
be normalized without a device.
Census of where colour lives in the native view tree — the diagnostic to reach
for when view_tree/1 reports nil colours and you need to know why.
Groups every native view by (view class, layer class, sublayer classes) and
reports, per group, how many views set each colour-bearing property:
Mob.Test.paint_debug(node)
#=> %{
# "total_views" => 443,
# "groups" => [
# %{"view" => "SwiftUI.CGDrawingView", "layer" => "SwiftUI.CGDrawingLayer",
# "sublayers" => ["CAShapeLayer"], "count" => 40,
# "view_bg" => 0, "layer_bg" => 0, "shape_fill" => 40,
# "gradient" => 0, "text_layer_fg" => 0, "uikit_text" => 0,
# "has_contents" => 40},
# ...
# ]
# }Read a row as: for these 40 views the only colour set is
CAShapeLayer.fillColor, so that is the property the extractor has to read.
A group where every tally is 0 but has_contents is high is a view that drew
itself into a bitmap — its colour is not recoverable without pixel sampling.
iOS only, debug builds only. Android raises :nif_error.
@spec pop(node()) :: :ok
Pop the current screen and return to the previous one. Synchronous.
Returns :ok once the navigation and re-render are complete, so it is safe
to call screen/1 or assigns/1 immediately after.
No-op (returns :ok) if already at the root of the stack.
Pop the stack until dest is at the top. Synchronous.
dest is a screen module or registered name atom. No-op if not in history.
@spec pop_to_root(node()) :: :ok
Pop all screens back to the root of the current stack. Synchronous.
Reduce a raw RGBA buffer to colour statistics. Pure — no device needed.
rgba is width * height pixels of 4 bytes each in R, G, B, A order (what
:mob_nif.sample_region/4 returns). Colours come back as 0xAARRGGBB
integers, alpha first, matching component props (guides/theming.md).
Mob.Test.reduce_rgba(<<0, 0, 255, 255, 0, 0, 255, 255>>, 2, 1)
#=> {:ok, %{average: 0xFF0000FF, dominant: 0xFF0000FF, dominant_share: 1.0,
# distinct: 1, pixels: 2}}:average is the per-channel mean (each channel independently, alpha
included, rounded to nearest). :dominant is the most frequent exact pixel
value, ties broken by the higher 0xAARRGGBB value so the result is
deterministic. :dominant_share is its fraction of all pixels and
:distinct counts distinct values — together they say whether :dominant
describes a flat fill or just the most common pixel of a gradient.
The capture path renders opaque, so alpha is 255 in practice; a buffer with
varying alpha is averaged channel-wise and not un-premultiplied.
{:error, :empty_region} for a non-positive dimension, {:error, :size_mismatch} when byte_size(rgba) != width * height * 4.
Replace the current navigation stack with a new root screen. Synchronous.
Use this to simulate auth transitions (e.g. login → home with no back button).
Pass transition: :push or transition: :pop to drive a directional reset,
matching Mob.Socket.reset_to/4. Pass scope: :all to discard every parked
stack as well as the active one.
@spec sample_color( node(), String.t() | atom() | {number(), number(), number(), number()} ) :: {:ok, map()} | {:error, term()}
What colour did the app actually draw in a region? Samples real pixels.
Address the region either by an element :id (resolved through
element_frames/1) or by an explicit {x, y, w, h} rect in logical points:
Mob.Test.sample_color(node, "my-card")
Mob.Test.sample_color(node, {24.0, 416.0, 327.0, 53.5})Returns {:ok, sample} where sample is the map reduce_rgba/3 produces:
{:ok, %{average: 0xFF2196F3, dominant: 0xFF2196F3, dominant_share: 0.94,
distinct: 37, pixels: 2400}}Why pixels and not view_tree/1
view_tree/1's :bg_color is nil for virtually all SwiftUI content — on
iOS 26 SwiftUI paints via SDFLayer or rasterises into contents, exposing no
readable paint property (measured: colour for 4 of 443 nodes; see
decisions/2026-08-09-view-tree-colour-needs-screenshot-sampling.md). Sampling
the rendered pixels is the only way to catch a regression like the glass theme
that discarded every Box background — under which a background: :primary Box
and a :surface_raised Box sample to the same colour, and that difference is
what this asserts.
Reading the result
A region is rarely one flat colour — a card has text, a border, antialiased
corners — so a bare mean can be misleading. :average is the mean, :dominant
is the most common exact pixel value (the background of a mostly-flat region),
and :dominant_share says how much to trust it: 0.9 is a flat fill, 0.2 is
a gradient or a busy region where only :average means much. Assert on
:dominant for solid fills, on :average for anything glassy.
Errors
{:error, :not_found}— no element with that:idhas a tracked frame (the element needs an:id, and must have laid out at least once){:error, :empty_frame}— the element's frame has zero width or height{:error, :offscreen}— the rect lies entirely outside the window{:error, :no_window}— app has no visible window (backgrounded){:error, :size_mismatch}— the buffer didn't match the reported dimensions, so no colour is reported rather than a wrong one{:error, {:badrpc, _}}— nosample_region/4on this platform; the NIF is iOS-only and debug-build only
A rect that only partly overlaps the window is clamped to the visible part and
:pixels reports what was actually sampled.
The payload is w * h * screen_scale^2 * 4 bytes — cropping happens in the
native render, so an element-sized region is tens to hundreds of KB, not a
framebuffer. Don't hand it a full-screen rect.
Return the current screen module.
Return screen geometry in logical units (points on iOS, dp on Android).
Mob.Test.screen_info(node)
#=> %{
# width: 393.0, height: 852.0, scale: 3.0,
# safe_area: %{top: 59.0, bottom: 34.0, left: 0.0, right: 0.0}
# }:scale is the device-pixel ratio (UIScreen.scale on iOS, displayMetrics.density
on Android). All other values are already in logical units; no further conversion
needed in the agent.
Capture the running app's own window in-process and return the image bytes.
Returns {:ok, binary} (PNG or JPEG) or {:error, reason}. The bytes come
back over Erlang distribution — no adb screencap / xcrun simctl io, so it
works against a remote device an agent can only reach over dist.
Options:
:format—:png(default) or:jpeg:quality—0..100, JPEG only (default90):scale— output scale factor (default1.0);0.5halves resolution
Captures only the app's own surface, not system layers or other processes.
Secure text fields (iOS) and FLAG_SECURE windows (Android) render blank by
OS policy. A backgrounded app has no live window, so this fails when the app
is not foregrounded.
{:ok, png} = Mob.Test.screenshot(node)
File.write!("/tmp/shot.png", png)
{:ok, jpg} = Mob.Test.screenshot(node, format: :jpeg, quality: 60, scale: 0.5)
@spec screenshot_tour(node(), String.t() | atom(), keyword()) :: [{{float(), float()}, binary()}] | {:error, term()}
Walk a scroll view top→bottom, capturing a screenshot at each page. Returns a
list of {offset, image_binary} pairs — the agent's "see the whole long
screen" path, entirely over dist.
Options:
:format/:quality/:scale— passed through toscreenshot/2:overlap—0.0..0.9, fraction of a viewport to overlap between pages (default0.0):settle_ms— pause after each scroll before capturing (default150)pages = Mob.Test.screenshottour(node, "feed", format: :jpeg, quality: 60) for {{_x, y}, bin} <- pages, do: File.write!("/tmp/page#{trunc(y)}.jpg", bin)
Read a scroll view's current offset and extent, addressed by its :id prop
(the same :id you set on a type: :scroll or type: :list node).
Returns a map, or {:error, reason}:
%{
offset: {x, y}, # current scroll position
content: {w, h}, # full scrollable content size
viewport: {w, h}, # visible area
max_offset: {x, y}, # offset at the bottom/right edge
kind: :pixel | :index
}:kind is :pixel for pixel-precise scroll views (iOS UIScrollView,
Android verticalScroll). It is :index for item-indexed lists (Android
LazyColumn), where the y components count items, not pixels, and viewport
height is the number of visible items. scroll_to/4 and screenshot_tour/3
work in whichever unit :kind reports, so paging stays coherent either way.
Mob.Test.scroll_info(node, "feed")
#=> %{offset: {0.0, 0.0}, content: {393.0, 2400.0}, viewport: {393.0, 756.0},
# max_offset: {0.0, 1644.0}, kind: :pixel}
Scroll a view (by :id) to a target position. Reads scroll_info/2 first to
resolve and clamp the absolute offset, then drives the native scroll view.
target:
{x, y}— absolute offset (pixels, or item index on an:indexlist):top/:bottom— the extremes{:page, n}—nviewport-heights down from the top (works on both:pixeland:indexviews)
Returns :ok or {:error, reason}.
Mob.Test.scroll_to(node, "feed", :bottom)
Mob.Test.scroll_to(node, "feed", {:page, 2})
Mob.Test.scroll_to(node, "feed", {0.0, 500.0})
@spec select(node(), atom(), non_neg_integer()) :: :ok
Select a row in a :list component by index.
list_id must match the :id prop on the type: :list node. index is
zero-based. Delivers {:select, list_id, index} to handle_info/2.
Fire-and-forget.
Mob.Test.select(node, :my_list, 0) # first row
Send an arbitrary message to the screen's handle_info/2. Fire-and-forget.
Use this to simulate results from device APIs without triggering real hardware:
# Permissions
Mob.Test.send_message(node, {:permission, :camera, :granted})
Mob.Test.send_message(node, {:permission, :notifications, :denied})
# Camera
Mob.Test.send_message(node, {:camera, :photo, %{path: "/tmp/photo.jpg", width: 1920, height: 1080}})
Mob.Test.send_message(node, {:camera, :cancelled})
# Location
Mob.Test.send_message(node, {:location, %{lat: 43.6532, lon: -79.3832, accuracy: 10.0, altitude: 80.0}})
Mob.Test.send_message(node, {:location, :error, :denied})
# Photos / Files
Mob.Test.send_message(node, {:photos, :picked, [%{path: "/tmp/photo.jpg", width: 800, height: 600}]})
Mob.Test.send_message(node, {:files, :picked, [%{path: "/tmp/doc.pdf", name: "doc.pdf", size: 4096}]})
# Audio / Motion / Scanner
Mob.Test.send_message(node, {:audio, :recorded, %{path: "/tmp/audio.aac", duration: 12}})
Mob.Test.send_message(node, {:motion, %{ax: 0.1, ay: 9.8, az: 0.0, gx: 0.0, gy: 0.0, gz: 0.0}})
Mob.Test.send_message(node, {:scan, :result, %{type: :qr, value: "https://example.com"}})
# Notifications
Mob.Test.send_message(node, {:notification, %{id: "n1", title: "Hi", body: "Hello", data: %{}, source: :push}})
Mob.Test.send_message(node, {:push_token, :ios, "abc123def456"})
# Biometric
Mob.Test.send_message(node, {:biometric, :success})
Mob.Test.send_message(node, {:biometric, :failure, :user_cancel})
# Custom
Mob.Test.send_message(node, {:my_event, %{key: "value"}})
Block until the app has finished processing and the current frame is on screen.
Drains the navigation owner and the screen process (twice, since an event
that navigates hands off to a different screen), then waits for
Mob.Sender to commit. All three are needed: the owner forwards the event,
the screen builds the tree, and the sender commits it — so a drained owner
mailbox alone does not mean the frame has been rendered.
Use after any fire-and-forget call (tap/2, back/1, send_message/2)
before reading the native side with view_tree/1, screenshot/2, tap_id/2
or element_frames/2.
Mob.Test.tap(node, :save)
Mob.Test.settle(node)
Mob.Test.view_tree(node)
Swipe from (x1, y1) to (x2, y2). Drives UIScrollView contentOffset on simulator; synthesises a drag gesture on real device.
Mob.Test.swipe(node, 195.0, 500.0, 195.0, 100.0) # scroll down
Switch to a named tab stack. Synchronous.
Pass transition: :push, transition: :pop, or transition: :reset to
exercise the same directional animation as Mob.Socket.switch_tab/3.
mount_params: %{...} is passed to a target root only on its first mount.
Send a tap event to the current screen by tag atom.
The tag comes from on_tap: {self(), :tag_atom} in the screen's render/1.
Check the screen's render function to find available tags.
Fire-and-forget — does not wait for the screen to finish processing. Follow
with settle/2 before reading the native side.
Mob.Test.tap(node, :save)
Mob.Test.tap(node, :open_detail)
Tap the element with id at the center of its tracked frame — driving by id
without a screenshot or coordinate guess. The element must carry an :id
(see element_frames/1).
Mob.Test.tap_id(node, "save")Inherits tap_xy/3's return contract, including its platform limits — read
those before treating a non-:ok result as a test failure.
Locate an element and tap it via the simulator's native UI mechanism.
Requires idb (iOS) to be installed. Exercises the full native gesture path
rather than sending a BEAM message — useful for testing gesture recognizers
or verifying that the native layer wired up the tap handler correctly.
Prefer tap/2 for testing Elixir logic; use tap_native/1 when you need
the native path.
Mob.Test.tap_native("Save") # by visible text
Mob.Test.tap_native(:save) # by accessibility_id (= tag atom name)
@spec tap_xy(node(), number(), number()) :: :ok | {:error, :no_view_at_point | :no_element_at_point | :no_effect | term()}
Tap at screen coordinates on the native app.
Mob.Test.tap_xy(node, 289.7, 518.8)Return values
:ok means the app reacted — an event reached the BEAM within 300ms of
the tap. Every other outcome is an error tuple; there is no "probably worked".
| Value | Meaning |
|---|---|
:ok | A tap/focus/change/submit/select event reached the BEAM. |
{:error, :no_view_at_point} | Hit-test found nothing — the coordinate is outside every visible window. |
{:error, :no_element_at_point} | iOS simulator only: a view is there but no accessibility element to activate. |
{:error, :no_effect} | Input was accepted by the OS but no handler ran. |
{:error, probe} | iOS device only: the private injection API is missing; probe lists which selectors resolved. |
Real capability per platform — read before trusting a result
- iOS simulator — activates the accessibility element under the point.
That works for
Button, and for text fields (the responder chain is walked to focus them). It does not work for Mob'sBox/Row/Columnwithon_tap:: SwiftUI gives a plain.onTapGestureno accessibility action, so activation is accepted and the handler never runs. ABoxwithaccessibility_role: "button"is a real AX element (.isButton) since #94, but it still has no activate action, so the result is the same. Those taps return{:error, :no_effect}. Usetap/2(by tag) to drive them. - iOS physical device — synthesises an
IOHIDEvent. As of iOS 26.5 UIKit accepts the event and delivers no touch, so this returns{:error, :no_effect}for every coordinate. Treat coordinate tapping as not working on device and usetap/2. Seedecisions/2026-08-09-ios-device-tap-injection-has-no-effect.md. - Android — not routed through this function;
adb shell input tapworks and is what the tooling uses.
:no_effect in sidecar mode
The check is "did an event reach the BEAM", so it only sees handlers Mob owns.
Driving a non-Mob app (sidecar mode), a genuinely successful tap still reports
{:error, :no_effect} because there is nothing for the NIF to observe.
Confirm those with ui_tree/1 or a screenshot instead.
The counter behind the check is process-wide, not per-tap: any Mob event that
reaches the BEAM inside the 300ms settle window (a scroll notification, a
timer-driven change, another handler) counts as the tap's effect and can
turn a miss into a false :ok. The check assumes a serial harness — one
synthetic interaction in flight at a time, no concurrent UI activity.
Toggle a switch by a label substring. SwiftUI exposes Toggle as a button
with an empty accessibility label and value "0" or "1" — so we find the
Text element matching label_match, then activate the next button below it.
Mob.Test.toggle(node, "Notifications")Known limitation (issues.md #8)
Mob's iOS Toggle component does not currently surface its label: prop as
a separate :text AX element, so find_label_y/2 returns
{:error, :label_not_found}. Workaround: use ax_action_at_xy/4 directly
with the toggle's frame from ui_tree/1 (filter for :button with value
"0" or "1"). Once issue #8 lands, this helper works as documented.
Return the current rendered tree (calls render/1 on the live assigns).
Type text into the currently focused text field.
Tap the field first to give it focus, then call this function.
Mob.Test.tap_xy(node, 195.0, 300.0)
Process.sleep(100)
Mob.Test.type_text(node, "hello@example.com")
Return the live accessibility tree from the running native app.
Each element is a tuple: {type, label, value, {x, y, w, h}}
Mob.Test.ui_tree(node)
#=> [{:button, "Increment", "", {164.0, 400.0, 54.0, 54.0}}, ...]
Return the live UI tree as a nested map, walking native views directly.
Unlike ui_tree/1 (which uses the accessibility subsystem and requires
VoiceOver activation on iOS), this walks UIView/View hierarchies directly:
no AX activation needed.
Coverage caveat
- UIKit apps (sidecar mode): full UIView hierarchy with labels and frames.
- SwiftUI apps (current Mob): shallow — SwiftUI doesn't expose its content
as separate UIView instances under the hosting view. You'll see containers
and scroll views but not individual buttons/text. For Mob apps, prefer
Mob.Test.tree/1(the logical render tree, which has all the semantic info) orMob.Test.ui_tree/1(AX walk, requires VoiceOver activation). - Android (planned): a registry populated via
onGloballyPositionedin Mob's Compose components — seefuture_developments.md"WireTap" section.
Returns a nested map:
%{
type: :root, class: nil, label: nil, value: nil,
frame: {0.0, 0.0, 393.0, 852.0},
bg_color: nil, text_color: nil,
children: [
%{type: :window, class: "UIWindow", ..., children: [
%{type: :scroll, ..., children: [
%{type: :button, class: "SwiftUI.CGDrawingView", label: "Roll Dice",
frame: {24.0, 416.0, 327.0, 53.5},
bg_color: 0xFF2196F3, text_color: 0xFFFFFFFF, children: []}
]}
]}
]
}:class is the concrete native view class. On SwiftUI it is usually the only
thing that identifies a node — :type collapses anything it doesn't recognise
to :view — and it's what tells you which renderer drew a node when a colour
comes back nil.
Colours
:bg_color and :text_color are the colours the view actually painted,
as 0xAARRGGBB integers — the same representation component props use
(guides/theming.md). nil means nothing paintable was found, or the colour
has no single RGBA value (a multi-stop gradient, a pattern fill).
UIKit puts colour on the view (UIView.backgroundColor, UILabel.textColor).
SwiftUI mostly does not — .background(Color, in: shape) and
.foregroundColor, which is what Mob's renderer uses for every Box and Text,
go through SwiftUI's own renderer and land on a CALayer (typically a
CAShapeLayer fill) under a structural view whose own backgroundColor stays
nil. So each node also harvests from its own layer subtree, excluding layers
owned by its subviews so a container never claims a child's paint.
Sources consulted per node, first match wins:
| Background | Text | |
|---|---|---|
| view | UIView.backgroundColor | UILabel/UITextField/UITextView/UIButton |
| layer subtree | CAShapeLayer.fillColor, single-stop CAGradientLayer, CALayer.backgroundColor | CATextLayer.foregroundColor |
Fully-transparent colours are treated as no colour, so a Color.clear
placeholder doesn't read as "painted black at alpha 0".
Because these are read back off UIView/CALayer rather than echoed from the
render tree, they are the way to catch a styling regression where a theme or
modifier silently drops a colour Elixir sent. Compare against tree/1 (what
Elixir asked for) to see the two diverge.
If colours come back nil across the board, don't guess at the reason —
call paint_debug/1, which reports which view/layer classes the renderer
produced and which colour properties they actually set. On iOS 26 SwiftUI that
is the expected outcome, and sample_color/2 (real pixels) is the way to
verify a drawn colour.
On Android, the JSON returned by mob_nif:ui_view_tree/0 is decoded here —
but no shipped MobBridge.kt implements uiViewTree(), so today Android
returns {:error, :not_loaded}.
@spec view_tree_flat(node()) :: [{[non_neg_integer()], map()}]
Return the view tree flattened to a list of {path, node} tuples.
path is the list of child indices from the root — e.g. [0, 2, 1] is
"the second child of the third child of the first child of the root."
Useful for filter/find — see find_view/2.
Mob.Test.view_tree_flat(node)
#=> [
# {[], %{type: :root, ...}},
# {[0], %{type: :window, ...}},
# {[0, 0], %{type: :scroll, ...}},
# ...
# ]
Wait until predicate returns true when called with the current ui_tree,
polling every interval_ms until timeout_ms elapses.
Mob.Test.wait_for(node, fn tree ->
Enum.any?(tree, fn {_, label, _, _} -> label == "Success" end)
end)
Wait until an element whose label or value contains text appears in the
accessibility tree.
Mob.Test.wait_for_text(node, "Welcome")
Mob.Test.wait_for_text(node, "Error", timeout_ms: 2000)