--- name: gripui-cli identifier: dosaygo-gripui-cli description: Use the `gripui` CLI (aliases: grip, gu) to browse, search, inspect, and act on live web pages from an agent workflow. Prefer this skill when a task needs browser state, web navigation, stable refs, form/text interaction, page text, or inspectable actions instead of screenshots. --- # gripui CLI SKILL A real, persistent, logged-in browser with stable numbered action refs. You **perceive** the live page, **decide**, and **act** — you are not writing brittle automation scripts. Treat every site as a problem-solving domain and the `gripui` CLI (aliases: `grip`, `gu`) as your toolkit. This skill is organized as: - **The loop** — hud, act, hud. This is everything. - **Don't panic** — state annotations are informational; snap/eval are last resorts. - **Mindset** — exploration over optimization; mistakes are recoverable. - **Discipline checkpoint** — anti-patterns and the cadence rule. - **Perceiving** — hud, find, read, and how to find what's not visible. - **Acting** — clicks, text, forms, selects, files. - **Obstacles** — overlays, blocked states, auth, getting unstuck, and reporting tool issues (see "When you're stuck" and `report-issue`). - **Context** — frames, tabs, profiles, waiting, recovery. - **Reference** — full command list, shell composition, diagnostics. --- ## The loop ```sh gripui hud # look gripui do # act gripui hud # look again ``` **This gets you through 99.95% of every situation.** Everything else in this skill is elaboration on what to do when the loop surfaces something unexpected — but the loop itself never stops. When in doubt: hud. Just acted: hud. Not sure what happened: hud. Trust the tooling. Trust the loop. **Note on SKILL versions**: you might have an outdated copy of gripui's SKILL.md but an up-to-date copy of the binary tool. Run `gripui teach --force` to overwrite agent relative skill files for gripui plus a current directory SKILL.md. Or use `gripui skill-md` to output the current binary's version. **ALWAYS: If you don't understand a command, or it isn't behaving the way you expect, run `gripui --help` before guessing.** It is the fastest way to see the exact flags and arguments a command accepts — and which it doesn't. Do not assume a flag exists; check. (For example, there is no `--tab` flag: every command always acts on the active tab.). If a page or situation is confusing or delaying you, read SKILL.md for guidance. --- ## Don't panic **Inspect output annotations are descriptive state, not alarms.** `[invalid]`, `[selected]`, `[unchecked]`, `[offscreen]`, `[error: "..."]` describe what the page currently is. React to them only when they persist after a deliberate action that should have changed them (a blur, a submit, a continue click). A mid-form `[invalid]` on an unfilled field just means the field is waiting for input — not that anything went wrong. **A click failure or unexpected result is not a crisis.** Re-hud. The page will tell you what changed (or didn't). One refresh the HUD resolves most apparent failures. **`gripui snap` and `gripui eval` have a high bar.** Reach for them only when: - you have refreshed the HUD at least once after the failure, and - the state is genuinely unchanged, and - you can state clearly why normal hud/do cannot proceed. If you haven't refreshed the HUD since the last failure, do that first. You haven't earned snap or eval yet. --- ## Mindset: explore, don't optimize Web navigation is inherently exploratory. Humans click the wrong link, back up, try a different path — and that's completely fine. The `gripui` CLI gives you the same freedom. **A wrong click is not a failure.** It costs one action and an hud to recover. Navigate back, try something else, scroll somewhere new. The page will tell you where you are. **You don't need the shortest path.** First contact with an unfamiliar site is always discovery. Try something reasonable, see what happens, adjust. An extra hud or a detour is far cheaper than paralysis from over-planning. **Uncertainty is a reason to act, not to wait.** If you're not sure which element to click, click a plausible one and refresh the HUD. The result is almost always more informative than reasoning about it from context alone. **The goal is the north star; the path is flexible.** A blocked step is not a blocked goal — most goals have more than one route. Try a different entry point, a different order, or a different part of the task while the stuck part waits. `hud` shows you what's actually reachable right now; act on those targets. Keep moving toward the goal from wherever you can. The discipline in this skill is about avoiding specific *tool misuse patterns* — not about being perfect. Navigation mistakes are normal and recoverable; the anti-patterns below are different because they bypass the page's own feedback mechanisms. --- ## Discipline checkpoint **If you notice yourself doing any of the following, stop and return to the core loop.** | Anti-pattern | Why it breaks | Correct move | |---|---|---| | `gripui read --full \| grep …` to find elements | Misses overlays, loading states, errors | Use `gripui hud` or `gripui find` | | `gripui eval` to click, type, or interact | `eval` is diagnostics only — never interaction | Use `gripui do`, `gripui type`, `gripui click` | | Chaining actions with `&&` | Ref epochs expire; second command acts on stale refs | One action → one hud → next action | | Navigating away from a CAPTCHA or bot check | Silently abandons the site; leaves the task incomplete | Call `gripui human-drives` immediately, explain, wait | | `gripui find` before the first `gripui hud` in a session | Skips full orientation; misses overlays and frame structure | Always `gripui hud` raw first | | Escalating to `gripui eval` because `find`/`do` didn't work | Eval can't recover from a wrong strategy | Work the obstacle ladder in **Obstacles** instead | | Ignoring the "active layer detected" hint in hud | The foreground panel/modal has different refs than the base page | Run `gripui hud --layer active` immediately | | Using `gripui snap` before `gripui hud --layer active` | Snap bypasses the ref system; layer hud is faster and gives actionable refs | `gripui hud --layer active` first | | `gripui go` or `gripui tab close` right after `gripui do` / submit / delete | Leaving the tab can miss async outcomes | `gripui hud` first; `gripui go` / `gripui tab close` may block once with full hud (exit 1) — repeat to proceed, or `--force` / `--skip-exit-check`. Tab **switch** is not gated. | | Jumping to a deep page via a URL recalled from training or assembled by pattern | Session not established; URL is likely stale; looks like bot traffic to the site | Navigate from the site's root or a search engine; use the site's own UI to reach deep pages | **Active-layer rule:** an expanded choice popup is an active action surface even if it covers no other control. Persistent navigation is not a blocking overlay. When hud output shows `Hint: active layer detected`, run `gripui hud --layer active` before your next `gripui do`. The hint identifies an expanded choice popup or a layer that occludes an actionable target. Other structural layers remain normal UI and do not require dismissal. **Cadence rule:** every ~5 actions, pause and ask: *when did I last hud?* If you've lost track, hud now. Re-reading the page is never wasted — acting on a stale or misread page is. **Anchor rule:** if you are spiralling — repeating commands, piping hud through grep, expecting something you can't find, feeling uncertain about the page state — stop. Run `gripui hud` with no flags, no grep, no piping. Read the full output. Always orient with an hud and read it in full before acting. `gripui hud` is your anchor; return to it whenever the situation is unclear. **Post-action rule (look before you cross the road):** after any click, submit, delete, or other action that changes page state, run `gripui hud` before `gripui go` or `gripui tab close` on that tab. The CLI blocks the first `go`/`close` attempt with a full hud preview (exit 1); repeating proceeds if you have seen it. Tab **switch** is not gated — rotate freely, but still hud each tab before meaningful work. Use `--force` or `--skip-exit-check` when you have already verified the outcome. **Presentation:** when an active layer is present (modal/dropdown/panel), `hud` automatically hides occluded elements from the ref list and shows a count hint — use `--show-occluded` to restore them. Pass `--hide-occluded` globally on `hud`/`find` to always suppress occluded refs; refs still exist for `gripui do` if you already know the number. **Navigation rule:** Never jump directly to a deep page using a URL recalled from training or assembled by pattern — search results, checkout, account areas, booking steps. Navigate from the site's root or a search engine, then click toward your goal using the site's own UI. A URL the site's own nav produced is reliable; one you assembled from memory is a guess — and it reads as bot traffic to site defenses. Exception: a URL the user gave you in their message, or one you navigated to yourself in this session. --- ## Core loop **Look → act → refresh the HUD. Be a human at a browser.** Don't see what you want? ```sh gripui scroll # page down — most things are just below the fold gripui scroll until text:"..." # or scan straight to what you're after gripui scroll --anchor # move a specific panel (ref onscreen) gripui hud # look again ``` **Scroll first, then hud.** Most things are below the fold, and `gripui scroll` is a trusted wheel that just works — lead with it. `--ignore-viewport` is a last resort, not a first move. **Efficiency is successful completion, not minimum commands.** On dynamic pages, re-reading after meaningful state changes is often faster than trying long chains against stale refs. **Complex widgets need room.** For large calendars, maps, editors, drag/drop surfaces, or multi-panel pickers, expand the browser viewport before interacting so the whole widget stays visible while the pointer moves and the page updates. After resizing, verify the viewport actually changed and refresh the HUD so refs are regenerated before using the widget. ### Session start checklist — every session, before your first action ``` -1. gripui status — confirm the running browser matches the owner's chosen headless or visible mode (see "Headless and visible sessions"). 0. gripui resize WxH --move-to XxY — position window if recording or demoing (as needed). gripui set demo-mode — when visual click tracing is wanted; confirm with `gripui status`. 1. gripui hud — raw, no grep. Orient completely. 2. gripui tab list — confirm which tab you are targeting. 3. gripui frame list — note which frames are present (`fN` labels stay stable for the same iframe). 4. Dismiss blocking overlays — structural panels/layers that occlude no action are normal UI. 5. Re-read Discipline checkpoint — confirm you know the anti-patterns before acting. 6. Settle handoff availability — if the task could hit a login/CAPTCHA/MFA/payment gate, ask the human NOW whether they'll be on hand to clear it, or whether to skip the parts that need a handoff (see "Settle handoff availability up front"). 7. Opening a site? Arrive via DuckDuckGo search → click the result. Never `go` straight to a remembered login/dashboard URL (see "Navigation philosophy"). ``` Skipping this is the single biggest cause of wasted actions. It costs 4 commands and saves 40. ### Refs, contexts, and sheet epochs **Single context rule:** you are always in one `(tab, frame)` context. Refs are valid **only** in that context (`t2:top`, `t2:f1`, …). Ref numbers are unique within a tab across all frames (tab-wide monotonic allocator), but you still act only in the current frame context. Tab switch restores that tab's stored refs/sheet — not a global wipe. Acting on a ref from another tab or frame fails with an honest wrong-context error. **OODA loop (no cross-frame magic):** 1. **Orient** — `gripui tab list`, `gripui frame list` / hud Frames section 2. **Decide** — `gripui frame switch fN` if the controls live in an iframe 3. **Observe** — `gripui hud` or `gripui find` in the **current** context 4. **Act** — `gripui do N` / `click N` using numbers from that observation 5. **Re-orient** after navigation, tab switch, or frame auto-drop (informational exit + `ok: false` means context reset to top — refresh the HUD before acting) **A ref number is tied to an element's identity, not its position.** Bare integers (`10`, not `r10`) are canonical element refs in the current context — `click`, `do`, `type`, and `focus` all resolve the same target. `do`, `click`, `find`, and `hud` all draw from one pool: the number `find` prints for an element is the same number `hud` prints, and the same number `do`/`click` accept. A given element keeps its `[N]` across HUD refreshes **and across unrelated page mutations** — open a dropdown, scroll, let a sibling appear, and the elements that survive keep the numbers they already had. Numbers are not contiguous and don't start at 1; that's expected. State/recovery actions (dialogs, file choosers) use a separate `s` namespace (`[s1] Press Escape`); tabs use `t`. These never collide with page refs. `hud` keeps the **same sheet id** and exits zero when the actionable set is unchanged, so `gripui hud && …` won't break just because you looked twice. The sheet epoch bumps — and the command returns a **non-zero exit** — only when the set of actionable elements genuinely changes (a ref you could have been holding was added or removed): ``` ↳ sheet s104 → s110 (action set changed; surviving refs retained) ↳ exit: non-zero because available actions changed; use the refs shown above for the next action ``` A completed action HUD acknowledges the preceding mutation even when it exits nonzero for changed actions or page warnings. Repeating a HUD is a valid new observation; it does not imply shell filtering. This is the guardrail for `&&` chains: a real change stops the chain before it acts on a stale map. It is honest now — a surviving element is never renumbered, so the notice reports an action-set change, not renumbering. A surfaced ref is always usable on your next command; if an element has genuinely detached, resolving it fails loudly ("refresh the HUD") rather than acting on the wrong node. If a flow still behaves strangely, fall back to the strict look → act → refresh the HUD loop. ### Closed shadow roots Dub already traverses open shadow roots. If a control is absent because a site put it in a **closed** shadow root, opt in to the compatibility shim, then reload the page: ```sh gripui set open-only-light gripui reload ``` This changes the page's shadow-root behavior and can break a site, so it is a targeted recovery tool rather than a default. After every click, **read the confirmation label**: ``` ok clicked tab=5 Settings ``` If it doesn't match what you intended, you clicked the wrong element. Stop, refresh the HUD, and correct before continuing. The label says *what* was clicked; the epoch notice says *the map changed* — two signals for the same event. > **Tip — refresh the HUD to see what *changed*, not to renumber what didn't.** > Refs are stable per element identity, so refreshing the HUD is about discovering > what the last action did to the page — new controls, dismissed overlays, > validation errors — not about refreshing numbers that are already valid. The > loop is still *one action → refresh the HUD → next action*: the exit code tells > you honestly whether the actionable set changed (non-zero, refresh the HUD before > chaining) or held (zero, act immediately). Trust that signal instead of > assuming either stability or expiry. --- ## Perceiving the page ### hud, find, and grep **`gripui hud` is the complete map of everything actionable on the page.** It is the default: use it when orienting, after navigating, when overlays or errors might be present, or whenever you haven't seen this page this session. Nothing replaces it for first contact. Action lines can include `context="…"`: nearby rendered row/card/field text that helps distinguish controls such as two “Get started” buttons. The action's own name remains separate. This context also appears in `form`, `find`, and `observe`; `find` and `form` can match it directly. Ambiguous surrounding text is omitted. JSON exposes the evidence under `surface.context` (text, source, selector, and optional parent scopes); context does not change a control's ref or activation target. `surface.context_status` explains selected evidence or abstention (`ambiguous`, `boundary`, `budget-exhausted`, `no-evidence`); `describe` also exposes it. HUD and form share control naming, with entered/selected values kept separate. Tree rows can summarize their own members without asserting an action-to-label association. Native disclosures appear as summary actions with `[collapsed]` or `[expanded]`; controls inside a closed disclosure appear after opening it. **Read the whole hud.** Page actions often follow the Tabs section, so a filter that stops at Tabs can make a working page look empty. Capture stderr too (`gripui hud 2>&1`): blocked states such as a pending file chooser are reported there. HUD and find show one primary action for observed aliases, such as a native label and its control. The reachable surface wins; matching text or overlapping rectangles alone do not merge actions. `hud --explode-pairs` shows the individual surfaces, and their existing refs remain usable. JSON `surface.presentation_owner` identifies the observed activation owner; surrounding `context` identifies the record or group independently of how many actions it contains. `hud` includes a conservative Chrome/CDP listener supplement by default in standard builds: controls discovered from registered click/pointer activation listeners join the same globally ordered action pool as baseline controls. Document/tree position decides where each action appears; provenance only breaks exact ordering ties. Delegation expands only canonical collection boundaries (lists, table rows/cells, and ARIA listbox/menu/tablist/tree/radiogroup children); it does not infer generic cards or treat cursor styling as actionability. Listener controls use the same text, surface, shape, state, and annotation metadata as baseline refs. Normal hud discovers listener controls in the current viewport; pass `--ignore-viewport` to include offscreen controls. The default presentation limits are 250 viewport actions and 1,000 document actions. `--limit N` changes only that presentation limit; `--unlimited` shows every discovered action. Baseline and listener discovery share one 100,000-action safety ceiling. Firefox/BiDi keeps the baseline sheet and reports listener discovery as unsupported. Disable it with `gripui unset listener-actions` or re-enable it with `gripui set listener-actions`; explicit settings persist across restarts. Existing profiles without this setting adopt the enabled default. Builds without the `cdp-listener-actions` feature default to off. **`gripui find` + `gripui do` is the fast path once you know a label** — usually quicker than a full hud for navigation: ```sh gripui find "Create" # locate by label (a different path than hud) gripui do 2 # act gripui find "Submit" --all # --all when there may be duplicates ``` After `find`, use **only** the refs that command returned — never mix them with refs from a prior hud. In `&&`-chained commands the first command's refs are already stale by the time the second runs. **Grepping hud output trades safety for brevity.** It shows only what you already expected — you miss overlays, loading states, unexpected controls, and errors that would change your next move, and one blind spot compounds into several wrong actions. Reach for it only when: - you already have a full hud of this same page load in context, and just need the current ref number for a known element, or - the page is large, stable, and familiar (repeat visit, no loading, no overlays) and you know exactly what label you want, or - context is tight and preserving it outweighs the orientation you'd gain. ```sh gripui hud | grep -i "submit" # acceptable only when already oriented gripui hud # after ANY grep, your next hud is raw ``` **The discipline:** never open a session with grep, never grep right after navigation. It's a focused second look, never a substitute for the first. ### Reading text ```sh gripui read # visible text gripui read --full # whole document gripui read --headings | --links # structure or link list ``` `read --main` identifies a region, not all page content. Its default scope is the viewport; output exposes `content_scope` and `loaded_content_only`. Use `read --main --ignore-viewport` for that region's currently rendered content, or read a specific observed region/selector with `--full`. Lazy-loaded comments may require scrolling first; no read mode silently loads all comments. ### When hud doesn't surface what you need Escalate in order; stop as soon as something appears: ```sh gripui hud --layer active # 1. FIRST if hud showed "active layer detected" gripui find "Submit" --all # 2. text search — different path than hud gripui hover # 3. hover a ref near where it should be … gripui scroll && gripui hud # 4. … scroll that panel and refresh the HUD gripui frame list # 5. is it inside a frame? gripui frame switch f1 # 6. switch and retry gripui hud --ignore-viewport # 7. last resort — offscreen elements ``` **`--layer active` is step 1, not step 6.** If hud reports an active layer, the action you want is almost certainly in that layer, not in the base page. Skipping this step and going straight to `find` or `snap` costs 5–10 wasted actions. **No find matches + frames visible = switch the frame first.** Content inside an iframe is invisible to `find` on the top frame. Bare numbers are refs; force text matching when a label *is* a number: ```sh gripui click text:"8" ``` ### Scrolling **`gripui scroll` just works.** It's a trusted CDP/BiDi wheel — the same gesture a real mouse fires. Don't overthink it; reach for it the moment something might be below the fold. ```sh gripui scroll # page down (gripui scroll up / left / right) gripui scroll --anchor 44 # wheel the panel ref 44 lives in — ref must be onscreen gripui scroll --anchor 44 until "New York" # scan inside that panel (listbox, virtual list) gripui scroll --anchor 44 --step 0.25 # finer wheel steps inside a small panel gripui scroll until text:"Submit" # page scan — returns ref for match found gripui scroll up until 3314 # same, upward gripui scroll --next until "Next" # skip current match, find the next one down gripui press ArrowDown # fine step inside a focused combobox/listbox gripui hud # refresh the HUD after scrolling — refs/layout may change ``` **To move a specific panel, name it: `gripui scroll --anchor `.** Anchor an *onscreen* ref **inside** the region you want to move (a visible listbox option, not the combobox trigger) — the wheel fires there and scroll-chaining moves that container. **Targeted reveal centers by default.** Centering avoids placing controls under sticky headers and footers. Use `--edge` only when nearest-edge placement is intentional; both modes verify hittability and make bounded attempts to move the target away from an occluder. ```sh gripui reveal 42 # centered, then checked for hittability gripui reveal 42 --edge # nearest edge, then checked and corrected if occluded ``` If bounded mitigation still leaves the target covered, use `gripui scroll --step 0.1` to position it manually. For a scrollable panel, anchor the wheel to an onscreen ref inside it: `gripui scroll --anchor --step 0.1`. **To find an item inside a panel: `gripui scroll --anchor until `.** Search stays inside the anchor's scroll host; prefers `role=option` matches so a city field with the same text doesn't false-positive. Combine with `--step 0.25` (or lower) when wheel steps jump too far; use `gripui press ArrowDown` / `ArrowUp` for one-item precision. **Shadow DOM / web components:** scoped scroll pierces open shadow roots (same as hud). Prefer a visible option ref inside the listbox; when hud surfaces only the listbox carrier (`#host >> #state-list`), anchoring that listbox ref also works for `until`. **Page scan: `gripui scroll until `.** Without `--anchor`, scans the page. Probes the viewport at each scroll step; elements with a `position:fixed` ancestor are excluded (they're chrome, never reachable by scrolling). On success, outputs the ref for the matched element — use it directly without a follow-up `gripui hud`. On exhaustion or occlusion, outputs an hud of the current page state so you can see where you ended up. Use `gripui scroll --next until ` to skip the current visible match and continue to the next occurrence (Ctrl-F / vim `n` style). Bare `gripui scroll` scrolls the page. If a listbox is open, feedback points you to `--anchor`. --- ## Acting on the page ### Clicks and refs ```sh gripui do ["text"] # run action n; optionally supply text/option value gripui click [--new] [--follow] # click a ref or a label; --new opens a new tab ``` Prefer `gripui do ` for hud refs — it takes a non-selector path that is more robust than label clicks. **Text-query disambiguation.** `gripui click ` returns matching refs when more than one element matches. Use this as a fast disambiguation flow when you know what you're looking for: ```sh gripui click Issues # returns all matching refs if ambiguous gripui do 5 # pick the right one ``` This is often faster than a full hud when the label is known. `--new` works with text queries too: `gripui click Issues --new` opens the best match in a new tab. `--new` reports the tab it opened on its own line (`new tab: t8 (background)`, or `no new tab` when the link stayed in place); JSON carries `new_tab` and `followed`. Add `--follow` to switch to that tab in the same command (`new tab: t8 (active)`), instead of a separate `tab list` and `tab switch`. ### Text input ```sh gripui type 3 "hello" # type into ref 3 (optional ref) — key events; clears the field first gripui type "hello" # no ref → types into the focused field gripui say "long text" # paste into the FOCUSED field (never takes a ref); clears first gripui type 3 "x" --no-clear # append instead of clearing first (works on type and say) gripui clear # clear the focused field gripui focus 3 # trusted CDP/BiDi click to activate ref 3 (same path as click/type) ``` **Activation is always trusted input.** `click`, `do`, `type `, and `focus` all resolve the ref, compute a click point (JS diagnostics only), then Rust dispatches real CDP/BiDi mouse events. No programmatic `element.focus()` or `element.click()` on user paths. `say` never activates — it pastes into whatever the browser already has focused. **`--try-anyway`** on `click`, `do`, `type `, and `focus`: when geometry preflight fails (e.g. `zero_size` on iframe fields with odd CSS wrappers), retry with the flag to bypass the preflight gate and still dispatch a trusted click at the element's bounding-box center. Costs: you opted out of the tool's hittability opinion — the click may miss or focus may land on a related control (e.g. label → input). `say` has no `--try-anyway`; establish focus with `focus`/`type`/`click` first. Failure hint on `zero_size` mentions this flag. **`type` and `say` clear the target field before writing, by default.** Typing into a pre-filled field used to append/merge unpredictably; now you don't need a separate `gripui clear` first. Pass `--no-clear` when you want to append to existing content. `gripui type ` targets a specific field; with no ref (and `gripui say` always) it writes into whatever is focused. **Pipe a file or heredoc into a field with `-`.** Anywhere text is expected, pass `-` to read it from stdin (read to EOF; a single trailing newline is trimmed, interior newlines are preserved): ```sh cat body.md | gripui do 4 - # type a file into the hud ref's field echo "$TITLE" | gripui say - # paste piped text into the focused field gripui type 3 - <<'EOF' # heredoc into a ref multi-line value stays intact EOF ``` `-` is only special when it is the *entire* value — `gripui say "a - b"` types a literal hyphen. **`gripui say` does not accept a ref.** It pastes text into whatever element is currently focused/active. Click to focus first if needed. `gripui type` optionally takes a ref to target a field; `gripui say` never does — paste only, into focus. `type` works without a ref when a field is already focused; `say` always uses the focused field. To empty a field manually use `gripui clear`; `ctrl+a`/`cmd+a` are unreliable in browser context (rich-text editors are the exception; see Resistant fields). Note you rarely need a manual `gripui clear` anymore — `type`/`say` clear the field first by default (use `--no-clear` to append). **Focus hygiene:** before typing into a filter or search box, check `gripui status` for `activeElement=` and heed `warn=active element differs from ref target` on `gripui type` output — when it appears, keystrokes may have gone elsewhere. The warning is suppressed for expected cases (combobox container gaining focus while an inner input was targeted; shadow-host/shadow-child focus transfer), so when it does appear it signals genuine drift. `gripui press Tab` can move focus; otherwise dismiss overlays, click the field, refresh the HUD. `gripui type ` and `gripui do ` use the same click-then-type path. **Typed text is verified.** `type`, `say`, `do ` and `clear` anchor the target field before the first key and read it back afterwards. Some UIs replace the input on every keystroke and move focus to a dialog; gripui follows the field across those re-renders, refocuses it, re-types what was lost and removes its own activation Space there. The output says what happened: `repaired: field re-rendered Nx, refocused Nx, re-typed N char(s)` means the text now landed intact; `warn=field value differs from typed text: ...` means the field rewrote the value itself (masks, formatters, chip inputs) — check it. If the text cannot be made to land, the command fails instead of reporting success. `landed in "": the page moved focus to that field` means a click opened a search or picker dialog whose own input took the text (Google Flights' "Where from?") — that is success; pick a suggestion from `hud --layer active` next. Values in `hud`/`form` mark edge whitespace as ``, like `doctor` does. **Native date and time inputs** (`type=date`, `datetime-local`, `month`, `week`, `time`) are segmented, not text. Give `type`/`do` an ISO value — `2027-10-18`, `2027-10-18T14:30`, `2027-10`, `2027-W42`, `14:30` — or a date in the field's displayed order (`10/18/2027` in en-US). gripui fills each segment in the browser locale's order and verifies the value (`strategy=date_segments`). Custom calendar widgets (flight and hotel sites) are not native inputs: open the picker and click the day cells, whose names read like `Thursday, November 12, 2026`. Rich-text editors are cleared through scoped native deletion, not DOM replacement. They do not receive the ordinary input's synthetic Space/Backspace activation (unless `--keep-final-space` explicitly asks for a final space). Multiline text preserves paragraphs; newline key events require an observed multiline field and never implicitly submit a single-line input. `clear` verifies the anchored field after the page has had a chance to restore its value. A restored or rejecting field fails with a diagnostic hint; it is not reported as `len=0`. Modifier chords such as `Meta+v` do not insert a bare `v`. **Comboboxes / custom dropdowns:** open, then hover-and-scroll to reveal options rather than trusting an in-list filter (filters are sometimes scoped to the current selection). Readonly inputs are `click` actions: `gripui do ` focuses or opens the field (for example, a calendar). Readonly alone does not establish that a calendar exists. Field errors appear beside their refs when native validation, ARIA message links, or an unambiguous local alert supplies evidence. Fix the indicated field and refresh after submitting; displayed text alone does not establish validity. Choice fields carry `opens=choices`, `[expanded]` / `[collapsed]`, and, when separately rendered, `display="…"`. The display can be a selected value or an empty-field prompt; opening a menu or highlighting an option does not commit a selection. A `click` ref can represent a select-only input: use `gripui do ` to open its visible field even when it is annotated `[muted] [transparent]`. Editable choice inputs remain `type` actions; use `gripui click ` to open without typing. Popup options carry their controlling field's context when the page supplies an unambiguous relationship. Refresh HUD after selecting to see dependent category/access fields and changed values. ```sh gripui do 23 # open the combobox gripui hud --layer active # grab option ref, or the listbox carrier in shadow gripui scroll --anchor 24 # coarse move inside the panel gripui scroll --anchor 24 until "Target" # or scan inside the panel gripui press ArrowDown # fine step if wheel jumps too far gripui hud gripui do 30 # select your target ``` ### Forms and selects ```sh gripui form # all fields, current values, and errors gripui form | grep -F "[error:" # validation errors only gripui doctor # why THIS field is rejecting input gripui type 4 "value" gripui choose 7 "Option name" # native ` and `datalist` options. - **the browser's own verdict** — `validity.*` flags and `validationMessage`, which catch constraints no attribute shows, such as `setCustomValidity`. - **whether a declared `pattern` is actually enforced.** HTML compiles `pattern` with the `v` flag, stricter than `u`; a pattern that fails to compile is silently *ignored*, so the field looks validated and accepts anything. Nothing else surfaces this. - **page error text** from `aria-errormessage`, `aria-describedby` and live alerts. - **other invalid fields in the same form** — a submit usually fails because of a field you were not looking at. - **whether an edit is staged but uncommitted**, naming the commit control. That last one matters for a specific trap: **a reload is not a verification step for a staged form.** Reloading discards pending edits, so "it reverted after I reloaded" and "the server rejected it" look identical. Run `doctor` before concluding a write failed. ```sh gripui type 12 "example.com" gripui doctor 12 # value, constraints, verdict, uncommitted edits ``` ### When you're stuck **First, notice it.** Signs: identical hud output twice, the same action failing repeatedly, rotating tabs without advancing any, rising uncertainty about what to try. The moment you spot one — stop and reorient. Then work this escalation, stopping as soon as something moves: 1. **Re-read this SKILL.md** — the relevant section, properly, not skimmed. 2. **`gripui --help` or `gripui --help`** — a forgotten flag is often the unlock. 3. **Try an unused command** — `read --full`, `observe`, `hud --layer active`, `hud --ignore-viewport`, `describe `, `eval` for diagnostics. 4. **Do something seemingly random** — reload, navigate back and in, hover a different ref, scroll the other way, move the mouse. State changes unstick UIs. 5. **Stop and think** — re-read the last hud carefully. What does the page structure tell you? What would a human do here? 6. **If the problem appears to be in gripui itself** (command broken, behavior inconsistent with docs or ladders, missing capability, repeated failure after recovery steps, tool reliability issue): consider `gripui report-issue`. **Report-issue workflow.** If gripui appears to have a bug, missing capability, or reliability issue, consider opening an issue on the project repository. This helps improve the tool and provides maintainers with real-world failure cases. Depending on the environment, you may ask the user for approval before creating the issue. The important thing is making "report issue" part of the obstacle ladder rather than treating failures as dead ends. `gripui report-issue` stages a pre-filled report (with session context) under the hood; review with `gripui status` or the pending files if needed. By default `gripui report-issue` navigates to the GitHub new-issue URL **in the current gripui session** (same browser). Pass `--system-browser` to open it in your OS default browser instead: ```sh gripui report-issue # opens GitHub in the active gripui tab gripui report-issue --system-browser # opens GitHub in your system browser gripui report-issue --batch # opens all pending issues (gripui session) ``` **Dead-end pages — exit to the front door.** Two signals together mean the current page is a broken dead end: (1) a persistent `[alert]` or `[assertive]` ARIA message that survives actions and HUD refreshes, and (2) zero ROI after a retry — hud output unchanged after clicking and submitting. When both persist after one retry, stop spending actions on this path. The escape: click the site's main branded logo or home link — almost always top-left in the nav. This resets you to the front door, where the site's primary flows are present and working. Re-hud and take a fresh path toward your goal from there. ```sh gripui find "logo" # or: gripui click "home" gripui hud # you're at the front door; navigate normally from here ``` A synthetic deep-link URL with no prior session navigation is a third signal you may be in the wrong place. One retry on any error is always warranted. Two persistent signals after that retry is the exit cue. **Never loop.** Same action twice with no result → change approach, not repeat. Two tries is the limit; a third is a loop. **Keep a lab book when overwhelmed.** Juggling many tabs/profiles, write a scratch file and reason from it — it prevents re-running dead ends: ```sh echo "## Persistent overlay on checkout - Tried: Escape x2, surfaced close control, click blank space - Observed: re-mounts after every dismiss, z-index 9999 - Not tried: pointermove loop, hud --layer active, reload - Hypothesis: JS listener re-mounts on pointer events" >> /tmp/labbook.md ``` **Don't abandon the goal.** If the site is the specified destination, keep working it — different command, entry point, frame, active layer; advance another part of the task if one step is stuck. If the site is just one path to the goal and it's genuinely not yielding after recovery, switching to a different path is goal-following, not abandonment. Only stop when the human says to. --- ## Context: frames, tabs, profiles ### Frames ```sh gripui hud # read the "Frames (visible)" section gripui frame switch f1 # sticky — all commands now target f1 gripui hud gripui frame switch top ``` Many single-page and portal-style apps render their real content inside an iframe, so the top frame looks nearly empty. Inspect tells you what's in each frame before you switch: ``` Frames (visible): [f1] "content-frame" https://... ↳ "Account settings" — 40 buttons — e.g. Create, Manage ``` If **Frames (visible)** is non-empty, consider `gripui frame switch fN` before filter/grid/type work. `--layer active` is the current frame's top layer only — it does not pierce iframes. Switch to the frame with the buttons you need before acting. **`fN` labels are stable** for the same iframe across HUD refreshes — `f1` today is the same frame as `f1` after a blade opens, until a full navigation (`gripui go`) or top-level URL change clears the registry. You can also `gripui frame switch ` using the protocol id from hud JSON when needed. ### Tabs ```sh gripui tab new https://example.org gripui tab list # tabs are listed canonically as [t1] [t2] [t3] gripui tab switch t2 | "partial title" gripui tab close t2 gripui quit --all # stop all profile daemons ``` Close tabs you are done with. A profile holds at most 15 open tabs by default (`gripui set tab-limit `, max 50); past that, commands that open a tab are refused until you `gripui tab close` one. Tabs are always referenced as `t` — the number shown by `gripui tab list` (and in the hud `Tabs:` section) is exactly what `switch`/`close`/`focus` accept (the bare number `2` also works). `t` tracks the tab's identity, not its visual order, so dragging tabs doesn't change it. Close unneeded tabs periodically, especially before a handoff. Closing or losing a tab can shift later `t` labels. The next output reports the observed disappearance and old→new labels (it does not guess closed versus crashed). Saved refs for shifted labels are invalidated; refresh the HUD. Full URLs are always available from `tab list`/`tab find`; truncated HUD URLs are display text, not copyable addresses. ### Headless and visible sessions Honor the owner's mode. `gripui set headless` runs the same persistent browser profile without displaying a window; `gripui unset headless` restores a visible window after restart. Human mode is enabled by default. Chromium headless uses BrowserBox's desktop presentation, including a version-derived user agent, client hints, and document-start mitigations. Sites can still request login or human verification; use the normal handoff when an actual gate appears. Headful human/agent handoff preserves the running browser without restarting. Headless handoff restarts into a visible window unless `--no-restart` is given; that flag preserves page state but keeps the browser headless. Agent commands pause until `agent-drives`. Mode changes and handoffs enable `restore-last-session` automatically. The same persistent Chrome or Firefox profile restores its open tabs; an explicit handoff URL opens in addition to the restored session. Repeating the current headless setting keeps the running browser intact. You can disable restoration with `gripui unset restore-last-session`; the next mode change or handoff enables it again. ```sh gripui set headless # choose unattended browsing gripui status # compare configured headless with browser_headless ``` `WEB_FORCE_HEADLESS` overrides the profile setting and prints a warning. Prefer the profile setting for normal operation. A human handoff needs a visible window. ### Audio Owned Chrome and Firefox launches mute native audio by default, including human handoffs. `gripui get mute` shows the saved policy; `gripui unset mute` opts into audio and `gripui set mute` restores silence. Changes apply on the **next owned browser launch**: quit and relaunch an existing browser to apply them. Running or externally attached browsers are unchanged; this setting does not report live audibility. Browser-setup files must not set `--mute-audio` or `media.volume_scale`; use the mute setting so there is one owner of the policy. ### Profiles ```sh export GRIPUI_JOB_ID=unique-job-id # stable across this logical job's invocations gripui attach work # once per project; saves its .gripui.json gripui hud # subsequent commands omit --profile gripui attach work --job-only # alternative: job attachment, no default write gripui @other status # explicit one-command shared-control override gripui detach # releases this job's claim; retains browser ``` There is no implicit global default profile. New jobs use the nearest ancestor `.gripui.json` (`{"version":1,"profile":"work"}`), stopping at a Git repository or worktree boundary. Without a marker, the current directory is the project. `attach` saves that project's default; `attach work --job-only` leaves it unchanged. Only the small `.gripui.json` selector lives in the project. Browser profiles, cookies, settings, job bindings, and daemon state stay under `~/.config/DOSAYGO/gripui/`; project discovery never relocates browser data. Child directories share the project attachment. The same job can attach different profiles in different projects. A binding stays pinned within its project despite later config edits; detach explicitly before changing it. Nested config files create separate scopes. Noninteractive unconfigured commands fail with setup instructions. `CODEX_THREAD_ID` or `TERM_SESSION_ID` can identify the job; otherwise the calling shell's process identity is used. Automation should set `GRIPUI_JOB_ID` explicitly. Prefer **attach once, then bare commands**. Repeating `--profile` for the attached profile emits a one-time nudge to omit it. `@profile` works in argv, REPL, and pipes; it never changes the following command's profile. `--profile` / `--persona` remain supported. Another job cannot attach an owned profile. An explicit override to another profile reports its persistent owner and leaves that ownership unchanged. Claims persist until the owning job detaches, or its recorded process identity is proven dead. For token-identified automation, set `GRIPUI_JOB_PID` to the long-lived agent/job process when attaching to enable safe dead-job reclamation. Without that evidence (including legacy claims), explicit detach is required. Shell-identified jobs record their parent process automatically. Heartbeat age never authorizes takeover; reused PIDs are distinguished by kernel process birth identity. A dead job's healthy browser is retained, not killed, when a new job claims the profile. Daemon/browser cleanup also records kernel process birth identity. Unknown legacy PID-only handoff records require closing that browser explicitly; repair/quit never treat a reused PID as the original process or unlink a live profile's locks. `set profile --name work` and `unset profile` remain aliases for attach/detach. Attachment changes must be separate invocations, not commands inside a pipe/REPL. When driving several profiles at once, rotate through all of them every 3–5 actions — a silent tab may be sitting on a blocker. **Each time you touch a tab, complete a unit of work** (at least one meaningful step); inspecting the same state again is a micro-loop. If you land on a blocked tab, resolve or hand it off before moving on. To drive a BrowserBox client and its engine browser side by side, keep each in an explicitly named profile. Open the login link normally in the client profile, then attach the engine profile through the authenticated CDP proxy: ```sh gripui --profile bbx-client go '' gripui --profile bbx-engine connect browserbox '' --engine gripui --profile bbx-engine hud ``` `--engine` derives exactly one endpoint: the login link's host on main port + 1. It does not scan ports or fall back to the BrowserBox session transport. To drive the browser a TDF Tree Grower capture is running in — so everything you page through is recorded into the archive — start the capture in Tree Grower, then attach a dedicated profile to the debugging port Grower shows (9222 unless it says otherwise): ```sh gripui --profile tdf connect tdf # or: connect tdf gripui --profile replay connect tdf --player # Player's port 9223 gripui --profile tdf go '' gripui --profile tdf scroll until "..." # lazy content is only captured once it loads gripui --profile tdf quit # disconnects; the capture browser keeps running ``` `connect tdf` refuses when nothing is listening on the port (start the capture first). gripui never intercepts requests, so it cannot interfere with TDF's capture; it only navigates, scrolls and reads. Page at a human pace: the capture records what the browser loads, and rate-limited sites notice crawlers. ### Waiting Prefer `sleep` + `hud`. Use `gripui wait` only for a known exact condition: ```sh gripui wait text:"Dashboard" gripui wait url:"/dashboard" gripui wait title:"Home" --timeout 10000 ``` One operation at a time per profile: a long `type` or `wait` holds the profile, and a second command fails with a `client.lock` error instead of queueing. Let the first finish (or cancel it) before sending the next. ### Cancelling a busy operation Run `gripui cancel` in a separate invocation for the same profile. It uses the daemon's control channel rather than queuing behind the active operation. `gripui daemon status` exposes the operation ID, start time, outcome and log path. Cancellation and the operation deadline stop further protocol work through the same token; neither closes the browser nor rolls back input already sent. `cancelled_partial` or `deadline_partial` requires inspecting the page before retrying. `recovery_required` means no terminal receipt was observed within the bounded cancellation wait; inspect status and use explicit repair only if needed. Interrupted text operations include a value-free extent receipt when readable: initial/intended/observed character counts, whether the field changed, and `rollback_verified=false`. Secret, replaced, or unreadable targets are explicitly unknown. Firefox sends small key batches and waits for each batch receipt before acknowledging cancellation; neither browser silently replays cancelled work. Daemon status includes both release version and source-build identity. An idle daemon from another build is replaced through the browser-preserving upgrade path, even when the release version is unchanged. Busy work is not replaced. A local debug port is not proof of profile ownership. Reconnect/recovery verifies the saved browser/session identity. An occupied or mismatched port is left alone; there is no automatic listener-kill-and-retry fallback. Healthy browsers orphaned by a dead daemon use the same ownership-preserving transfer path as upgrades. Scroll-until yields a continuation checkpoint after 30 pages or approximately 10 seconds between probes/steps. JSON reports elapsed time and budget; in-flight protocol calls retain their own timeouts. Inspect, then repeat to continue from the current viewport. A probe that exceeds its own 1.5s scan budget yields `checkpoint_reason=probe_time_budget` with `match_status=unknown`. It does not return partial matches or prove absence, and skips an immediate duplicate HUD scan. Narrow the query or scroll explicitly before continuing; do not interpret an incomplete scan as "not found". ### Recovery from degraded states - `state=degraded:page.error` — often still partially usable. `gripui reload`, then refresh the HUD; `find` and `read` on visible text usually still work. - `state=degraded:page.action_occluded` — don't retry the same ref. Re-hud, dismiss the cover, or `gripui snap` to diagnose. - ARIA live regions in hud (`[polite]`/`[assertive]`) report real-time state even when the document has errors — read them. - A confirmation toast's ref (e.g. X's "View") can expire before you use it. That is not a failed submit: re-hud (`--verbose` shows hrefs), open the real result URL and verify it. Never resubmit only because the toast disappeared. --- ## Navigation philosophy **Navigate like a person.** Start from a homepage or main section and click toward your target. Constructed deep-link URLs are fragile — they skip app initialization, break session state, and trip bot detection. Use a direct URL only when you know the app produces it as a stable, shareable link. **Arrive by search, then click.** When opening a site — especially one with a login, a dashboard, or a bot check — go to DuckDuckGo, search for the site, and click the result: ```sh gripui go https://duckduckgo.com gripui hud # find the search box gripui do "cloudflare dashboard" gripui press Enter gripui hud # click the official result, not an ad ``` Arriving through a search result is how a person gets there, so any Cloudflare/bot gate activates in front of you and gets crossed on the way in. Jumping straight to a remembered login or dashboard URL hides that gate: the challenge never runs properly, the session never becomes trusted, and the login fails. Never navigate directly to a remembered login URL. The ladders here are high-signal starting points, not scripts. Real pages are messy: overlays misbehave, content loads late, structures differ from spec. When a pattern doesn't fit, improvise the way a human with a browser would. Prefer core commands first, then advanced, then debug tools. --- ## Reference ### Shell composition Pipe high-volume **structured** output — not `hud`: ```sh gripui form | grep -F "[error:" gripui status --json | jq '.data.states[] | select(.kind=="form.error") | .details.errors' ``` `gripui hud --json` is for diagnostics, not for jq-filtering refs — use `gripui find` for that. After any grep of hud, your next hud must be raw. ### Diagnostics ```sh gripui status # session + page state + blocked conditions gripui describe # tag, role, selector, visibility gripui doctor # why a field is rejecting input (see below) gripui eval "document.title" # JS diagnostics ONLY — never for interaction gripui transcript --path # full .jsonl history path ``` Screenshots are a last resort and token-heavy (`gripui snap` = `gripui eval`) — use only when text/hud/read can't answer. (`gripui capture` is an alias for `gripui snap`.) The receipt prints the absolute path written (wrapper scripts may change the working directory), the image size, the CSS viewport and the scale between them: `image=2560x1366px viewport=1280x683css scale=2.00`. Snap pixels are not CSS pixels — divide image coordinates by `scale` before `click --coords`. ### Recording and demos - `gripui resize WxH --move-to XxY` — position the browser window before starting (e.g. `gripui resize 1280x900 --move-to 1280x0` for the right half of a dual monitor). - `gripui click --trace-pointer` on important actions — the pointer trail shows intent. - `gripui highlight ` to call attention to what you see. - Don't `gripui snap` for recordings — the terminal + browser already show everything. ### Command reference ``` Core go Navigate; also accepts history keywords hud [--ignore-viewport] [--verbose] [--layer active] [--layers] [--show-occluded] do ["text"] Run action n; optionally supply text / option value ("-" = stdin) click [--new [--follow]] [--trace-pointer] [--coords X,Y] [--try-anyway] type "text" Key events — framework-managed / validated fields, comboboxes ("-" = stdin) say "text" Paste into the focused field (no key events; "-" = stdin) focus Trusted click to activate (same path as click/type) clear [] Clear a field read [--full|--ignore-viewport] [--headings|--links] find "text" [--all] [--ignore-viewport|--within-viewport] [--verbose] disambiguate "text" All elements matching a label, each with href form [--ignore-viewport] Form fields with current values and errors submit Submit the nearest form press Enter, Escape, Tab, ctrl+a, … choose Native scroll [down|up| [up]] Scroll page, or hover a ref then scroll its panel reveal [--edge] Center a ref and verify hittability; --edge uses nearest edge hover Hover (triggers tooltips and menus) back / forward / reload status Session + page state + blocked conditions tab new|list|switch|close|find frame list|switch quit Handoff human-drives [url] [--no-restart] / agent-drives / pause / resume Configure resize WxH [--move-to XxY] Resize and/or reposition the browser window set / unset / get Sticky settings: profile, headless, ref-shape, … use teach Write skill file into agent config directories mcp Serve gripui as an MCP tool on stdio (Claude Code, Codex) Advanced wait Wait for text, URL, title, or selector observe --json [--ignore-viewport] Emergency: status + read + hud + form refs / last focus / caret / select-text snap [] Screenshot to a file (alias: capture) search connect / disconnect Debug eval "" Diagnostics only — never for interaction describe highlight transcript [--path] report-issue [--system-browser] version / readme ``` **Stuck on syntax? `gripui --help`.** ## Driving the Web This project uses the gripui CLI through `gripui` (aliases: `grip`, `gu`) for browser automation and real web interaction. The gripui CLI SKILL is installed at: - `.claude/skills/gripui-cli/SKILL.md` - `.grok/skills/gripui-cli/SKILL.md` - `.gemini/skills/gripui-cli/SKILL.md` - `.copilot/skills/gripui-cli/SKILL.md` - `.codex/skills/gripui-cli/SKILL.md` Load the relevant file before browser work. Run `gripui teach` to install or refresh it.