monitoring: vanilla Android GUI smoke + AUTOMATED-GUI-NOTES

uiautomator multi-round taps for goa/stage; document legitimate
deep-link shortcuts vs pure CLI install tests.
This commit is contained in:
Hernâni Marques 2026-07-17 21:00:56 +02:00
parent b1c8e5ddb2
commit f1b6eb8b45
No known key found for this signature in database
6 changed files with 509 additions and 5 deletions

View file

@ -2,3 +2,5 @@ apks/*.apk
out/ out/
*.png *.png
out-source/ out-source/
out-gui/
out-source-gui/

View file

@ -0,0 +1,119 @@
# Automated GUI notes — Android wallet (vanilla level)
Goal: tests for **two systems** (GOA / hacktivism and stage TESTPAYSAN) that
exercise the wallet **graphically in the UI** as far as practical, not only
CLI/`adb` deep-links into wallet-core.
## Vanilla level (this tree)
| Layer | What we do | Tooling |
|-------|------------|---------|
| **A — Install** | Install published F-Droid APK **or** APK built from `taler-android` master | `adb install` |
| **B — Entry (shortcut)** | Deliver withdraw / pay as `taler://…` intents | `adb shell am start -a VIEW -d …` |
| **C — Graphical UI** | Dump hierarchy, tap Confirm / ToS / Pay-like buttons, dismiss ANR | `uiautomator dump` + `input tap` (`lib_ui.py`) |
| **D — Evidence** | Screenshots + UI XML + logcat + JSON status | `android-test/out-gui/` |
Scripts:
| Script | Role |
|--------|------|
| `run-android-pay-smoke.sh` | Install + deep-link + light taps (hybrid / CLI-heavy) |
| `run-android-gui-smoke.sh` | **Vanilla GUI path**: deep-link entry + multi-round UI driving |
| `run-android-build-and-smoke.sh` | Build latest source APK → same smoke (set `GUI=1` to call GUI script) |
| `lib_ui.py` | Shared dump/tap/ANR helpers |
```bash
cd scripts/taler-monitoring/android-test
# Published APK, graphical drive, both stacks
STACK=goa ./run-android-gui-smoke.sh
STACK=stage ./run-android-gui-smoke.sh
# From-source APK + GUI
APK_PATH=~/taler/taler-android/wallet/build/outputs/apk/fdroid/debug/wallet-fdroid-debug.apk \
PKG=net.taler.wallet.fdroid.debug \
STACK=goa ./run-android-gui-smoke.sh
```
## Legitimate shortcuts (document, do not hide)
These are **not** full human UX, but they are the same contracts as QR scan /
paywall open on the landings:
| Shortcut | Replaces | Why legitimate |
|----------|----------|----------------|
| `adb` `VIEW` `taler://withdraw/…` | Camera QR scan of communal / demo withdraw | Identical URI payload as landings (`demo-withdraw.json`) |
| `adb` `VIEW` `taler://pay…` / `pay-template/…` | Scan shop/paywall QR or open “pay with Taler” | Same URI merchant/backend would show |
| Host-side `POST …/templates/{id}` then pay URI | Browser shop UI to create order | Stage farmer templates are public; order create is API, pay is still wallet UI |
| Multi-language label list for taps | Human reading the button | Same visible buttons; brittle if copy changes |
| Skip full ToS reading | User scrolling long TOS | Tap “Accept” if shown; no legal review automation |
**Not automated yet (out of vanilla scope):**
- Typing personal bank passwords into the wallet webview
- NFC
- Real camera / vision-based QR
- Biometric unlock
- Full onboarding wizard every time (we force-stop + re-intent)
## Two systems under test
| Stack | Withdraw source | Pay source |
|-------|-----------------|------------|
| **GOA** (`STACK=goa`) | `https://bank.hacktivism.ch/intro/demo-withdraw.json` | `taler://pay-template/taler.hacktivism.ch/instances/goa-shop/paivana` (Paivana) |
| **stage** (`STACK=stage`) | `https://stage.bank.lefrancpaysan.ch/intro/demo-withdraw.json` | Public template e.g. `fermes-des-collines` / `panier-legumes``taler://pay/…` |
Both use the **same** GUI driver (`lib_ui.py`); only URLs and package under test differ.
## GUI driver behaviour (`lib_ui.gui_drive`)
1. Every round: `uiautomator dump` + optional screencap.
2. If **“System UI isnt responding”** → tap **Wait** (prefer over Close).
3. Else try exact taps on Confirm / Withdraw / Accept / Pay / DE/FR synonyms.
4. Else regex partial match on those words.
5. Repeat `GUI_ROUNDS` times (default 10), `GUI_SLEEP` seconds apart.
Success signals (layered):
- **Intent delivered**`am start` OK
- **wallet-core** — logcat `prepareBankIntegratedWithdrawal` / pay-related ops
- **GUI taps** — non-empty `taps` in `gui-*-status.json`
- **Screenshots** — human-readable progress under `gui-withdraw/` / `gui-pay/`
A run can be “vanilla-partial” if intents + some taps succeed but final
balance/settlement is not asserted (settlement still better covered by
`taler-monitoring.sh e2e` CLI wallet).
## Host limits (clementine, 2026-07)
| Constraint | Effect |
|------------|--------|
| ~3.7GiB RAM | Emulator ANRs; GUI rounds often only dismiss ANR |
| Prefer physical phone or ≥68GiB host | Real Confirm/Pay taps complete more often |
| No root required | User SDK + adb + F-Droid/source APK |
## Relation to CLI e2e
| Suite | Strength |
|-------|----------|
| `taler-monitoring.sh e2e` / `ladder` | Full withdraw+pay **settlement** via `taler-wallet-cli` |
| `run-android-gui-smoke.sh` | **Android app UI** surface for the same stacks |
| Combined | CLI = money path green; GUI = app UX green as far as taps go |
## Future upgrades (not required for vanilla)
- Maestro / Appium flows with stable resource-ids
- Explicit wait for text “Withdrawal complete” / balance
- Separate jobs: F-Droid vs nightly source vs Play flavor
- Record video (`adb shell screenrecord`) on failure
## Checklist — “vanille erreicht?”
- [x] Two stacks parameterized (goa / stage)
- [x] Published APK path
- [x] Source-build path (same smoke after build)
- [x] Deep-link shortcuts documented
- [x] Multi-round **graphical** taps (uiautomator)
- [x] Screenshots + UI XML + logcat artifacts
- [ ] Fully reliable unattended Confirm→settled on low-RAM emulator (host-limited)
- [ ] Assert final balance in app UI (optional next step)

View file

@ -27,4 +27,17 @@ TALER_ANDROID_SRC=/path/to/taler-android ./run-android-build-and-smoke.sh
Built package id: `net.taler.wallet.fdroid.debug`. Built package id: `net.taler.wallet.fdroid.debug`.
Needs `ANDROID_HOME` (e.g. `$HOME/Android/Sdk`), Java 17+, network for Maven. Needs `ANDROID_HOME` (e.g. `$HOME/Android/Sdk`), Java 17+, network for Maven.
See parent [GIT-AUTOMATION-NOTES.md](../GIT-AUTOMATION-NOTES.md). ## 3) Graphical UI (vanilla)
Deep-link entry (≡ QR) **plus** multi-round uiautomator taps:
```bash
STACK=goa ./run-android-gui-smoke.sh
STACK=stage ./run-android-gui-smoke.sh
# after source build:
GUI=1 ./run-android-build-and-smoke.sh
```
Documented shortcuts and limits: **[AUTOMATED-GUI-NOTES.md](./AUTOMATED-GUI-NOTES.md)**.
See also parent [GIT-AUTOMATION-NOTES.md](../GIT-AUTOMATION-NOTES.md).

View file

@ -0,0 +1,192 @@
#!/usr/bin/env python3
"""Minimal adb + uiautomator helpers for Taler wallet GUI smoke (no root).
Does not replace Maestro/Appium; uses only adb shell + uiautomator dump.
"""
from __future__ import annotations
import re
import subprocess
import time
from pathlib import Path
from typing import Iterable, List, Optional, Sequence, Tuple
Bounds = Tuple[int, int, int, int]
def adb(serial: str, *args: str, check: bool = False, timeout: Optional[float] = 60) -> subprocess.CompletedProcess:
cmd = ["adb", "-s", serial, *args]
return subprocess.run(cmd, check=check, capture_output=True, text=True, timeout=timeout)
def dump_ui(serial: str, dest: Path) -> str:
adb(serial, "shell", "uiautomator", "dump", "/sdcard/ui.xml", timeout=90)
adb(serial, "pull", "/sdcard/ui.xml", str(dest), timeout=30)
return dest.read_text(errors="replace") if dest.is_file() else ""
def screencap(serial: str, dest: Path) -> None:
p = subprocess.run(
["adb", "-s", serial, "exec-out", "screencap", "-p"],
capture_output=True,
timeout=60,
)
if p.returncode == 0 and p.stdout:
dest.write_bytes(p.stdout)
def texts(xml: str) -> List[str]:
return [t for t in re.findall(r'text="([^"]*)"', xml) if t.strip()]
def find_bounds(xml: str, label: str) -> List[Bounds]:
"""Return list of (x1,y1,x2,y2) for nodes whose text or content-desc matches label (exact)."""
out: List[Bounds] = []
esc = re.escape(label)
patterns = [
rf'(?:text|content-desc)="{esc}"[^>]*bounds="\[(\d+),(\d+)\]\[(\d+),(\d+)\]"',
rf'bounds="\[(\d+),(\d+)\]\[(\d+),(\d+)\]"[^>]*(?:text|content-desc)="{esc}"',
]
for pat in patterns:
for m in re.finditer(pat, xml, flags=re.I):
out.append(tuple(map(int, m.groups()))) # type: ignore[arg-type]
return out
def find_bounds_re(xml: str, pattern: str) -> List[Tuple[str, Bounds]]:
out: List[Tuple[str, Bounds]] = []
for m in re.finditer(
rf'(?:text|content-desc)="([^"]*{pattern}[^"]*)"[^>]*bounds="\[(\d+),(\d+)\]\[(\d+),(\d+)\]"',
xml,
flags=re.I,
):
label = m.group(1)
b = tuple(map(int, m.group(2, 3, 4, 5)))
out.append((label, b)) # type: ignore[arg-type]
return out
def center(b: Bounds) -> Tuple[int, int]:
x1, y1, x2, y2 = b
return (x1 + x2) // 2, (y1 + y2) // 2
def tap(serial: str, x: int, y: int) -> None:
adb(serial, "shell", "input", "tap", str(x), str(y))
def tap_bounds(serial: str, b: Bounds) -> None:
x, y = center(b)
tap(serial, x, y)
def dismiss_anr(serial: str, xml: str) -> bool:
"""If System UI isn't responding, prefer Wait over Close."""
if "isn't responding" not in xml and "reagiert nicht" not in xml.lower():
return False
for label in ("Wait", "Warten", "OK"):
bs = find_bounds(xml, label)
if bs:
tap_bounds(serial, bs[0])
return True
# fallback: right-ish button often Wait
adb(serial, "shell", "input", "keyevent", "22")
adb(serial, "shell", "input", "keyevent", "66")
return True
# Labels seen on wallet UI (EN/DE/FR) — expand as needed
CONFIRM_LABELS = [
"Confirm",
"Confirm withdrawal",
"Withdraw",
"Accept",
"I accept",
"Agree",
"Continue",
"Next",
"OK",
"Pay",
"Pay now",
"Bestätigen",
"Abheben",
"Akzeptieren",
"Zustimmen",
"Weiter",
"Bezahlen",
"Confirmer",
"Retirer",
"Accepter",
"Payer",
]
TOS_LABELS = [
"I accept",
"Accept",
"Agree",
"Akzeptieren",
"Zustimmen",
"Accepter",
]
def try_tap_any(serial: str, xml: str, labels: Sequence[str]) -> Optional[str]:
for label in labels:
bs = find_bounds(xml, label)
if bs:
tap_bounds(serial, bs[0])
return label
return None
def gui_drive(
serial: str,
out_dir: Path,
*,
rounds: int = 8,
sleep_s: float = 3.0,
phase: str = "gui",
) -> dict:
"""Poll UI, dismiss ANR, tap confirm/ToS-like buttons. Returns status dict."""
out_dir.mkdir(parents=True, exist_ok=True)
seen_texts: List[str] = []
taps: List[str] = []
anr = 0
for i in range(1, rounds + 1):
xml_path = out_dir / f"{phase}-{i:02d}-ui.xml"
png_path = out_dir / f"{phase}-{i:02d}.png"
try:
xml = dump_ui(serial, xml_path)
except Exception as e:
taps.append(f"dump-fail:{e}")
time.sleep(sleep_s)
continue
try:
screencap(serial, png_path)
except Exception:
pass
ts = texts(xml)
seen_texts.extend(ts[:20])
if dismiss_anr(serial, xml):
anr += 1
time.sleep(sleep_s)
continue
hit = try_tap_any(serial, xml, CONFIRM_LABELS)
if hit:
taps.append(hit)
time.sleep(sleep_s)
continue
# partial regex for amount confirm rows
for label, b in find_bounds_re(xml, r"Confirm|Withdraw|Pay|Accept|Bestätig|Abheben|Payer"):
tap_bounds(serial, b)
taps.append(f"re:{label}")
time.sleep(sleep_s)
break
else:
time.sleep(sleep_s)
return {
"taps": taps,
"anr_dismissals": anr,
"sample_texts": seen_texts[-40:],
"rounds": rounds,
}

View file

@ -82,12 +82,18 @@ echo "copied → $STAMP_APK"
echo "$COMMIT" >"$ROOT/out/build-commit.txt" 2>/dev/null || mkdir -p "$ROOT/out" && echo "$COMMIT" >"$ROOT/out/build-commit.txt" echo "$COMMIT" >"$ROOT/out/build-commit.txt" 2>/dev/null || mkdir -p "$ROOT/out" && echo "$COMMIT" >"$ROOT/out/build-commit.txt"
echo "$APK" >"$ROOT/out/build-apk-path.txt" echo "$APK" >"$ROOT/out/build-apk-path.txt"
# Same smoke as published F-Droid path # Same smoke as published F-Droid path (CLI hybrid or GUI)
export APK_PATH="$STAMP_APK" export APK_PATH="$STAMP_APK"
export APK_NAME="$(basename "$STAMP_APK")" export APK_NAME="$(basename "$STAMP_APK")"
export PKG export PKG
export OUT_DIR="${OUT_DIR:-$ROOT/out-source}" export OUT_DIR="${OUT_DIR:-$ROOT/out-source}"
mkdir -p "$OUT_DIR" mkdir -p "$OUT_DIR"
echo "=== smoke with built APK (PKG=$PKG) ===" SMOKE_CMD="$SMOKE"
exec env APK_DIR="$ROOT/apks" APK_NAME="$APK_NAME" PKG="$PKG" OUT_DIR="$OUT_DIR" \ if [ "${GUI:-0}" = "1" ]; then
"$SMOKE" SMOKE_CMD="$ROOT/run-android-gui-smoke.sh"
OUT_DIR="${OUT_DIR%-source}-source-gui"
export OUT_DIR
fi
echo "=== smoke with built APK (PKG=$PKG · GUI=${GUI:-0}) ==="
exec env APK_DIR="$ROOT/apks" APK_NAME="$APK_NAME" APK_PATH="$APK_PATH" PKG="$PKG" OUT_DIR="$OUT_DIR" \
"$SMOKE_CMD"

View file

@ -0,0 +1,172 @@
#!/usr/bin/env bash
# GUI-oriented Android smoke (vanilla level): deep-link entry + UI taps.
#
# Entry shortcut (legitimate — same as scanning a QR on the landing):
# adb VIEW taler://withdraw/… and taler://pay…
# Then drives the **graphical UI** via uiautomator dump + taps (Confirm / ToS / Pay).
#
# Systems: STACK=goa | stage | auto
# APK: published F-Droid or APK_PATH=… (from-source)
#
# See AUTOMATED-GUI-NOTES.md
#
set -euo pipefail
ROOT=$(cd "$(dirname "$0")" && pwd)
# shellcheck source=run-android-pay-smoke.sh
# We re-use install/URI resolution by calling common pieces inline.
export OUT_DIR="${OUT_DIR:-$ROOT/out-gui}"
export GUI_ROUNDS="${GUI_ROUNDS:-10}"
export GUI_SLEEP="${GUI_SLEEP:-3}"
# Install + deliver intents via existing smoke, but skip its weak single-tap
# by setting GUI_MODE=1 which we implement here more fully.
# Simpler: do install/URI here by sourcing patterns from pay-smoke via env + python.
# Delegate install + URI start to pay-smoke with GUI_AFTER=1 if we patch it;
# instead run a self-contained flow calling pay-smoke pieces.
APK_DIR="${APK_DIR:-$ROOT/apks}"
APK_URL="${APK_URL:-https://f-droid.org/repo/net.taler.wallet.fdroid_854.apk}"
APK_NAME="${APK_NAME:-net.taler.wallet.fdroid_854.apk}"
APK_PATH="${APK_PATH:-}"
PKG="${PKG:-net.taler.wallet.fdroid}"
STACK="${STACK:-auto}"
SERIAL="${SERIAL:-}"
mkdir -p "$OUT_DIR"
if ! command -v adb >/dev/null 2>&1; then
[ -x "$HOME/Android/Sdk/platform-tools/adb" ] && export PATH="$HOME/Android/Sdk/platform-tools:$PATH"
fi
command -v adb >/dev/null || { echo "adb missing" >&2; exit 2; }
if [ -z "$SERIAL" ]; then
SERIAL=$(adb devices | awk '/\tdevice$/{print $1; exit}')
fi
[ -n "$SERIAL" ] || { echo "no adb device" >&2; exit 3; }
ADB=(adb -s "$SERIAL")
echo "device: $SERIAL mode: GUI (uiautomator)"
case "$STACK" in
auto)
if curl -sfS -m 5 -o /dev/null https://bank.hacktivism.ch/config; then STACK=goa; else STACK=stage; fi
;;
esac
case "$STACK" in
goa)
BANK=https://bank.hacktivism.ch
MERCHANT=https://taler.hacktivism.ch
WITHDRAW_JSON="$BANK/intro/demo-withdraw.json"
;;
stage)
BANK=https://stage.bank.lefrancpaysan.ch
MERCHANT=https://stage.monnaie.lefrancpaysan.ch
WITHDRAW_JSON="$BANK/intro/demo-withdraw.json"
;;
*) echo "unknown STACK" >&2; exit 2 ;;
esac
echo "stack: $STACK"
if [ -n "$APK_PATH" ] && [ -f "$APK_PATH" ]; then
:
elif [ -f "$APK_DIR/$APK_NAME" ]; then
APK_PATH="$APK_DIR/$APK_NAME"
else
APK_PATH="$APK_DIR/$APK_NAME"
mkdir -p "$APK_DIR"
curl -fL --retry 3 -o "$APK_PATH" "$APK_URL"
fi
echo "apk: $APK_PATH pkg: $PKG"
"${ADB[@]}" wait-for-device
"${ADB[@]}" install -r "$APK_PATH"
"${ADB[@]}" shell settings put global window_animation_scale 0 || true
"${ADB[@]}" shell settings put global transition_animation_scale 0 || true
"${ADB[@]}" shell settings put global animator_duration_scale 0 || true
WURI=$(curl -sfS -m 25 "$WITHDRAW_JSON" | python3 -c 'import json,sys; d=json.load(sys.stdin); print(d.get("taler_withdraw_uri") or d.get("qr_payload") or "")')
[ -n "$WURI" ] || { echo "no withdraw uri" >&2; exit 4; }
echo "withdraw: $WURI"
echo "$WURI" >"$OUT_DIR/last-withdraw.uri"
"${ADB[@]}" shell am force-stop "$PKG" 2>/dev/null || true
"${ADB[@]}" logcat -c 2>/dev/null || true
# --- Shortcut A: deep-link withdraw (≡ QR scan on landing) ---
"${ADB[@]}" shell am start -a android.intent.action.VIEW -d "$WURI" | tee "$OUT_DIR/start-withdraw.txt"
sleep 5
# --- Graphical: drive UI ---
python3 - "$ROOT/lib_ui.py" "$SERIAL" "$OUT_DIR" "$GUI_ROUNDS" "$GUI_SLEEP" <<'PY'
import json, sys, importlib.util
from pathlib import Path
lib_path, serial, out, rounds, sleep_s = (
Path(sys.argv[1]), sys.argv[2], Path(sys.argv[3]), int(sys.argv[4]), float(sys.argv[5])
)
spec = importlib.util.spec_from_file_location("lib_ui", lib_path)
lib = importlib.util.module_from_spec(spec)
spec.loader.exec_module(lib)
st = lib.gui_drive(serial, out / "gui-withdraw", rounds=rounds, sleep_s=sleep_s, phase="wd")
(out / "gui-withdraw-status.json").write_text(json.dumps(st, indent=2))
print("gui-withdraw taps:", st.get("taps"))
print("gui-withdraw anr:", st.get("anr_dismissals"))
PY
# Pay URI
PAY_URI=""
if [ "$STACK" = "stage" ]; then
if RESP=$(curl -sfS -m 20 -X POST -H 'Content-Type: application/json' -d '{}' \
"$MERCHANT/instances/fermes-des-collines/templates/panier-legumes" 2>/dev/null); then
PAY_URI=$(printf '%s' "$RESP" | python3 -c 'import json,sys;d=json.load(sys.stdin);oid=d.get("order_id")or"";tok=d.get("token")or"";
print("taler://pay/stage.monnaie.lefrancpaysan.ch/instances/fermes-des-collines/%s/?c=%s"%(oid,tok) if oid and tok else "")')
fi
else
PAY_URI="taler://pay-template/taler.hacktivism.ch/instances/goa-shop/paivana"
fi
if [ -n "$PAY_URI" ]; then
echo "pay: $PAY_URI"
echo "$PAY_URI" >"$OUT_DIR/last-pay.uri"
# Shortcut B: deep-link pay (≡ scan pay QR / open paywall link)
"${ADB[@]}" shell am start -a android.intent.action.VIEW -d "$PAY_URI" | tee "$OUT_DIR/start-pay.txt"
sleep 5
python3 - "$ROOT/lib_ui.py" "$SERIAL" "$OUT_DIR" "$GUI_ROUNDS" "$GUI_SLEEP" <<'PY'
import json, sys, importlib.util
from pathlib import Path
lib_path, serial, out, rounds, sleep_s = (
Path(sys.argv[1]), sys.argv[2], Path(sys.argv[3]), int(sys.argv[4]), float(sys.argv[5])
)
spec = importlib.util.spec_from_file_location("lib_ui", lib_path)
lib = importlib.util.module_from_spec(spec)
spec.loader.exec_module(lib)
st = lib.gui_drive(serial, out / "gui-pay", rounds=rounds, sleep_s=sleep_s, phase="pay")
(out / "gui-pay-status.json").write_text(json.dumps(st, indent=2))
print("gui-pay taps:", st.get("taps"))
print("gui-pay anr:", st.get("anr_dismissals"))
PY
fi
"${ADB[@]}" logcat -d -t 400 >"$OUT_DIR/logcat.txt" || true
grep -iE 'taler-wallet|prepareBank|preparePay|confirm|Error|success' "$OUT_DIR/logcat.txt" \
| tail -100 >"$OUT_DIR/logcat-taler.txt" || true
echo
echo "=== GUI smoke summary ($STACK) ==="
echo "package: $PKG"
echo "shortcut withdraw URI: delivered"
echo "GUI withdraw: see $OUT_DIR/gui-withdraw-status.json + screenshots"
[ -n "$PAY_URI" ] && echo "shortcut pay URI: delivered"
[ -n "$PAY_URI" ] && echo "GUI pay: see $OUT_DIR/gui-pay-status.json + screenshots"
if grep -q prepareBankIntegratedWithdrawal "$OUT_DIR/logcat.txt" 2>/dev/null; then
echo "wallet-core: prepareBankIntegratedWithdrawal YES"
else
echo "wallet-core: prepareBankIntegratedWithdrawal no (ANR/slow UI?)"
fi
if grep -qiE 'preparePay|confirmPay|pay-template|preparePurchase' "$OUT_DIR/logcat.txt" 2>/dev/null; then
echo "wallet-core: pay path YES"
else
echo "wallet-core: pay path not confirmed in logcat"
fi
echo "artifacts: $OUT_DIR/"
ls -la "$OUT_DIR" | sed 's/^/ /'
echo "Documented shortcuts → AUTOMATED-GUI-NOTES.md"