docs: define production acceptance matrix

2026-07-12 03:35:02 -05:00
parent bb375f0795
commit a5467882b1
1 changed files with 123 additions and 0 deletions
@@ -0,0 +1,123 @@
 # Production Acceptance Matrix
 This document is the release contract for Rabby. It combines the Rabby Wallet inventory in `docs/rabbyhub-mvp-feature-inventory.md` and the Tabby inventory in `docs/tabby-comparison.md` into one auditable production baseline.
 ## Status vocabulary
 | Status | Meaning |
 | --- | --- |
 | `PASS` | The complete user path exists and its required automated and manual evidence is recorded for the release candidate. |
 | `PARTIAL` | A real slice exists, but at least one required runtime, UI, integration, security, or platform behavior is missing. |
 | `MODELED` | Types, validation, or unit tests exist; this is not production behavior. |
 | `UI PROTOTYPE` | A visible workflow exists, but it is not backed by the complete runtime path. |
 | `NOT STARTED` | No meaningful implementation exists. |
 | `BLOCKED` | Acceptance cannot proceed because a named dependency or decision is unresolved. |
 `Implemented`, `RuntimeImplemented`, modeled catalog coverage, a passing unit test, and a visible mock are not synonyms for `PASS`. A row becomes `PASS` only when every acceptance statement in that row is demonstrated end to end and the applicable global gates below pass on the same release candidate.
 ## Clean baseline
 Baseline inspected on `main` at `bb375f0` (`docs: update runtime parity status`). The worktree was clean except for the intentionally untracked `.hermes/` directory. The authoritative inventories were:
 - Rabby Wallet: `docs/rabbyhub-mvp-feature-inventory.md`
 - Tabby terminal: `docs/tabby-comparison.md`
 `./scripts/check.sh` passed at the baseline: formatting, all workspace tests (79 total: 7 `rabby`, 50 `rabby-core`, 22 `rabby-runtime`), workspace check, and release build. This proves the build baseline only; it does not satisfy the security, E2E, smoke, or packaging gates.
 ## Rabby Wallet acceptance matrix
 | ID | Capability | Production acceptance criteria | Current evidence | Status |
 | --- | --- | --- | --- | --- |
 | W01 | Create, lock, unlock, recover | A user can create a wallet with a confirmed password, lock automatically and manually, unlock after restart, and follow a safe forgotten-password/reset path. Wallet state is authenticated-encrypted at rest; blank and incorrect credentials fail without leaking secrets. | AES-256-GCM/Argon2id demo dashboard vault and negative tests; no production key lifecycle or complete onboarding UI. | PARTIAL |
 | W02 | Create/import accounts | Create mnemonic and import mnemonic, private key, JSON keystore, and watch-only account flows derive or load the correct addresses, require confirmation, redact sensitive input, persist safely, survive restart, and reject malformed/duplicate material. Test vectors and destructive-flow E2E tests pass. | Input validation/redaction models only; no key derivation or production persistence. | MODELED |
 | W03 | Hardware and institutional accounts | Ledger, Trezor, Keystone, GridPlus, OneKey, BitBox02, Gnosis Safe, Coinbase, WalletConnect, and Cobo Argus can connect, enumerate/select accounts, sign through the external authority, recover from disconnect/rejection, and never copy external private keys into Rabby storage. Supported platform/device matrix is tested and documented. | Account-source enum coverage only. | MODELED |
 | W04 | Multi-chain networks | Built-in, custom RPC, custom testnet, and offline-capable chains can be added, edited, selected, health-checked, and removed safely. Chain IDs cannot silently mismatch RPC responses; invalid/insecure endpoints are rejected or explicitly warned; per-chain state remains isolated. | Chain configuration validation only. | MODELED |
 | W05 | Portfolio dashboard | Real token, NFT, and DeFi balances sync for the selected accounts/chains, include source and freshness state, reconcile retries/reorgs, handle partial provider failure, avoid double counting, and render empty/loading/error/offline states. Totals are deterministic against fixtures. | Static/demo dashboard summaries; no real portfolio providers. | UI PROTOTYPE |
 | W06 | Send and receive | Receive displays/copies/verifies the selected chain/account address. Native token, ERC-20, and NFT sends estimate fees, validate destination/amount/balance/chain, show a final human-readable review, sign, broadcast, track replacement/failure/confirmation, and prevent accidental duplicate submission. Hardware/watch-only restrictions are enforced. | Send request validation only; no signing/broadcast. | MODELED |
 | W07 | Swap and bridge | Quotes come from real configured providers with expiry, route, fees, minimum received, price impact, allowance, and slippage shown. The user approves the exact route, signs/broadcasts, and can track source and destination completion. Stale quotes, provider failure, unsupported assets, and bridge partial failure are handled safely. | Quote model validation only; no provider execution. | MODELED |
 | W08 | Token/NFT approvals | Current approvals are fetched per account/chain, spender identity and risk are displayed, revoke/batch-revoke estimates fees and requires review, transactions are signed/broadcast/tracked, and stale or changed allowance data is refreshed before action. | Approval/risk records only. | MODELED |
 | W09 | Dapp provider and permissions | A browser-compatible provider exposes the required EIP-1193 behavior, isolates origins, prompts for accounts/chains/methods, persists revocable least-privilege grants, rejects malformed/unauthorized requests, handles chain/account changes, and passes a compatibility suite against representative dapps. Desktop/browser integration and threat boundaries are documented. | Permission model only; no `window.ethereum` injection or request bridge. | MODELED |
 | W10 | Signing and security preview | Personal messages, typed data, and transactions show origin, account, chain, decoded intent, asset/value changes, spender/recipient, simulation, and risk warnings before confirmation. Blind/unparseable signing is prominently gated. User rejection, simulation failure, phishing inputs, chain mismatch, and malicious typed data are covered by tests. | Signing request/risk model only; no signing or simulation engine. | MODELED |
 | W11 | Activity and history | Local and on-chain activity is persisted and reconciled across restart, account, chain, pending/confirmed/failed/replaced/dropped states, and reorgs. Transaction hashes/signature identifiers link to verifiable detail; filtering and pagination remain correct under provider failures. | Activity record model only. | MODELED |
 | W12 | Contacts and whitelist | Contacts are add/edit/delete/searchable; addresses are chain-aware where required, normalized and duplicate-safe. Whitelist changes require clear confirmation and cannot bypass send/signing review. Spoofing, clipboard alteration, and look-alike names are warned or prevented. | Contact/whitelist model only. | MODELED |
 | W13 | Settings and customization | Theme, language, currency, auto-lock, and default-wallet/MetaMask compatibility settings persist across restart and apply without corrupting state. Invalid values recover safely; sensitive changes require reauthentication; accessibility and keyboard operation are verified. | Theme/config persistence slice plus incomplete UI. | PARTIAL |
 | W14 | Notifications, guides, points, feedback | Notifications have stable read/dismiss state, deep links are safe, guides are dismissible and versioned, points data is authenticated and failure-tolerant, and feedback submission clearly states what data leaves the device. No notification can trigger signing or navigation without user intent. | Dashboard shell only. | UI PROTOTYPE |
 ## Tabby terminal acceptance matrix
 | ID | Capability | Production acceptance criteria | Current evidence | Status |
 | --- | --- | --- | --- | --- |
 | T01 | VT terminal emulation | Interactive PTY output is continuously streamed and rendered with a proven VT engine covering cursor movement, scrollback, alternate screen, colors, modes, resize, selection, and common full-screen apps. Golden/conformance tests and real shell/editor smoke tests pass without data loss or hangs. | Real local PTY registry plus small `TerminalGrid`/ANSI parser; no continuous UI renderer or full VT engine. | PARTIAL |
 | T02 | Nested split panes | Users can create horizontal/vertical nested splits, focus, resize, close, and move panes without losing sessions. Layout and active pane restore after restart; minimum sizes and last-pane behavior are deterministic. | Recursive model and serialization tests; no complete live pane UI lifecycle. | MODELED |
 | T03 | Tabs and restoration | Tabs can be opened, closed, reordered, placed on supported sides, renamed, and restored with profile/layout/cwd metadata. Crashed or unavailable sessions recover gracefully and never restore secret input. | Workspace tab model only. | MODELED |
 | T04 | Quake console/global hotkey | A configurable global shortcut shows/hides the intended window on every supported OS, handles focus/multi-monitor/fullscreen correctly, avoids shortcut collisions, and restores prior state. Startup and permission failures are actionable. | `quake_mode` state only. | MODELED |
 | T05 | Progress/process notifications | Process completion/progress detection is accurate enough to avoid notification spam, obeys per-profile/user preferences, redacts command secrets, and deep-links only to the originating pane. Permission-denied and background cases are tested. | Tab progress field only. | MODELED |
 | T06 | Paste/copy safety | Bracketed paste works; multiline/control-character pastes require an explicit safe review policy; right-click paste and copy-on-select are configurable; clipboard operations preserve Unicode and never execute without the terminal receiving deliberate input. | Parser tracks bracketed-paste mode; UI policies absent. | MODELED |
 | T07 | Unicode, width, ligatures | Grapheme clusters, combining marks, emoji, CJK/double-width cells, bidi policy, fallback fonts, ligatures, selection, cursor placement, and resize render consistently against fixtures on supported platforms. | UTF-8 model tests only. | MODELED |
 | T08 | Local shell profiles | Real interactive shells launch with configured executable, args, cwd, environment, and platform discovery; input/output/resize/signal/exit work; invalid profiles fail safely. Supported PowerShell/WSL/Git-Bash/Cygwin/MSYS2/Cmder/CMD/platform matrix is tested where applicable. | Real generic local PTY sessions; no full profile UI/discovery/cross-platform evidence. | PARTIAL |
 | T09 | SSH client/manager | Saved and ad-hoc SSH sessions support interactive PTY, authentication choices, reconnect/error UX, keepalive, host selection, and session lifecycle without shell injection or secret logging. Real-server integration tests cover success, rejection, disconnect, and timeout. | System `ssh` bounded command and interactive `ssh -tt` entrypoints; incomplete manager/auth UX/integration evidence. | PARTIAL |
 | T10 | SSH trust, keys, forwarding | Known-host verification defaults to fail closed and supports explicit fingerprint review/change handling. Identity selection, passphrases/agent, jump hosts, local/remote/dynamic and X11 forwarding, agent forwarding, and login scripts are validated and cancellable. Injection, bind exposure, and trust-change tests pass. | Options/argument modeling only; key selection, trust management, and real forwarding incomplete. | MODELED |
 | T11 | Telnet | Interactive Telnet negotiation, input/output, resize where supported, reconnect, profile persistence, and error states work against a real test server. UI clearly warns that Telnet is unencrypted and prevents secret-store assumptions. | Options model only. | MODELED |
 | T12 | Serial terminal | Device selection, baud/data/parity/stop/flow settings, readline, hex/hexdump, newline conversion, reconnect, unplug/replug, permission errors, and binary-safe logging are tested with loopback or hardware fixtures on supported OSes. | Options model only. | MODELED |
 | T13 | SFTP/Zmodem transfers | Upload/download, overwrite policy, cancellation, progress, resume policy, permissions, large files, Unicode paths, and failure cleanup work against real protocol fixtures. Local and remote traversal/symlink escapes are blocked; checksums verify completed transfers where available. | Request/path validation only. | MODELED |
 | T14 | Encrypted SSH secrets/config | Passwords, key passphrases, tokens, and protected config use an OS keychain or reviewed encrypted fallback; logs/debug/crash reports redact them; lock/unlock/rotation/migration/corruption behavior is tested. No plaintext secret is written to profiles, argv, temp files, or telemetry. | Redacting wrapper and `SecretStore` boundary only. | MODELED |
 | T15 | Themes and color schemes | Built-in and imported themes validate all tokens, preview/apply/revert atomically, persist, remain readable under contrast/accessibility checks, and cannot load active content or unsafe paths. Terminal ANSI colors and application chrome stay coherent. | Theme model and persisted app config; incomplete import/live UI/accessibility evidence. | PARTIAL |
 | T16 | Shortcuts and command palette | All commands are discoverable; single and multi-chord bindings can be edited, conflict-checked, reset, and scoped. Reserved OS shortcuts and text input are not intercepted unexpectedly; keyboard-only operation and persistence pass E2E tests. | Shortcut parser/duplicate validation only. | MODELED |
 | T17 | Plugins | Signed/trusted-source policy, manifest/schema validation, explicit least-privilege permissions, install/update/remove/rollback, compatibility checks, isolation, resource limits, and failure recovery are implemented. Malicious plugin fixtures cannot escape granted capabilities or access secrets. | Manifest/permission model only; execution intentionally absent. | MODELED |
 | T18 | Portable and web surfaces | Portable builds keep all intended state within the portable root and leave no host residue; upgrade/uninstall preserve or remove data as documented. Any web surface has authenticated encrypted transport, origin/session/CSRF controls, rate limits, and feature-specific E2E coverage. | Portable path resolver and static/Tauri shell only; no web service. | UI PROTOTYPE |
 ## Mandatory security gates
 Every gate is release-blocking; `N/A` requires a written rationale approved in the release record.
 | Gate | Required evidence | Baseline status |
 | --- | --- | --- |
 | Threat model and trust boundaries | Reviewed data-flow/threat model for wallet keys, signing, dapp/provider bridge, terminal input/output, protocol adapters, plugins, updates, and local persistence. | FAIL — not present |
 | Key and secret handling | Reviewed key lifecycle; production KDF/encryption parameters; zero plaintext persistence/logging/argv leakage; memory/lock/rotation/corruption tests; OS keychain or reviewed fallback. | FAIL — demo vault and boundaries only |
 | Transaction/signing safety | Deterministic test vectors, chain/origin binding, decoded preview and simulation policy, hardware flow coverage, user-rejection behavior, and no-sign/no-broadcast-by-default negative tests. | FAIL — no production signing |
 | Input and command safety | Fuzz/property tests for parsers and validation; shell/SSH/login-script argument safety; paste controls; URL/origin/path validation; traversal and symlink tests. | FAIL — partial unit validation only |
 | Network and trust safety | TLS policy, SSH known-host fail-closed behavior, RPC chain-ID verification, redirect/proxy policy, timeouts/retries/rate limits, certificate/trust-change tests. | FAIL |
 | Dependency and supply chain | Locked dependencies, license policy, vulnerability audit, secret scan, reproducible provenance/SBOM, pinned CI actions/toolchain, and release artifact signing. | FAIL — lockfile and pinned Rust toolchain alone are insufficient |
 | Desktop/web isolation | Tauri allowlist/CSP review, no unsafe remote content, permission minimization, provider origin isolation, CSRF/session controls for web, plugin sandbox before execution. | FAIL |
 | Privacy and telemetry | Documented collected/transmitted data, opt-in/opt-out behavior, log/crash redaction, retention/deletion policy, and tests preventing wallet/terminal secrets in telemetry. | FAIL |
 | Independent review | Security-sensitive changes reviewed by another qualified person; critical/high findings resolved or explicitly release-blocked. | FAIL |
 ## Mandatory E2E and reliability gates
 | Gate | Required evidence | Baseline status |
 | --- | --- | --- |
 | Wallet happy paths | Automated create/import/lock/unlock, connect dapp, review/sign/broadcast, send/receive, swap/bridge, approval revoke, restart/recovery against deterministic fixtures or test networks. | FAIL |
 | Wallet adverse paths | Wrong password, malformed import, insufficient funds, stale quote, simulation/provider/RPC failure, chain mismatch, rejected hardware action, dropped/reorged transaction, offline restart. | FAIL |
 | Terminal happy paths | Automated launch, local interactive shell, stream input/output, resize, full-screen app, tabs/splits/restore, SSH, transfer, settings, shortcuts, and close/restart on each supported OS. | FAIL |
 | Terminal adverse paths | Invalid executable/profile, PTY child crash, huge/rapid output, malformed VT bytes, disconnect/reconnect, host-key change, auth failure, transfer interruption, device unplug. | FAIL |
 | Persistence and migration | Fresh install, upgrade from every supported schema, rollback policy, corrupted/truncated state, backup/restore, concurrent write/crash consistency. | FAIL |
 | Performance/resource | Published budgets and measured startup, idle memory/CPU, terminal throughput/latency, scrollback limits, large portfolio, many accounts/tabs/panes, long-running leak/handle tests. | FAIL |
 | Accessibility/UX | Keyboard-only flows, focus order, screen-reader labels, contrast, scaling, reduced motion, localization overflow, destructive confirmations, actionable errors. | FAIL |
 | Linux GUI smoke | Release binary launches under Xvfb (or equivalent), remains healthy, and exits cleanly with logs retained as CI evidence. | FAIL — not run by `scripts/check.sh` |
 | Supported-platform matrix | CI/build/E2E evidence for every declared Linux, macOS, and Windows target; unsupported combinations are explicitly documented. | FAIL |
 ## Mandatory packaging and release gates
 | Gate | Required evidence | Baseline status |
 | --- | --- | --- |
 | Versioned release build | Clean checkout at a signed/tagged commit; locked release build; no untracked inputs; version/changelog match. | FAIL |
 | Native packages | Install, launch, upgrade, repair where supported, and uninstall tested for every advertised format/OS/architecture; desktop integration and permissions are correct. | FAIL |
 | Artifact integrity | SHA-256 checksums, signatures/notarization, provenance attestation, SBOM, and verification instructions are published over authenticated channels. | FAIL |
 | Update safety | Signed update metadata/artifacts, rollback/failed-update recovery, migration compatibility, and channel separation tested. | FAIL |
 | CI parity | CI runs formatting, full tests, checks, release build, security scans, E2E/smoke, and packaging from a clean checkout with retained logs/artifacts. | FAIL — CI currently runs only `scripts/check.sh` |
 | Release notes/support | Known limitations, security/privacy notes, supported platforms, data locations, backup/recovery, upgrade/uninstall, and vulnerability reporting contact are documented. | FAIL |
 | Release-candidate sign-off | Product, engineering, QA, security, and release owners sign the immutable evidence manifest; no unexplained skipped/flaky tests or open critical/high defects. | FAIL |
 ## Definition of production-ready
 A specific immutable release candidate is production-ready only when all of the following are true:
 1. Every required Rabby Wallet and Tabby row is `PASS`, or an explicitly out-of-scope row is removed from the advertised product with an approved rationale and no exposed incomplete control.
 2. Every security, E2E/reliability, and packaging/release gate above is `PASS`; no critical or high security defect and no release-blocking functional defect remains open.
 3. `./scripts/check.sh` and all platform E2E, GUI smoke, migration, security, performance, and package install/upgrade/uninstall suites pass from a clean checkout of the exact release commit with zero unexplained skips or flakes.
 4. Wallet cryptography, key handling, signing, provider isolation, and update mechanisms have independent review; terminal protocol trust, command/paste safety, transfer paths, and plugin isolation have threat-driven tests.
 5. Release artifacts are generated by CI from the tagged commit, signed/notarized as applicable, accompanied by checksums, SBOM and provenance, and verified on every supported OS/architecture.
 6. The evidence manifest records commit, toolchain, dependency lock hash, test reports, security reports, package identifiers/checksums, approvals, known limitations, and rollback/recovery procedure.
 7. Documentation accurately describes supported behavior and risks. Catalog or model coverage is never presented as runtime or production parity.
 At this baseline, Rabby is **not production-ready**. The passing Rust quality gate establishes a clean development baseline; the matrices and gates above identify the remaining production work without upgrading modeled or prototype behavior into unsupported claims.