Orbit 1.0 test matrix

This matrix defines platform evidence required for release. Cross-compilation alone is not platform support.

Evidence status

Record what has been exercised, on which platform, and when. An unmarked criterion is untested, not passing.

Last audit: 2026-08-01, macOS arm64 with clean-OS checks on linux/amd64 containers, at 4ff1ac8 (the history rewrite of 2026-08-02 renumbered every commit; this is the rewritten identity of the audited tree).

Verified in that audit:

• Clean-user setup through init --yes → up → status → open → down, including the mini-shop success path, the no-side-effect failure path, and stock compensation (scripts/smoke.sh in the external mini-shop example, then hosted in the retired orbit-examples repository and since folded into orbit-demo).
• Project-local orbit.yaml discovery, the documented ten-minute config verbatim, and the project/shared-environment switch warning.
• Doctor relevance (a Python+Redis environment reports neither .NET, Node, pnpm, nor SQL Server) and failure-first ordering with per-service remedies.
• Crash, port-conflict, missing-runtime, malformed-config, and abrupt daemon-exit recovery paths.
• Secrets absent from status, inspect, doctor, logs, and daemon.log with the container running.
• Dependency vulnerabilities (make vulncheck, pnpm --dir ui audit --prod).
• License review of all 101 shipped dependencies: every Go module and dashboard runtime package is permissive (MIT, Apache-2.0, BSD, ISC) except unmodified indirect hashicorp/yamux (MPL-2.0, file-level obligation satisfied upstream) and gsap, whose standard license carries an attribution requirement. Nothing conflicts with distributing Orbit under MIT. Attributions ship in NOTICE, regenerated by make notice.
• Dashboard serves loopback and answers a foreign Host header with 403; the daemon socket is srw-------, owner only.
• Unauthenticated access to everything a new user fetches: git clone of this repository and of the demo environment, the README installer URL, and the latest release with its 8 assets.
• Clean-OS install on a ubuntu:24.04 host that had never run Orbit: the README block installs the then-current preview (v0.4.0) linux binary with a verified SHA-256, orbit resolves in the same shell, --version and version agree, and init --yes → up → down brings all five demo resources healthy and back down.
• The SQL Server surface stays hidden in an environment that does not declare it, and sqlserver reset is labelled destructive behind a confirmation.

Automated by CI, so evidence is produced per release rather than recorded here:

• All six platforms above run the clean-user journey, the installer lifecycle (install, downgrade, failure, rollback), and native uninstall via .github/workflows/platform-smoke.yml.
• The SQL Server workflow runs against a real containerized instance via .github/workflows/sqlserver-smoke.yml, covering publish, state guards, procedure add/replay/deletion, table edit, external drift, and a baseline reset that asserts local rows are discarded. Those cases exercise the publish engine directly and not the CLI layer in front of it; the job comment in that workflow carries the boundary and why it matters, and is the copy to trust if this line ever disagrees with it.
• Publishing is gated on both: release.yml runs them after manual approval and before the release job, so a tag cannot be published without them passing.
• Dependency vulnerabilities run on every push and pull request.

Outstanding before v1.0.0:

• Repeat the clean-OS install below against the final release candidate, since it is the one check that exercises the published artifact rather than the source tree.
• Gate the contracts added after this audit, none of which have recorded evidence yet: orbit env info --json (readable with no daemon; observed values withheld on config_match: false; values require --show-secrets), the confirmation_required refusal when switch would stop running resources, and data.failed_resources evidence on a failed orbit up --json; named runtime isolation, resolved endpoints, instance- targeted recommended actions, and cleanup that preserves peer instances.

Known coverage gap, accepted for 1.0:

• No environment in this repository declares a type: dotnet service, so no automated run reaches the .NET paths in Doctor or the engine. Python, Node, and Go are all covered by the demo or an e2e fixture. A real-world config pointing path at a .csproj surfaced a Doctor bug that every existing test missed. Unit tests now cover that specific path; the broader gap remains.

Support levels

Platform	1.0 level	Release policy
macOS arm64	Supported	Failure blocks release
macOS amd64	Supported	Failure blocks release
Linux amd64	Supported	Failure blocks release
Linux arm64	Supported	Failure blocks release
Windows amd64	Beta	Smoke failure blocks the Beta artifact; non-safety parity gaps may be documented
Windows arm64	Beta	Build and basic launch coverage; limitations documented

Clean-user setup

Run with no existing Orbit home, daemon, environment files, or cached settings. When this test runs beside another Orbit runtime, give it a unique named instance. Verify that its daemon, Docker ownership, volumes, network, and resolved endpoints are independent, then clean it without changing the peer.

• Install from the release instructions.
• With a normal clean PATH that excludes ~/.local/bin, pasting the primary README install block unchanged launches orbit init in the same shell. The installer gives one executable next step and does not require a terminal restart; Windows updates both the current process and user PATH.
• Before setup, orbit status points directly to orbit init; it never recommends an orbit up command that must fail.
• Before setup, agent-first orbit inspect --json reports setup_required, does not invent a selected environment, and recommends only initialization.
• After setup but before startup, inspect lists configured resources as stopped and recommends only orbit up; users do not need to start the daemon separately.
• orbit --version and orbit version match the release.
• Replacing a running daemon's binary makes status, inspect, doctor, and mutations recommend only orbit daemon restart; one restart clears the update state even when another Orbit installation exists elsewhere.
• Normal orbit update replaces the binary that was actually invoked, reconnects a running environment, and restores exactly the resources that were running; update and rollback require no separate daemon command or second orbit up.
• With another Orbit installation first on PATH, invoking the active binary by absolute path recommends that exact binary, doctor reports the shadowing install, and the suggested restart clears the state in one step.
• orbit init --yes succeeds from an empty directory.
• From a project root or nested project directory, the nearest orbit.yaml is used without repeating -c; explicit -c still wins, and status reports the project source and exact path.
• A service or container with one declared port can omit the duplicate health port. An http/https service can also omit url; status, open, dashboard, and dependency injection use the selected runtime port.
• Default interactive orbit init accepts the public demo source and selected environment without waiting for input, and does not persist the empty current directory as workspace_root. Its human output presents a value-first quickstart and demo preparation; repository URL, ref, sync destination, and file count remain implementation details.
• A custom environment is asked for a project workspace only when its selected config references one. Non-interactive init returns the exact offline workspace-setting action instead of guessing a directory.
• An unresolved custom path variable keeps its name and recommends the persistent settings set-env action. It never mentions workspace root, recommends Doctor before remediation, or creates a recovery loop.
• The default environment syncs and is selected.
• Human Doctor reports Docker and Python for the demo; it does not report Git, Node, pnpm, .NET, SQL Server, or an inactive daemon. Git is diagnosed only when a shared-environment sync actually needs it.
• Human Doctor leads with failed and warning checks plus their exact remedies, then summarizes passing checks instead of making users scan a success list. The dashboard uses the same priority, exposes copyable command remedies, and keeps passing and informational checks collapsed until requested.
• Once required checks pass for a stopped environment, human and JSON Doctor point directly to the normal orbit up path rather than sending the user through Status or back through Doctor.
• orbit up --json reaches healthy for Redis and all four host Python services.
• Successful startup identifies one healthy application URL as the primary value-first action; the dashboard remains visible as a secondary option. When no started resource has a URL, the dashboard is the single fallback.
• The healthy dashboard leads with the environment result and one application action, not competing lifecycle controls. Stop remains secondary and infrastructure-only startup is available through progressive disclosure.
• The storefront and API URLs respond.
• orbit status --json reports all five resources healthy.
• Human status leads with the selected environment, not daemon process state or source-repository provenance, or an internal version identifier. Daemon and source details remain in the JSON contract (and source provenance remains in orbit env list) and appear in daily human output only when recovery requires them.
• orbit logs shop-order-api returns useful checkout-path output.
• orbit open opens or prints the dashboard URL.
• orbit down --json stops the environment.
• Repeating human or JSON down after Orbit has stopped succeeds as a no-op with one normal orbit up next action.
• Daemon stop leaves no managed host process or occupied demo port.

Existing-team setup

• Sync a non-default environment repository.
• Re-syncing identical environment contents reports Environment is up to date, does not rewrite files, and never asks the user to apply or restart anything.
• Normal command discovery presents sync as the complete refresh operation; the separate apply command appears only after a user explicitly defers an update or when Orbit detects pending changes.
• A changed active environment is described as an update to apply, without exposing daemon lifecycle. Applying validates the new config before stopping anything, leaves unchanged resources uninterrupted, restarts affected resources when needed, and keeps previously stopped resources stopped.
• A downloaded-but-deferred update remains discoverable and can be applied by a later orbit env apply, even when a second sync writes no files.
• Resources removed by the new environment are reported explicitly; invalid updates leave the current daemon and resources running.
• Authentication failure identifies the repository and login remedy.
• A missing Git executable is explained at the shared-environment sync boundary; project-local environments and daily Doctor output do not present Git as a runtime prerequisite.
• Invalid YAML points to the field and source line.
• Unknown top-level, core, and registered-extension fields fail before startup; schema mismatches provide exactly one advancing action: source sync when the shared environment is older, Orbit update when the binary is older. A project-local schema 2 file instead gives the exact schema 3 edit and migration URL; Doctor and status never recommend one another in a loop.
• Missing host runtime identifies every affected service, including a Go service whose command starts with go.
• Project runtime version files, including Go's go.mod, are checked without installing or switching a runtime; mismatches and conflicts name every declaration but remain non-blocking warnings.
• Missing Node package installation is distinguished from a missing runtime for package-manager commands and direct commands such as node server.js; nearest-workspace packageManager or lockfile selects one explicit setup command, and that package manager is itself checked before startup. If the manager is missing, Orbit leads with that prerequisite and does not yet recommend an install command that cannot run. Services sharing one workspace are named together and produce one package-install action, not one duplicate action per service.
• A Python service with an unsatisfied project-root requirements.txt fails Doctor before any resource starts. The check names the service and provides one explicit setup command for the exact configured interpreter; Orbit never runs that command implicitly. After the user executes it, Doctor recommends only normal startup.
• A PostgreSQL-only environment exposes orbit query postgres, uses the container's POSTGRES_USER and POSTGRES_DB by default, and does not expose the optional SQL Server orbit sqlserver workflow.
• One matching Redis, MongoDB, or PostgreSQL container is selected without ceremony. Multiple matching containers produce a stable candidate list and require --container <name>; target choice never depends on map order.
• Container seeds use one in-container command with files on standard input; core config owns no SQL Server or MongoDB credential fields. Changing the command or file content invalidates the recorded seed without applying it implicitly.
• A host service with one declared port receives that value as both PORT and the label-specific variable.
• Declared ports never move. Occupying one fails the start with a port conflict instead of a silently different address.
• Port conflict identifies the port, Orbit resource, and observed owner where the platform permits it.
• Doctor uses the same loopback bind semantics as startup, so an IPv4 listener cannot pass diagnostics and then fail up. A stored port conflict is revalidated live: while occupied, Doctor recommends only owner inspection; after release, it reports the port available and recommends one targeted retry without restarting healthy peers.
• Requesting logs for a resource that never started does not return an empty success. Human and JSON output explain why no logs exist and route directly to live conflict inspection or the targeted retry, depending on current port ownership.
• A container port claimed after Doctor's availability check is reported as the same structured port conflict; Docker endpoint and networking-driver errors never reach human or JSON output, and recovery recommends inspection rather than logs or a blind restart.
• A dependency-blocked resource with no buffered output does not offer an empty Logs action. When buffered output exists, opening Logs hydrates the daemon snapshot before continuing with live streamed lines.
• Invalid switch targets leave the selected environment and its running resources unchanged.
• Switching while Orbit is stopped only records the selection and checks its prerequisites. It does not start background state, bind the dashboard port, or introduce a second startup path before orbit up.
• Switching environment updates the required-tool checks before startup.
• A successful switch describes the selected environment rather than daemon lifecycle, and human and JSON output point directly to the normal orbit up path.

Lifecycle and recovery

• A new user can sync environment sources, set a source workspace before the daemon starts, switch, run Doctor, and start without learning daemon lifecycle.
• An unresolved or missing host-service directory is named as the root cause before any container or process starts. JSON uses service_working_directory_missing with one exact recovery action.
• Infrastructure-only startup ignores unrelated host-service directories.
• Repeating up, down, and restart is safe and convergent.
• Fifty sequential targeted host-service restarts complete without a false port conflict or owner-inspection action. Each success means the prior process has exited and released its listener before the replacement becomes healthy; unrelated healthy dependencies are unchanged.
• After an abrupt daemon exit, status describes previously live resources as reconciling rather than stopped and recommends only normal orbit up. Startup adopts matching namespaced containers and persisted live host processes (or safely replaces stale owned resources); it never reports their ports as externally owned. A genuinely unrelated listener remains a normal port conflict with one owner-inspection action.
• After down, human and JSON status recommend only the normal orbit up path; they never lead a first-time user into an infrastructure-only partial environment. The completion message explains that Orbit remains ready for the next start without teaching daemon lifecycle.
• Leaving a project directory does not replace its still-active context with a previously selected managed environment. Status and Doctor identify the active project and give its directory or explicit config path without exposing a daemon restart; an explicit --config still fails closed on a mismatch.
• A valid environment with no enabled resources makes human and JSON up succeed immediately, explains the no-op, and never prints starting 0 or waits for a health timeout.
• up --group and up --infra wait only for the resources the daemon actually selected, including required dependencies.
• Human up narration describes the user's selection mode (environment, infrastructure, group, or named resource); it never labels a container as a service.
• Normal up, down, open, and interrupted-start help or recovery text describes the user operation without exposing daemon lifecycle.
• A restart blocked by a terminal dependency fails promptly, names the dependency, and leads to its logs instead of waiting for the global timeout.
• Failed lifecycle actions exclude healthy or still-converging peers when a terminal root failure is known; pending dependency chains lead directly to that root resource's logs.
• Ctrl-C during startup leaves no orphan host process.
• A service crash is visible in status, logs, and dashboard. Human CLI, JSON, and agent inspect expose one linear recovery path: status/Doctor/inspect lead to logs, logs lead to a targeted restart, and no command recommends itself. The targeted restart leaves healthy dependencies unchanged.
• The dashboard environment action follows the same root-cause chain: start a stopped root dependency, open logs for a degraded root, then return to the healthy application's open action after recovery.
• An empty healthy trace receiver does not add an unrelated dashboard workspace. The first received trace reveals Tracing, a receiver failure reveals it for recovery, and a direct tracing link remains usable.
• A deterministic process exit keeps its exit status and last meaningful application-log evidence in status, Doctor, JSON, and dashboard. Follow-on context cancellation never replaces the specific cause.
• When a supported dependency check explains the crash, Logs recommends setup followed by a targeted restart as one advancing action; it never recommends an unchanged blind restart.
• A recovered health endpoint returns to healthy without restarting Orbit.
• Docker restart is detected and resource state converges.
• System sleep/resume does not leave a false healthy state.
• A stale daemon state file or socket is recovered automatically. A live, unrelated process whose PID was reused from stale metadata is never terminated; Orbit proves daemon ownership before retiring a process.
• Two isolated Orbit homes/namespaces can run without state collision.

Optional SQL Server workflow

• The release commit passes the live SQL Server smoke workflow, not only mocked or package-level tests.
• It is completely absent when not configured.
• Explicit configuration reveals CLI, Doctor, and dashboard surfaces.
• Missing .NET, sqlpackage, project, credential, target, or port has one specific remedy.
• Diff is non-destructive.
• Publish preserves data by default.
• Possible data loss is blocked without --allow-data-loss.
• Reset and forced publish require confirmation.
• Secrets are absent from all output and retained logs.

Installation lifecycle

• First install does not overwrite a newer binary.
• Update preserves settings and environments.
• Rollback to a named release is documented and verified.
• Checksum mismatch aborts before replacing the binary.
• Interrupted download leaves the previous binary usable.
• Uninstall distinguishes binary/daemon removal from optional user-data removal.

Release and security

• Full preflight and platform jobs pass on the release commit.
• The dashboard binds loopback only, rejects non-loopback Host headers, and rejects cross-site browser mutations without blocking Unix-socket CLI calls.
• The daemon socket is accessible only to its owning operating-system user.
• Every artifact reports the same version and commit.
• Checksums verify for every binary.
• SBOM and provenance refer to the published artifacts.
• Secret and reachable-vulnerability scans pass.
• Anonymous clone, installer, release download, and demo sync pass after the repositories are public and before creating v1.0.0.

Windows Beta additions

• PowerShell or documented shell installation succeeds.
• --version, init, doctor, config parsing, and JSON output smoke tests pass.
• Host process stop behavior and path quoting are exercised.
• Docker Desktop integration limitations are explicit.
• Unsupported behavior says Beta limitation; it does not fail with a POSIX implementation detail.