Marshal
Sign in Get started free

Docs

Design

Agreed in the design session of 2026-10-10. Change this file when a decision changes.

Scope

  • Patching only. Marshal decides what gets installed and when, reboots in maintenance windows and checks servers come back healthy. It does not manage apps or services, inventory, monitoring or remote access (that work was tried and archived on 2026-10-14: tag archive/enterprise).
  • Windows Server 2019 and newer, and Linux servers (Ubuntu, Debian, RHEL family; Linux servers), in every edition. Windows 10/11 endpoints with Pro (client_os, Licensing): the same agent and Windows Update flow; without Pro they are kept and scanned but not patched.
  • Linux servers have no catalogue or approvals: each customer’s Linux policy (security upgrades only, or all) says what the install window applies. Everything else (windows, order, limits, reboots, health checks) is shared.
  • About 300 machines across many customers, mixing domains and workgroups.
  • Servers download from Microsoft Windows Update directly. They can all reach the internet.
  • OS updates only: Security, Critical, Update Rollups, Servicing Stack, Updates, .NET Framework, MSRT. Excluded: Defender definitions, drivers, feature upgrades, Microsoft Update products (SQL, Exchange…).
  • Replaces WSUS. Groups are rebuilt by hand; nothing is imported from WSUS.

Model

  • Customer = group. Every server belongs to exactly one customer. A test ring is just another customer.
  • Tags describe a server’s role (DC, SQL, IIS…) and carry: reboot order (priority), max rebooting at once per customer, and required services for the health check.
  • Approvals work like WSUS: per KB, per customer. New updates start as not approved; there are bulk approvals; superseded updates are declined automatically; a KB can be declined globally. (Phase 2)
  • Maintenance windows: separate install and reboot windows. Precedence is server override > tag override > customer default. Each customer has its own timezone. Windows recur weekly or monthly (“2nd Tue + N days”), and one-off windows and blackout dates can be added. (Phase 2)

Patching behaviour (Phase 2)

  • Approved updates install in the install window; the reboot happens in the reboot window.
  • The app reboots only for its own updates. If a reboot is already pending when the install window opens, that server is skipped and an alert is raised.
  • Health check after reboot: the server is reachable again and the tag’s required services are running, within 20 minutes (configurable). On failure the server is marked failed and the rest of that customer’s run stops; other customers carry on.
  • Update mode: servers in an active run are polled often; otherwise there is one touch a day (the daily scan, which also refreshes reachability and pending-reboot state).
  • Admins get buttons for scan now, run now, reboot now and abort.

Connectivity and security

  • Agents (agent edition, decided 2026-10-12; replaces Ansible/PSRP, Kerberos/NTLM, jump hosts and the Windows credential vault). A Go agent runs on each server as the MarshalAgent Windows service (LocalSystem, delayed automatic start, restarts on failure), installed from MarshalAgent.msi.
    • Communication (phase D): every check-in carries the server’s status and gets back the next interval and any jobs. Between check-ins an agent (0.4.0+) long-polls POST /agent/v1/wait: the console answers at once when there is work (queued job, removal, update offer), otherwise after 120 s, and the agent checks in again. Wake-ups travel as Postgres NOTIFY marshal_agent '<server id>' ('*' for all), sent in the same transaction as the change (services.jobs.wake_agents), and are received by a LISTEN task in the agent API. Waiting holds no thread and no database connection. The agent also checks in at once when a 30-second watcher sees a meaningful change (pending reboot or its reasons, a boot, the WU service, the WSUS policy, the OS build, free space moving 1 GB or crossing 5 GB) and after every job. While a run or job is going it checks in every 10 s. Older agents poll every 5 min (60 s while awaiting approval). A server is unreachable after 3 missed check-ins (about 6 min with long-poll).
    • Install links (phase D; key pin since 0.6.0): the one-liner fetches the per-token script from /agent/v1/install/<token> accepting only the agent API’s own public key (sha256 of the raw key as .NET’s GetPublicKey() returns it; the key is kept across certificate renewals, and through a proxy the console’s own certificate still arrives). The TLS handshake proves possession of that key, so no other certificate from the Marshal CA and no forged chain can pass. Or, when an admin says servers can reach the web address with a trusted certificate, irm <web address>/install/<token> | iex. Every token (mrt_…, mrp_…) is masked in the access logs.
    • Agent logs (phase D): one agent-YYYY-MM-DD.log per day in the logs folder next to the agent’s exe (C:\Program Files\Marshal Agent\logs, SYSTEM and Administrators only), split at 20 MB, lines stamped with local time; job steps are in the same file, tagged job=<id> type=<type>. 30 days kept, 200 MB at most. Idle check-ins are not logged, apart from an hourly alive line.
    • The console decides, the agent finishes. Approvals, windows, ordering and halts stay in the console; jobs are fixed verbs (scan now; install, reboot, self-update, uninstall later). The agent never runs scripts it is sent. Jobs are written to disk on receipt and results kept until the console has them, so a lost link or a restart loses nothing. Agent jobs live in the same jobs table (the worker never claims them); a job the agent took but never answered fails after a timeout (scan: 2 h).
    • Enrollment: an operator makes an install script for a customer: a reusable enrollment token (1 day by default, optional limit on uses, revocable; auto-approve is admin-only) that also carries the tags. The script downloads the MSI from the agent port under the token (/agent/v1/package/<token>/MarshalAgent.msi; without a token only agents and proxies get it, by certificate), trusting only the Marshal CA, then checking the file hash, and runs msiexec with SERVER_URL, TOKEN, CONSOLE_NAME, CA_FINGERPRINT; the same MSI works through GPO/Intune/RMM. New agents wait for approval. A server name already taken in the customer (exact, case-insensitive) is refused, except for a reinstall on the same machine (MachineGuid), which reattaches to its record with a new identity. The MachineGuid is only what the agent says, so a reattach goes back to pending (unless the token auto-approves), is audited as agent.reenrolled and raises the agent_reenrolled notification. Proxies the same.
    • Trust: Marshal runs its own CA (ECDSA P-256, data/keys/agent-api/ca.key). It signs the agent API’s server certificate (serverAuth; names from AGENT_URL + AGENT_TLS_NAMES) and the agents’ client certificates (clientAuth only, CN = agent id, 90 days, renewed when a third is left; the old one works until the new one is used). Agents pin the CA by its SHA-256 and check the host name and serverAuth, so no agent can pose as the console. Every request checks that the certificate is still the server’s current one: deleting a server or re-enrolling locks the old agent out.
    • The agent API is its own service on port 8443; the app terminates mutual TLS itself (client certificates optional at the TLS layer for enrollment, required by every other route). uvicorn does not hand TLS details to the app, so a small protocol subclass puts the verified certificate into the request scope. nginx must pass the port through, never terminate it. TLS 1.3 for agents and proxies (their minimum); TLS 1.2 only with ECDHE + AEAD ciphers, for the PowerShell one-liner on Windows without TLS 1.3. Bodies over 32 MB are refused while being read (chunked too). Routes without a certificate are limited to 600 requests a minute per address, and refused enrollments are audited at most 10 times per 10 minutes per address. It is the part on the internet, so it holds the least: no vault key and no signing private key (mounts data/keys/agent-api and data/agent-releases read-only), and its own database role marshal_agent (services/dbroles.py, refreshed by the api at every start) without access to users, sessions, OIDC state or notification channels, and only INSERT on the audit log.
    • Reachability: public HTTPS, a private address (per-customer agent URL), or Marshal proxies (below).
    • Self-update (Ed25519; agents get the public key at enrollment). Builds are signed where the image is built (agent/tools/sign, the key as the BuildKit secret signing_key from SIGNING_KEY_FILE; never in the image or ./data): <build>.sig.json holds a signature over the sha256 (agents before 0.6.0) and one over the manifest marshal-release-v1 / product / platform / version / sha256 / allow_downgrade. The console pins the public key once (data/keys/agent-api/agent-signing.pub, from the old private key on upgrade, else from the image’s signing.pub) and registers a build only if its signatures check out against that key; a build without them is not offered (online signing with data/keys/signing/agent-signing.key only with ONLINE_SIGNING=true, for development). Agents and proxies from 0.6.0 check the manifest and refuse a version that isn’t newer unless it was signed with allow_downgrade (a deliberate rollback). Each console image carries its agent build, registered as a release at start (copied to data/agent-releases/<version>/). Rings: Pilot and Broad follow the version an admin sets for them; Manual customers only get the version pinned on the customer. An agent (0.3.0 or newer) is offered its target only while idle (no jobs, not in a run). It downloads over mutual TLS, checks the sha256 and signature, renames its running exe to marshal-agent.previous.exe, puts the new one in place, schedules a rollback task (as SYSTEM, task XML so the time is locale-independent) and restarts the service. The new version disarms the task at its first good check-in; otherwise the task restores the previous exe and records the failure, which the agent reports until it updates. A version that failed on a server isn’t offered to it again until an admin retries.
    • Removal: Remove marks the server removing (disabled, queued jobs cancelled). At its next check-in the agent is told to uninstall: it says goodbye (certificate revoked, server archived as removed: out of lists and counts, update states dropped, run history kept), forgets its identity and starts msiexec /x detached. An agent that never checks in leaves the server removing until someone deletes it. A server removed before its agent ever checked in is archived at once. Re-enrolling the same machine brings the archived record back; another machine of the same name gets a new record and the archive is renamed. (A local uninstall from Programs and Features does not tell the console; the server then goes unreachable.)
  • Credential vault: AES-256-GCM, with the master key in data/keys/vault/master.key (kept outside the DB and its backups; mounted by the api and, read-only, the worker). It holds MFA seeds, notification and Entra secrets, and enrollment tokens (so the script can be shown again). Secrets are write-only through the API. A proxy token’s corporate proxy credentials are encrypted with a key derived from the token itself, so the agent API can write them into the installer it serves under that token; everywhere else (API, check-ins, logs) the proxy URL appears without them.
  • Keys (0.6.0): one folder per container, data/keys/vault (api, worker), data/keys/agent-api (api, agent API), data/keys/signing (only while an old signing key is still there; Settings warns). Only the api creates keys; the api moves a pre-0.6.0 flat data/keys into these folders at its first start.
  • Users: local accounts with mandatory TOTP and an argon2 password hash. Accounts lock for 15 minutes after 5 failures, an address is refused for 15 minutes after 20 failed passwords or codes, and TOTP codes can’t be replayed. Changing your password signs you out everywhere else. Roles are Admin / Operator / Viewer.
  • Customer limits: operators and viewers can be limited to customers. The limit is enforced in every API; other customers return 404 rather than 403, so they can’t be probed. Limited users can’t create customers, edit tags or tag windows, or decline globally. Admins are never limited. Customers have no access.
  • Microsoft (Entra ID) sign-in: OIDC authorization code flow with PKCE, state (also bound to the browser by a SameSite=Lax cookie) and nonce. The ID token’s signature (JWKS), issuer, audience and tenant are checked. Only users an admin pre-created as Microsoft accounts get in; they are linked by Entra object id at first sign-in. Entra enforces their MFA. They cannot use the password form.
  • Sessions use an httpOnly, SameSite=Strict cookie, __Host-marshal_session (Secure) over HTTPS. Only its sha256 is stored. Idle timeout 2 h, absolute 12 h. State-changing API calls also need X-Requested-With: marshal (CSRF).
  • Behind nginx: the UI port is published on 127.0.0.1. uvicorn trusts no forwarded headers; the app takes X-Forwarded-For/-Proto only from requests carrying X-Marshal-Proxy-Secret = PROXY_SECRET (app/proxyheaders.py), because Docker’s userland proxy makes every client look like the network gateway, so trusting by address would trust everyone. COOKIE_SECURE forces the Secure flag.
  • The first admin needs a one-time setup token from the API log.
  • Notifications:
    • Mail through Microsoft Graph sendMail (client credentials) or SMTP (STARTTLS, TLS or plain, optional sign-in), Teams Workflows webhook (Adaptive Card), Slack, and a generic HTTPS webhook (optional HMAC: X-Marshal-Signature-V2 over <X-Marshal-Timestamp>.<body>, plus the older X-Marshal-Signature over the body). Webhook URLs must be HTTPS.
    • Events go to an outbox table in the same transaction as the change. The worker delivers them every 10 s, up to 5 attempts, and records each channel’s last error.
    • Events: run halted, run with problems (one message per run listing the servers), every finished run (off by default), server became unreachable or refused login (on transition only), weekly approvals digest (Monday 08:00), MSRC sync failed.
    • Channels can be limited to customers. Those channels get a per-customer approvals digest instead of the fleet-wide one.
  • Keys and backups: data/keys (vault key, CA, signing public key) stays out of the automated database backups on purpose; it is backed up separately, and the signing private key lives on the build machine. Dumps are encrypted with age to BACKUP_AGE_RECIPIENT before they reach the disk (a warning is logged without it).
  • Vault key mismatch (data moved without master.key): secrets that can’t be decrypted raise a clear error where they’re used. The sign-in page and the rest of the app keep working.
  • Compliance: a server is compliant when it’s up to date, or updated and only waiting for the reboot. Declined updates don’t count. Other states: unreachable, not scanned in 3 days, missing critical or important, missing other. CSV exports (servers, missing updates, audit log) start with a UTF-8 BOM so Excel opens them correctly.

Proxies (phase E, decided 2026-10-12)

  • A proxy is a Linux agent (agent/cmd/marshal-proxy, static Go, linux-amd64/arm64) in a customer network that can reach the console. It belongs to one customer, or is shared (admins only) and usable by any customer. It replaces the phase C relay and the customer HTTP-proxy field.
  • Install: a proxy install token (kind proxy; ports; optional corporate HTTP proxy for its own way out). The dialog gives curl -fsSL -k --pinnedpubkey 'sha256//<agent API key>' …/agent/v1/proxy-install/<token> | sudo sh (the pin makes it safe without trusting the Marshal CA system-wide; the agent API keeps its key across certificate renewals), or docker run / compose with the image tarball the console serves (docker load, built in the image build by agent/tools/mkimage: one layer, non-root user, host networking). The systemd install runs the proxy as its own user; a root path unit runs the uninstall when the proxy drops a flag file after removal.
  • Lifecycle like endpoint agents, on its own API (/proxy/v1/…; a proxy certificate is refused on /agent/v1 and the other way round): enrollment, approval (nothing is forwarded before), check-ins with long-poll (wake key p<id>), certificate renewal (proxy certificates also carry serverAuth, for the cache), self-update (proxy rings; Linux: the binary is swapped and the process exits for systemd/Docker to restart it; a version that doesn’t confirm within 10 min exits and is rolled back at the next start), removal, and a proxy unreachable notification that lists the servers whose current route is that proxy.
  • Forwarding (port 8443 by default): blind TCP to the console’s agent port (directly, or CONNECT through the corporate proxy); endpoint TLS ends at the console. Optional allowed endpoint networks (CIDRs) are pushed from the console (the UI warns while there are none). Cache (8444, HTTPS with the proxy’s certificate, TLS 1.3): /cache/<sha256> for exactly the files the console lists at check-in (cache_allow: sha256 -> console path of the agent package and the agent releases; the client’s src is ignored then), only for approved proxies and allowed networks; fetched once with the proxy’s identity, checked against the sha256 before serving, and verified again by endpoints (pinned hash in the install script, signed updates). Unused files go after 30 days; at most 2 GB.
  • Proxy downloads: the install script fetches the binary under its token (/agent/v1/proxy-binary/<token>/<platform>); docker load goes through /agent/v1/proxy-docker/<token>, a script that downloads the image with the Marshal CA and checks its sha256 first.
  • Routes: ordered entries proxy:<id> / direct on a customer (default), a tag (shared proxies only) or a server; precedence server > tag > customer, none = direct. Check-ins return the resolved list (proxy endpoint address and ports) and the console’s name. Agents (0.5.0+) dial a route’s address but check the certificate for the console’s name (CONSOLE_NAME), test a new first route before switching (/healthz through it; otherwise they keep the old list and report route_error), fail over down the list on connection errors, return to the first route every 10 min when it answers, report the route in use, and download updates from the route’s cache (falling back to the console). Install links follow the customer’s/tags’ first route.
  • The address endpoints use is reported by the proxy (real interfaces, not container bridges) and editable (DNS, NAT).

Catalogue

  • Built from scans: each server runs a search-only IUpdateSearcher query against Windows Update (ServerSelection = ssWindowsUpdate, so leftover WSUS policies are ignored). It reports missing and installed software updates.
  • Trust between customers (0.6.0): each customer’s latest report of an update is kept (update_reports). The shared row takes what servers of two customers report identically (verified); while one customer alone reports it, its report is used; when customers disagree and none agree, the row stays and is disputed (shown in the catalogue). Supersedence declines updates for a customer only if the row is verified or reported only by that customer’s servers, so one customer’s agents can’t get others’ updates declined. Reports older than 60 days go.
  • Enriched from the public MSRC CVRF API (last 12 months, synced daily): KB → CVEs, severity (highest across products), impact, and “exploited”. The effective severity is the higher of MSRC and the update’s own MsrcSeverity.

Platform

  • Docker Compose on the Ubuntu control host: FastAPI + SQLAlchemy/Alembic + PostgreSQL 17, the agent API, a worker, and a React + Mantine UI served by the API. The image build also compiles the Go agent and packs it as an MSI (wixl).
  • Jobs live in Postgres (FOR UPDATE SKIP LOCKED), with 6 running in parallel by default.
  • Retention: detailed job logs are kept 180 days; job summaries and the audit log are kept forever. Nightly pg_dump with 14-day rotation.
  • Meant to sit behind nginx (deploy/nginx-marshal.conf): the UI port (HTTP_PORT, 8081) is published on BIND_ADDRESS, 127.0.0.1 by default; use 0.0.0.0 only if nginx runs on another host. The agent port is published on AGENT_BIND_ADDRESS.

Phase 2 details (decided while building)

  • Runs are per customer, per window occurrence. Servers whose tag or own windows differ from the customer’s form their own run, and reboot ordering applies within a run.
  • Tag conflicts: if a server has several tags with windows of the same kind, the tag that reboots first (lowest priority number, then name) wins.
  • Install (agent edition): the run queues an install job with the Windows Update ids of the approved updates the server’s last scan reported missing. The agent searches Microsoft Windows Update (ignoring a leftover WSUS policy), picks exactly those ids, and downloads and installs them one at a time, posting progress (“Installing 2/5 | KB…”). An empty list is refused by both the console and the agent. Updates no longer offered (installed or replaced meanwhile) are reported, not failed. If the agent restarts mid-install (service restart or a foreign reboot), it reports a failure instead of running again; the next scan shows what got installed.
  • Pre-checks use the last check-in: a server whose agent is not checking in fails (install) or is skipped (reboot) without a job being queued.
  • Agent job timeouts: a job the agent does not take within 3 polls + 2 min is cancelled (an idle agent polls every 5 min; agents switch to 10 s polling from 6 min before a window with work for them opens, and as soon as a manual run includes them). The install limit and the unreachable limit (agent silent for 20 min during the install) end the wait; a closed job’s late result is not taken.
  • Reboot (agent edition): one reboot job per attempt. The agent saves the job state, restarts Windows (shutdown /r /t 15, reason “planned, OS update”), and after the boot waits for the tag’s required services and reports health (missing services, stopped automatic services, reboot still pending). A server rebooting in a run is not flagged unreachable; the run’s limit (reboot timeout + service wait + 2 min) decides. A reboot that doesn’t happen within 15 min (cancelled with shutdown /a) fails.
  • Skips without failing: a pending reboot the app didn’t cause, a reboot the app still owes, less than 5 GB free, or a server that’s unreachable before its reboot. Install failures never halt a run; a reboot or health failure does.
  • Reboot: up to 2 reboots if a reboot is still pending afterwards. The server must be back online within 20 min, then its required services running within 5 min (both configurable). Manual “Reboot now” can force a reboot with nothing pending.
  • Concurrency: per customer, 10 installs / 2 reboots at once by default; tag max_concurrency applies on top. The worker runs install/reboot runs in their own pool (up to 50) so long runs don’t block other jobs; waiting on agents costs no more than a database read every Progress check seconds.
  • Abort stops new servers from starting. Servers already installing or rebooting finish, because Windows Update can’t be stopped halfway.
  • Worker restart mid-run: servers in flight are marked failed (Windows Update may have carried on, and the next scan shows what happened), and the run resumes with the remaining servers.
  • Live progress uses 3-second polling instead of WebSockets: simpler, and fast enough for runs measured in minutes.
  • Defender: the Microsoft Defender Antivirus product (definitions and platform) is excluded from the catalogue; Defender updates itself.

Licensing

Free and Pro; Pro also as a Marshal Cloud license, at fixed prices. Free has all the core patching for 1 customer, 25 machines and 3 users; Pro is unlimited and adds endpoints, team and sign-in features, chat notifications, reports, exports and the API. Licenses are Ed25519-signed and delivered online: a console bound to its organisation on our website picks its license up at a daily check-in (unbound or air-gapped consoles run Free). 30 days’ grace after expiry, then read-only. Details: Licensing.

Agent behaviour worth keeping

  • Windows Update COM calls run as SYSTEM (the agent service’s account).
  • Reboot-pending logic: CBS and WU flags count; PendingFileRename is only reported.
  • Clock skew against the console is reported with every check-in.
  • If a job is cut off, the next scan shows what really happened; there is no Windows Update history fallback.