Upgrade Guide

Docker Compose Upgrade

1. Back up the database

   ./scripts/backup.sh --db

   Database migrations are forward-only. Rolling the container tag back does not roll the schema back. See Backup & Restore.

2. Diff your Compose files against the tagged release and merge the required changes before pulling images; follow Compose file changes, preserving your local configuration. Back up .env, the active Compose files, and overrides alongside the database backup.

3. Update the version in .env

   # Set to the version you want (e.g. 0.11.3)
   BREEZE_VERSION=0.11.3

   This also selects the agent release your fleet is offered, so set it to a version that is published on GitHub, not one that has only been tagged. See Binary Distribution.

4. Pull and restart

   docker compose pull
   docker compose up -d

   This pulls the new API, web, and binaries images and restarts all services. Database migrations run automatically on API startup when AUTO_MIGRATE=true (the default).

   Expected downtime: Typically 5–15 seconds while containers restart. If the new version includes database migrations, the API container will take additional time to apply them before accepting traffic – usually under 60 seconds for routine schema changes, but potentially several minutes for large data migrations. Agents continue running independently during this window and reconnect automatically.

5. Verify

   # Check API health
   curl -s https://breeze.yourdomain.com/health | jq .version
   
   # Check all running image versions
   docker compose images
   
   # Check agent version (on the device)
   breeze-agent version

   Also sign in through your public URL and confirm agents resume checking in. /health bypasses the HTTPS redirect and can stay healthy while a stale proxy address sends other requests into a redirect loop.

Tip

The binaries-init container uses pull_policy: always, so docker compose up -d automatically pulls the latest binaries image and refreshes the volume — no manual steps needed.

Compose file changes

docker compose pull updates images; it does not download a new docker-compose.yml, override file, or .env.example. Current images can therefore run with old network settings or missing environment mappings. Review the configuration shipped with every target release, including when skipping versions; the entries below highlight changes that need operator action, not a complete Compose changelog.

Fetch a separate copy of the target release’s Compose file and compare it with the one you actually deploy. Replace the placeholder with the published release tag matching your intended BREEZE_VERSION:

release_tag="v<target-version>"
release_compose=$(mktemp)
curl --fail --show-error --location \
  "https://raw.githubusercontent.com/LanternOps/breeze/${release_tag}/docker-compose.yml" \
  --output "$release_compose" && diff -u docker-compose.yml "$release_compose"

diff exits with status 1 when differences exist; a failed download must be resolved before continuing. If you deploy deploy/docker-compose.prod.yml or additional override files, compare those same paths at the same tag too. Review the tagged .env.example and any configuration files referenced by changed services.

Merge the required changes into your backed-up local files, preserving custom services, domains, mounts, secrets, and proxy topology. Do not overwrite .env with .env.example. Add required variables to both .env and the appropriate service’s environment: mapping; setting .env alone does not pass a variable to the container. For digest-pinned deployments, update the image references to the target release as well as BREEZE_VERSION.

Validate the merged configuration before the pull/restart step:

docker compose config --quiet

Use the same -f options and profiles you normally deploy for validation, pull, and restart so your overrides are included. Validation checks the Compose configuration; it does not prove that a trusted proxy address matches the running proxy.

v0.95.0 — stable Caddy address and proxy trust

The v0.95.0 Compose file introduced these coupled settings. If your deployed file predates them, apply all three even if your images are already newer:

Compose setting	Required action
networks.breeze.ipam.config.subnet uses BREEZE_DOCKER_SUBNET	Merge the IPAM configuration. If the shipped subnet overlaps another Docker network, host route, or VPN, choose a non-overlapping subnet in .env.
services.caddy.networks.breeze.ipv4_address uses BREEZE_CADDY_IP	Merge the static address pin. If you customize the subnet, choose an unused Caddy address inside it and set BREEZE_CADDY_IP alongside BREEZE_DOCKER_SUBNET.
services.api.environment.TRUSTED_PROXY_CIDRS defaults to BREEZE_CADDY_IP with a /32 mask	Merge the derived default. An explicit TRUSTED_PROXY_CIDRS in .env or an override takes precedence: remove a stale override when using bundled Caddy, or set it to the actual immediate proxy’s stable address for a custom topology.

Keep TRUST_PROXY_HEADERS=true when using the bundled proxy. Do not widen trust to an entire private network to work around an address mismatch. A proxy’s dynamically assigned address can move after a container recreation or host reboot.

Changing an existing network’s IPAM/subnet requires recreating that network during a maintenance window; docker compose restart cannot apply it. For a standalone stack whose network is used only by that stack, stop/remove its containers and network with docker compose down, then recreate them with docker compose up -d after pulling the target images. Do not use down -v, which deletes named data volumes. If other containers share the network, coordinate their migration before recreating it. This network migration takes longer than the routine image-only restart described above.

Before upgrading a proxy-fronted deployment to v0.103.0 or later, verify that TRUSTED_PROXY_CIDRS matches the proxy that directly connects to the API. That release’s TRANSPORT-001 enforcement only honors forwarded HTTPS headers from trusted peers; a stale address can cause repeated 308 redirects for the UI and agents while /health still returns 200. After restarting, check public sign-in and agent check-ins as well as API logs for [proxy-trust] or [force-https] warnings (available from v0.104.0).

Version-specific notes

v0.113.0 — remote desktop stability, AI-authored scripts, service deliverables

No breaking API changes. Two settings deserve a decision before you pull: APP_ENCRYPTION_KEY_ID (now a boot refusal for split-role deployments and a 503 on integration credential saves everywhere else) and BREEZE_AI_SCRIPT_AUTHORING_ENABLED, which is now on by default. Read Action required before you pull.

If remote desktop has been dropping every 30–90 seconds since early August, take this release

Since late July the orphan-session sweeper was ending healthy peer-to-peer viewer sessions because it only recognised a WebSocket-owned session as alive — and the default P2P transport never opens that socket. The fix is entirely server-side; no agent update is needed for it. The separate secondary-monitor fix (input and cursor landing on the wrong monitor on Windows hosts with mixed DPI scaling) does need the updated agent and helper.

Pull api, web and portal together, in one pass (plus worker if you run the worker split). The customer portal gains two new sections and the web dashboard gains new navigation, so a mixed set of images shows menu entries that 404. Afterwards assert version parity on every ghcr.io/lanternops/breeze/* container rather than trusting the service list.

Action required in v0.113.0

1. Set APP_ENCRYPTION_KEY_ID if you have not already. The API refuses to boot without it when BREEZE_ROLE is api or worker or when M365_GRAPH_ACTIONS_TOOLS_ENABLED=true. Every other deployment boots with a warning, but saving integration credentials returns 503 — Discord, Teams, Grafana, OpsGenie, PagerDuty, monitoring webhooks, ticketing and PSA settings — until the key id is set, because those credentials are now sealed at rest with key-bound ciphertext and no longer fall back to unbound ciphertext. Add any [A-Za-z0-9._-]+ identifier to .env and map it in the api (and worker) service’s environment: block; installs whose guided-setup.sh run predates the key never had it generated. Setting it does not rotate APP_ENCRYPTION_KEY. See Environment Variables.
2. AI script authoring is on by default. The chat assistant and background AI agents can now write a script as a reviewed proposal and, once a human approves it on a card, run it. Set BREEZE_AI_SCRIPT_AUTHORING_ENABLED=false to keep the feature dark. Two related permission changes: approving a proposal that matched a Strict danger pattern requires the approver to hold scripts:write in the proposal’s organization and to have completed MFA (the same bar the script library already applies), and four-eyes requests for such proposals fan out only to approvers who hold scripts:write — grant it to whoever decides AI script runs. The unattended lane ships off: it needs a partner to open the ceiling and each organization to opt in, and no environment variable controls it. See AI Script Authoring.
3. Two new permissions, documents:read and documents:write, gate the new organization document library and the Documents tab. The migration grants both to the built-in Partner Technician, Org Admin and Org Technician roles; custom roles get nothing until you add them under Settings → Roles. Service deliverables themselves use the existing contracts:read / contracts:write.
4. If you take online payments through Stripe, resetting a payment link, recording a manual payment, voiding an invoice or replacing the Stripe key now refuses with 503 STRIPE_REVOCATION_PENDING when Breeze cannot prove the customer’s open Checkout session is dead. A 60-second sweep retries, so the refusal is brief; if it ever blocks invoice work during an incident, set STRIPE_SESSION_REVOCATION_MODE=observe rather than rolling back.
5. Customers with Microsoft 365 Graph-read consent see a re-consent prompt. The customer-graph-read manifest moves to v3 with four additional Graph permissions for the upcoming posture program. The prompt is non-interrupting — existing connections keep working until the customer administrator re-consents.

Database migrations in v0.113.0

22 migrations (2026-10-15-170300 through 2026-10-16-170700). Take a database backup first. All are idempotent, apply automatically on API startup unless AUTO_MIGRATE=false, and have been dry-run as a non-superuser role. None rewrites a large table. The ones with operator-visible effects:

Migration	Effect
2026-10-15-170400-documents-permissions.sql	Seeds documents:read / documents:write and grants them to the built-in Partner Technician, Org Admin and Org Technician roles (counts logged as Postgres WARNINGs).
2026-10-16-100000-script-versions-immutable.sql	Creates one head-version row per existing script and makes script versions append-only — every save now cuts a new immutable version.
2026-10-16-100300-scripts-origin.sql	Adds origin (human / ai / system) to scripts and backfills system on the built-in ones.
2026-10-16-160200-checkout-session-revocation.sql	Marks aged-out pending Stripe Checkout session mappings revoked and pre-existing ones legacy_unbounded; adds the revocation-intent bookkeeping.
2026-10-16-170100-mfa-enrollment-grace.sql	Adds the enrolment-deadline columns on users. No rows written.
2026-10-16-110000-portal-branding-service-documents-flags.sql	Adds the portal Service and Documents visibility flags, both false.

The rest — organization documents, script proposals and policies, monitor definitions, the remote-session start generation, M365 tenant-sync foundation, Fleet Designer and device-function tables — are DDL-only additions with no backfill.

New settings in v0.113.0

Only APP_ENCRYPTION_KEY_ID (item 1 above) is required, and only for the configurations listed there. Full descriptions are in Environment Variables.

Variable	Default	Why you might set it
BREEZE_AI_SCRIPT_AUTHORING_ENABLED	true	Set false to keep AI script authoring dark.
BREEZE_AI_SCRIPT_REVIEWER_MODEL	—	Platform-default model for the independent script reviewer; a partner or organization policy can override it.
STRIPE_SESSION_REVOCATION_MODE	enforce	observe is the incident de-escalation lever for item 4 above.
M365_SYNC_CONTINUATION_KEY, M365_SYNC_MAX_IN_FLIGHT, M365_MAX_IN_FLIGHT	— / 4 / 32	Read by the separate m365-graph-read-executor sidecar only. Without a continuation key the executor mints one per process and a sync resumes from page 1 after a restart — self-healing, so existing executors need nothing.
BACKUP_GC_ORPHAN_MANIFEST_MAX_AGE_MS, BACKUP_BASE_LEASE_MS, BACKUP_RESTORE_PIN_LINGER_MS, BACKUP_PUBLISH_MARGIN_MS	9 d / 7 d / 7 d / 1 h	Lab/test knobs for the backup storage sweep. Production enforces floors and logs when a lower value is ignored. Leave unset.

New infrastructure: one new queue, script-verify, consumed by the API replica that owns the agent socket. No new service, no new port.

Rolling back v0.113.0

Prefer roll-forward. Every schema change is additive, so rolling the images back needs no schema rollback; scripts saved on v0.113.0 keep their version rows. Do not revert the Checkout-session migration to escape a revocation refusal — flip STRIPE_SESSION_REVOCATION_MODE=observe instead, otherwise the older code ignores the pending intents and re-opens the duplicate-charge window.

Behaviour changes in v0.113.0

• Navigation. Automations now live under a top-level Jobs entry (bookmarked /automations/* links redirect); the sidebar’s Network Monitor entry is now Monitoring, a hub with Monitors, Network, Delivery and Legacy rules tabs; Devices is now Devices & Assets, with your last five devices under it and a Cmd+K palette that opens on recent devices and pages. See Automations and Monitors.
• Monitors are the new single object a technician authors — condition, severity, delivery and responses in one place — compiled into the alert templates, alert rules and automations that already run. Existing alert rules keep working; Legacy rules offers one-click conversion.
• Service deliverables and the organization document library ship with two new customer-portal sections, Service and Documents, both off per organization until you turn them on. A daily sweep opens tickets for deliverables that come due and for key-date reminders. See Service deliverables and Customer Portal.
• AI Fleet Designer is a fourth AI-agent kind, designer, that reads the fleet and writes a Fleet Design report. It is inert until an administrator creates one for an organization, and nothing in its design is applied automatically. It also assesses a Function for each device (domain controller, kiosk, …), shown next to the billable Device Role. See AI Agents.
• Consecutive supervised Tier-3 approvals need one step-up ceremony — a 120-second grant bound to the approver’s session — instead of one per decision. Four-eyes approvals still take a fresh ceremony each time.
• MFA enrolment grace window. A user who lands in a Force MFA role and has never held a factor gets a nonrenewable 14-day window (banner plus email notices) before the 428 enrolment wall applies. Inert unless MFA_FORCE_FOR_PARTNER_ADMIN=true, whose default is unchanged (false). See Users & Roles.
• Backup storage usage starts to drop. The reclamation sweep now deletes retired restore points’ exclusive data and orphaned upload prefixes, on top of the loose-object grace rules. See Storage.
• Integration credentials are sealed and masked. Discord, Teams, Grafana, OpsGenie, PagerDuty and monitoring-webhook secrets are encrypted at rest and never echoed back to the settings form; re-enter a secret to change it.
• Alert-correlation views honour the site ceiling and alerts:read. A site-restricted or permission-stripped reader no longer sees the whole organization’s correlation graph.
• Approver devices now distinguish Hardware-attested from Not attested platform-bound mobile devices. See Approval Security.
• M365 tenant-sync foundation lands schema and executor plumbing only — nothing new in the dashboard this release.
• Remote desktop start and stop commands are now strictly ordered per session on both server and agent. No operator action; a late start can no longer win against a terminal decision.

v0.112.0 — Windows MSI fix, security remediations, bare-metal recovery

No breaking API changes and no new required environment variables, but this release removes implicit access and pauses some scheduled work the moment the migrations run. Read Action required before you pull.

If you enroll Windows devices with the MSI, take this release now

The agent MSI published with v0.110.0 through v0.111.1 refused every fresh install on Windows 10, 11 and Server 2016+ with “Breeze Agent requires Windows 10 or Windows Server 2016 or later”. Already-enrolled agents were unaffected (self-update never runs the MSI); only new MSI enrollments were blocked. v0.112.0 reads the Windows version from the registry instead of the value Windows Installer reports. Replace any installer links or MSIs you handed out from those releases. See Supported Windows versions.

Pull api, web and portal together, in one pass (plus worker if you run the worker split). Two changes in this release have no flag-level rollback: session-binding enforcement (below) and the remote desktop lease, which expects every replica on the new protocol. Sequence the API restart as stop-then-start rather than rolling: stop the old API replicas, let the migrations apply, then start only the new ones — an older API must not resume serving after the installer-token cutover (item 6 below). Afterwards assert version parity on every ghcr.io/lanternops/breeze/* container rather than trusting the service list.

Action required in v0.112.0

1. PAM auto-approve rules are suspended. Every rule with verdict Auto-approve is switched to Require approval and badged “Auto-approve suspended”. Matching elevations wait for a human until an admin holding the new pam:manage_policy permission clicks Re-approve on the rule (MFA). See PAM → Rules.
2. New permissions replace implicit access. pam:approve, pam:manage_policy, accounting:read, accounting:manage, workspace:read|write|credentials|execute and connected_apps:read|manage are granted to the built-in Org Admin only (Partner Admin holds *:*). Every other built-in role and every custom role loses QuickBooks, Workspace, connected-apps and PAM-approval access until you grant the keys in Settings → Roles. A Partner Technician or Partner Billing user who could browse or sync QuickBooks yesterday gets a 403 today. See Built-in permissions.
3. Recurring sensitive-data scan policies are switched off and any queued or running scheduled scan is failed. There is no badge: each affected policy simply shows as inactive on the Scan Policies tab. A user who holds devices:write and devices:execute, with MFA satisfied, re-approves it by turning Active back on (or re-saving it), which binds the schedule to that user’s authority. Manual scans are unaffected. See Sensitive Data.
4. Recurring network-baseline scans are paused with the banner “Recurring scan paused: re-approval required”. Re-saving the schedule arms it under the saving user; Scan Now still works. Newly auto-created baselines now start with recurrence off. See Network Baselines.
5. Office add-in technician bindings are revoked. Every technician who uses the Outlook/Office add-in technician pane repeats the one-time sign-in; the account must now have an authenticator-app (TOTP) factor. See AI for Office.
6. Outstanding installer bootstrap tokens are invalidated. Unused bootstrap-derived enrollment keys are deleted. Regenerate any installer links or copies you handed out before the upgrade.
7. Remote desktop needs an updated agent and is capped at 12 hours. Only an agent that reports lease support can start a desktop session; older agents show “Remote desktop needs an agent update on this device” (HTTP 503 agent_upgrade_required). Expect up to one heartbeat interval (about a minute) per device while agents self-update; if you pin agent versions, desktop sessions are unavailable until you move the pin. Max session duration is clamped to 1–12 hours (0 no longer means unlimited), a membership/role/site-scope/MFA change ends a live session within about 25 s, and a Redis outage longer than 90 s ends live desktop sessions. Terminal, Files and VNC are unaffected. See Session lease and revocation.
8. AUTH_BROWSER_TRANSITIONS_ENFORCED is removed. Session binding is enforced on every issuance path (password, SSO, Cloudflare Access, MFA, passkey, refresh, registration, invitation) with no opt-out. Delete the line from .env; the safety net is an image rollback, not a flag. Existing browser sessions self-heal on their next refresh. Mobile app builds older than v0.109.0 cannot sign in until updated.
9. Legacy policy execution routes are gone. POST /policies/:id/evaluate, /remediate and /activate are retired; use POST /automations/:id/trigger. /deactivate is unchanged.
10. Site-restricted technicians can no longer create or edit organization-wide governance objects (webhooks, notification channels, software and peripheral policies, configuration policies, PAM config, backup destinations and profiles) or manage PSA and DNS-security connections. They can still assign an existing policy to their own site.
11. Check TRUSTED_PROXY_CIDRS. With proxy trust unset, client-IP attribution now records the raw socket peer instead of unknown. Behind a real proxy that peer is the proxy, so make sure the variable is set on every deployment that fronts Breeze with one.

Database migrations in v0.112.0

20 migrations. Take a database backup first. All are idempotent, none abort on pre-existing data, and all apply automatically on API startup unless you have set AUTO_MIGRATE=false. The ones with operator-visible effects:

Migration	Effect
2026-10-15-150200-pam-dedicated-permissions.sql	Suspends every auto_approve PAM rule (original verdict kept for re-approval); seeds pam:approve / pam:manage_policy and grants them to each built-in Org Admin.
2026-10-15-150051-office-addin-binding-mfa-epoch.sql	Revokes every existing Office add-in technician binding.
2026-10-15-140001 / 140002 / 140003-sensitive-data-*.sql	Deactivate every recurring sensitive-data policy and fail queued/running scheduled scans.
2026-10-15-141001-installer-bootstrap-credential-generation.sql	Deletes unused bootstrap-derived enrollment keys and bumps the generation on their parents.
2026-10-15-150030 / 150031 / 150032 and 150500-accounting-dedicated-permissions.sql	Seed workspace:*, connected_apps:*, accounting:read, accounting:manage; grant them to the built-in Org Admin only.
2026-10-15-140004-backup-snapshot-lineage-fk-set-null.sql	Makes restore_jobs.snapshot_id and recovery_tokens.snapshot_id nullable and moves five foreign keys onto backup_snapshots to ON DELETE SET NULL, so retention can finally delete a snapshot that was ever restored, verified or tokened.
2026-10-15-150600-network-baseline-recurring-authority.sql	Pauses every enabled recurring baseline scan (reapproval_required) until an authorized user re-saves the schedule.
2026-10-15-150001-bound-webhook-retries.sql	Normalizes legacy webhook retryCount values into 0–2; the count is logged as a Postgres WARNING.
2026-10-15-150050-mobile-refresh-family-binding.sql	Adds a partial index on refresh_token_families in-transaction (not CONCURRENTLY). Instant at typical sizes; run off-peak if yours is very large.

The rest (150010, 150020, 150040, 150100, 150300, 160010) are additive column adds with no data change. A non-superuser migrating role is sufficient, as before.

New optional settings in v0.112.0

None are required. Full descriptions are in Environment Variables.

Variable	Why you might set it
LENOVO_WARRANTY_ENABLED=true and/or LENOVO_API_KEY	Lenovo warranty lookup works for the first time — credential-free via Lenovo’s public lookup endpoint, or through the official Warranty API. DELL_CLIENT_ID / DELL_CLIENT_SECRET and both Lenovo knobs are now mapped through every shipped Compose file, so a value in .env is enough. See Warranty lookups.
RECOVERY_MEDIA_WORK_DIR	Already defaulted to /data/recovery-work by the shipped Compose files; moves bare-metal recovery media builds off the container’s 64 MB /tmp. Override only to relocate it.
BACKUP_GC_GRACE_MS	Lab/test only. Production enforces a one-hour floor; the 48 h default is unchanged.

Removed: AUTH_BROWSER_TRANSITIONS_ENFORCED (see item 8 above).

Fresh installs and image pinning in v0.112.0

.env.example and deploy/docker-compose.prod.yml now expect signed, digest-pinned image references (ghcr.io/lanternops/breeze/api@sha256:…) resolved from the release’s signed image inventory. The guided installer resolves and verifies them itself for 0.112.0 and later (fail-closed), and falls back to tag-tracking refs for older releases. Existing installs whose .env still carries …/api:${BREEZE_VERSION} refs keep working with the upgrade steps at the top of this page. If you have already switched to digest pins, update the four BREEZE_*_IMAGE_REF lines from the new release’s inventory on every upgrade — bumping BREEZE_VERSION alone changes nothing. See Signed, digest-pinned image references.

Rotating RELEASE_ARTIFACT_MANIFEST_PUBLIC_KEYS is a fleet re-enrollment: agents pin the key they enrolled under and there is no remote revocation. See Binary Distribution.

Rolling back v0.112.0

Prefer roll-forward. If you must roll back, downgrade agents before the API: a v0.112.0 agent talking to an older API gets no lease answer and ends its desktop session after about 150 s. The added columns are additive and need no schema rollback. Rolling back the network-baseline change resumes every paused recurring baseline under no authority — clear each affected baseline’s schedule first.

Other behaviour changes in v0.112.0

• Bare-metal recovery now restores system state on Linux, every system_image backup records a disk-layout manifest with a restorability verdict, and a breeze-backup rebuild command can provision a disk from a snapshot. See Bare Metal Recovery.
• Backup exclude patterns with a leading / are root-anchored (gitignore semantics) instead of matching at any depth. File backups now capture symlinks, directories, ownership and setuid/setgid/sticky bits; manifests carrying them use a new format that older agents cannot read, so restore with a v0.112.0 agent. See Backup Policies.
• Retention GC no longer stops on the first snapshot that was ever restored, verified or tokened. Bucket usage still does not shrink — storage reclamation lands in a later release.
• Webhooks make one HTTP attempt per invocation; the queue owns retries, retryCount is capped at 2, and permanent failures (bad configuration, SSRF-blocked URL, ordinary 4xx) dead-letter immediately. /test and /retry on a paused webhook return 409.
• Org-scoped sessions are now evaluated against the partner IP allowlist.
• MCP tool get_fleet_status is renamed get_invite_funnel. The old name dispatches for one more release but no longer appears in tools/list.
• HP warranty lookup is de-registered — HP’s API is not available to RMM vendors, so HP devices stay at unknown.

v0.111.1 — hotfix for a v0.111.0 upgrade-blocking migration

Install v0.111.1, not v0.111.0

If you have not yet upgraded to v0.111.0, install v0.111.1 instead — it contains everything in v0.111.0 plus this fix, and skips straight past the problem below. If you are already on v0.111.0 and it started up normally, your database was not affected and you can upgrade to v0.111.1 at your leisure. Everything else in this guide’s v0.111.0 section still applies.

One of the migrations that shipped in v0.111.0, 2026-10-14-100100-discovered-assets-manual-source.sql, backfills a new provenance column on discovered network assets. On a database where those discovered_assets rows span more than one organization or partner, the backfill’s two update statements can request the partner-export lock hierarchy out of order, which Postgres rejects. The migration transaction rolls back, autoMigrate retries it on the next boot, and the API never finishes starting — a crash loop, not a degraded mode. A single-partner or empty install is not affected.

v0.111.1 adds a new migration, 2026-10-14-100050-discovered-assets-source-backfill-prelock.sql, which sorts immediately before the failing one. It acquires every partner and organization lock the backfill will need, up front and in the correct order, so that by the time the original migration replays, it has nothing left to do and completes normally.

If your API is already crash-looping on v0.111.0, you have two ways out:

1. Upgrade to v0.111.1 (recommended). The failed migration’s transaction rolled back cleanly, so nothing is half-applied — pulling v0.111.1 and restarting is sufficient on its own.

2. Apply the fix-forward SQL below by hand against your database, using the same role your migrations run as, then restart the API on v0.111.0 before upgrading further:

   BEGIN;
   
   -- The failed migration rolled back its own CREATE TYPE / ADD COLUMN, so in the
   -- crash-loop state the `source` column does NOT exist yet: create it first
   -- (guarded, so this is also safe where it already exists).
   DO $$ BEGIN
     CREATE TYPE public.discovered_asset_source AS ENUM ('scan', 'unifi', 'manual');
   EXCEPTION WHEN duplicate_object THEN NULL;
   END $$;
   ALTER TABLE public.discovered_assets ADD COLUMN IF NOT EXISTS source public.discovered_asset_source;
   ALTER TABLE public.discovered_assets ADD COLUMN IF NOT EXISTS url text;
   
   -- Required: the lock helpers read organizations, and discovered_assets is
   -- FORCE ROW LEVEL SECURITY. Without this the discovery SELECTs return no rows
   -- and the UPDATEs match nothing, silently.
   SELECT set_config('breeze.scope', 'system', true);
   
   -- Acquire every affected partner (shared) and organization (exclusive) export
   -- lock in ascending order BEFORE the backfill, so the per-row triggers find
   -- them already held.
   SELECT public.breeze_partner_export_lock_partners_shared(ARRAY(
     SELECT DISTINCT o.partner_id
       FROM public.discovered_assets d
       JOIN public.organizations o ON o.id = d.org_id
      WHERE d.source IS NULL AND o.partner_id IS NOT NULL
      ORDER BY 1));
   
   SELECT public.breeze_partner_export_lock_orgs_exclusive(ARRAY(
     SELECT DISTINCT d.org_id
       FROM public.discovered_assets d
      WHERE d.source IS NULL AND d.org_id IS NOT NULL
      ORDER BY 1));
   
   UPDATE public.discovered_assets a
      SET source = 'unifi'
    WHERE a.source IS NULL
      AND (a.detected_type_source = 'unifi_controller'
           OR EXISTS (SELECT 1 FROM public.unifi_devices u
                       WHERE u.discovered_asset_id = a.id));
   
   UPDATE public.discovered_assets SET source = 'scan' WHERE source IS NULL;
   
   COMMIT;

   This is idempotent: on a database where the migration already succeeded, every statement is a no-op.

Recovery for a deployment that is already stuck in the crash loop:

1. Set BREEZE_VERSION=0.111.1 in .env (or apply the fix-forward SQL above against v0.111.0 first, if you cannot upgrade immediately).
2. docker compose pull api web portal && docker compose up -d binaries-init api.
3. Wait for the API to report healthy (curl -sf https://yourdomain/health).
4. Once the API is healthy, bring the rest of the stack back up: docker compose up -d web portal.

No schema shape changes beyond v0.111.0: the same enum, the same two columns, the same constraints. The new migration only moves the backfill earlier and wraps it in the correct lock order.

v0.111.0 — offline work queue, script cancellation, organization record page

No breaking API changes and no new required environment variables. But five defaults change on upgrade, and two of them will alter what your customers and your technicians see the moment the API restarts. Read Behaviour changes before you pull.

Pull the portal container too, alongside api and web — and worker if you have opted into the worker split.

Do these two things before you pull

1. If you have QuickBooks connected and are not ready for Breeze to write payments into a customer’s books, turn Payment push off first (Integrations → QuickBooks). It ships on. See Payment push.
2. If you run network monitors with alert rules, review those rules, your notification channels and your escalation policies. Monitor alerts now notify and fire automations where before they only sat in the inbox. See Network Monitors.

Database migrations in v0.111.0

35 migrations. Take a database backup first. All are idempotent and apply automatically on API startup unless you have set AUTO_MIGRATE=false.

Two migrations deliberately abort the deploy on bad pre-existing data

Both concern custom fields, and both are intentional — the alternative is silently picking a winner among rows that are supposed to be unique. Each ships with a preflight query so you can find out before you pull rather than during:

Migration	Aborts on	Preflight
2026-10-10-100300-custom-field-definition-integrity.sql	Duplicate or ownerless custom-field definitions	apps/api/migrations/preflight/2026-10-10-100300-custom-field-definition-integrity-preflight.sql
2026-10-11-141000-custom-field-no-cross-axis-shadowing.sql	A custom-field key defined at both the organization and the partner level for the same device	apps/api/migrations/preflight/2026-10-11-141000-custom-field-shadowing-preflight.sql

Run both against your database before you upgrade and reconcile anything they report. See Custom Fields → Uniqueness and shadowing.

Six indexes are built concurrently, so writes are not blocked while they run — on a large audit_logs or process-sample history they are the long pole and the API does not serve until they finish. An interrupted build leaves an invalid index that the next boot reports loudly; recover with DROP INDEX CONCURRENTLY <name> and restart.

Three migrations touch existing rows and record what they did as a Postgres WARNING. Read these in the API log after the first boot:

Log line	What it means
device_custom_field_values backfill: copied N value(s), minted N definition(s) for previously undefined keys, N stored key(s) do not match the enforced key pattern	Custom-field values move into a table of their own. Sized by devices × custom-field keys. Keys written before the naming rules were enforced are deliberately kept in the old projection only, so nothing is lost or renamed under you.
backfilled N discovered_assets rows to source=unifi / source=scan	Every discovered asset gains a provenance column. Sized by your discovery inventory, not your device count — on a large scanned estate this is the longest single statement in the set.
The Partner Admin forced-MFA stamp	Records the reconcile described under Behaviour changes. Enforcement is off this release, so nothing changes for your admins at upgrade time.

One hot-table lock worth planning around

The offline-work-queue migration adds a foreign key and a partial index to the agent command table without the concurrent/deferred options, so it holds an exclusive lock on the agent command path for one sequential scan. At ordinary sizes that is seconds. If you retain millions of device commands, build both by hand before you pull — add the constraint NOT VALID then VALIDATE, and create the index CONCURRENTLY under the name the migration uses (idx_device_commands_deliver_by) — and the migration’s guards will no-op at boot.

A non-superuser migrating role is sufficient, as in every previous release: the range creates no roles, extensions, or schema-level grants.

New optional settings in v0.111.0

None are required and none fail boot when missing — except PLAY_INTEGRITY_SERVICE_ACCOUNT, which is validated when set. Full descriptions are in Environment Variables.

Variable	Why you might set it
DEVICE_COMMAND_OFFLINE_QUEUE_ENABLED	Already on. Set false only to restore the old hard-failure behaviour while you adjust your automations — this escape hatch is removed in a later release.
DEVICE_COMMAND_QUEUE_TTL_HOURS, DEVICE_COMMAND_QUEUE_SHORT_TTL_HOURS, DEVICE_COMMAND_QUEUE_POWER_STATE_TTL_HOURS	Shorten or lengthen how long queued work waits for an offline device (7 days / 24 hours / 24 hours).
MFA_FORCE_FOR_PARTNER_ADMIN	Now defaults to false. Set true to enforce forced MFA for Partner Admins immediately.
AI_OPERATOR_TASKS_ENABLED, AI_OPERATOR_RECIPE_SERVICE_RECOVERY_ENABLED	Turn on the opt-in AI Operator preview. Both ship off.
READINESS_CACHE_TTL_MS, READINESS_PROBE_TIMEOUT_MS	Tune the new readiness contract. Defaults suit every deployment we know of.
OFFLINE_DETECTOR_WORKER_CONCURRENCY	Raise offline-sweep concurrency on a very large fleet.
PLAY_INTEGRITY_SERVICE_ACCOUNT	Verify Android approver devices with Play Integrity as well as hardware Key Attestation. A malformed value stops the boot rather than silently reporting “not configured” forever.

Readiness versus liveness

/health is unchanged and remains liveness — it answers 200 before the instance can actually serve. /ready and /health/ready now answer a real aggregate readiness question, and API admission is gated on the consumers the process role requires. If your load balancer or Compose healthcheck should hold traffic until Postgres, Redis and the background consumers are genuinely up, point it at /ready. See Health Monitoring.

Behaviour changes in v0.111.0

• ⚠️ QuickBooks starts pushing payments into every connected realm. The Payment push switch ships on, so a connected realm in automatic push mode begins creating QuickBooks Payments as soon as the API restarts — no operator action. Only payments recorded after the switch became active are pushed, and a payment deletion always propagates regardless of the switches. Turn it off first if you are not ready. See Payment push.
• ⚠️ Network-monitor alerts now notify, escalate and automate. ICMP/TCP/HTTP/DNS monitor alerts previously landed in the Alerts inbox only. Expect a step change in notification and automation volume if you run monitors with alert rules. The five-minute cooldown, the deduplication and the auto-resolve on recovery all still apply. See Network Monitors.
• Work aimed at an offline device queues instead of failing. Patch jobs, automation Run script / Run command actions and scan/rollback commands are persisted with a delivery deadline and claimed on the device’s next heartbeat. A waiting step reads “Queued — device offline” instead of Failed. The other side of that coin: work you previously saw fail fast now sits pending for up to a week. Existing automations have no stored offline preference and therefore queue — set a step to Skip if it is only meaningful against a live device. See Devices → Queued actions.
• ⚠️ On Linux and macOS, breeze-agent service stop is now temporary while the watchdog is installed. The watchdog restarts the agent after its standby grace period. For a durable stop, stop or uninstall the watchdog too. See Agent Watchdog.
• service install now restarts the service on an already-enrolled host instead of leaving it stopped — the old behaviour stranded a host Offline until someone started it by hand. See Agent Installation. The Windows equivalent is fixed too, including installing over an existing service, which previously failed while reporting the service as running when it was not.
• “Remove device” now uninstalls the agent by default. The removal flow asks about the agent and defaults to uninstalling it; previously the uninstall was effectively never sent and agents were left running on removed devices. A permanent delete now refuses until that uninstall has been collected.
• Forced MFA for Partner Admins is reconciled but NOT enforced. A migration marks every existing system Partner Admin role as requiring MFA — reconciling installs that predate the rule — but MFA_FORCE_FOR_PARTNER_ADMIN now defaults to false, because acting on it at upgrade time would lock existing Partner Admins into enrolment with no warning. Set it to true to enforce now. A notification-period rollout — grace window, banner, deadline — ships next release, at which point enforcement returns to on by default. Separately, an MFA policy change that would lock out already-enrolled users is now refused outright with the number of users affected.
• Maintenance windows suppress every script path. A device inside a window with Suppress scripts is now skipped by the AI assistant, automation Run script / Run command, fleet-fix remediation and edition auto-migration — not just a manual Run Script. Each is recorded as a skip, not a failure, so an automation run stays green and its trailing actions still run. A window that cannot be evaluated deliberately keeps the ordinary failure treatment, so a maintenance-config outage can never render as a fleet of green runs. See Maintenance Windows.
• Script cancellation kills the whole process tree. Cancel and timeout now terminate descendants on Windows and Unix. A script that intentionally detaches a child will now have that child killed too — worth a canary group if you rely on that pattern. See Stopping a running script.
• Configuration-policy inheritance changes what a “linked” policy delivers. A policy marked as inheriting could previously deliver nothing to its devices; every reader now resolves the real inherited settings. If you have such a policy, devices under it will start receiving configuration they were not receiving before. See Configuration Policies.
• Service Management is a partner-level module. native is the default and existing partners are unchanged. Turning it off hides the Service Desk and Billing sections and refuses new ticket creation everywhere. See Service Management mode.
• The unified device list is on by default. PUBLIC_ENABLE_NETWORK_DEVICES_IN_LIST now defaults to true. It is a build-time variable baked into the web image, so the published image ships with the All / Agent / Network / Manual segments on and there is nothing to change in .env or Compose. Build the web image yourself and pass false to keep the agent-only list.
• Metric-anomaly detection looks back 15 minutes instead of 30, with a statement timeout, so overlapping runs stop piling up on the rollup tables. The trade-off: a skipped tick no longer self-heals and a gap needs a manual backfill.
• Cross-organization device moves may now drop custom-field values that are not visible in the target organization. Previously they were kept silently; now they are dropped and logged.
• Still off by default: AI agents and the AI Operator preview, reboot deferral (per policy), and the removed-device purge retention policy.

Known wrinkle on Linux

systemctl stop breeze-agent runs its full 30-second stop timeout and ends in a kill, so each service install takes roughly 32 seconds and the unit is briefly reported as failed before it comes back. This is pre-existing and cosmetic — the agent ends up running.

v0.110.0 — reboot deferral, device-set billing, QuickBooks payments, portal visibility

No breaking changes and no new required environment variables. Everything new is either off by default or optional. A number of new optional variables were added, though — see New optional settings below and the full Environment Variables reference.

Pull the portal container too. The customer portal runs in its own container and gained several new pages in this release. A bare docker compose pull covers it; if you pull services by name, make sure portal is on the list alongside api and web. If you have opted into the worker split, include worker as well.

Refresh your Compose file from the release

docker compose pull updates images — it does not update your docker-compose.yml. New optional variables only reach the container if they are mapped in the api service’s environment: block, so a value added to .env alone silently does nothing. Compare your deployed Compose file against the one published at the release tag and merge in what you need, keeping your own domains, mounts, and overrides.

Database migrations

54 migrations. Take a database backup first. All are idempotent and apply automatically on API startup (unless you have set AUTO_MIGRATE=false). Every index on a table that can be large is built concurrently, so writes are not blocked while they run — the trade-off is that an interrupted build leaves an invalid index behind. If that happens the next boot reports it loudly; recover with DROP INDEX CONCURRENTLY <name> and restart.

Three migrations clean up or stamp existing rows and record what they touched as a Postgres WARNING. Read these in the API log after the first boot:

Log line	What to do if the count is above zero
cleaned N contract_lines rows whose site belonged to another org	Re-scope those contract lines in the contract editor before your next billing run.
The site-name stamp on contract_lines	This records how many existing site-scoped lines were protected. A line whose site was deleted before this release cannot be recovered and keeps billing organization-wide until a technician re-scopes it.
rls-scoped replay	Informational. Five earlier migrations wrote rows without the right database scope and are replayed correctly here; the counts tell you how many rows were fixed.

New optional settings in v0.110.0

None of these are required and none of them fail boot when missing. Full descriptions are in Environment Variables.

Variable	Why you might set it
QBO_WEBHOOK_VERIFIER_TOKEN	Makes recorded QuickBooks payments come back within seconds instead of on the 15-minute sweep.
FIREBASE_SERVICE_ACCOUNT	Enables push notifications to the Android app.
AUDIT_CHAIN_VERIFY_MODE, AUDIT_CHAIN_VERIFY_RESCAN_SLICES	Nightly audit-chain verification is now incremental by default rather than a full-table walk. AUDIT_CHAIN_VERIFY_ENABLED=false now also stops a sweep that is already running.
TICKET_OUTBOX_RETENTION_DAYS, INTENT_OUTBOX_RETENTION_DAYS, METRIC_ANOMALY_INCIDENT_RETENTION_DAYS (and their batch knobs)	Tune how long the new nightly retention sweeps keep delivery and incident history. Defaults to 14 days.
SMTP_TIMEOUT_MS, MAILGUN_TIMEOUT_MS	Bound how long a wedged mail host can hold an outbound send open.
BREEZE_AUTHENTICATOR_ATTESTATION_ENFORCED	Already on by default. Only set it if you need to accept an approver device’s unverified platform-bound claim.
PARTNER_TRUST_MODE and friends	Hosted service only — inert on a self-hosted install.

Behaviour changes to know about

• End-user restart prompts ship off. Turn them on per policy on the Patch tab with Let users postpone the restart. The deadline is the guarantee; the postponement count is a courtesy budget. Linux endpoints need zenity installed to show a dialog the user can postpone from — without it they get a plain notification and the restart proceeds on schedule. See Patch Management.
• Unrated patches are never treated as Low and are excluded from auto-approval unless an update ring opts in. Expect a backlog of pending patches to appear that were previously auto-approved as Low. See Update Rings.
• Deleting a site used by a site-scoped contract line now fails invoice generation with a clear error instead of quietly billing every device in the organization. Re-scope or remove the line, then generate again.
• Audit-log retention now runs for every organization that has a policy. New organizations get a default policy automatically; existing organizations have no policy until an operator saves one, and nothing is deleted until then. See Audit Logs → Retention.
• New partners are created with inbound email-to-ticket turned off. Existing partners are unchanged.
• Windows 10 and Windows Server 2016 are the declared agent OS floor. Older Windows builds are refused by the installer rather than enrolling and failing later.

AI agent graduation and impact accounting

Both features run under the flags you already have — BREEZE_AI_AGENTS_ENABLED for the nightly impact rollup, and BREEZE_AI_AGENTS_POLICY_DECIDE_ENABLED (still off, still not generally available) for promoting an operation to pre-authorized execution. Automatic demotion is not gated by anything and is always on.

Behaviour change: partner-level pre-authorized operations stop applying on their own

A partner-wide agent’s supervisedActionKeys are now a ceiling — what an organization may be granted — not an inherited grant. An organization whose agent has no override row of its own now resolves to an empty set of pre-authorized operations, whatever the partner baseline lists.

Nothing is silently revoked and no data changes, but any organization that was relying on the partner list has to be granted the operations explicitly: either through the Graduation panel’s approval flow, or by editing that organization’s own agent. Do this before you turn BREEZE_AI_AGENTS_POLICY_DECIDE_ENABLED on, not after. See AI Agents → Graduation.

Three new daily background jobs are registered. They run in the container that owns background work — the api container by default, or the worker container if you have enabled the worker split:

Job	Runs daily at (UTC)	What it does
Graduation evaluation	18:28	Re-evaluates which agent operations have earned enough clean evidence to be promoted. Read-only with respect to what agents may do — it never grants or revokes anything.
Impact rollup	18:33	Rebuilds the last 7 complete UTC days of the AI Impact report per organization, or the full 90 days the first time it sees an organization. Skipped entirely while BREEZE_AI_AGENTS_ENABLED is off.
Evidence retention	18:48	Prunes agent outcome evidence older than 400 days.

All three sit in the existing daily lane and are deliberately spaced apart from each other and from the retention sweeps, so they do not add a new nightly spike.

Note

The graduation view returns 404 — “No active agent policy for this organization/kind” for an organization that has no effective agent, which includes an organization-only agent with no partner-wide baseline behind it. That is by design, not a fault: without a baseline there is no ceiling to graduate against.

v0.109.0 — AI agents, ticket attachments and org archiving

Mostly additive. Nothing in this release requires a new setting before you upgrade, but note the following:

• Migrations. 53 idempotent migrations auto-apply on boot. Two touch tables that can be very large on a long-running self-hosted instance: one builds an index on remote-session history, and one adds a validated constraint to time entries. Both take a brief lock while they run, so expect a slightly longer boot than a routine upgrade — proportional to how much remote-session and time-entry history you hold. Take the database backup in step 1 above and upgrade during a quiet window.
• New AI agent capabilities are off by default. BREEZE_AI_AGENTS_ENABLED remains false, and even with it on, each agent starts in Off — an agent only acts after you deliberately move it to Act and confirm with multi-factor. See AI Agents.
• New partners start with inbound email-to-ticket turned off. Existing partners are unchanged. See Ticketing.
• Optional worker split. You can now move background jobs to their own container. Entirely opt-in and off unless you enable it — see Worker Split.

v0.95.0 — authentication & tenancy hardening

This release resets the authentication surface. Plan for a short low-traffic window and expect the following the first time the new API boots:

• Everyone is signed out once. Existing web and mobile sessions are invalidated and everyone signs in again — there is no silent-refresh recovery and no kill switch. Confirm you and your admins have a working multi-factor device or recovery codes before upgrading, since signing in is the only way back in. Agents, the Helper, the portal, and installers are unaffected.
• Connected AI / automation apps must re-authorize. Legacy OAuth refresh tokens are removed; any MCP/OAuth client reconnects and re-consents once.
• Microsoft 365 ticket mailboxes need re-consent. Non-disabled mailboxes drop to a re-auth state; a Partner Admin re-approves each one. Mail arriving during the gap is picked up after re-consent, not lost.
• New required database credential. Production now runs request handlers as an unprivileged, RLS-enforced role and will not start unless one of DATABASE_URL_APP, BREEZE_APP_DB_PASSWORD, or POSTGRES_PASSWORD is set, and it refuses to run as a SUPERUSER/BYPASSRLS role. See Environment Variables → Database. This check runs even when AUTO_MIGRATE=false.
• Behind Cloudflare? Set TRUST_CF_CONNECTING_IP=true (see Environment Variables).
• Migrations. 26 idempotent migrations auto-apply on boot. Most are metadata-only, but one backfills the refresh_token_families table — on a large, long-running deployment, check that table’s size first and expect a slightly longer boot while it applies.

Caution

Any value you set in .env only takes effect if it is also mapped in the api service’s environment: block of your compose file. The shipped compose files map the database-credential variables; if you maintain your own compose file, add them there too.

Agent Updates

Agents update themselves automatically. On each heartbeat (~60s), the API checks if a newer version is available and signals the agent to upgrade. The agent:

1. Downloads the new binary and verifies its SHA-256 checksum
2. Backs up the current binary
3. Replaces the executable and restarts the service
4. Rolls back automatically if the new binary fails to start
5. Swaps the bundled breeze-backup helper to the matching version in the same upgrade; a periodic reconcile re-fetches it later if the swap was skipped or the helper is missing or stale

Since v0.105.0, release manifests can label each asset with an agent edition (self-host or hosted); the updater refuses an asset whose declared edition doesn’t match the running build, and accepts manifests without edition labels for backward compatibility. See Self-Host Agent Migration.

During an update, the device status changes to Updating in the dashboard so you can distinguish active updates from offline agents. If the update takes longer than 5 minutes, the device reverts to Offline. After a successful restart, the agent immediately reports its new version (bypassing the normal 0–60 second heartbeat jitter).

Linux Binary-In-Use Handling

On Linux, the updater handles the case where the agent binary is still in use (common with manual installs that don’t use a package manager). The updater removes the old binary before writing the new one to avoid ETXTBSY (“text file busy”) errors. If a file lock error still occurs, it is treated as transient and retried on the next heartbeat.

Watchdog Recovery

The watchdog service automatically restarts the agent after a failed update or crash. If an update leaves the agent in a bad state, the watchdog detects the failure and restarts the previous (backed-up) binary.

The binary source depends on your BINARY_SOURCE setting:

Mode	Agent downloads from
github (default)	GitHub Releases CDN
local	Your API server (or S3 if configured)

See Binary Distribution for details.

Disabling Auto-Update on Specific Devices

Some devices — production database servers, devices in a change-controlled window — should not self-update on the normal heartbeat cadence. Toggle auto-update per device without editing config on the machine:

This endpoint requires a signed-in user session with MFA satisfied, not an API key — $TOKEN below is a Breeze access token, not a partner API key.

# Disable auto-update for one device
curl -X POST "https://breeze.example.com/api/v1/devices/<device-id>/auto-update" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"enabled": false}'

# Re-enable later
curl -X POST "https://breeze.example.com/api/v1/devices/<device-id>/auto-update" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"enabled": true}'

The agent persists this setting across restarts. This is also the way to recover agents that were left with auto-update disabled by a manual or recovery update — see the set_auto_update command.

Controlled agent fleet rollout

By default (AGENT_AUTO_PROMOTE=true), syncing a new agent release immediately makes it the fleet’s upgrade target — every auto-updating device moves to it on its next heartbeat. Boot-time sync only ever registers the release matching your own BREEZE_VERSION (or BINARY_VERSION if set), so a newly published GitHub release does not move your fleet until you upgrade your server images — the fleet follows the server, not the public release feed. If you’d rather stage rollouts (canary first, then the fleet), set AGENT_AUTO_PROMOTE=false to decouple registering a release from promoting it.

1. Turn on controlled rollout. Set AGENT_AUTO_PROMOTE=false in .env and map it in the api service environment: block of your compose file, then restart the API. (Compose only passes through variables you list there.)

2. Publish or sync the release. New binaries register as downloadable but are not promoted — the fleet stays on the current version. Existing devices are unaffected.

3. Canary it. Update a few non-production devices to the new version and verify (you can target an individual device’s update directly).

4. Promote to the fleet. When you’re satisfied, promote the version. This is a platform-admin action and requires MFA:

   curl -X POST "https://breeze.example.com/api/v1/agent-versions/promote" \
     -H "Authorization: Bearer $TOKEN" \
     -H "Content-Type: application/json" \
     -d '{"version": "0.84.0"}'

   Auto-updating devices then move to the promoted version on their next heartbeat. To promote a single component (e.g. just the helper or watchdog) while holding the rest, add "component": "helper" to the body.

Tip

Leave AGENT_AUTO_PROMOTE unset (or true) to keep the default behavior where the release matching your server version rolls out as soon as the API syncs it. The promote action is audit-logged either way.

Pinning Agent and Watchdog Versions

Instead of always tracking the latest promoted release, you can pin the fleet — or a single customer — to a specific version. Settings → Partner Defaults and Settings → Org Defaults each expose an Update version targets card with independent Agent target and Watchdog target selectors. Each selector offers every registered version plus Latest promoted (track the globally promoted release, the default behavior).

Pin precedence is inherit-with-override — deliberately different from the lock model used by the update policy and maintenance window below:

• A partner pin is the inherited default for all of that partner’s organizations.
• Any org can override the partner pin with its own value — including explicitly re-selecting Latest promoted to move that org back to the promoted release while the rest of the fleet stays pinned.
• Agent and watchdog pins resolve independently.

This is what lets a partner pilot a new version on one customer: pin the fleet to the known-good version at the partner level, then override a single org to the candidate version (or to Latest promoted).

Caution

Pins fail closed. If the resolved pin has no registered build for a device’s platform and architecture, the update — including watchdog-recovery pushes — is withheld: the device simply stays on its current version and retries on the next heartbeat. This is surfaced only in server logs and error tracking; there is no dashboard badge for a frozen device. Before pinning broadly, verify the pinned version has builds registered for every platform/architecture in your fleet.

Pins gate alongside the update policy and maintenance window, not instead of them — a pinned update still waits for the maintenance window and respects the update policy.

Agent Update Scheduling

In addition to the per-device auto-update toggle and the partner-level controlled rollout, Breeze supports a structured maintenance window for agent updates. When a maintenance window is configured, agents only accept update signals from the server during the allowed window — outside of it, update prompts are deferred until the next eligible window.

Setting a maintenance window

Agent update maintenance windows are set in Settings → Org Defaults (and optionally in Settings → Partner Defaults to apply fleet-wide). The window configuration includes:

Field	Description
Day	A single day of the week the window applies to, or left unset to apply the window every day — you cannot select more than one specific day
Start time	Window start time in UTC
End time	Window end time in UTC

Note

All times are specified and stored in UTC. Communicate the UTC equivalent to your team to avoid scheduling confusion across time zones.

The update policy and maintenance window follow the same partner-locks precedence that governs other policy settings: a field set at the Partner level wins and locks — child organizations cannot override it. The org value applies only where the partner has left that field unset, and each field is merged independently. The update gate that agents check on heartbeat honors this merged result.

Note

The maintenance window gates when the update signal is acted on, not when the binary is downloaded. Agents can pre-download an available update during any window but will only apply it (restart the service) during the allowed window.

When no window is configured

If no maintenance window is set, agents update on their normal heartbeat cadence (~60 seconds after a new version is promoted). Configure a window when you need updates to avoid business hours or to concentrate restarts into a known change window.

Migration Failure Recovery

If the API container fails to start after an upgrade due to a migration error, follow these steps:

1. Check the container logs to identify the failing migration.

   docker compose logs api --tail 100

   Look for lines starting with [auto-migrate]. Common failure messages include:

   • Migration checksum mismatch – a previously applied migration file was modified. This should never happen in normal operation.
   • SQL errors during a specific migration file (e.g., constraint violations, missing columns).

2. Do not manually edit migration files. Migrations are tracked by checksum. Modifying a previously applied migration will cause all future startups to fail.

3. Fix and restart. The most common causes and resolutions:

   • Incompatible extension (e.g., TimescaleDB not installed): Install the required extension and restart.
   • Permissions issue: Ensure the database user has sufficient privileges.
   • Data conflict: Manually resolve the conflicting data in the database, then restart the API container.

   docker compose restart api

4. If unrecoverable, restore your database from a pre-upgrade backup and re-attempt the upgrade after resolving the underlying issue.

Caution

Migrations run inside a transaction. If a migration fails partway through, the transaction is rolled back and no partial changes are applied. The API will not start until all migrations succeed.

Remote-access lease cutover

The first release that enables shared remote-session ownership is a barrier deployment. Do not use a rolling or blue-green handoff for that release: legacy API processes can mint credentials that the new pre-upgrade admission path must not accept.

1. Set REMOTE_ACCESS_ADMISSION_MODE=closed, REMOTE_WS_AUTH_MODE=post_upgrade, and REMOTE_WS_REDIS_TOPOLOGY=standalone-single-primary. Recreate Caddy and verify every terminal, desktop, VNC, WebSocket-tunnel, and HTTP-tunnel issuer/consumer returns 503 with Retry-After.

2. Stop the entire old API pool. Verify no old API process, WebSocket, or tunnel-HTTP consumer remains reachable. Record the verified UTC stop times in REMOTE_WS_LEGACY_TICKET_WRITER_DRAINED_AT and REMOTE_WS_LEGACY_VIEWER_ISSUER_DRAINED_AT.

3. Keep admission closed for the independent V0 credential drains: 60 seconds for legacy WebSocket tickets and 300 seconds for tunnel-HTTP tickets/cookies. The 60-second milestone does not permit reopen. At exactly 300 seconds, a uniform new pool may reopen in post_upgrade.

4. Deploy the new pool, verify every API instance uses the release digest, and run the Redis preflight (standalone primary, cluster disabled, AOF enabled, maxmemory-policy=noeviction). Only then set REMOTE_ACCESS_ADMISSION_MODE=open and recreate Caddy.

5. Independently wait 7,200 seconds plus a final 60 seconds for legacy viewer tokens and their last compatibility ticket. Only after that boundary may the complete pool switch to REMOTE_WS_AUTH_MODE=pre_upgrade. Startup rejects this mode when either verified timestamp is missing, malformed, future-dated, or too recent.

Danger

A health check is not cutover proof. Any failed probe, mixed image digest, old process/socket, reachable tunnel-HTTP consumer, unsupported Redis mode, or incomplete drain must leave remote-access admission closed.

Zero-Downtime Upgrades

For production deployments where any downtime is unacceptable, you can perform a blue-green upgrade using two parallel Compose stacks:

Caution

Do not use this procedure for the first shared-lease-aware remote-access release. That release requires the closed admission barrier and full-pool replacement above.

1. Deploy the new version alongside the old one. Use a separate Compose project name or directory with the new BREEZE_VERSION and point it at the same database and Redis.

   COMPOSE_PROJECT_NAME=breeze-new BREEZE_VERSION=0.12.0 \
     docker compose -f docker-compose.yml -f docker-compose.override.yml up -d

2. Wait for migrations and health check. The new API container applies any pending migrations on startup. Once the health endpoint returns the expected version, the new stack is ready.

   curl -s http://localhost:NEW_PORT/health | jq .version

3. Switch traffic. Update your reverse proxy (Caddy, nginx, Cloudflare tunnel) to route traffic to the new stack.

4. Tear down the old stack. Once traffic is fully migrated, stop the old containers.

Rolling Back

1. Revert the version

   # In .env
   BREEZE_VERSION=0.11.2

2. Pull and restart

   docker compose pull
   docker compose up -d

Danger

Database migrations are forward-only. If the newer version ran a migration, reverting the API version may cause schema mismatches. Restore from a database backup if needed.

Checking Versions

# API version
curl -s https://breeze.yourdomain.com/health | jq .version

# Agent version (on the device)
breeze-agent version

# Docker image versions
docker compose images

Pre-Release Testing

You can test a release candidate before making it available to users:

1. Tag a prerelease

   git tag v0.11.3-rc.1 && git push --tags

   CI builds all images but does not tag them as latest.

2. Deploy the RC locally

   # In .env
   BREEZE_VERSION=0.11.3-rc.1
   
   docker compose pull
   docker compose up -d

3. Sync agents to the RC using the sync-github endpoint. This fetches the release assets from GitHub and registers them in the Breeze database so agents know a new version is available. It requires a signed-in admin session with MFA satisfied — $TOKEN is a Breeze access token, not an API key.

   curl -X POST "https://your-domain/api/v1/agent-versions/sync-github?version=v0.11.3-rc.1" \
     -H "Authorization: Bearer $TOKEN"

   The response includes the version that was synced and the list of platform/architecture targets that were registered:

   {
     "version": "0.11.3-rc.1",
     "synced": ["agent:linux/amd64", "agent:macos/arm64", "agent:windows/amd64", "helper:windows/amd64"]
   }

   Agents will pick up the new binary on their next heartbeat (~60 seconds). Omit the ?version= parameter to sync the latest stable release instead of a specific version. This endpoint requires system-level (admin) authentication.

4. Test, then promote

   git tag v0.11.3 && git push --tags

   This pushes latest tags — all users get the update.