Firewatch architecture
Status and guiding constraint
Section titled “Status and guiding constraint”This repository began as an architectural skeleton and now includes the first incident-response product slice. It proves process, persistence, queue, API, client-generation, and deployment boundaries while incrementally adding the on-call domain. Schedules, escalation execution, alert ingestion, safe message templates, and email, Twilio, Slack DM, and signed-webhook delivery are implemented.
Firewatch has one product implementation with multiple independently deployable processes. The public repository is the source of truth for the domain, application behavior, HTTP API, worker, web application, and portable deployment artifacts. Self-hosted Firewatch and managed Firewatch Cloud run those same artifacts. There is no Community/Enterprise fork and no cloud-specific copy of the product.
The arrows describe the durable asynchronous path, not separate product implementations. The API and worker share libraries and a database schema but run, scale, restart, and deploy independently.
Repository boundaries
Section titled “Repository boundaries”| Project | Responsibility | May depend on |
|---|---|---|
Firewatch.Domain |
Pure business types and rules | Nothing outside the base class library |
Firewatch.Contracts |
Versioned HTTP and message DTOs | Base class library only |
Firewatch.Application |
Use cases, ports, and orchestration | Domain and contracts |
Firewatch.Infrastructure |
EF Core context, PostgreSQL persistence, outbox, repositories, DI | Application, contracts, queue abstractions, EF Core/Npgsql |
Firewatch.Queueing.Abstractions |
Small publishing/consuming contracts and message identity conventions | Base class library and stable contracts as needed |
Firewatch.Queueing.Postgres |
Portable PostgreSQL queue/outbox processing | Queue abstractions and PostgreSQL infrastructure |
Firewatch.Queueing.Sqs |
SQS publishing, long polling, acknowledgement, and visibility leases | Queue abstractions and AWS SDK for SQS |
Firewatch.Storage.S3 |
Managed runbook-image object storage | Application asset-store port and AWS SDK for S3 |
Firewatch.Email.Ses |
Managed transactional email through the SESv2 API | Application email port, infrastructure options, AWS SDK |
Firewatch.Api |
REST endpoints, HTTP concerns, health, OpenAPI, composition root | Application and concrete runtime adapters |
Firewatch.Worker |
Hosted consumers, retries, outbox work, composition root | Application and concrete runtime adapters |
The API never exposes EF Core entities. Versioned types from
Firewatch.Contracts form the wire boundary. Endpoint registration is kept out
of Program.cs so the composition root stays readable.
This is a modular monolith: modules share a release and relational data store, while dependencies point inward toward domain and application code. It avoids network boundaries inside the product until there is evidence that one is needed.
Deployable processes
Section titled “Deployable processes”Firewatch.Api owns synchronous HTTP handling under /api/v1, OpenAPI,
exception translation, request/correlation IDs, structured JSON logs, and liveness
and readiness endpoints. It is stateless apart from PostgreSQL and can be scaled
horizontally after migrations have been run.
Worker
Section titled “Worker”Firewatch.Worker owns asynchronous execution. It polls or consumes through the
selected queue provider, dispatches durable outbox work, records completion, and
honors cancellation during shutdown. API replicas do not run worker loops, and
worker replicas do not serve the public API.
apps/web is a standalone React/Vite application. It communicates only through
the public HTTP API by using packages/api-client; it does not reference .NET
projects or persistence types. The production web container serves static files
and proxies same-origin /api, /health, and /openapi requests to the API
upstream selected at deployment time.
Authentication and tenant authorization
Section titled “Authentication and tenant authorization”ASP.NET Core Identity owns local user credentials and account-security behavior. A user has a first name, last name, and normalized unique email address. Users do not own tenant data directly: an organization membership connects a user to an organization and records whether that member is an organization administrator. Organization-scoped use cases resolve the active tenant from an authenticated membership and enforce that boundary in application and persistence queries; they do not trust an arbitrary organization identifier supplied by a client.
The browser signs in with an HTTP-only, Secure authentication cookie. Requests that change state also require a CSRF token, including requests made through the same-origin web proxy. Cookie contents are protected by the ASP.NET Core Data Protection key ring stored in the shared PostgreSQL database, so horizontally scaled API replicas and replacement containers share the same keys. Production operators remain responsible for protecting that key ring at rest and rotating its deployment-managed protecting key.
Organization API keys are tenant-scoped machine credentials rather than user sessions. An organization administrator can create and revoke them. The full secret is returned only at creation, while Firewatch stores a non-reversible hash for constant-time verification. Authorization uses the key’s organization and active/expiry state and never treats an API key as a global administrator credential. Cloud control-plane provisioning uses a separate workload-token policy and does not reuse tenant API keys.
Services and incidents
Section titled “Services and incidents”A monitored service is an organization-scoped thing that can degrade or fail.
Its normalized name is unique within the organization. A service always begins
with at least one owner: the member who created it as a primary owner. Owners
carry a persisted primary or secondary label; multiple owners may use either
label, API responses order primary owners first, and both labels grant the same
service-management permission. Service owners and
organization administrators can edit the service, add or remove active members
as owners, and archive it. The domain rejects removal of the final owner, and
organization-member removal is blocked when it would orphan an active service.
Archival is non-destructive: archived services retain their owners and incident
history but cannot receive new incidents or be edited. A service may select one
escalation policy for newly created incidents. Future alert-source and runbook
modules will reference this same service identifier rather than introducing
separate copies of the service concept.
An incident belongs to exactly one service and organization. It has a global,
human-readable INC-000001 reference, severity (SEV1, SEV2, or SEV3),
status (triggered, acknowledged, or resolved), and a source kind. Manual
declaration accepts the actual incident start time rather than assuming the API
request time. Resolution likewise records its selected time and an optional
reason. Manual declaration is the first supported source; alert ingestion will create the same
aggregate with an alert reference later.
Every incident owns a chronological timeline. Creation, acknowledgement, resolution (including its optional reason), reopening, responder notes, severity changes, escalation stages, notification deliveries, failed attempts, and skipped notifications append new immutable entries containing the actor, status transition, optional severity transition, message, and timestamp. Reopening begins a new response cycle without removing the prior cycle’s events. The domain exposes no mutation or removal operation for entries, and the EF context rejects modified or deleted timeline rows. Incident listing performs tenant-scoped server-side search, filtering, and pagination; dashboard counts and recent incidents are computed from the same persisted data. Tenant identity is repeated on timeline rows and enforced through composite foreign keys so an incident, service, and timeline cannot cross organization boundaries.
Time zones, schedules, and escalations
Section titled “Time zones, schedules, and escalations”An organization stores a default IANA timezone for new configuration, while every user stores their own display timezone. Neither value silently overrides a schedule: each on-call schedule explicitly owns the IANA timezone that defines its wall-clock rotation start and handoff cadence. This keeps a 09:00 handoff at 09:00 through daylight-saving transitions. Notification content converts dates to the recipient’s personal timezone.
A schedule contains an ordered rotation of organization members. A rotation can
use one fixed segment length or a repeating pattern such as 4,3, allowing one
schedule to alternate Sun–Wed and Thu–Sat coverage. Schedules also store the
weekdays that provide coverage; excluded weekdays create explicit gaps and do
not advance the pattern. Each participant can have optional effective-from and
effective-until instants, so planned hires, leave, and removals are reflected in
future calendars without a precisely timed manual edit. Membership changes are
treated as schedule transitions and the active roster is recalculated at the
effective instant.
Non-overlapping overrides use exact start and end instants and temporarily supersede both the rotation and weekday exclusions. Organization member removal is blocked while that member remains referenced by a rotation.
An escalation policy contains up to ten ordered steps. Step one is immediate; each later step stores a delay relative to the preceding step and targets an on-call schedule. When an incident is created for a service with a policy, the API snapshots every step into a delayed outbox message in the same transaction. The worker resolves the current responder when each step becomes due, so rotation changes and active overrides are honored at delivery time. Acknowledgement or resolution causes remaining stages to be audited as skipped.
Registration is deployment-configurable. Self-hosted Compose selects Open, so
users start by verifying email ownership, then choose a passkey (recommended) or
password. A pending row contains a hashed 30-minute verification secret and a
stable prospective user ID; no user, organization, or membership exists until
ASP.NET Core Identity verifies the selected credential. Firewatch then creates
the confirmed user, organization, administrator membership, credential, and
cookie session in one transaction.
Firewatch Cloud selects Provisioned, which removes public
registration while allowing its authenticated control plane to create the paid
customer’s organization and initial administrator, then send a single-use
invitation. Disabled is the fail-closed product default. The same web build
reads the registration capability from the API rather than embedding a cloud
edition switch.
ASP.NET Core Identity 10 owns passkey attestation, assertion, replay counters, and credential storage. Public passkeys are discoverable and require local user verification, enabling username-less sign-in. The explicit RP ID is inferred from the canonical public URL unless an operator deliberately overrides it. An authenticated user can keep up to ten named passkeys. Credential removal is serialized per user, and a passkey-only account cannot remove its final passkey.
Local Identity credentials are one authentication method, not a permanent identity-provider boundary. External login records are modeled by stable provider and provider-subject identifiers. The public API exposes a trusted, explicitly configured extension loader plus a provider-neutral federated sign-in service. That service owns account linking, organization-boundary enforcement, optional JIT membership creation, invitation completion, and the normal Identity cookie. Protocol adapters validate their upstream response and submit only the resulting identity to this service.
The private Firewatch Cloud repository supplies the first adapter for enterprise SAML. It is copied into an additive API overlay and loaded from an explicit assembly path at startup; a private web overlay fills the generic product route registry with its administration UI. The public login page discovers its button through the normal authentication-capability response. Public seams also permit an extension to veto local credential login and to register a namespaced PostgreSQL context whose migrations participate in the one-off migration mode. A missing, invalid, or duplicate extension fails API startup. Self-hosted deployments load no extension by default and contain no SAML-specific package, endpoint, UI, or configuration.
Durable work and delivery semantics
Section titled “Durable work and delivery semantics”The PostgreSQL outbox closes the failure window between changing application state and requesting asynchronous work:
- An application use case stores its state change and a uniquely identified outbox record in one database transaction.
- A worker claims pending work without allowing another worker to own the same lease concurrently.
- The selected provider publishes or handles the message.
- Completion is recorded durably. Notification deliveries store every attempt, recipient, channel, outcome, and error. Failures use exponential backoff and become dead letters after the configured maximum.
Delivery is at least once. A stable message identifier travels with every message; it is not regenerated during retry. Consumers must use that identifier as an idempotency key and make their state change plus processed-message marker atomic wherever possible. A handler must assume it can run again after a timeout, lost acknowledgement, process crash, or queue visibility expiry.
The abstractions intentionally cover only publishing and consuming Firewatch
messages. They are not a general-purpose event bus. PostgreSQL is the default and
requires no additional broker beyond the database Firewatch already needs. Its
FOR UPDATE SKIP LOCKED claim path supports multiple worker replicas. In
managed Cloud, that same claim path relays committed outbox rows to SQS; SQS
workers long-poll and use a durable processed-message record plus PostgreSQL
advisory lock for idempotent redelivery. Email delivery uses IEmailSender;
phone verification and delivery use IPhoneSender. The public composition
supplies SMTP, SES, and optional Twilio adapters without changing incident
behavior.
Slack delivery uses the immutable user ID captured by verified-email mapping.
Each delivered Slack alert stores its bot-message reference. Signed Block Kit
actions mutate the incident in the API transaction, and the resulting outbox
event makes the worker update the original message and append a lifecycle
thread reply. Outbound webhooks use a stable delivery ID plus timestamped HMAC
signature. Organization template records are shared through PostgreSQL,
avoiding divergent template files across worker replicas.
Architectural smoke flow
Section titled “Architectural smoke flow”The asynchronous processing smoke flow is authenticated system-test infrastructure:
POST /api/v1/system/test-jobscreates a test-job row and its outbox message in one PostgreSQL transaction.- The API returns
202 Accepted, including the stable job identifier and a status location. - The separately running worker claims and processes the outbox item, logs its message identifier, and marks the test job complete.
- A duplicate delivery observes the processed identifier and does not perform duplicate completion.
GET /api/v1/system/test-jobs/{id}reports the state to the web application or a developer.
This flow validates the architecture; it is not a model for alerts, incidents, or escalation policies.
Health and observability
Section titled “Health and observability”GET /health/livereports that the API process can serve requests. It does not fail merely because PostgreSQL is unavailable.GET /health/readyverifies required dependencies, including PostgreSQL, and is the deployment readiness gate.- API and worker logs are structured JSON and include message or request context plus correlation/request identifiers where available. Deployments keep the process streams distinct and may add service-name enrichment in their log collection layer.
- OpenTelemetry registration is an extension point for traces and metrics. An exporter or full observability backend is deliberately not bundled.
Liveness should restart a stuck process. Readiness should remove an unhealthy API from traffic. Neither endpoint should leak credentials or detailed exception data.
One product, two deployment models
Section titled “One product, two deployment models”For self-hosting, Compose runs PostgreSQL, API, worker, and web, with the PostgreSQL queue provider. A future public Helm chart may package the same containers.
For Firewatch Cloud, the private firewatch-cloud repository pins the
public image digests and supply AWS infrastructure, SQS, managed PostgreSQL,
secrets, networking, provisioning, billing, and operational automation. It may
configure or surround the product, but it must not duplicate domain or product
behavior. The precise handoff is documented in
Cloud deployment contract.
Future iOS client
Section titled “Future iOS client”A native iOS application will be another client of the same versioned REST API; the web app is not a backend-for-frontend. The checked OpenAPI document can be fed to a maintained Swift OpenAPI generator (for example Apple’s Swift OpenAPI Generator) when iOS work begins. Authentication and API version rules live at the public API boundary rather than in a web-only endpoint; future compatibility changes must preserve that boundary.