# Firewatch architecture

> Understand Firewatch's processes, persistence, queueing, authentication, and deployment boundaries.

This is the machine-readable version of this Firewatch documentation page.

## Status and guiding constraint

Firewatch is a modular on-call and incident-alerting product. Schedules,
escalation execution, alert ingestion, incidents, runbooks, safe message
templates, and email, Twilio, Slack DM, and signed-webhook delivery are
implemented across independently deployable processes.

Firewatch has one product implementation with multiple independently deployable
processes. The public repository is the source of truth for the domain,
application behavior, HTTP API, worker, web application, and portable
deployment artifacts. Self-hosted and managed installations use the same
product behavior.

state + outbox"]
    Db --> Worker[".NET Worker"]
    Worker --> Delivery["Delivery-provider<br/>abstractions"]
    Delivery --> Channels["Email · SMS · voice<br/>Slack · signed webhooks"]
    Worker --> Queue["Queue-provider<br/>abstraction"]
    Queue --> PgQueue["PostgreSQL queue<br/>self-hosted default"]
    Queue -.-> Sqs["AWS SQS<br/>optional provider"]
    Api --> Slack["Slack OAuth<br/>member mapping"]
    Slack -.-> Channels`}
  caption="Firewatch keeps API, worker, and provider boundaries consistent across deployment models."
/>

The arrows describe the durable asynchronous path, not separate product
implementations. The API and worker share libraries and a database schema but
run, scale, restart, and deploy independently.

## Repository boundaries

| Project                           | Responsibility                                                        | May depend on                                              |
| --------------------------------- | --------------------------------------------------------------------- | ---------------------------------------------------------- |
| `Firewatch.Domain`                | Pure business types and rules                                         | Nothing outside the base class library                     |
| `Firewatch.Contracts`             | Versioned HTTP and message DTOs                                       | Base class library only                                    |
| `Firewatch.Application`           | Use cases, ports, and orchestration                                   | Domain and contracts                                       |
| `Firewatch.Infrastructure`        | EF Core context, PostgreSQL persistence, outbox, repositories, DI     | Application, contracts, queue abstractions, EF Core/Npgsql |
| `Firewatch.Queueing.Abstractions` | Small publishing/consuming contracts and message identity conventions | Base class library and stable contracts as needed          |
| `Firewatch.Queueing.Postgres`     | Portable PostgreSQL queue/outbox processing                           | Queue abstractions and PostgreSQL infrastructure           |
| `Firewatch.Queueing.Sqs`          | SQS publishing, long polling, acknowledgement, and visibility leases  | Queue abstractions and AWS SDK for SQS                     |
| `Firewatch.Storage.S3`            | Optional runbook-image object storage                                 | Application asset-store port and AWS SDK for S3            |
| `Firewatch.Email.Ses`             | Optional transactional email through the SESv2 API                    | Application email port, infrastructure options, AWS SDK    |
| `Firewatch.Api`                   | REST endpoints, HTTP concerns, health, OpenAPI, composition root      | Application and concrete runtime adapters                  |
| `Firewatch.Worker`                | Hosted consumers, retries, outbox work, composition root              | Application and concrete runtime adapters                  |

The API never exposes EF Core entities. Versioned types from
`Firewatch.Contracts` form the wire boundary. Endpoint registration is kept out
of `Program.cs` so the composition root stays readable.

This is a modular monolith: modules share a release and relational data store,
while dependencies point inward toward domain and application code. It avoids
network boundaries inside the product until there is evidence that one is
needed.

## Deployable processes

### API

`Firewatch.Api` owns synchronous HTTP handling under `/api/v1`, OpenAPI,
exception translation, request/correlation IDs, structured JSON logs, and liveness
and readiness endpoints. It is stateless apart from PostgreSQL and can be scaled
horizontally after migrations have been run.

### Worker

`Firewatch.Worker` owns asynchronous execution. It polls or consumes through the
selected queue provider, dispatches durable outbox work, records completion, and
honors cancellation during shutdown. API replicas do not run worker loops, and
worker replicas do not serve the public API.

### Web

`apps/web` is a standalone React/Vite application. It communicates only through
the public HTTP API by using `packages/api-client`; it does not reference .NET
projects or persistence types. The production web container serves static files
and proxies same-origin `/api`, `/health`, and `/openapi` requests to the API
upstream selected at deployment time.

## Authentication and tenant authorization

ASP.NET Core Identity owns local user credentials and account-security behavior.
A user has a first name, last name, and normalized unique email address. Users do
not own tenant data directly: an organization membership connects a user to an
organization and records whether that member is an organization administrator.
Organization-scoped use cases resolve the active tenant from an authenticated
membership and enforce that boundary in application and persistence queries;
they do not trust an arbitrary organization identifier supplied by a client.

The browser signs in with an HTTP-only, Secure authentication cookie. Requests
that change state also require a CSRF token, including requests made through the
same-origin web proxy. Cookie contents are protected by the ASP.NET Core Data
Protection key ring stored in the shared PostgreSQL database, so horizontally
scaled API replicas and replacement containers share the same keys. Production
operators remain responsible for protecting that key ring at rest and rotating
its deployment-managed protecting key.

Organization API keys are tenant-scoped machine credentials rather than user
sessions. An organization administrator can create and revoke them. The full
secret is returned only at creation, while Firewatch stores a non-reversible hash
for constant-time verification. Authorization uses the key's organization and
active/expiry state and never treats an API key as a global administrator credential.
Automated organization provisioning uses a separate workload-token policy and
does not reuse tenant API keys.

## Services and incidents

A monitored service is an organization-scoped thing that can degrade or fail.
Its normalized name is unique within the organization. A service always begins
with at least one owner: the member who created it as a primary owner. Owners
carry a persisted `primary` or `secondary` label; multiple owners may use either
label, API responses order primary owners first, and both labels grant the same
service-management permission. Service owners and
organization administrators can edit the service, add or remove active members
as owners, and archive it. The domain rejects removal of the final owner, and
organization-member removal is blocked when it would orphan an active service.
Archival is non-destructive: archived services retain their owners, incidents,
alert events, and runbook associations but cannot receive new incidents or be
edited. A service may select one escalation policy for newly created incidents.
Alert sources and runbooks reference this same service identifier rather than
introducing separate copies of the service concept.

An incident belongs to exactly one service and organization. It has a global,
human-readable `INC-000001` reference, severity (`SEV1`, `SEV2`, or `SEV3`),
status (`triggered`, `acknowledged`, or `resolved`), and a source kind. Manual
declaration accepts the actual incident start time rather than assuming the API
request time. Resolution likewise records its selected time and an optional
reason. Manual declarations and normalized alert ingestion create the same
incident aggregate; alert-created incidents retain their source events and
correlation identity.

Every incident owns a chronological timeline. Creation, acknowledgement,
resolution (including its optional reason), reopening, responder notes, severity
changes, escalation stages, notification deliveries, failed attempts, and skipped
notifications append new immutable
entries containing the actor, status transition, optional severity transition,
message, and timestamp. Reopening begins a new response cycle without removing
the prior cycle's events. The domain exposes no mutation or removal operation for
entries, and the EF context rejects modified or deleted timeline rows. Incident
listing performs tenant-scoped server-side search, filtering, and pagination;
dashboard counts and recent incidents are computed from the same persisted data. Tenant
identity is repeated on timeline rows and enforced through composite foreign
keys so an incident, service, and timeline cannot cross organization boundaries.

## Time zones, schedules, and escalations

An organization stores a default IANA timezone for new configuration, while
every user stores their own display timezone. Neither value silently overrides a
schedule: each on-call schedule explicitly owns the IANA timezone that defines
its wall-clock rotation start and handoff cadence. This keeps a 09:00 handoff at
09:00 through daylight-saving transitions. Notification content converts dates
to the recipient's personal timezone.

A schedule contains an ordered rotation of organization members. A rotation can
use one fixed segment length or a repeating pattern such as `4,3`, allowing one
schedule to alternate Sun–Wed and Thu–Sat coverage. Schedules also store the
weekdays that provide coverage; excluded weekdays create explicit gaps and do
not advance the pattern. Each participant can have optional effective-from and
effective-until instants, so planned hires, leave, and removals are reflected in
future calendars without a precisely timed manual edit. Membership changes are
treated as schedule transitions and the active roster is recalculated at the
effective instant.

Non-overlapping overrides use exact start and end instants and temporarily
supersede both the rotation and weekday exclusions. Organization member removal
is blocked while that member remains referenced by a rotation.

An escalation policy contains up to ten ordered steps. Step one is immediate;
each later step stores a delay relative to the preceding step. A step targets
an on-call schedule, the service's primary owners, or all service owners. When
an incident is created for a service with a policy, the API snapshots every
step into a delayed outbox message in the same transaction. The worker resolves
the current schedule responder or service owners when each step becomes due, so
rotation changes and active overrides are honored at delivery time.
Acknowledgement or resolution causes remaining stages to be audited as skipped.

Registration is deployment-configurable. Self-hosted Compose selects
`Bootstrap`, so exactly the first administrator can create an organization and
choose a passkey (recommended) or password without needing an outbound mail
server. Registration then closes automatically and additional members join by
administrator invitation. `Open` deliberately keeps public registration
available and may require email ownership verification according to deployment
configuration.
`Provisioned` removes public registration while allowing an authenticated
provisioning service to create an organization and its initial administrator.
It can support a self-service checkout flow without exposing an unauthenticated
organization-creation endpoint. `Disabled` is the fail-closed product default.
The same web build reads the registration capability from the API rather than
embedding a deployment-edition switch.

ASP.NET Core Identity 10 owns passkey attestation, assertion, replay counters,
and credential storage. Public passkeys are discoverable and require local user
verification, enabling username-less sign-in. The explicit RP ID is inferred
from the canonical public URL unless an operator deliberately overrides it.
An authenticated user can keep up to ten named passkeys. Credential removal is
serialized per user, and a passkey-only account cannot remove its final passkey.

Local Identity credentials are one authentication method, not a permanent
identity-provider boundary. External login records are modeled by stable provider
and provider-subject identifiers. The public API exposes a trusted, explicitly
configured extension loader plus a provider-neutral federated sign-in service.
That service owns account linking, organization-boundary enforcement, optional
JIT membership creation, invitation completion, and the normal Identity cookie.
Protocol adapters validate their upstream response and submit only the resulting
identity to this service.

Trusted authentication extensions can validate an external identity, veto local
credential login, add product routes, and register a namespaced PostgreSQL
context whose migrations participate in the one-off migration mode. A missing,
invalid, or duplicate configured extension fails API startup. Self-hosted
deployments load no extension by default.

## Durable work and delivery semantics

The PostgreSQL outbox closes the failure window between changing application
state and requesting asynchronous work:

1. An application use case stores its state change and a uniquely identified
   outbox record in one database transaction.
2. A worker claims pending work without allowing another worker to own the same
   lease concurrently.
3. The selected provider publishes or handles the message.
4. Completion is recorded durably. Notification deliveries store every attempt,
   recipient, channel, outcome, and error. Failures use exponential backoff and
   become dead letters after the configured maximum.

Delivery is **at least once**. A stable message identifier travels with every
message; it is not regenerated during retry. Consumers must use that identifier
as an idempotency key and make their state change plus processed-message marker
atomic wherever possible. A handler must assume it can run again after a timeout,
lost acknowledgement, process crash, or queue visibility expiry.

The abstractions intentionally cover only publishing and consuming Firewatch
messages. They are not a general-purpose event bus. PostgreSQL is the default and
requires no additional broker beyond the database Firewatch already needs. Its
`FOR UPDATE SKIP LOCKED` claim path supports multiple worker replicas. When SQS
is selected, committed outbox rows are relayed to SQS; workers long-poll and use
a durable processed-message record plus a PostgreSQL advisory lock for
idempotent redelivery. Email delivery uses `IEmailSender`; phone verification
and delivery use `IPhoneSender`. SMTP, SES, and optional Twilio adapters do not
change incident behavior.
Slack delivery uses the immutable user ID captured by verified-email mapping.
Each delivered Slack alert stores its bot-message reference. Signed Block Kit
actions mutate the incident in the API transaction, and the resulting outbox
event makes the worker update the original message and append a lifecycle
thread reply. Outbound webhooks use a stable delivery ID plus timestamped HMAC
signature. Organization template records are shared through PostgreSQL,
avoiding divergent template files across worker replicas.

## Architectural smoke flow

The asynchronous processing smoke flow is authenticated system-test infrastructure:

1. `POST /api/v1/system/test-jobs` creates a test-job row and its outbox message
   in one PostgreSQL transaction.
2. The API returns `202 Accepted`, including the stable job identifier and a
   status location.
3. The separately running worker claims and processes the outbox item, logs its
   message identifier, and marks the test job complete.
4. A duplicate delivery observes the processed identifier and does not perform
   duplicate completion.
5. `GET /api/v1/system/test-jobs/{id}` reports the state to the web application
   or a developer.

This flow validates the architecture; it is not a model for alerts, incidents,
or escalation policies.

## Health and observability

- `GET /health/live` reports that the API process can serve requests. It does not
  fail merely because PostgreSQL is unavailable.
- `GET /health/ready` verifies required dependencies, including PostgreSQL, and
  is the deployment readiness gate.
- API and worker logs are structured JSON and include message or request context
  plus correlation/request identifiers where available. Deployments keep the
  process streams distinct and may add service-name enrichment in their log
  collection layer.
- OpenTelemetry registration is an extension point for traces and metrics. An
  exporter or full observability backend is deliberately not bundled.

Liveness should restart a stuck process. Readiness should remove an unhealthy API
from traffic. Neither endpoint should leak credentials or detailed exception
data.

## One product, two deployment models

For self-hosting, Compose runs PostgreSQL, API, worker, and web, with the
PostgreSQL queue provider. A future public Helm chart may package the same
containers.

Managed installations configure the same provider interfaces and product
contracts while operating infrastructure, secrets, backups, and delivery
providers for customers. Private infrastructure does not introduce a second
copy of Firewatch's domain behavior.

## Future iOS client

A native iOS application will be another client of the same versioned REST API;
the web app is not a backend-for-frontend. The checked OpenAPI document can be
fed to a maintained Swift OpenAPI generator (for example Apple's Swift OpenAPI
Generator) when iOS work begins. Authentication and API version rules live at
the public API boundary rather than in a web-only endpoint; future compatibility
changes must preserve that boundary.