Operations

A runbook for the failure modes that have actually occurred, in the order you are likely to meet them. Each entry names the symptom first, because that is what you have when something breaks.

Health and status

kubectl -n mobilesigner-demo get pods
kubectl -n mobilesigner-demo logs -l app=backend --tail=50
kubectl -n mobilesigner-demo logs -l app=frontend --tail=50

The backend serves SmallRye health endpoints:

Endpoint	Reports
/q/health	everything
/q/health/live	process is alive
/q/health/ready	datasource reachable

The Deployments declare no probes, so nothing consumes them automatically. Use them by hand:

kubectl -n mobilesigner-demo exec deploy/backend -- \
  curl -s localhost:8090/q/health/ready

The quickest end-to-end check is the dashboard, which touches four tables in one request:

curl -s http://webauthn-demo.localhost/backend/dashboard

A response with totalAuditEvents of 12 and nothing else means the database was just reseeded — see below.

The database

PostgreSQL 17 runs as a StatefulSet with its own PersistentVolumeClaim, so the working set outlives the pods. Destroying the entire backend tier does not lose data:

kubectl -n mobilesigner-demo delete pod -l app=backend
kubectl -n mobilesigner-demo rollout status deploy/backend

Every credential, approval and audit row is still there afterwards. This is the behaviour to verify after any change to the datasource configuration, because the failure is silent — a misconfigured URL falls back to nothing, it just starts empty and reseeds.

Check what is actually stored:

kubectl -n mobilesigner-demo exec postgres-0 -- \
  psql -U mobilesigner -d mobilesigner -c "\\dt"

kubectl -n mobilesigner-demo exec postgres-0 -- psql -U mobilesigner -d mobilesigner -t \
  -c "select 'files='||count(*) from gov_file union all select 'approvals='||count(*) from approval;"

Column names have no underscores — Hibernate's implicit naming lowercases the field, so it is filenumber, not file_number. The Approval entity has no @Table, so its table is approval.

Symptom: the audit trail is back to twelve rows and your enrolled key no longer works.

Cause: the database is empty, so DemoDataSeeder reseeded. Either the PVC was deleted, or the backend is pointed somewhere unintended. Check what it resolved:

kubectl -n mobilesigner-demo get pod -l app=backend \
  -o jsonpath='{.items[0].spec.containers[0].env}'
kubectl -n mobilesigner-demo get pvc

data-postgres-0 should be Bound. If it is Pending, no storage class satisfied the claim and PostgreSQL never started.

Multiple replicas

The backend runs two replicas. Confirm both are serving and agree:

kubectl -n mobilesigner-demo get pods -l app=backend -o wide
npm run verify:distributed        # from the frontend repository

That script establishes a real session through the ingress, then replays the cookie against each pod directly. If one pod rejects it, the replicas do not share SESSION_ENCRYPTION_KEY — check the Secret is mounted into both:

kubectl -n mobilesigner-demo get secret mobilesigner-session

A missing Secret makes the container fall back to the default in application.properties. All pods would then still agree with each other, but the key is a published literal, so rotate it and redeploy.

On startup both pods race to seed. One wins, and the other logs:

Seed data not written — another instance seeded this database first (...)

That line is expected and not an error. The unique constraint on gov_file.filenumber is what makes it safe.

Deliberately resetting

Locally:

cd mobilesigner-webauthn-backend-demo
rm -rf data/
./gradlew quarkusDev

In the cluster a restart no longer resets anything — that is the point of the volume. Truncate the tables instead, then restart the backend so the seeder runs again:

kubectl -n mobilesigner-demo exec postgres-0 -- psql -U mobilesigner -d mobilesigner -c \
  'truncate file_note, approval, audit_event, signing_device, webauthncredential, user_account, gov_file cascade;'
kubectl -n mobilesigner-demo rollout restart deploy/backend

cascade matters because file_note references gov_file. Truncating in one statement avoids ordering the tables by hand.

To discard the storage entirely, delete the StatefulSet and its claim — the claim is not removed with the StatefulSet, which is deliberate on Kubernetes' part and a common surprise:

kubectl -n mobilesigner-demo delete statefulset postgres
kubectl -n mobilesigner-demo delete pvc data-postgres-0
kubectl -n mobilesigner-demo apply -f k8s/postgres.yaml

Nothing at http://webauthn-demo.localhost

Work outward from the browser.

1. Is the host-to-cluster bridge running? It is a plain Docker container, not a cluster resource, and it does not come back with the cluster. It has stopped on its own before.

docker ps --filter name=webauthn-demo-ingress-bridge
docker start webauthn-demo-ingress-bridge

It must be in the same Docker context as the Kind cluster and attached to the kind network. A bridge in a different context cannot reach the node containers at all, and looks perfectly healthy while doing so. That was the hardest failure to diagnose in this project — see Deployment.

2. Does the Ingress have a class?

kubectl -n mobilesigner-demo get ingress mobilesigner-ingress \
  -o jsonpath='{.spec.ingressClassName}'

Must print traefik. Without it no controller adopts the Ingress and every request 404s with nothing logged to explain it.

3. Are the Services finding pods?

kubectl -n mobilesigner-demo get endpoints

Empty ENDPOINTS means the label selectors match nothing, or the pods are not ready.

4. Can you reach the pods directly?

kubectl -n mobilesigner-demo exec deploy/frontend -- wget -qO- localhost:3000/
kubectl -n mobilesigner-demo exec deploy/backend  -- curl -s localhost:8090/api/dashboard

If these work and the hostname does not, the problem is the bridge or the Ingress, not the application.

Frontend running but unreachable

Symptom: the pod is Running, the log says Ready in 412ms, and every request through the Service times out. Nothing appears in the frontend log because no connection is ever accepted.

Cause: HOSTNAME. Kubernetes sets it to the pod name; Next.js standalone binds to exactly that address and refuses connections to the cluster IP.

Fix: the env var, which is already in k8s/deploy.yaml:

- name: HOSTNAME
  value: "0.0.0.0"

Confirm it survived any edit:

kubectl -n mobilesigner-demo get deploy frontend \
  -o jsonpath='{.spec.template.spec.containers[0].env}'

This is the single most confusing failure in the stack, because every signal says healthy.

WebAuthn fails in the browser

"The Quarkus WebAuthn ceremony client did not load"

SDK_NOT_LOADED — window.WebAuthn is undefined, so /q/webauthn/webauthn.js did not arrive.

curl -sI http://webauthn-demo.localhost/q/webauthn/webauthn.js

Expect 200 and JavaScript. A 404 means the Ingress /q/webauthn rule is missing or ordered after /, so the catch-all sent it to the frontend.

The key is rejected, or the browser offers no credential

Almost always an RP ID mismatch. A credential is bound to the relying party id it was created under, and the browser will not offer it under a different one.

kubectl -n mobilesigner-demo get deploy backend \
  -o jsonpath='{.spec.template.spec.containers[0].env}'

WEBAUTHN_RP_ID must equal the browser's hostname exactly, and WEBAUTHN_ORIGIN must equal scheme plus host. A credential enrolled at localhost in local development cannot be used against webauthn-demo.localhost, and vice versa — enrol once per environment.

"This browser does not support WebAuthn"

UNSUPPORTED_BROWSER, or a secure-context failure. WebAuthn and crypto.subtle both need a secure context. http://localhost and http://*.localhost qualify; any other plain-HTTP origin does not.

Reaching the app by pod IP or NodePort over plain HTTP will fail for this reason. Use the hostname.

The ceremony times out

CEREMONY_CANCELLED. WebAuthn deliberately conflates "user declined", "timeout elapsed" and "no matching credential" into one NotAllowedError, so the message covers all three. Usually the officer did not touch the key in time. Retry.

Session problems

Symptom: decision controls stay disabled after an apparently successful enrolment.

The gate is server truth:

curl -s http://webauthn-demo.localhost/backend/session

{"authenticated":false,"username":null} means the session cookie is absent or undecryptable.

quarkus.webauthn.cookie-same-site=strict means the cookie is not sent on cross-site navigations. Same-origin is preserved by both the Next rewrite and the Ingress, so this should not bite — but it will if you reach the frontend and the backend on different hostnames.

A cookie that was valid and suddenly is not means the encryption key changed, which for this deployment means the pod restarted with a different key. Which brings us to the one to fix first.

Setting the session key

SESSION_ENCRYPTION_KEY is not set in k8s/deploy.yaml, so every deployment uses the default committed in application.properties:

quarkus.http.auth.session.encryption-key=${SESSION_ENCRYPTION_KEY:local-demo-session-key-32-bytes!}

The backend authorises requests from the cookie rather than by re-running an assertion, so the cookie is what needs protecting. Supply a real key before the deployment is reachable by anyone else.

kubectl -n mobilesigner-demo create secret generic mobilesigner-session \
  --from-literal=key="$(openssl rand -base64 32)"

env:
  - name: SESSION_ENCRYPTION_KEY
    valueFrom:
      secretKeyRef:
        name: mobilesigner-session
        key: key

Rotating it invalidates every existing session, which is the intended effect.

Audit log growth

POST /api/verify is public by design, and is now rate limited to 20 calls per minute per caller. An anonymous verification writes no audit row at all, so a read that anyone can make can no longer grow the ledger.

The table therefore grows with verification traffic, not just with approvals. Worth watching if you expose the endpoint beyond a demo.

Watch for it:

curl -s 'http://webauthn-demo.localhost/backend/audit?type=SIGNATURE_VERIFIED&limit=200' \
  | grep -o '"id"' | wc -l

Hitting 200 means you are at the API cap: limit sets a page size with no page index, so the endpoint returns the most recent rows and nothing older. Query PostgreSQL directly for a true count.

Mitigation is authentication on the endpoint, or a rate limit, or not writing an audit row for anonymous reads. Verification is a read; recording it as a ledger entry is what creates the exposure.

Log noise and what is missing

quarkus.hibernate-orm.log.sql=false
quarkus.log.level=INFO

SQL logging is off. Turn it on temporarily to see the N+1 in FileResource.list():

kubectl -n mobilesigner-demo set env deploy/backend \
  QUARKUS_HIBERNATE_ORM_LOG_SQL=true

You will see one COUNT per file row, because toSummary() calls FileNote.count() per file.

There is no access log, no structured JSON logging, no metrics endpoint and no tracing. AuditEvent.correlationId is a fresh UUID per row rather than a per-request value, so it identifies a row rather than grouping a flow. Diagnosing anything beyond a single request means reading raw logs.

Inspecting the database

psql ships in the PostgreSQL image, so the database is directly queryable — including the rows the API caps:

kubectl -n mobilesigner-demo exec -it postgres-0 -- psql -U mobilesigner -d mobilesigner

# true audit count, past the 200-row API cap
kubectl -n mobilesigner-demo exec postgres-0 -- psql -U mobilesigner -d mobilesigner -t \
  -c 'select count(*) from audit_event;'

Remember the column naming: filenumber, currentholder, initiatedat — Hibernate lowercases the field name rather than inserting underscores.

Backup

There is still no automated backup. The volume makes data durable across pod loss, which is not the same as recoverable — a deleted claim, a corrupted cluster or a bad migration all remain unrecoverable.

A manual dump works today:

kubectl -n mobilesigner-demo exec postgres-0 -- \
  pg_dump -U mobilesigner -d mobilesigner --clean > mobilesigner-$(date +%F).sql

The minimum for anything shared is a CronJob running pg_dump to object storage, and for anything carrying real approvals, managed PostgreSQL with point-in-time recovery. Moving to a managed instance is a change of DB_URL and the Secret — the entity layer is untouched.

Both audit hardening steps are now in place. V2__audit_ledger.sql installs triggers that refuse UPDATE, DELETE and TRUNCATE — a trigger blocks even the table owner, which a GRANT could not — and chains each row to its predecessor with a SHA-256 of its own contents. Check it at any time:

kubectl -n mobilesigner-demo exec postgres-0 -- \
  psql -U mobilesigner -d mobilesigner -c 'select * from audit_event_verify_chain();'

A null broken_at means every row still hashes to its successor's previoushash. The same check is exposed at GET /api/audit/integrity for the auditor role, and the Audit Trail screen has a button for it.