> ## Documentation Index
> Fetch the complete documentation index at: https://docs.safeonlineexam.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Upgrading and Rolling Back Safe Online Exam Deployments

> Upgrade procedure for Docker Compose and Cloud Run, covering the mandatory migration job, staged candidate revision checks, and schema-aware rollback.

Safe Online Exam upgrades follow a staged procedure on both supported platforms: a migration job runs first, a candidate revision is verified before it receives traffic, and rollback is always available without reversing database migrations. This page covers the pre-upgrade checklist, the migration requirement, platform-specific upgrade and rollback procedures, and the npm commands that underpin each step.

## Pre-Upgrade Checklist

Before upgrading any environment, complete these steps:

<Steps>
  <Step title="Review the changelog and release notes">
    Read the [CHANGELOG](https://github.com/JSB2010/safe-online-exam/blob/main/CHANGELOG.md) for the target version. Note whether the release adds a database migration, changes Canvas LTI registration URLs, adds OAuth scopes, or modifies public SEB compatibility endpoints — each requires additional post-upgrade steps.
  </Step>

  <Step title="Verify the image digest and attestation">
    Production must pin the immutable digest from the GitHub Release, never a mutable tag:

    ```text theme={null}
    ghcr.io/jsb2010/safe-online-exam@sha256:...
    ```

    Verify the checksum of the downloaded bundle before extracting:

    ```bash theme={null}
    sha256sum --check "safe-online-exam-X.Y.Z-<platform>.tar.gz.sha256"
    ```

    Then verify the image attestation using the exact repository, signer workflow, source commit, and source tag printed in the release notes.
  </Step>

  <Step title="Run acceptance on a test environment first">
    Apply the upgrade to a non-production or testbed environment and complete the relevant acceptance checks — including the [Canvas and SEB acceptance sequence](operations/testing) — before promoting to production.
  </Step>

  <Step title="Create a database backup">
    Both upgrade helpers create a backup automatically before any migration or traffic change. For Compose, also ensure the named `postgres_data` volume is intact. For Cloud Run, the upgrade waits for the Cloud SQL backup operation to report `SUCCESSFUL` before proceeding.
  </Step>
</Steps>

## Migration Requirement

<Warning>
  The migration job **must** complete successfully before the new application image receives any traffic. The `/ready` endpoint fails until all migrations have been applied. Do not manually route traffic to a new revision before its migration job has succeeded.
</Warning>

Every upgrade helper enforces this order automatically. If you are managing a custom container platform, run the migration as a one-shot pre-traffic job:

```bash theme={null}
node dist/server/server/data/migrate.js
```

The equivalent npm script for local or source-based environments:

```bash theme={null}
npm run db:migrate
```

The cleanup job runs on a schedule to remove expired sessions, state, and other transient records:

```bash theme={null}
npm run db:cleanup
```

<Note>
  Multiple application instances share all state through PostgreSQL. No sticky sessions are required. You can run any number of application replicas against the same database and they will remain consistent without any additional coordination.
</Note>

***

## Docker Compose Upgrade

<Steps>
  <Step title="Download and verify the new bundle">
    Download the next Compose release bundle and verify its checksum before extraction:

    ```bash theme={null}
    export VERSION="X.Y.Z"
    curl -fLO "https://github.com/JSB2010/safe-online-exam/releases/download/v${VERSION}/safe-online-exam-${VERSION}-compose.tar.gz"
    curl -fLO "https://github.com/JSB2010/safe-online-exam/releases/download/v${VERSION}/safe-online-exam-${VERSION}-compose.tar.gz.sha256"
    sha256sum --check "safe-online-exam-${VERSION}-compose.tar.gz.sha256"
    tar -xzf "safe-online-exam-${VERSION}-compose.tar.gz"
    ```

    Read the extracted `README.md` before proceeding.
  </Step>

  <Step title="Preserve your existing environment and secrets">
    Keep the existing protected environment file, secret files, client identity record, and database volume in their durable installation directory. Merge any new template keys from the new bundle into your existing configuration — do not overwrite local configuration wholesale.
  </Step>

  <Step title="Run the upgrade helper with your existing environment path">
    Invoke the new bundle's `upgrade.sh` with the path to the existing environment file. Relative secret and backup paths resolve from the directory containing that file:

    ```bash theme={null}
    NEW_BUNDLE=/opt/safe-online-exam-X.Y.Z
    EXISTING_ENV=/srv/safe-online-exam/.env.secrets
    "$NEW_BUNDLE/upgrade.sh" "$EXISTING_ENV"
    ```

    The helper will:

    1. create a PostgreSQL custom-format backup and validate it;
    2. pull the exact pinned images;
    3. apply checked forward migrations;
    4. restart the topology; and
    5. verify readiness at `/ready`.
  </Step>

  <Step title="Copy the backup to off-host storage">
    Move the newly created backup to encrypted, access-controlled off-host storage immediately. The named `postgres_data` volume is not a substitute for a verified backup.
  </Step>

  <Step title="Run post-upgrade smoke checks">
    Verify the key endpoints after the upgrade completes:

    ```bash theme={null}
    curl -fsS "${TOOL_URL}/health"
    curl -fsS "${TOOL_URL}/ready"
    curl -fsS "${TOOL_URL}/.well-known/jwks.json"
    curl -fsS "${TOOL_URL}/lti/config"
    curl -fsS "${TOOL_URL}/js/canvas-seb-detector.js" | head
    ```

    Then complete the [Canvas and SEB acceptance sequence](operations/testing) for the roles affected by the release.
  </Step>
</Steps>

### Compose Rollback

Application rollback does not undo database migrations. Restore an older image only after confirming it supports the current schema. Data rollback requires a reviewed restore into a controlled target:

1. Stop the current topology.
2. Confirm schema compatibility between the prior image and the migrated database schema.
3. Update the image digest in your environment file to the prior pinned release digest.
4. Restart the topology.

For data rollback, restore a backup into a controlled target first. Do not overwrite the active database as the first diagnostic action.

***

## Cloud Run Upgrade

<Steps>
  <Step title="Download and verify the new Cloud Run bundle">
    ```bash theme={null}
    export VERSION="X.Y.Z"
    curl -fLO "https://github.com/JSB2010/safe-online-exam/releases/download/v${VERSION}/safe-online-exam-${VERSION}-cloud-run.tar.gz"
    curl -fLO "https://github.com/JSB2010/safe-online-exam/releases/download/v${VERSION}/safe-online-exam-${VERSION}-cloud-run.tar.gz.sha256"
    sha256sum --check "safe-online-exam-${VERSION}-cloud-run.tar.gz.sha256"
    tar -xzf "safe-online-exam-${VERSION}-cloud-run.tar.gz"
    ```
  </Step>

  <Step title="Preserve existing environment and bootstrap records">
    Keep `cloudrun.env`, bootstrap state, client identity, and `.state` files in a durable installation directory outside source control. Merge new template keys from the new bundle instead of overwriting local configuration.
  </Step>

  <Step title="Run the upgrade helper">
    Invoke the new bundle's `upgrade.sh` with the path to the existing environment file:

    ```bash theme={null}
    NEW_BUNDLE=/opt/safe-online-exam-X.Y.Z-cloud-run
    EXISTING_ENV=/srv/safe-online-exam/cloudrun.env
    "$NEW_BUNDLE/upgrade.sh" "$EXISTING_ENV"
    ```

    The helper will:

    1. validate the current environment and bootstrap contract, create any newly required numbered Secret Manager versions, and grant the existing runtime identity access;
    2. create an on-demand Cloud SQL backup, wait for its operation to finish, and require the resulting backup status to be `SUCCESSFUL`;
    3. run the new migration job;
    4. update the cleanup job and apply current environment and secret bindings;
    5. deploy a no-traffic candidate revision;
    6. temporarily enable a previously disabled generated URL so the candidate can be verified;
    7. verify candidate readiness and JWKS;
    8. cut traffic over explicitly; and
    9. verify the custom origin and restore the prior generated-URL policy.
  </Step>

  <Step title="Run post-upgrade smoke checks">
    After traffic cutover, verify:

    ```bash theme={null}
    curl -fsS "${TOOL_URL}/health"
    curl -fsS "${TOOL_URL}/ready"
    curl -fsS "${TOOL_URL}/.well-known/jwks.json"
    curl -fsS "${TOOL_URL}/lti/config"
    curl -fsS "${TOOL_URL}/js/canvas-seb-detector.js" | head
    ```

    Also inspect:

    * the active revision and exact image digest;
    * migration and cleanup job execution history;
    * Cloud Scheduler's last execution;
    * Cloud SQL backup/PITR status, storage, and connections; and
    * numbered secret-version references.
  </Step>
</Steps>

### Cloud Run Rollback

If a candidate check fails during `upgrade.sh`, the previous revision retains traffic automatically. Completed forward migrations remain in place.

For an explicit rollback after traffic has been cut over:

1. Confirm the prior revision supports the current database schema (application rollback never reverses migrations).
2. Use the bundle's explicit schema-compatibility confirmation before proceeding.
3. Route traffic back to the prior revision using the bundle's rollback helper or `gcloud run services update-traffic`.
4. The rollback helper verifies `TOOL_URL` when configured and re-disables the generated URL if applicable.

For data recovery, restore a Cloud SQL backup into a controlled target first. Do not overwrite the active database as the first diagnostic action.

***

## OAuth Token Encryption Migration

If upgrading from a version that stored OAuth tokens in plaintext (pre-1.1), follow the two-phase mode transition before enforcing encryption.

<Tabs>
  <Tab title="Docker Compose">
    <Steps>
      <Step title="Deploy in compatibility mode first">
        Set `OAUTH_TOKEN_ENCRYPTION_MODE=compat` in `.env.secrets` and run the upgrade. The compatibility revision can read both plaintext and encrypted tokens while writing only encrypted rows.
      </Step>

      <Step title="Enforce and rewrite existing tokens">
        Change the mode to `enforce`, recreate the application, then run the one-shot rewrite:

        ```bash theme={null}
        docker compose --env-file .env.secrets -f compose.yaml -f compose.secrets.yaml \
          --profile maintenance run --rm encrypt-oauth-tokens
        ```

        Run the command a second time and require `0` updates before retiring the old plaintext key.
      </Step>
    </Steps>
  </Tab>

  <Tab title="Cloud Run">
    <Steps>
      <Step title="Deploy in compatibility mode first">
        Set `OAUTH_TOKEN_ENCRYPTION_MODE=compat` in `cloudrun.env` and run `upgrade.sh`. The upgrade verifies the current serving revision already reports `compat` or `enforce` before accepting an `enforce` request.
      </Step>

      <Step title="Enforce and rewrite existing tokens">
        After the `compat` revision is healthy, set `OAUTH_TOKEN_ENCRYPTION_MODE=enforce` and re-run `upgrade.sh`. Then run the one-shot Cloud Run rewrite job:

        ```bash theme={null}
        ./encrypt-oauth-tokens.sh cloudrun.env
        ```

        Run it a second time and require `0` updates before removing a retired key.
      </Step>
    </Steps>
  </Tab>
</Tabs>

<Warning>
  A pre-encryption application revision is not a valid rollback target after encrypted writes begin. The staged `compat` deployment is mandatory for any existing installation — the upgrade helper rejects an `enforce` request when the current serving revision has not yet reported `compat` or `enforce`.
</Warning>

***

## npm Commands Reference

The following npm scripts are used during upgrades and maintenance. The `built` variants run the compiled output directly without requiring TypeScript tooling, matching the commands the deployment helpers invoke internally.

| Script                                  | What it does                                                                  |
| --------------------------------------- | ----------------------------------------------------------------------------- |
| `npm run db:migrate`                    | Applies all pending database migrations (source/dev environments)             |
| `npm run db:migrate:built`              | Applies migrations from the compiled output — used by deployment helpers      |
| `npm run db:cleanup`                    | Drains expired sessions, state, and transient records (source/dev)            |
| `npm run db:cleanup:built`              | Drains expired records from the compiled output — used by deployment helpers  |
| `npm run db:encrypt-oauth-tokens`       | One-shot rewrite of legacy and retired-key OAuth token rows (source/dev)      |
| `npm run db:encrypt-oauth-tokens:built` | One-shot token rewrite from compiled output — used by deployment helpers      |
| `npm run release:check`                 | Validates synchronized release metadata before tagging                        |
| `npm run verify`                        | Full non-browser gate: typecheck, lint, format, coverage, and build           |
| `npm run verify:postgres`               | PostgreSQL integration gate: isolated schema migrations and concurrency tests |

For production deployments managed by the release bundles, the `built` variants invoke the compiled output directly without requiring TypeScript tooling. The migration and cleanup jobs call `npm run db:migrate:built` and `npm run db:cleanup:built` respectively, which resolve to `node dist/server/server/data/migrate.js` and `node dist/server/server/data/cleanup.js --drain`.