Security Regression Testing

Begin with evidence from a permitted test: the request, the actor, the response, and the security property the fix should enforce. Do not encode a response status just because it appeared once; choose the condition that means the vulnerability is blocked.

1. Save the flow with an explicit expectation

Tests live under .h5i-tests/tests/ and use version: h5i.test/v1. This example assumes Alice can read post 41 but must not read Bob's post 42. Supply the token as a CI secret rather than committing it.

version: h5i.test/v1
id: post-ownership
name: Alice cannot read Bob
actors:
  alice: {}
requests:
  own_post:
    path: /api/posts/41
    headers:
      authorization: Bearer ${env.ALICE_TOKEN}
  other_post:
    path: /api/posts/42
    headers:
      authorization: Bearer ${env.ALICE_TOKEN}
flow:
  - send: own_post
    as: alice
    expect:
      status: 200
  - send: other_post
    as: alice
    expect:
      status: 403

The first request confirms the actor is authenticated and the route works; the second checks ownership. For a property the built-in response checks cannot express, use an oracle. A test with neither an expectation nor an oracle is refused.

2. Run it against staging

h5i plugin install test
h5i test .h5i-tests/tests/post-ownership.yaml --target https://staging.example.com

Commit the test with the application fix. Give CI a staging target and an identity with only the permissions needed. For example, add this step to a job that deploys a disposable staging build and waits for it to become ready:

# Inside a GitHub Actions job that can reach staging:
- name: Replay security regression tests
  env:
    ALICE_TOKEN: ${{ secrets.STAGING_ALICE_TOKEN }}
  run: |
    curl -fsSL https://h5i.dev/install.sh | sh -s -- --test
    h5i test .h5i-tests/tests/post-ownership.yaml --target https://staging.example.com

A failed property is different from an execution error; h5i does not turn an undecidable result into a pass. Review the recorded result and response artifacts when a job fails. Keep the staging token in the CI secret store and ensure the target uses test data that can be reset between runs.

3. Check API coverage deliberately

When the project has an OpenAPI description, h5i test --openapi openapi.json reports which declared operations the completed tests cover. Add --min-coverage only after reviewing the denominator and the operation mapping. A test that errored does not count as covered; a response assertion still has to express the security property.

4. Keep the test maintainable

Use named actors and separate cookie jars for multi-user flows. Extract values from earlier responses when IDs change across runs. Keep setup and cleanup distinct from the verdict, and let cleanup failures remain visible. The h5i test manual defines the schema, oracle environment, and CI output. A complete checked-in example is the board authorization test.