Skip to content

test(ci): audit.log posture helper + jest suites for backendKind allow-list (#540)#577

Merged
shaun0927 merged 1 commit intodevelopfrom
test/540-audit-backend-kind
Apr 16, 2026
Merged

test(ci): audit.log posture helper + jest suites for backendKind allow-list (#540)#577
shaun0927 merged 1 commit intodevelopfrom
test/540-audit-backend-kind

Conversation

@shaun0927
Copy link
Copy Markdown
Owner

Summary

  • New src/ci/audit-log-posture.ts — tolerant parser that pulls every backendKind value out of ~/.opensafari/audit.log (or \$OPENSAFARI_AUDIT_LOG) whether it lives at the top level or inside a stringified args_summary, and exposes scanAuditLog() / assertAuditLogPosture().
  • New tests/unit/audit-log-posture.test.ts — 10 fixture-driven cases covering all branches (10/10 pass locally).
  • New tests/integration/audit-log-backend-kind.live.test.ts — runs against the real audit log produced by the live smoke jobs; default npm test skips it (excluded in jest.config.js).

Refs: #540 — moves the acceptance-criteria box "Telemetry: backendKind in {'flutter-vm','simhid','webkit'} only (applescript = 0)" into a shape that can be asserted in CI. Workflow wiring is a follow-up so this PR stays a pure library addition.

Why

Unit B (PR #576) already centralises the "no applescript" check. This PR adds the positive-side constraint — every recorded backend must come from the allowed set — with a testable helper that the smoke jobs can call after their live steps land.

Test plan

  • `npx jest tests/unit/audit-log-posture.test.ts --no-coverage` → 10/10 pass.
  • Parser handles `"backendKind":"simhid"` and `\"backendKind\":\"simhid\"` shapes.
  • `scanAuditLog('/does/not/exist')` returns a zero-entry report (treats absent log as vacuous pass).
  • After merge, follow-up PR wires the live suite into each smoke job so the assertion runs on every scheduled cron.

Known red CI

`develop` currently fails `npm run build` because of a duplicate `wsToHttpUrl` in `src/flutter/vm-service-discovery.ts` (introduced by commit `5527d3f7`). That is unrelated to this PR but will break this PR's CI until a hotfix lands. I'm opening that hotfix as a separate PR immediately after.

🤖 Generated with Claude Code

shaun0927 added a commit that referenced this pull request Apr 16, 2026
Commit 5527d3f ("Stabilize the stacked headless smoke jobs") accidentally
re-declared `wsToHttpUrl` in `src/flutter/vm-service-discovery.ts` — the
documented implementation at lines 99-102 is preceded by an identical
undocumented copy at lines 89-92. ts-loader surfaces this as TS2323 +
TS2393 during `npm run build`, which in turn fails the `lint` and `test`
jobs in `.github/workflows/ci.yml` (both run `npm run build` through the
`prepare` hook).

Remove the undocumented duplicate; keep the documented one.

Verified:
- `npm run build:src` now compiles cleanly (0 errors).
- `npx jest tests/unit/flutter-vm-service.test.ts` — 21/21 pass.
- `git blame` confirms the duplicate was introduced on 2026-04-16 and not
  used by any caller outside the file itself.

This hotfix unblocks CI on develop and on every in-flight PR (#575, #576, #577).

Refs: #540

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…w-list (#540)

Adds `src/ci/audit-log-posture.ts` with `scanAuditLog()` /
`assertAuditLogPosture()` — extracts `backendKind` values from
`~/.opensafari/audit.log` (or `$OPENSAFARI_AUDIT_LOG`) regardless of
whether they live at the top level or inside a stringified
`args_summary`, and asserts the set is a subset of
`{flutter-vm, simhid, webkit}`.

Two test suites exercise it:

- `tests/unit/audit-log-posture.test.ts` — 10 fixture-driven cases covering
  absent log, empty log, top-level + nested extraction, disallowed kinds,
  case-insensitive applescript detection, malformed JSONL tolerance, and
  the assert wrapper.
- `tests/integration/audit-log-backend-kind.live.test.ts` — runs against
  the real audit log produced by earlier live smoke steps in the same CI
  job. Default `npm test` skips it (integration tests are excluded in
  `jest.config.js`); the smoke workflow invokes it explicitly via
  `--runTestsByPath`.

Refs: #540 acceptance criterion
"Telemetry: backendKind in {'flutter-vm','simhid','webkit'} only
(`applescript` = 0)".

Follow-up: wire the live suite into each smoke job in
`.github/workflows/headless-smoke.yml` in a separate PR so the workflow
change stays reviewable.

Verified:
- `npx jest tests/unit/audit-log-posture.test.ts --no-coverage` → 10/10 pass.
- Helper tolerates both `"backendKind":"simhid"` and `\"backendKind\":\"simhid\"`.

Refs: #540

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@shaun0927 shaun0927 force-pushed the test/540-audit-backend-kind branch from 33054e9 to 6a2607b Compare April 16, 2026 13:51
Copy link
Copy Markdown
Owner Author

@shaun0927 shaun0927 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code review: P0=0, P1=0, P2=2 (vacuous pass semantics + path traversal — both CI-only). Well-structured posture helper with 10 thorough unit tests. LGTM.

@shaun0927 shaun0927 merged commit 41c1411 into develop Apr 16, 2026
3 checks passed
@shaun0927 shaun0927 deleted the test/540-audit-backend-kind branch April 16, 2026 14:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant