feat: Automate eval tests setup

[re-pixel]

What changed

Automates local agent eval setup: a bootstrap script talks to the running SuperPlane app (owner setup or password login), ensures an eval canvas and service account, then prints or merges EVAL_ORG_ID, EVAL_CANVAS_ID, SUPERPLANE_API_TOKEN, and SUPERPLANE_BASE_URL into agent/.env. Adds make agent.evals.bootstrap, httpx for the script, and updates agent/.env.example / agent/README.md.

How

• New agent/scripts/bootstrap_eval_env.py (cookie session + public HTTP API).
• Makefile runs it in the agent container with host-owned user, UV_CACHE_DIR and UV_PROJECT_ENVIRONMENT under /tmp so uv sync does not fight a root-owned bind-mounted .env / .venv.

Related Issues

#3666

[superplanehq-integration]

👋 Commands for maintainers:

• /sp start - Start an ephemeral machine (takes ~30s)
• /sp stop - Stop a running machine (auto-executed on pr close)

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

[cursor]

Casbin ptype NOT NULL constraint accidentally removed

Medium Severity

The casbin_rule.ptype column changed from NOT NULL to nullable, and a new unique index idx_casbin_rule was added on all-nullable columns. These changes are unrelated to eval test automation and likely a side effect of the gorm-adapter's AutoMigrate being captured in a schema dump. The original migration explicitly defined ptype VARCHAR(100) NOT NULL. Removing NOT NULL on ptype weakens constraints on the authorization table. Additionally, the new unique index on (ptype, v0, v1, v2, v3, v4, v5) is largely ineffective because PostgreSQL treats NULLs as distinct in B-tree unique indexes — since v0–v5 are commonly NULL, duplicate Casbin rules won't be prevented in most cases.

Additional Locations (1)

• db/structure.sql#L1199-L1200

 

[shiroyasha]

@re-pixel to bootstrap, I should just run make agent.evals.bootstrap, or I have to do something else as well?

[re-pixel]

@re-pixel to bootstrap, I should just run make agent.evals.bootstrap, or I have to do something else as well?

you have to have the app running, run the make agent.evals.bootstrap, paste variables it printed to .env and you are set to run evals. this is stated more clearly now in the README