Support parallel step execution for independent crew members

[landigf]

Problem

In a crew pipeline, some reviewers are independent — they all read the same paper and produce independent reviews. Currently they run sequentially, which means an 8-reviewer crew takes 8x the time of a single reviewer.

For the IMC paper crew, reviewers 1-6 are independent (they each read the paper and write a review). Only reviewers 7-8 (meta-reviewer, revision-planner) depend on prior outputs. Running 1-6 in parallel would cut wall-clock time from ~60 minutes to ~10 minutes.

Suggested design

steps:
  - id: independent-reviews
    parallel:
      - id: security-expert
        prompt: "Review for security..."
      - id: methodology-expert
        prompt: "Review for methodology..."
      - id: related-work-expert
        prompt: "Review for related work..."

  - id: meta-review
    prompt: "Synthesize: {security-expert} {methodology-expert} {related-work-expert}"

Or in TypeScript:

const crew = pipeline("review")
  .parallel([
    step("security").prompt("..."),
    step("methodology").prompt("..."),
    step("related-work").prompt("..."),
  ])
  .step(step("meta").prompt("Synthesize: {security} {methodology} {related-work}"));

Constraints

• Each parallel step needs its own Claude CLI process (the CLI is single-session)
• With Claude Max subscription, parallel processes share the same rate limit
• Need to handle partial failures (what if 2/6 succeed and 4 timeout?)

Impact

For crew pipelines, this is the difference between "run overnight" and "run during lunch." Critical for iteration speed when using crews for paper improvement.

Labels

enhancement, crew, performance