[Spike] Pipeline generation: Core abstractions and GitHub Actions scheduling#15643
Draft
mitchdenny wants to merge 37 commits intomainfrom
Draft
[Spike] Pipeline generation: Core abstractions and GitHub Actions scheduling#15643mitchdenny wants to merge 37 commits intomainfrom
mitchdenny wants to merge 37 commits intomainfrom
Conversation
Contributor
|
🚀 Dogfood this PR with:
curl -fsSL https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.sh | bash -s -- 15643Or
iex "& { $(irm https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.ps1) } 15643" |
Contributor
|
Re-running the failed jobs in the CI workflow for this pull request because 1 job was identified as retry-safe transient failures in the CI run attempt.
|
Contributor
|
Re-running the failed jobs in the CI workflow for this pull request because 1 job was identified as retry-safe transient failures in the CI run attempt.
|
…uling Add core pipeline environment abstractions (IPipelineEnvironment, IPipelineStepTarget, PipelineEnvironmentCheckAnnotation) to Aspire.Hosting, along with a new Aspire.Hosting.Pipelines.GitHubActions package implementing workflow resource model and scheduling resolver. Key changes: - IPipelineEnvironment marker interface for CI/CD environments - IPipelineStepTarget for scheduling steps onto workflow jobs - PipelineStep.ScheduledBy property for step-to-job assignment - GetEnvironmentAsync() with annotation-based environment resolution - GitHubActionsWorkflowResource and GitHubActionsJob types - SchedulingResolver projecting step DAG onto job dependency graph - 29 unit tests covering environment resolution and scheduling Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Add WorkflowYaml model types (WorkflowYaml, JobYaml, StepYaml, etc.) - Add hand-rolled WorkflowYamlSerializer (no external dependencies) - Add WorkflowYamlGenerator: scheduling result → complete workflow YAML - Boilerplate steps: checkout, setup-dotnet, install Aspire CLI - State artifact upload/download between dependent jobs - Per-job aspire do --continue --job <jobId> execution - Add TryRestoreStepAsync on PipelineStep for CI/CD state restore - Executor calls restore callback before Action - If restore returns true, step is skipped (already completed) - Add 9 YAML generator unit tests - Add 4 Verify snapshot tests with complete YAML output - Add 5 step state restore integration tests - Update spec doc with YAML generation and state restore design Total: 47 tests passing (12 hosting + 35 GitHub Actions) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Adds ScheduleStep(string stepName, IPipelineStepTarget target) to IDistributedApplicationPipeline. This allows consumers to schedule built-in steps (e.g., WellKnownPipelineSteps.Build) onto CI/CD jobs without having to create them — useful when integrations register steps and the AppHost just needs to assign them to workflow jobs. - Interface and implementation in Aspire.Hosting - 7 unit tests: basic scheduling, override, built-in steps, null guards, error on missing step Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Revert IDistributedApplicationPipeline to its original signature to preserve binary compatibility. The interface method AddStep retains its original 4-parameter signature. New members (ScheduleStep, GetEnvironmentAsync, and the 5-param AddStep overload with scheduledBy) are on DistributedApplicationPipeline only. Adding an optional parameter to an interface method changes the IL signature, which is a binary breaking change even though it's source compatible. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
mitchdenny
force-pushed
the
spike/pipeline-generation
branch
from
d1185d9 to
3a360ae
Compare
…urce
- GitHubActionsJob → GitHubActionsJobResource (extends Resource)
- New GitHubActionsStageResource (extends Resource) for logical job grouping
- workflow.AddStage("build-stage") → stage.AddJob("build")
- workflow.AddJob("build") still works for direct job creation
- Jobs created via stages are also registered on the workflow
- 6 new tests for stage/resource APIs
- Total: 60 tests passing
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…jobs
- GitHubActionsWorkflowResource and GitHubActionsStageResource now
implement IPipelineStepTarget, so steps can be scheduled onto them
- When a step targets a workflow, a default stage + default job are
auto-created to host it
- When a step targets a stage, a default job is auto-created within
that stage (named '{stage}-default')
- SchedulingResolver.ResolveJobForStep uses pattern matching to
resolve workflow/stage/job targets down to concrete jobs
- GetOrAddDefaultStage() on workflow, GetOrAddDefaultJob() on stage
- DefaultJob in SchedulingResult is now nullable (not created when
all steps are explicitly scheduled)
- 6 new resolver tests covering workflow/stage scheduling targets
- Total: 66 tests passing (19 hosting + 47 GH Actions)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- PipelineCommand: group command (like 'mcp') at 'aspire pipeline' - PipelineInitCommand: subcommand extending PipelineCommandBase - Passes --operation publish --step pipeline-init to AppHost - Follows same pattern as deploy/publish commands - Added WellKnownPipelineSteps.PipelineInit constant - Registered in DI (Program.cs) and RootCommand - Resource strings (.resx + .Designer.cs) with localization xlf files - HelpGroup.Deployment for discoverability Usage: aspire pipeline init [--apphost <path>] [--output-path <path>] Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Fix DI resolution failure: 'Unable to resolve service for type PipelineCommand while attempting to activate RootCommand'. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Register a built-in 'pipeline-init' step in DistributedApplicationPipeline that discovers IPipelineEnvironment resources and calls their PipelineWorkflowGeneratorAnnotation to generate CI/CD workflow files. - Add PipelineWorkflowGeneratorAnnotation and PipelineWorkflowGenerationContext - Register generator annotation in AddGitHubActionsWorkflow extension - Pipeline-init step iterates environments and invokes generators Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Detect repository root via git rev-parse, fall back to walking up directories for aspire.config.json, and confirm with the user via IInteractionService. The resolved root is passed to generators as RepositoryRootDirectory (replaces OutputDirectory). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Move ScheduleStep from concrete class only to the interface so it is discoverable via builder.Pipeline. Add convenience AddStep extension method with scheduledBy parameter. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ActionsWorkflowResource> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ng default Unscheduled steps (ScheduledBy=null) are now resolved by BFS through reverse dependencies to find the nearest explicitly-scheduled consumer, and placed on that consumer's job. This eliminates unnecessary default stages/jobs when explicit stages exist. Fallback behavior: if no consumer is found and explicit targets exist, orphan steps go to the first available stage/job. If no explicit targets exist at all, the original default job behavior is preserved. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- PipelineScopeAnnotation: auto-detects execution scope from CI env vars
(GITHUB_RUN_ID + GITHUB_RUN_ATTEMPT for RunId, GITHUB_JOB for JobId)
- PipelineScopeMapAnnotation: maps scope IDs to step names from scheduling
- ContinuationStateManager: per-scope writes to .aspire/state/{RunId}/{JobId}.json,
merged reads from all files in RunId directory for fan-in scenarios
- Executor enters continuation mode automatically when scope is detected:
in-scope steps execute, out-of-scope steps skip (or restore from state)
- YAML generator simplified: 'aspire do' (no --continue --job flags),
artifact names use 'aspire-do-state-' prefix, paths include run ID
- 17 new tests: scope annotation, continuation state manager, scope filtering
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…steps Two issues fixed: 1. The pipeline-init step was passing only well-known steps (_steps) to the workflow generator, missing resource annotation steps that create the actual cross-job dependencies. Fixed by using _lastResolvedSteps which contains all steps after annotation collection and RequiredBy normalization. 2. The pull-based scheduling assigned orphan steps (no consumer chain to an explicit target) to the first available job, which could create spurious cross-job dependencies leading to cycles. For example, publish-prereq (orphan) went to publish-default but depended on process-parameters in deploy-default, while deploy-default already depended on publish-default. Fixed by adding Phase 2b: orphan steps are now co-located with their dependencies instead of the first available job. Uses iterative resolution to handle chains of orphan-to-orphan dependencies. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…stomization - Add WellKnownDependencyTags class (requires-dotnet, requires-nodejs, requires-docker, requires-azure-cli, requires-aspire-cli) that declare what tooling a pipeline step needs on the CI machine - Tag existing step creation sites: ProjectResource and Container build/push steps get requires-dotnet/requires-docker, Azure steps get requires-azure-cli - WorkflowYamlGenerator now scans dependency tags per job and conditionally emits setup steps (setup-dotnet, setup-node, azure/login) only when needed - Channel-aware Aspire CLI install: reads aspire.config.json for channel (stable/preview/dev/daily) and emits curl|bash with quality arg - Make WorkflowYaml/JobYaml/StepYaml/etc public with XML docs and [Experimental] attributes for developer customization - Add ConfigureWorkflow extension method on IResourceBuilder<GitHubActionsWorkflowResource> with WorkflowCustomizationAnnotation for post-generation YAML mutation - Add 14 new tests: tag collection, conditional emission, channel install, ConfigureWorkflow callback tests - Update 4 snapshot tests for new install command format Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- URL: aspire.dev/install.sh (not aka.ms/install-aspire.sh) - Quality flags: -q dev (daily channel), -q staging (staging channel) - Unknown channels (e.g. 'preview') fall back to default (stable) - Remove unused PrInstallScriptUrl constant (PR install is future work) - Add staging channel test, rename preview test to unknown channel test - Update 4 snapshots with correct URL Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Compute terminal steps per job in SchedulingResolver and emit 'aspire do <step-name>' instead of bare 'aspire do' which requires a step argument - Add pull_request trigger to generated YAML (targeting main) - Add PullRequestTrigger to YAML model and serializer - Make CLI install step PR-aware: on pull_request events, use get-aspire-cli-pr.sh with PR number; on push/manual, use channel-based install from aspire.dev/install.sh - Update 4 snapshot tests and 4 unit tests for new behavior Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Add ExistingSteps property to PipelineStepFactoryContext so annotation factories can see previously collected pipeline steps - Use two-pass step collection: non-pipeline-environment resources first, pipeline environments second (full visibility of all steps) - Register PipelineStepAnnotation on workflow resource that creates one synthetic no-op step per job (e.g., gha-deploy-default-stage-default-job) depending on the terminal steps for that job - YAML generator emits aspire do <synthetic-step-name> per job instead of chaining multiple aspire do commands with && - Remove github.event_name conditional from CLI install step; instead parse channel from aspire.config.json (pr-<N> uses PR script, daily=-q dev, staging=-q staging, else default stable install) - Include synthetic steps in scope map for continuation mode - Move scope map creation from generator callback to annotation factory so it's available during both init and aspire do Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
|
Re-running the failed jobs in the CI workflow for this pull request because 1 job was identified as retry-safe transient failures in the CI run attempt.
|
Steps like 'publish-manifest', 'diagnostics', and 'pipeline-init' are standalone CLI tools with no connection to the deployment DAG. Including them in the CI scheduler made them terminal steps in the job, causing the synthetic step to depend on them. This led to 'aspire do' executing publish-manifest which fails because --output-path is not provided in CI. Fix: FilterConnectedSteps() excludes steps with no DependsOnSteps, no RequiredBySteps, not depended upon by other steps, and no explicit scheduling. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
NuGet.config supports %HOME% environment variable expansion cross-platform. Use it for hive package source paths instead of hardcoded absolute paths so the config is portable across machines (local dev → CI runner). Also use descriptive key names (aspire-hive-pr-NNNNN) instead of the path as the packageSources key, eliminating the key mismatch bug where packageSources and packageSourceMapping keys could diverge on different machines. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
When the channel is a PR channel (pr-<number>), the generated GitHub
Actions workflow step for installing the Aspire CLI now includes
GH_TOKEN: ${{ github.token }} in its env block. This is required
for the get-aspire-cli-pr.sh script which uses the GitHub CLI (gh)
to download the PR build artifact.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Append `echo "\$HOME/.aspire/bin" >> \$GITHUB_PATH` after the curl install command in all channels. This ensures the aspire CLI is on PATH for all subsequent workflow steps in GitHub Actions. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
|
Re-running the failed jobs in the CI workflow for this pull request because 1 job was identified as retry-safe transient failures in the CI run attempt.
|
Replace the config-file-based channel detection with assembly version
introspection. The AssemblyInformationalVersionAttribute contains
"-pr.{number}" for PR builds (set by ci.yml VersionSuffix), so the
pipeline generator is now self-aware about its build origin.
- PR build (version contains -pr.NNNNN): emits PR install script + GH_TOKEN
- Prerelease build (any - suffix): emits install with -q dev quality
- Stable build (no prerelease): emits default install
Removes dependency on aspire.config.json for CLI install decisions.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- GitHelper: wraps git commands (init, remote, commit, push) - GitHubApiClient: uses gh CLI auth + REST API for repo creation - GitHubRepositoryBootstrapper: orchestrates repo detection, init, GitHub remote creation, and post-generation commit/push - GitIgnoreTemplate: baked-in .NET/Aspire .gitignore - Wire bootstrapper into GHA extension callback (runs before YAML gen) - Make RepositoryRootDirectory optional (extensions can override) - Central repo detection becomes best-effort default Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- GitHelperTests: integration tests using real git in temp directories (init, repo detection, remote management, commit) - GitIgnoreTemplateTests: verifies content contains expected patterns Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ationAsync The CLI-to-AppHost backchannel only proxies InputsInteractionInfo (from PromptInputAsync) and NotificationInteractionInfo. It does NOT handle MessageBoxInteractionInfo (from PromptConfirmationAsync), causing the AppHost to hang forever waiting for a response that never arrives. - Replace all PromptConfirmationAsync calls with PromptInputAsync using InputType.Boolean, which IS supported by the backchannel - Extract PromptBooleanAsync helper for consistent yes/no prompting - Skip redundant git detection when RepositoryRootDirectory is already set by the central pipeline - Extract InitGitRepoAsync to reduce duplication Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
|
Re-running the failed jobs in the CI workflow for this pull request because 1 job was identified as retry-safe transient failures in the CI run attempt.
|
On second run, synthetic scheduling-target steps (gha-*-job) from the
first run were included in existingSteps, passed through the filter
(they have ScheduledBy set), and got fed into SchedulingResolver
alongside real steps — corrupting the job dependency graph.
Fix: FilterConnectedSteps now excludes synthetic steps matching the
gha-{workflowName}-*-job pattern before scheduling.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…pace-relative
The YAML generator hardcoded '.aspire/state/...' (workspace-relative) for
artifact upload/download, but FileDeploymentStateManager writes state to
$HOME/.aspire/deployments/{sha}/{env}.json (global path). The upload step
found nothing because the runtime never writes to the workspace path.
Changed StatePathExpression to '${{ env.HOME }}/.aspire/deployments/'
which resolves to the actual runtime state directory in GitHub Actions.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
YamlQuote() was wrapping values containing ${{ }} in single quotes,
which causes GitHub Actions to treat them as literal strings instead
of evaluating the expression. This broke artifact upload/download
paths like ${{ env.HOME }}/.aspire/deployments/ — the path was
interpreted literally, causing uploads from the workspace root.
Now values containing ${{ are returned unquoted so the runner
evaluates the expression before passing it to the action.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The SchedulingResolver only inspects DependsOnSteps when building its reverse dependency map. Steps using RequiredBy (like docker-compose publish/deploy steps) had invisible edges, causing incorrect job assignments. When combined with explicit stage scheduling, this led to circular job dependencies (deploy-default → build-default → deploy-default). The fix normalizes RequiredBy relationships into DependsOn before calling the scheduler, matching what the pipeline does later in ExecuteAsync. This gives the scheduler full visibility of the DAG. Added test covering the docker-compose two-stage scenario. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
|
🎬 CLI E2E Test Recordings — 52 recordings uploaded (commit View recordings
📹 Recordings uploaded automatically from CI run #23741542522 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Spike/proof-of-concept for pipeline generation — the ability to generate CI/CD workflow YAML files (e.g., GitHub Actions) from the Aspire application model.
What's Included
Core Abstractions (
Aspire.Hosting)IPipelineEnvironment— Marker interface extendingIResourcefor CI/CD environmentsIPipelineStepTarget— Interface for scheduling targets (workflow jobs implement this)PipelineEnvironmentCheckAnnotation— Annotation-based environment relevance checkLocalPipelineEnvironment— Internal fallback for local executionPipelineStep.ScheduledBy— Property linking steps to workflow jobsGetEnvironmentAsync()— Resolution of active pipeline environment from modelGitHub Actions Package (
Aspire.Hosting.Pipelines.GitHubActions)GitHubActionsWorkflowResource—Resource + IPipelineEnvironmentrepresenting a workflowGitHubActionsJob— Job within a workflow, implementsIPipelineStepTargetGitHubActionsWorkflowExtensions—builder.AddGitHubActionsWorkflow(name)extensionSchedulingResolver— Projects step DAG onto job dependency graph with cycle detectionTests (29 total)
Spec Document
docs/specs/pipeline-generation.md— Full architecture, API primitives, and future roadmapExample Usage
Not Yet Implemented
aspire pipeline init)aspire do --continue --job <jobId>CLI supportRelated
See
docs/specs/pipeline-generation.mdfor full design details and open questions.