Skip to content

cleanup-acr-images-official pipeline is broken #2056

@lbussell

Description

@lbussell

The cleanup-acr-images-official pipeline has been broken for some time. This issue should be closed only once the pipeline runs successfully.

Current failure modes:

1. Ghost manifest crashes GenerateEolAnnotationDataForAllImagesCommand - #2055.
GetAllManifestPropertiesAsync lists a manifest in build-staging/2881074/dotnet/nightly/aspnet that no longer exists - GetManifestAsync returns 404 every time. Because the exception is unhandled, it crashes the entire Annotations job. This has been blocking the cleanup pipeline from completing at all.

2. oras CLI is not authenticated in the Clean job - #2052.
The pruneEol action calls IsDigestAnnotatedForEol, which shells out to oras discover. The oras CLI has no credentials in the Clean job - every call returns 401 Unauthorized.

3. Duplicate annotations keep staging repos perpetually fresh.
#2045 caused ImageBuilder to think that no image digests had EOL lifecycle annotations. This causes ImageBuilder to attach lots of duplicate annotations (See dotnet/dotnet-docker#7121). Furthermore, staging image repos are pruned based on their LastUpdatedOn date. Months of duplicate annotation writes have updated LastUpdatedOn on staging repos, so the Delete build-staging/* step's 15-day age check never passes. 1,840 staging repos have accumulated. This also inflates the publish pipeline's generateEolAnnotationDataForPublish to 78 minutes and 5M log lines, as it enumerates and checks all the accumulated digests and their bloated referrer lists.

List of related issues/PRs:

Metadata

Metadata

Assignees

Type

Projects

Status

In Progress

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions