Add support for syncing BUILD_VERSION=3 images

[ShelbyZ]

Summary

Add additional sync task steps for BUILD_VERSION=3 with failure handling to allow syncing newer images when available in ecr/dockerhub. Added some sync skips related to missing images in dockerhub/ecr to avoid failures and prevent stable syncing on BUILD_VERSION=3.

Testing

Local pipeline testing of sync task to ensure using the new BUILD_VERSION=3 with nonexistent images in ecr/dockerhub should not fail

Description for the changelog

Enhancement: Add support for syncing BUILD_VERSION=3 images

Licensing

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[mye956]

Just curious, do we want to eventually return/exit with a non-zero code here?

[ShelbyZ]

I think post launch we may revisit these workarounds and either remove entirely or error

[prateekchaudhry]

I think the way this script evaluates latest and is plugged in publish.sh, this will never actually sync?

Is that right? Looks like the get_latest_dockerhub_version.py would always assign latest to numerically highest 🤔

I wonder how it affects which version we end up syncing with respect to AWS_FOR_FLUENT_BIT_VERSION_PUBLIC_ECR var. Maybe we need to re-write some logic here?

[ShelbyZ]

I think the way this script evaluates latest and is plugged in publish.sh, this will never actually sync?

Is that right? Looks like the get_latest_dockerhub_version.py would always assign latest to numerically highest 🤔

I wonder how it affects which version we end up syncing with respect to AWS_FOR_FLUENT_BIT_VERSION_PUBLIC_ECR var. Maybe we need to re-write some logic here?

Tested offline, lines 1-57 of publish.sh, and it should not be an issue:

--- BUILD VERSION 2 ---
=== BUILD_VERSION=2 Test Results ===
BUILD_VERSION=2
AWS_FOR_FLUENT_BIT_VERSION=2.34.0
AWS_FOR_FLUENT_BIT_STABLE_VERSION=2.33.2
PUBLISH_LATEST=true
AWS_FOR_FLUENT_BIT_VERSION_PUBLIC_ECR=2.34.0

--- BUILD VERSION 3 ---
=== BUILD_VERSION=3 Test Results ===
BUILD_VERSION=3
AWS_FOR_FLUENT_BIT_VERSION=3.0.0
AWS_FOR_FLUENT_BIT_STABLE_VERSION=2.33.2
PUBLISH_LATEST=false
AWS_FOR_FLUENT_BIT_VERSION_PUBLIC_ECR=2.34.0
PUBLISH_NON_LATEST=false

--- BUILD VERSION UNSET ---
=== BUILD_VERSION=unset Test Results ===
BUILD_VERSION=unset
AWS_FOR_FLUENT_BIT_VERSION=2.34.0
AWS_FOR_FLUENT_BIT_STABLE_VERSION=2.33.2
PUBLISH_LATEST=true
AWS_FOR_FLUENT_BIT_VERSION_PUBLIC_ECR=2.34.0

PUBLISH_NON_LATEST=false for BUILD_VERSION=3 because the version did not exist in dockerhub which makes sense for now (3.0.0) does not exist when it is published that value would be true and would also be AWS_FOR_FLUENT_BIT_VERSION_PUBLIC_ECR 3.0.0.

I did run an example with an older version for BUILD_VERSION=3

AWS_FOR_FLUENT_BIT_VERSION=2.31.12.20231011
AWS_FOR_FLUENT_BIT_STABLE_VERSION=2.33.2
PUBLISH_LATEST=false
AWS_FOR_FLUENT_BIT_VERSION_PUBLIC_ECR=2.31.12.20231011
PUBLISH_NON_LATEST=true