Skip to content

Docker push to Vertex not compatible with newer Docker clients #59

Open
Open
Docker push to Vertex not compatible with newer Docker clients#59
@kyleneblett

Description

@kyleneblett
kyleneblett
opened on Aug 29, 2025

I am trying to run an experiment on Vertex, but the Docker push to Google Cloud registry fails due to unexpected formatting of the Docker client logs. The push includes a log that looks like this:

...
{"status":"20250828-133109-717399: digest: sha256:19b21afe2d41c616fd5b218648e8fb440498e72eb2b084027b3226cb7e6993c9 size: 9746"}

Which causes a Runtime error that looks like this:

RuntimeError: Expected docker push to return a string with `status: Pushed` and a Digest. This is probably a temporary issue with --xm_build_image_locally and you should try again

I believe the issue is that "digest" is not capitalized in my client logs.

I am using MacOS with Docker version 28.2.2, build e6534b4eb7
I have seen similar issues with Docker version 27.x on Ubuntu.

This is a potential fix that solved the issue for me:

diff --git a/xmanager/cloud/docker_lib.py b/xmanager/cloud/docker_lib.py
index 01c5600..adecf49 100644
--- a/xmanager/cloud/docker_lib.py
+++ b/xmanager/cloud/docker_lib.py
@@ -124,7 +124,7 @@ def push_docker_image(image: str) -> str:
   repository, tag = docker_utils.parse_repository_tag(image)
   push = docker_client.images.push(repository=repository, tag=tag)
   logging.info(push)
-  if not isinstance(push, str) or '"Digest":' not in push:
+  if not isinstance(push, str) or '"digest":' not in push.lower():
     raise RuntimeError(
         'Expected docker push to return a string with `status: Pushed` and a '
         'Digest. This is probably a temporary issue with '

I will share details of a repro using the Dopamine example below.

I run the following command. Note that the gin file is at a slightly different path than the one in the documentation, because it has since been moved in the main branch.

GOOGLE_CLOUD_BUCKET_NAME="<bucket_name>" xmanager launch examples/dopamine/launcher.py -- --gin_file=https://raw.githubusercontent.com/google/dopamine/refs/heads/master/dopamine/tf/agents/dqn/configs/dqn_mountaincar.gin --xm_build_image_locally=false

Here are my output logs. I have hidden some of my Google Cloud resource names:
(.venv) kyle@Kyles-MacBook-Pro xmanager_deepmind % GOOGLE_CLOUD_BUCKET_NAME="<>" xmanager launch examples/dopamine/launcher.py -- --gin_file=https://raw.githubusercontent.com/google/dopamine/refs/heads/master/dopamine/tf/agents/dqn/configs/dqn_mountaincar.gin --xm_build_image_locally=false
I0828 13:31:09.473802 8456036544 migration.py:155] Context impl SQLiteImpl.
I0828 13:31:09.473908 8456036544 migration.py:158] Will assume non-transactional DDL.
I0828 13:31:09.474538 8456036544 migration.py:155] Context impl SQLiteImpl.
I0828 13:31:09.474561 8456036544 migration.py:158] Will assume non-transactional DDL.
I0828 13:31:09.496369 8456036544 docker_lib.py:77] Local docker: {'Platform': {'Name': 'Docker Desktop 4.41.2 (191736)'}, 'Components': [{'Name': 'Engine', 'Version': '28.1.1', 'Details': {'ApiVersion': '1.49', 'Arch': 'arm64', 'BuildTime': '2025-04-18T09:52:08.000000000+00:00', 'Experimental': 'false', 'GitCommit': '01f442b', 'GoVersion': 'go1.23.8', 'KernelVersion': '6.10.14-linuxkit', 'MinAPIVersion': '1.24', 'Os': 'linux'}}, {'Name': 'containerd', 'Version': '1.7.27', 'Details': {'GitCommit': '05044ec0a9a75232cad458027ca83437aae3f4da'}}, {'Name': 'runc', 'Version': '1.2.5', 'Details': {'GitCommit': 'v1.2.5-0-g59923ef'}}, {'Name': 'docker-init', 'Version': '0.19.0', 'Details': {'GitCommit': 'de40ad0'}}], 'Version': '28.1.1', 'ApiVersion': '1.49', 'MinAPIVersion': '1.24', 'GitCommit': '01f442b', 'GoVersion': 'go1.23.8', 'Os': 'linux', 'Arch': 'arm64', 'KernelVersion': '6.10.14-linuxkit', 'BuildTime': '2025-04-18T09:52:08.000000000+00:00'}
Dockerfile:

FROM gcr.io/deeplearning-platform-release/base-cu110

RUN if ! id 1000; then useradd -m -u 1000 clouduser; fi

RUN apt update && apt install -y python3-opencv
RUN pip install dopamine-rl
COPY dopamine/ workdir
WORKDIR workdir
RUN wget -O ./dqn_mountaincar.gin https://raw.githubusercontent.com/google/dopamine/refs/heads/master/dopamine/tf/agents/dqn/configs/dqn_mountaincar.gin

COPY entrypoint.sh ./entrypoint.sh
RUN chown -R 1000:root ./entrypoint.sh && chmod -R 775 ./entrypoint.sh

ENTRYPOINT ["./entrypoint.sh", "--gin_files=dqn_mountaincar.gin"]

Size of Docker input: 7.9 kB
Building Docker image, please wait...
Cloud Build link: https://console.cloud.google.com/cloud-build/builds/2d3b9bb0-f312-4b0d-87c7-0c9e6af3039c?project=<>
Cloud Build status: QUEUED
Cloud Build status: WORKING
Cloud Build status: WORKING
Cloud Build status: WORKING
Cloud Build status: WORKING
Cloud Build status: WORKING
Cloud Build status: WORKING
Cloud Build status: WORKING
Cloud Build status: WORKING
Cloud Build status: WORKING
Cloud Build status: WORKING
Cloud Build status: WORKING
Cloud Build status: WORKING
Cloud Build status: WORKING
Cloud Build status: WORKING
Cloud Build status: WORKING
Cloud Build status: SUCCESS
Your image URI is: gcr.io/<>/dopamine:20250828-133109-717399@sha256:19b21afe2d41c616fd5b218648e8fb440498e72eb2b084027b3226cb7e6993c9
You can run your image locally via:
docker run gcr.io/<>/dopamine:20250828-133109-717399@sha256:19b21afe2d41c616fd5b218648e8fb440498e72eb2b084027b3226cb7e6993c9

I0828 14:36:14.126254 8456036544 docker_lib.py:77] Local docker: {'Platform': {'Name': 'Docker Desktop 4.41.2 (191736)'}, 'Components': [{'Name': 'Engine', 'Version': '28.1.1', 'Details': {'ApiVersion': '1.49', 'Arch': 'arm64', 'BuildTime': '2025-04-18T09:52:08.000000000+00:00', 'Experimental': 'false', 'GitCommit': '01f442b', 'GoVersion': 'go1.23.8', 'KernelVersion': '6.10.14-linuxkit', 'MinAPIVersion': '1.24', 'Os': 'linux'}}, {'Name': 'containerd', 'Version': '1.7.27', 'Details': {'GitCommit': '05044ec0a9a75232cad458027ca83437aae3f4da'}}, {'Name': 'runc', 'Version': '1.2.5', 'Details': {'GitCommit': 'v1.2.5-0-g59923ef'}}, {'Name': 'docker-init', 'Version': '0.19.0', 'Details': {'GitCommit': 'de40ad0'}}], 'Version': '28.1.1', 'ApiVersion': '1.49', 'MinAPIVersion': '1.24', 'GitCommit': '01f442b', 'GoVersion': 'go1.23.8', 'Os': 'linux', 'Arch': 'arm64', 'KernelVersion': '6.10.14-linuxkit', 'BuildTime': '2025-04-18T09:52:08.000000000+00:00'}
I0828 14:36:16.032333 8456036544 docker_lib.py:126] {"status":"The push refers to repository [gcr.io/<>/dopamine]"}
{"status":"Waiting","progressDetail":{},"id":"aa99269ca9c6"}
{"status":"Waiting","progressDetail":{},"id":"6aa50616693c"}
{"status":"Waiting","progressDetail":{},"id":"9b20cf9ab870"}
{"status":"Waiting","progressDetail":{},"id":"86484995220a"}
{"status":"Waiting","progressDetail":{},"id":"7be49a306f97"}
{"status":"Waiting","progressDetail":{},"id":"0c0c98653f69"}
{"status":"Waiting","progressDetail":{},"id":"b4c054bf4c8a"}
{"status":"Waiting","progressDetail":{},"id":"01f811674e02"}
{"status":"Waiting","progressDetail":{},"id":"e43c2058e496"}
{"status":"Waiting","progressDetail":{},"id":"806e39781bfe"}
{"status":"Waiting","progressDetail":{},"id":"5dcd803c1072"}
{"status":"Waiting","progressDetail":{},"id":"c81ffb54a680"}
{"status":"Waiting","progressDetail":{},"id":"80822cb9dc80"}
{"status":"Waiting","progressDetail":{},"id":"653af9c01b1c"}
{"status":"Waiting","progressDetail":{},"id":"655ed0df26cf"}
{"status":"Waiting","progressDetail":{},"id":"0ddefd2f3721"}
{"status":"Waiting","progressDetail":{},"id":"ab4b13d9633a"}
{"status":"Waiting","progressDetail":{},"id":"2ee807e9c8c6"}
{"status":"Waiting","progressDetail":{},"id":"96d54c3075c9"}
{"status":"Waiting","progressDetail":{},"id":"6d96052ed287"}
{"status":"Waiting","progressDetail":{},"id":"1731132f4b49"}
{"status":"Waiting","progressDetail":{},"id":"51c57cc69f67"}
{"status":"Waiting","progressDetail":{},"id":"848b95ad96b5"}
{"status":"Waiting","progressDetail":{},"id":"7198f93013a3"}
{"status":"Waiting","progressDetail":{},"id":"1944daf745af"}
{"status":"Waiting","progressDetail":{},"id":"b765012fa1f2"}
{"status":"Waiting","progressDetail":{},"id":"ad18d32cea80"}
{"status":"Waiting","progressDetail":{},"id":"f3c08fdd0d2b"}
{"status":"Waiting","progressDetail":{},"id":"de02cd42e516"}
{"status":"Waiting","progressDetail":{},"id":"70b5915c5a02"}
{"status":"Waiting","progressDetail":{},"id":"ab3d2e901fa4"}
{"status":"Waiting","progressDetail":{},"id":"f245f7c58813"}
{"status":"Waiting","progressDetail":{},"id":"d9084fa20108"}
{"status":"Waiting","progressDetail":{},"id":"59f6381879f6"}
{"status":"Waiting","progressDetail":{},"id":"a740c1dae985"}
{"status":"Waiting","progressDetail":{},"id":"4f4fb700ef54"}
{"status":"Waiting","progressDetail":{},"id":"5ae971429117"}
{"status":"Waiting","progressDetail":{},"id":"daf967c88e2a"}
{"status":"Waiting","progressDetail":{},"id":"57c57e84aec2"}
{"status":"Waiting","progressDetail":{},"id":"fe08c22d81fb"}
{"status":"Waiting","progressDetail":{},"id":"8d7c468eff18"}
{"status":"Waiting","progressDetail":{},"id":"b7cc51e7527f"}
{"status":"Waiting","progressDetail":{},"id":"c5cc5e507c13"}
{"status":"Waiting","progressDetail":{},"id":"db34eb92afe0"}
{"status":"Waiting","progressDetail":{},"id":"80822cb9dc80"}
{"status":"Waiting","progressDetail":{},"id":"653af9c01b1c"}
{"status":"Waiting","progressDetail":{},"id":"806e39781bfe"}
{"status":"Waiting","progressDetail":{},"id":"5dcd803c1072"}
{"status":"Waiting","progressDetail":{},"id":"c81ffb54a680"}
{"status":"Waiting","progressDetail":{},"id":"2ee807e9c8c6"}
{"status":"Waiting","progressDetail":{},"id":"96d54c3075c9"}
{"status":"Waiting","progressDetail":{},"id":"655ed0df26cf"}
{"status":"Waiting","progressDetail":{},"id":"0ddefd2f3721"}
{"status":"Waiting","progressDetail":{},"id":"ab4b13d9633a"}
{"status":"Waiting","progressDetail":{},"id":"848b95ad96b5"}
{"status":"Waiting","progressDetail":{},"id":"7198f93013a3"}
{"status":"Waiting","progressDetail":{},"id":"1944daf745af"}
{"status":"Waiting","progressDetail":{},"id":"b765012fa1f2"}
{"status":"Waiting","progressDetail":{},"id":"ad18d32cea80"}
{"status":"Waiting","progressDetail":{},"id":"6d96052ed287"}
{"status":"Waiting","progressDetail":{},"id":"1731132f4b49"}
{"status":"Waiting","progressDetail":{},"id":"51c57cc69f67"}
{"status":"Waiting","progressDetail":{},"id":"f3c08fdd0d2b"}
{"status":"Waiting","progressDetail":{},"id":"f245f7c58813"}
{"status":"Waiting","progressDetail":{},"id":"de02cd42e516"}
{"status":"Waiting","progressDetail":{},"id":"70b5915c5a02"}
{"status":"Waiting","progressDetail":{},"id":"ab3d2e901fa4"}
{"status":"Waiting","progressDetail":{},"id":"4f4fb700ef54"}
{"status":"Waiting","progressDetail":{},"id":"5ae971429117"}
{"status":"Waiting","progressDetail":{},"id":"d9084fa20108"}
{"status":"Waiting","progressDetail":{},"id":"59f6381879f6"}
{"status":"Waiting","progressDetail":{},"id":"a740c1dae985"}
{"status":"Waiting","progressDetail":{},"id":"8d7c468eff18"}
{"status":"Waiting","progressDetail":{},"id":"b7cc51e7527f"}
{"status":"Waiting","progressDetail":{},"id":"c5cc5e507c13"}
{"status":"Waiting","progressDetail":{},"id":"db34eb92afe0"}
{"status":"Waiting","progressDetail":{},"id":"daf967c88e2a"}
{"status":"Waiting","progressDetail":{},"id":"57c57e84aec2"}
{"status":"Waiting","progressDetail":{},"id":"fe08c22d81fb"}
{"status":"Waiting","progressDetail":{},"id":"86484995220a"}
{"status":"Waiting","progressDetail":{},"id":"7be49a306f97"}
{"status":"Waiting","progressDetail":{},"id":"0c0c98653f69"}
{"status":"Waiting","progressDetail":{},"id":"b4c054bf4c8a"}
{"status":"Waiting","progressDetail":{},"id":"aa99269ca9c6"}
{"status":"Waiting","progressDetail":{},"id":"6aa50616693c"}
{"status":"Waiting","progressDetail":{},"id":"9b20cf9ab870"}
{"status":"Waiting","progressDetail":{},"id":"e43c2058e496"}
{"status":"Waiting","progressDetail":{},"id":"01f811674e02"}
{"status":"Waiting","progressDetail":{},"id":"b7cc51e7527f"}
{"status":"Waiting","progressDetail":{},"id":"c5cc5e507c13"}
{"status":"Waiting","progressDetail":{},"id":"db34eb92afe0"}
{"status":"Waiting","progressDetail":{},"id":"daf967c88e2a"}
{"status":"Waiting","progressDetail":{},"id":"57c57e84aec2"}
{"status":"Waiting","progressDetail":{},"id":"fe08c22d81fb"}
{"status":"Waiting","progressDetail":{},"id":"8d7c468eff18"}
{"status":"Waiting","progressDetail":{},"id":"7be49a306f97"}
{"status":"Waiting","progressDetail":{},"id":"0c0c98653f69"}
{"status":"Waiting","progressDetail":{},"id":"b4c054bf4c8a"}
{"status":"Waiting","progressDetail":{},"id":"aa99269ca9c6"}
{"status":"Waiting","progressDetail":{},"id":"6aa50616693c"}
{"status":"Waiting","progressDetail":{},"id":"9b20cf9ab870"}
{"status":"Waiting","progressDetail":{},"id":"86484995220a"}
{"status":"Waiting","progressDetail":{},"id":"e43c2058e496"}
{"status":"Waiting","progressDetail":{},"id":"01f811674e02"}
{"status":"Waiting","progressDetail":{},"id":"653af9c01b1c"}
{"status":"Waiting","progressDetail":{},"id":"806e39781bfe"}
{"status":"Waiting","progressDetail":{},"id":"5dcd803c1072"}
{"status":"Waiting","progressDetail":{},"id":"c81ffb54a680"}
{"status":"Waiting","progressDetail":{},"id":"80822cb9dc80"}
{"status":"Waiting","progressDetail":{},"id":"96d54c3075c9"}
{"status":"Waiting","progressDetail":{},"id":"655ed0df26cf"}
{"status":"Waiting","progressDetail":{},"id":"0ddefd2f3721"}
{"status":"Waiting","progressDetail":{},"id":"ab4b13d9633a"}
{"status":"Waiting","progressDetail":{},"id":"2ee807e9c8c6"}
{"status":"Waiting","progressDetail":{},"id":"7198f93013a3"}
{"status":"Waiting","progressDetail":{},"id":"1944daf745af"}
{"status":"Waiting","progressDetail":{},"id":"b765012fa1f2"}
{"status":"Waiting","progressDetail":{},"id":"ad18d32cea80"}
{"status":"Waiting","progressDetail":{},"id":"6d96052ed287"}
{"status":"Waiting","progressDetail":{},"id":"1731132f4b49"}
{"status":"Waiting","progressDetail":{},"id":"51c57cc69f67"}
{"status":"Waiting","progressDetail":{},"id":"848b95ad96b5"}
{"status":"Waiting","progressDetail":{},"id":"f3c08fdd0d2b"}
{"status":"Waiting","progressDetail":{},"id":"f245f7c58813"}
{"status":"Waiting","progressDetail":{},"id":"de02cd42e516"}
{"status":"Waiting","progressDetail":{},"id":"70b5915c5a02"}
{"status":"Waiting","progressDetail":{},"id":"ab3d2e901fa4"}
{"status":"Waiting","progressDetail":{},"id":"5ae971429117"}
{"status":"Waiting","progressDetail":{},"id":"d9084fa20108"}
{"status":"Waiting","progressDetail":{},"id":"59f6381879f6"}
{"status":"Waiting","progressDetail":{},"id":"a740c1dae985"}
{"status":"Waiting","progressDetail":{},"id":"4f4fb700ef54"}
{"status":"Waiting","progressDetail":{},"id":"b765012fa1f2"}
{"status":"Waiting","progressDetail":{},"id":"ad18d32cea80"}
{"status":"Waiting","progressDetail":{},"id":"6d96052ed287"}
{"status":"Waiting","progressDetail":{},"id":"1731132f4b49"}
{"status":"Waiting","progressDetail":{},"id":"51c57cc69f67"}
{"status":"Waiting","progressDetail":{},"id":"848b95ad96b5"}
{"status":"Waiting","progressDetail":{},"id":"7198f93013a3"}
{"status":"Waiting","progressDetail":{},"id":"1944daf745af"}
{"status":"Waiting","progressDetail":{},"id":"f3c08fdd0d2b"}
{"status":"Waiting","progressDetail":{},"id":"f245f7c58813"}
{"status":"Waiting","progressDetail":{},"id":"de02cd42e516"}
{"status":"Waiting","progressDetail":{},"id":"70b5915c5a02"}
{"status":"Waiting","progressDetail":{},"id":"ab3d2e901fa4"}
{"status":"Waiting","progressDetail":{},"id":"5ae971429117"}
{"status":"Waiting","progressDetail":{},"id":"d9084fa20108"}
{"status":"Waiting","progressDetail":{},"id":"59f6381879f6"}
{"status":"Waiting","progressDetail":{},"id":"a740c1dae985"}
{"status":"Waiting","progressDetail":{},"id":"4f4fb700ef54"}
{"status":"Waiting","progressDetail":{},"id":"db34eb92afe0"}
{"status":"Waiting","progressDetail":{},"id":"daf967c88e2a"}
{"status":"Waiting","progressDetail":{},"id":"57c57e84aec2"}
{"status":"Waiting","progressDetail":{},"id":"fe08c22d81fb"}
{"status":"Waiting","progressDetail":{},"id":"8d7c468eff18"}
{"status":"Waiting","progressDetail":{},"id":"b7cc51e7527f"}
{"status":"Waiting","progressDetail":{},"id":"c5cc5e507c13"}
{"status":"Waiting","progressDetail":{},"id":"b4c054bf4c8a"}
{"status":"Waiting","progressDetail":{},"id":"aa99269ca9c6"}
{"status":"Waiting","progressDetail":{},"id":"6aa50616693c"}
{"status":"Waiting","progressDetail":{},"id":"9b20cf9ab870"}
{"status":"Waiting","progressDetail":{},"id":"86484995220a"}
{"status":"Waiting","progressDetail":{},"id":"7be49a306f97"}
{"status":"Waiting","progressDetail":{},"id":"0c0c98653f69"}
{"status":"Waiting","progressDetail":{},"id":"e43c2058e496"}
{"status":"Waiting","progressDetail":{},"id":"01f811674e02"}
{"status":"Waiting","progressDetail":{},"id":"653af9c01b1c"}
{"status":"Waiting","progressDetail":{},"id":"806e39781bfe"}
{"status":"Waiting","progressDetail":{},"id":"5dcd803c1072"}
{"status":"Waiting","progressDetail":{},"id":"c81ffb54a680"}
{"status":"Waiting","progressDetail":{},"id":"80822cb9dc80"}
{"status":"Waiting","progressDetail":{},"id":"96d54c3075c9"}
{"status":"Waiting","progressDetail":{},"id":"655ed0df26cf"}
{"status":"Waiting","progressDetail":{},"id":"0ddefd2f3721"}
{"status":"Waiting","progressDetail":{},"id":"ab4b13d9633a"}
{"status":"Waiting","progressDetail":{},"id":"2ee807e9c8c6"}
{"status":"Waiting","progressDetail":{},"id":"96d54c3075c9"}
{"status":"Waiting","progressDetail":{},"id":"655ed0df26cf"}
{"status":"Waiting","progressDetail":{},"id":"0ddefd2f3721"}
{"status":"Waiting","progressDetail":{},"id":"ab4b13d9633a"}
{"status":"Waiting","progressDetail":{},"id":"2ee807e9c8c6"}
{"status":"Waiting","progressDetail":{},"id":"b765012fa1f2"}
{"status":"Waiting","progressDetail":{},"id":"ad18d32cea80"}
{"status":"Waiting","progressDetail":{},"id":"6d96052ed287"}
{"status":"Waiting","progressDetail":{},"id":"1731132f4b49"}
{"status":"Waiting","progressDetail":{},"id":"51c57cc69f67"}
{"status":"Waiting","progressDetail":{},"id":"848b95ad96b5"}
{"status":"Waiting","progressDetail":{},"id":"7198f93013a3"}
{"status":"Waiting","progressDetail":{},"id":"1944daf745af"}
{"status":"Waiting","progressDetail":{},"id":"f3c08fdd0d2b"}
{"status":"Waiting","progressDetail":{},"id":"f245f7c58813"}
{"status":"Waiting","progressDetail":{},"id":"de02cd42e516"}
{"status":"Waiting","progressDetail":{},"id":"70b5915c5a02"}
{"status":"Waiting","progressDetail":{},"id":"ab3d2e901fa4"}
{"status":"Waiting","progressDetail":{},"id":"5ae971429117"}
{"status":"Waiting","progressDetail":{},"id":"d9084fa20108"}
{"status":"Waiting","progressDetail":{},"id":"59f6381879f6"}
{"status":"Waiting","progressDetail":{},"id":"a740c1dae985"}
{"status":"Waiting","progressDetail":{},"id":"4f4fb700ef54"}
{"status":"Waiting","progressDetail":{},"id":"db34eb92afe0"}
{"status":"Waiting","progressDetail":{},"id":"daf967c88e2a"}
{"status":"Waiting","progressDetail":{},"id":"57c57e84aec2"}
{"status":"Waiting","progressDetail":{},"id":"fe08c22d81fb"}
{"status":"Waiting","progressDetail":{},"id":"8d7c468eff18"}
{"status":"Waiting","progressDetail":{},"id":"b7cc51e7527f"}
{"status":"Waiting","progressDetail":{},"id":"c5cc5e507c13"}
{"status":"Waiting","progressDetail":{},"id":"b4c054bf4c8a"}
{"status":"Waiting","progressDetail":{},"id":"aa99269ca9c6"}
{"status":"Waiting","progressDetail":{},"id":"6aa50616693c"}
{"status":"Waiting","progressDetail":{},"id":"9b20cf9ab870"}
{"status":"Waiting","progressDetail":{},"id":"86484995220a"}
{"status":"Waiting","progressDetail":{},"id":"7be49a306f97"}
{"status":"Waiting","progressDetail":{},"id":"0c0c98653f69"}
{"status":"Waiting","progressDetail":{},"id":"e43c2058e496"}
{"status":"Waiting","progressDetail":{},"id":"01f811674e02"}
{"status":"Waiting","progressDetail":{},"id":"653af9c01b1c"}
{"status":"Waiting","progressDetail":{},"id":"806e39781bfe"}
{"status":"Waiting","progressDetail":{},"id":"5dcd803c1072"}
{"status":"Waiting","progressDetail":{},"id":"c81ffb54a680"}
{"status":"Waiting","progressDetail":{},"id":"80822cb9dc80"}
{"status":"Waiting","progressDetail":{},"id":"b7cc51e7527f"}
{"status":"Waiting","progressDetail":{},"id":"c5cc5e507c13"}
{"status":"Waiting","progressDetail":{},"id":"db34eb92afe0"}
{"status":"Waiting","progressDetail":{},"id":"daf967c88e2a"}
{"status":"Waiting","progressDetail":{},"id":"57c57e84aec2"}
{"status":"Waiting","progressDetail":{},"id":"fe08c22d81fb"}
{"status":"Waiting","progressDetail":{},"id":"8d7c468eff18"}
{"status":"Waiting","progressDetail":{},"id":"7be49a306f97"}
{"status":"Waiting","progressDetail":{},"id":"0c0c98653f69"}
{"status":"Waiting","progressDetail":{},"id":"b4c054bf4c8a"}
{"status":"Waiting","progressDetail":{},"id":"aa99269ca9c6"}
{"status":"Waiting","progressDetail":{},"id":"6aa50616693c"}
{"status":"Waiting","progressDetail":{},"id":"9b20cf9ab870"}
{"status":"Waiting","progressDetail":{},"id":"86484995220a"}
{"status":"Waiting","progressDetail":{},"id":"e43c2058e496"}
{"status":"Waiting","progressDetail":{},"id":"01f811674e02"}
{"status":"Waiting","progressDetail":{},"id":"653af9c01b1c"}
{"status":"Waiting","progressDetail":{},"id":"806e39781bfe"}
{"status":"Waiting","progressDetail":{},"id":"5dcd803c1072"}
{"status":"Waiting","progressDetail":{},"id":"c81ffb54a680"}
{"status":"Waiting","progressDetail":{},"id":"80822cb9dc80"}
{"status":"Waiting","progressDetail":{},"id":"96d54c3075c9"}
{"status":"Waiting","progressDetail":{},"id":"655ed0df26cf"}
{"status":"Waiting","progressDetail":{},"id":"0ddefd2f3721"}
{"status":"Waiting","progressDetail":{},"id":"ab4b13d9633a"}
{"status":"Waiting","progressDetail":{},"id":"2ee807e9c8c6"}
{"status":"Waiting","progressDetail":{},"id":"7198f93013a3"}
{"status":"Waiting","progressDetail":{},"id":"1944daf745af"}
{"status":"Waiting","progressDetail":{},"id":"b765012fa1f2"}
{"status":"Waiting","progressDetail":{},"id":"ad18d32cea80"}
{"status":"Waiting","progressDetail":{},"id":"6d96052ed287"}
{"status":"Waiting","progressDetail":{},"id":"1731132f4b49"}
{"status":"Waiting","progressDetail":{},"id":"51c57cc69f67"}
{"status":"Waiting","progressDetail":{},"id":"848b95ad96b5"}
{"status":"Waiting","progressDetail":{},"id":"f3c08fdd0d2b"}
{"status":"Waiting","progressDetail":{},"id":"f245f7c58813"}
{"status":"Waiting","progressDetail":{},"id":"de02cd42e516"}
{"status":"Waiting","progressDetail":{},"id":"70b5915c5a02"}
{"status":"Waiting","progressDetail":{},"id":"ab3d2e901fa4"}
{"status":"Waiting","progressDetail":{},"id":"5ae971429117"}
{"status":"Waiting","progressDetail":{},"id":"d9084fa20108"}
{"status":"Waiting","progressDetail":{},"id":"59f6381879f6"}
{"status":"Waiting","progressDetail":{},"id":"a740c1dae985"}
{"status":"Waiting","progressDetail":{},"id":"4f4fb700ef54"}
{"status":"Waiting","progressDetail":{},"id":"5ae971429117"}
{"status":"Waiting","progressDetail":{},"id":"d9084fa20108"}
{"status":"Waiting","progressDetail":{},"id":"59f6381879f6"}
{"status":"Waiting","progressDetail":{},"id":"a740c1dae985"}
{"status":"Waiting","progressDetail":{},"id":"4f4fb700ef54"}
{"status":"Waiting","progressDetail":{},"id":"b7cc51e7527f"}
{"status":"Waiting","progressDetail":{},"id":"c5cc5e507c13"}
{"status":"Waiting","progressDetail":{},"id":"db34eb92afe0"}
{"status":"Waiting","progressDetail":{},"id":"daf967c88e2a"}
{"status":"Waiting","progressDetail":{},"id":"57c57e84aec2"}
{"status":"Waiting","progressDetail":{},"id":"fe08c22d81fb"}
{"status":"Waiting","progressDetail":{},"id":"8d7c468eff18"}
{"status":"Waiting","progressDetail":{},"id":"7be49a306f97"}
{"status":"Waiting","progressDetail":{},"id":"0c0c98653f69"}
{"status":"Waiting","progressDetail":{},"id":"b4c054bf4c8a"}
{"status":"Waiting","progressDetail":{},"id":"aa99269ca9c6"}
{"status":"Waiting","progressDetail":{},"id":"6aa50616693c"}
{"status":"Waiting","progressDetail":{},"id":"9b20cf9ab870"}
{"status":"Waiting","progressDetail":{},"id":"86484995220a"}
{"status":"Waiting","progressDetail":{},"id":"e43c2058e496"}
{"status":"Waiting","progressDetail":{},"id":"01f811674e02"}
{"status":"Waiting","progressDetail":{},"id":"653af9c01b1c"}
{"status":"Waiting","progressDetail":{},"id":"806e39781bfe"}
{"status":"Waiting","progressDetail":{},"id":"5dcd803c1072"}
{"status":"Waiting","progressDetail":{},"id":"c81ffb54a680"}
{"status":"Waiting","progressDetail":{},"id":"80822cb9dc80"}
{"status":"Waiting","progressDetail":{},"id":"96d54c3075c9"}
{"status":"Waiting","progressDetail":{},"id":"655ed0df26cf"}
{"status":"Waiting","progressDetail":{},"id":"0ddefd2f3721"}
{"status":"Waiting","progressDetail":{},"id":"ab4b13d9633a"}
{"status":"Waiting","progressDetail":{},"id":"2ee807e9c8c6"}
{"status":"Waiting","progressDetail":{},"id":"7198f93013a3"}
{"status":"Waiting","progressDetail":{},"id":"1944daf745af"}
{"status":"Waiting","progressDetail":{},"id":"b765012fa1f2"}
{"status":"Waiting","progressDetail":{},"id":"ad18d32cea80"}
{"status":"Waiting","progressDetail":{},"id":"6d96052ed287"}
{"status":"Waiting","progressDetail":{},"id":"1731132f4b49"}
{"status":"Waiting","progressDetail":{},"id":"51c57cc69f67"}
{"status":"Waiting","progressDetail":{},"id":"848b95ad96b5"}
{"status":"Waiting","progressDetail":{},"id":"f3c08fdd0d2b"}
{"status":"Waiting","progressDetail":{},"id":"f245f7c58813"}
{"status":"Waiting","progressDetail":{},"id":"de02cd42e516"}
{"status":"Waiting","progressDetail":{},"id":"70b5915c5a02"}
{"status":"Waiting","progressDetail":{},"id":"ab3d2e901fa4"}
{"status":"Layer already exists","progressDetail":{},"id":"8d7c468eff18"}
{"status":"Waiting","progressDetail":{},"id":"b7cc51e7527f"}
{"status":"Waiting","progressDetail":{},"id":"c5cc5e507c13"}
{"status":"Layer already exists","progressDetail":{},"id":"db34eb92afe0"}
{"status":"Layer already exists","progressDetail":{},"id":"daf967c88e2a"}
{"status":"Layer already exists","progressDetail":{},"id":"57c57e84aec2"}
{"status":"Layer already exists","progressDetail":{},"id":"fe08c22d81fb"}
{"status":"Layer already exists","progressDetail":{},"id":"86484995220a"}
{"status":"Layer already exists","progressDetail":{},"id":"7be49a306f97"}
{"status":"Layer already exists","progressDetail":{},"id":"0c0c98653f69"}
{"status":"Layer already exists","progressDetail":{},"id":"b4c054bf4c8a"}
{"status":"Waiting","progressDetail":{},"id":"aa99269ca9c6"}
{"status":"Layer already exists","progressDetail":{},"id":"6aa50616693c"}
{"status":"Waiting","progressDetail":{},"id":"9b20cf9ab870"}
{"status":"Layer already exists","progressDetail":{},"id":"e43c2058e496"}
{"status":"Waiting","progressDetail":{},"id":"01f811674e02"}
{"status":"Layer already exists","progressDetail":{},"id":"80822cb9dc80"}
{"status":"Layer already exists","progressDetail":{},"id":"653af9c01b1c"}
{"status":"Layer already exists","progressDetail":{},"id":"806e39781bfe"}
{"status":"Layer already exists","progressDetail":{},"id":"5dcd803c1072"}
{"status":"Layer already exists","progressDetail":{},"id":"c81ffb54a680"}
{"status":"Waiting","progressDetail":{},"id":"2ee807e9c8c6"}
{"status":"Layer already exists","progressDetail":{},"id":"96d54c3075c9"}
{"status":"Waiting","progressDetail":{},"id":"655ed0df26cf"}
{"status":"Layer already exists","progressDetail":{},"id":"0ddefd2f3721"}
{"status":"Layer already exists","progressDetail":{},"id":"ab4b13d9633a"}
{"status":"Waiting","progressDetail":{},"id":"848b95ad96b5"}
{"status":"Layer already exists","progressDetail":{},"id":"7198f93013a3"}
{"status":"Waiting","progressDetail":{},"id":"1944daf745af"}
{"status":"Layer already exists","progressDetail":{},"id":"b765012fa1f2"}
{"status":"Waiting","progressDetail":{},"id":"ad18d32cea80"}
{"status":"Layer already exists","progressDetail":{},"id":"6d96052ed287"}
{"status":"Layer already exists","progressDetail":{},"id":"1731132f4b49"}
{"status":"Waiting","progressDetail":{},"id":"51c57cc69f67"}
{"status":"Layer already exists","progressDetail":{},"id":"f3c08fdd0d2b"}
{"status":"Layer already exists","progressDetail":{},"id":"f245f7c58813"}
{"status":"Waiting","progressDetail":{},"id":"de02cd42e516"}
{"status":"Layer already exists","progressDetail":{},"id":"70b5915c5a02"}
{"status":"Layer already exists","progressDetail":{},"id":"ab3d2e901fa4"}
{"status":"Layer already exists","progressDetail":{},"id":"4f4fb700ef54"}
{"status":"Waiting","progressDetail":{},"id":"5ae971429117"}
{"status":"Waiting","progressDetail":{},"id":"d9084fa20108"}
{"status":"Waiting","progressDetail":{},"id":"59f6381879f6"}
{"status":"Waiting","progressDetail":{},"id":"a740c1dae985"}
{"status":"Layer already exists","progressDetail":{},"id":"de02cd42e516"}
{"status":"Layer already exists","progressDetail":{},"id":"59f6381879f6"}
{"status":"Layer already exists","progressDetail":{},"id":"a740c1dae985"}
{"status":"Layer already exists","progressDetail":{},"id":"5ae971429117"}
{"status":"Layer already exists","progressDetail":{},"id":"d9084fa20108"}
{"status":"Layer already exists","progressDetail":{},"id":"b7cc51e7527f"}
{"status":"Layer already exists","progressDetail":{},"id":"c5cc5e507c13"}
{"status":"Layer already exists","progressDetail":{},"id":"9b20cf9ab870"}
{"status":"Layer already exists","progressDetail":{},"id":"aa99269ca9c6"}
{"status":"Layer already exists","progressDetail":{},"id":"01f811674e02"}
{"status":"Layer already exists","progressDetail":{},"id":"2ee807e9c8c6"}
{"status":"Layer already exists","progressDetail":{},"id":"655ed0df26cf"}
{"status":"Layer already exists","progressDetail":{},"id":"51c57cc69f67"}
{"status":"Layer already exists","progressDetail":{},"id":"848b95ad96b5"}
{"status":"Layer already exists","progressDetail":{},"id":"1944daf745af"}
{"status":"Layer already exists","progressDetail":{},"id":"ad18d32cea80"}
{"status":"20250828-133109-717399: digest: sha256:19b21afe2d41c616fd5b218648e8fb440498e72eb2b084027b3226cb7e6993c9 size: 9746"}

W0828 14:36:16.035428 8456036544 core.py:869] No work units were added to this experiment, which is usually not intended.
Traceback (most recent call last):
File "/Users/kyle/repos/xmanager_deepmind/.venv/bin/xmanager", line 8, in
sys.exit(entrypoint())
~~~~~~~~~~^^
File "/Users/kyle/repos/xmanager_deepmind/xmanager/cli/cli.py", line 68, in entrypoint
app.run(main)
~~~~~~~^^^^^^
File "/Users/kyle/repos/xmanager_deepmind/.venv/lib/python3.13/site-packages/absl/app.py", line 316, in run
_run_main(main, args)
~~~~~~~~~^^^^^^^^^^^^
File "/Users/kyle/repos/xmanager_deepmind/.venv/lib/python3.13/site-packages/absl/app.py", line 261, in _run_main
sys.exit(main(argv))
~~~~^^^^^^
File "/Users/kyle/repos/xmanager_deepmind/xmanager/cli/cli.py", line 43, in main
app.run(m.main, argv=argv)
~~~~~~~^^^^^^^^^^^^^^^^^^^
File "/Users/kyle/repos/xmanager_deepmind/.venv/lib/python3.13/site-packages/absl/app.py", line 316, in run
_run_main(main, args)
~~~~~~~~~^^^^^^^^^^^^
File "/Users/kyle/repos/xmanager_deepmind/.venv/lib/python3.13/site-packages/absl/app.py", line 261, in _run_main
sys.exit(main(argv))
~~~~^^^^^^
File "/Users/kyle/repos/xmanager_deepmind/examples/dopamine/launcher.py", line 55, in main
[executable] = experiment.package(
~~~~~~~~~~~~~~~~~~^
[
^
...<7 lines>...
]
^
)
^
File "/Users/kyle/repos/xmanager_deepmind/xmanager/xm/core.py", line 892, in package
return cls._async_packager.package(packageables)
~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^
File "/Users/kyle/repos/xmanager_deepmind/xmanager/xm/async_packager.py", line 134, in package
executables = self._package_batch(packageables)
File "/Users/kyle/repos/xmanager_deepmind/xmanager/xm_local/packaging/router.py", line 79, in package
_packaging_router(built_targets, packageable)
~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/kyle/repos/xmanager_deepmind/xmanager/xm_local/packaging/router.py", line 32, in _packaging_router
return cloud_packaging.package_cloud_executable(
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
built_targets,
^^^^^^^^^^^^^^
packageable,
^^^^^^^^^^^^
packageable.executable_spec,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/Users/kyle/repos/xmanager_deepmind/xmanager/xm_local/packaging/cloud.py", line 208, in package_cloud_executable
return _package_python_container(
bazel_outputs,
packageable,
python_container,
)
File "/Users/kyle/repos/xmanager_deepmind/xmanager/xm_local/packaging/cloud.py", line 151, in _package_python_container
build_image.push(image)
~~~~~~~~~~~~~~~~^^^^^^^
File "/Users/kyle/repos/xmanager_deepmind/xmanager/cloud/build_image.py", line 168, in push
return docker_lib.push_docker_image(image)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^
File "/Users/kyle/repos/xmanager_deepmind/xmanager/cloud/docker_lib.py", line 128, in push_docker_image
raise RuntimeError(
...<3 lines>...
)
RuntimeError: Expected docker push to return a string with status: Pushed and a Digest. This is probably a temporary issue with --xm_build_image_locally and you should try again
(.venv) kyle@Kyles-MacBook-Pro xmanager_deepmind %
(.venv) kyle@Kyles-MacBook-Pro xmanager_deepmind % docker --version
Docker version 28.2.2, build e6534b4eb7
(.venv) kyle@Kyles-MacBook-Pro xmanager_deepmind % git diff
(.venv) kyle@Kyles-MacBook-Pro xmanager_deepmind % GOOGLE_CLOUD_BUCKET_NAME="<>" xmanager launch examples/dopamine/launcher.py -- --gin_file=https://raw.githubusercontent.com/google/dopamine/refs/heads/master/dopamine/tf/agents/dqn/configs/dqn_mountaincar.gin --xm_build_image_locally=false

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions