Add IaaS IaC to run NVIDIA CUDA Optimized images on GCE L4 to B100

[obriensystems]

https://www.techpowerup.com/gpu-specs/l4.c4091

ObrienlabsDev/blog#160

GPT-OSS:20b Tokens/sec for various M series and NVidia Ampere, Ada and Grace Blackwell GPUs

20260120 - ObrienlabsDev/blog#160

see 2024 analysis in - https://github.com/ObrienlabsDev/blog/wiki/CUDA-based-%E2%80%90-High-Performance-Computing-%E2%80%90-LLM-Training-%E2%80%90-Ground-to-GCP-Cloud-Hybrid#cuda-on-gcp - https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/issues/747 - https://github.com/GoogleCloudPlatform/pbmm-on-gcp-onboarding/issues/252

When running on nane1 after getting quota - the terraform marketplace deploy failed on a 2nd run with a locked statefile - I deleted the deployment and switched to the default us-east-1c

connect via RDP after resetting password

PS C:\Users\michael> nvidia-smi
Mon Nov 17 21:36:26 2025
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 581.42                 Driver Version: 581.42         CUDA Version: 13.0     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                  Driver-Model | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA L4                    WDDM  |   00000000:00:03.0 Off |                    0 |
| N/A   41C    P8             13W /   72W |     176MiB /  23034MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A            2528    C+G   C:\Windows\explorer.exe               N/A      |
|    0   N/A  N/A            7304    C+G   ...h_cw5n1h2txyewy\SearchApp.exe      N/A      |
|    0   N/A  N/A            7540    C+G   ...5n1h2txyewy\TextInputHost.exe      N/A      |
+-----------------------------------------------------------------------------------------+

[obriensystems]

northamerica-northeast1/2 nane1/2
https://docs.cloud.google.com/compute/docs/regions-zones/gpu-regions-zones#view-using-table

northamerica-northeast1-a | Montréal, Québec, North America | • N1+P4
-- | -- | --
northamerica-northeast1-b | Montréal, Québec, North America | • N1+P4
northamerica-northeast1-c | Montréal, Québec, North America | • N1+T4  • N1+P4
northamerica-northeast2-a | Toronto, Ontario, North America | • G2
northamerica-northeast2-b | Toronto, Ontario, North America | • G2
northamerica-northeast2-c | Toronto, Ontario, North America | • A3 Edge

[obriensystems]

Prototype M11

michaelobrien@mbp7 m11-nvidia-gce-terraform-ubuntu % pwd
/Users/michaelobrien/wse_github/ObrienlabsDev/gcp-infrastructure-as-code/m10-3rdparty/m11-nvidia-gce-terraform-ubuntu

gcloud config set project  gce-nvidia-olx

michaelobrien@mbp7 m11-nvidia-gce-terraform-ubuntu % gcloud services list                     
NAME                                 TITLE
analyticshub.googleapis.com          Analytics Hub API
bigquery.googleapis.com              BigQuery API
bigqueryconnection.googleapis.com    BigQuery Connection API
bigquerydatapolicy.googleapis.com    BigQuery Data Policy API
bigquerydatatransfer.googleapis.com  BigQuery Data Transfer API
bigquerymigration.googleapis.com     BigQuery Migration API
bigqueryreservation.googleapis.com   BigQuery Reservation API
bigquerystorage.googleapis.com       BigQuery Storage API
cloudapis.googleapis.com             Google Cloud APIs
cloudtrace.googleapis.com            Cloud Trace API
dataform.googleapis.com              Dataform API
dataplex.googleapis.com              Cloud Dataplex API
datastore.googleapis.com             Cloud Datastore API
logging.googleapis.com               Cloud Logging API
monitoring.googleapis.com            Cloud Monitoring API
servicemanagement.googleapis.com     Service Management API
serviceusage.googleapis.com          Service Usage API
sql-component.googleapis.com         Cloud SQL
storage-api.googleapis.com           Google Cloud Storage JSON API
storage-component.googleapis.com     Cloud Storage
storage.googleapis.com               Cloud Storage API
websecurityscanner.googleapis.com    Web Security Scanner API

michaelobrien@mbp7 m11-nvidia-gce-terraform-ubuntu % gcloud services enable compute.googleapis.com
Operation "operations/acf.p2-213593325912-f19b28ca-8827-427a-a5f2-e0b9c03b7c1a" finished successfully.


michaelobrien@mbp7 m11-nvidia-gce-terraform-ubuntu % terraform apply -var-file="marketplace_test.tfvars"

google_compute_instance.instance: Creating...
google_compute_instance.instance: Still creating... [00m10s elapsed]
╷
│ Error: Error waiting for instance to create: Quota 'GPUS_ALL_REGIONS' exceeded.  Limit: 0.0 globally.
│ 	metric name = compute.googleapis.com/gpus_all_regions
│ 	limit name = GPUS-ALL-REGIONS-per-project
│ 	dimensions = map[global:global]
│ 
│ 
│   with google_compute_instance.instance,
│   on main.tf line 22, in resource "google_compute_instance" "instance":
│   22: resource "google_compute_instance" "instance" {
│ 
╵


granted 1 min
| GPUS_ALL_REGIONS |            | GLOBAL |               4 |              4 |


│ Error: Error waiting for instance to create: The zone 'projects/gce-nvidia-olx/zones/us-central1-b' does not have enough resources available to fulfill the request.  '(resource type:compute)'.
│ 
│ 
│ A g2-standard-4 VM instance with 1 nvidia-l4 accelerator(s) is currently unavailable in the us-central1-b zone. Alternatively, you can try your request again with a different VM hardware configuration or at a later time. For more information, see the troubleshooting documentation.

finding region with L4s

trying zone a

VM creation error

A g2-standard-4 VM instance with 1 nvidia-l4 accelerator(s) is currently unavailable in the us-central1-b zone. Consider trying your request in the us-central1-a zone(s), which currently has capacity to accommodate your request. Alternatively, you can try your request again with a different VM hardware configuration or at a later time. For more information, see the troubleshooting documentation.

gcloud compute instances create instance-20260108-022956 --project=gce-nvidia-olx --zone=us-central1-a --machine-type=g2-standard-4 --network-interface=network-tier=PREMIUM,stack-type=IPV4_ONLY,subnet=default --metadata=enable-osconfig=TRUE --maintenance-policy=TERMINATE --provisioning-model=STANDARD --service-account=213593325912-compute@developer.gserviceaccount.com --scopes=https://www.googleapis.com/auth/devstorage.read_only,https://www.googleapis.com/auth/logging.write,https://www.googleapis.com/auth/monitoring.write,https://www.googleapis.com/auth/service.management.readonly,https://www.googleapis.com/auth/servicecontrol,https://www.googleapis.com/auth/trace.append --accelerator=count=1,type=nvidia-l4 --create-disk=auto-delete=yes,boot=yes,device-name=instance-20260108-022956,image=projects/debian-cloud/global/images/debian-12-bookworm-v20251209,mode=rw,size=10,type=pd-balanced --no-shielded-secure-boot --shielded-vtpm --shielded-integrity-monitoring --labels=goog-ops-agent-policy=v2-x86-template-1-4-0,goog-ec-src=vm_add-gcloud --reservation-affinity=any && printf 'agentsRule:\n  packageState: installed\n  version: latest\ninstanceFilter:\n  inclusionLabels:\n  - labels:\n      goog-ops-agent-policy: v2-x86-template-1-4-0\n' > config.yaml && gcloud compute instances ops-agents policies create goog-ops-agent-v2-x86-template-1-4-0-us-central1-a --project=gce-nvidia-olx --zone=us-central1-a --file=config.yaml && gcloud compute resource-policies create snapshot-schedule default-schedule-1 --project=gce-nvidia-olx --region=us-central1 --max-retention-days=14 --on-source-disk-delete=keep-auto-snapshots --daily-schedule --start-time=20:00 && gcloud compute disks add-resource-policies instance-20260108-022956 --project=gce-nvidia-olx --zone=us-central1-a --resource-policies=projects/gce-nvidia-olx/regions/us-central1/resourcePolicies/default-schedule-1


A g2-standard-4 VM instance with 1 nvidia-l4 accelerator(s) is currently unavailable in the us-central1-a zone. Alternatively, you can try your request again with a different VM hardware configuration or at a later time. For more information, see the troubleshooting documentation.

[obriensystems]

us-east1-c works via console

rerunning terraform

michaelobrien@mbp7 m11-nvidia-gce-terraform-ubuntu % terraform apply -var-file="marketplace_test.tfvars" --auto-approve

of course - increasing quota

google_compute_instance.instance: Still creating... [00m10s elapsed]
╷
│ Error: Error waiting for instance to create: Quota 'NVIDIA_L4_GPUS' exceeded.  Limit: 1.0 in region us-east1.
│ 	metric name = compute.googleapis.com/nvidia_l4_gpus
│ 	limit name = NVIDIA-L4-GPUS-per-project-region
│ 	limit = 1
│ 	dimensions = map[region:us-east1]
│ 
│ 
│   with google_compute_instance.instance,
│   on main.tf line 22, in resource "google_compute_instance" "instance":
│   22: resource "google_compute_instance" "instance" {

no quota increase 4 or 2 - sticking with 1

redeploy working

Expanded Security Maintenance for Applications is not enabled.

121 updates can be applied immediately.
60 of these updates are standard security updates.
To see these additional updates run: apt list --upgradable

Enable ESM Apps to receive additional future security updates.
See https://ubuntu.com/esm or run: sudo pro status



The following GCP CLI version has been pre-installed. Begin using the GCP CLI by first configuring your credentials using 'gcloud init'

Google Cloud SDK 541.0.0


Welcome to the NVIDIA GPU Cloud image.  This image provides an optimized
environment for running the deep learning and HPC containers from the
NVIDIA GPU Cloud Container Registry.  Many NGC containers are freely
available.  However, some NGC containers require that you log in with
a valid NGC API key in order to access them.  This is indicated by a
"pull access denied for xyz ..." or "Get xyz: unauthorized: ..." error
message from the daemon.

Documentation on using this image and accessing the NVIDIA GPU Cloud
Container Registry can be found at
  http://docs.nvidia.com/ngc/index.html


The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.

Error while loading conda entry point: conda-libmamba-solver (cannot import name 'Spinner' from 'conda.common.io' (/opt/miniforge/lib/python3.12/site-packages/conda/common/io.py))
michael@test-deployment-vm:~$ nvidia-smi
Thu Jan  8 02:44:52 2026       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.172.08             Driver Version: 570.172.08     CUDA Version: 12.8     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA L4                      On  |   00000000:00:03.0 Off |                    0 |
| N/A   54C    P8             13W /   72W |       0MiB /  23034MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

[obriensystems]

test out running my machine-learning python tensorflow container

michael@test-deployment-vm:~$ ls
README.txt
michael@test-deployment-vm:~$ df
Filesystem      1K-blocks    Used Available Use% Mounted on
/dev/root       128917488 7480448 121420656   6% /
tmpfs             8187124       0   8187124   0% /dev/shm
tmpfs             3274852    1044   3273808   1% /run
tmpfs                5120       0      5120   0% /run/lock
efivarfs               56      24        27  48% /sys/firmware/efi/efivars
/dev/nvme0n1p16    901520  136248    702144  17% /boot
/dev/nvme0n1p15    106832    6250    100582   6% /boot/efi
tmpfs             1637424      20   1637404   1% /run/user/1001
michael@test-deployment-vm:~$ git clone https://github.com/ObrienlabsDev/machine-learning.git
Cloning into 'machine-learning'...
remote: Enumerating objects: 408, done.
remote: Counting objects: 100% (218/218), done.
remote: Compressing objects: 100% (163/163), done.
remote: Total 408 (delta 103), reused 110 (delta 39), pack-reused 190 (from 1)
Receiving objects: 100% (408/408), 98.24 KiB | 3.51 MiB/s, done.
Resolving deltas: 100% (194/194), done.
michael@test-deployment-vm:~$ cd machine-learning/
michael@test-deployment-vm:~/machine-learning$ ls
LICENSE    build.sh  docs          gpus.md   pyproject.toml  requirements.txt  stats.md
README.md  cuda      environments  links.md  pytorch.py      src               tensorflow-version.py
michael@test-deployment-vm:~/machine-learning$ docker --version
Docker version 29.1.3, build f52814d
michael@test-deployment-vm:~/machine-learning$ vi environments/windows/Dockerfile 
michael@test-deployment-vm:~/machine-learning$ vi build.sh 


michael@test-deployment-vm:~/machine-learning$ git diff
diff --git a/build.sh b/build.sh
index 5091b0f..69c2112 100755
--- a/build.sh
+++ b/build.sh
@@ -1,8 +1,8 @@
 # https://forums.developer.nvidia.com/t/unable-to-work-with-tensorflow-in-docker-on-dgx-spark/350419
-#export DOCKER_DEFAULT_PLATFORM=linux/amd64
-export DOCKER_DEFAULT_PLATFORM=linux/arm64
+export DOCKER_DEFAULT_PLATFORM=linux/amd64
+#export DOCKER_DEFAULT_PLATFORM=linux/arm64
 cd environments/windows
-docker build -t ml-tensorflow-arm .
+docker build -t ml-tensorflow .
 cd ../../
-docker run --rm --gpus all --name ml-tensorflow-win ml-tensorflow-arm
+docker run --rm --gpus all --name ml-tensorflow ml-tensorflow
 
diff --git a/environments/windows/Dockerfile b/environments/windows/Dockerfile
index 8dfa7d4..ed66347 100644
--- a/environments/windows/Dockerfile
+++ b/environments/windows/Dockerfile
@@ -1,9 +1,9 @@
 #FROM tensorflow/tensorflow:latest-gpu
 # https://hub.docker.com/layers/tensorflow/tensorflow/2.14.0-gpu/images/sha256-64602abcd8cc4f4bdd6268ca0abc39e6d37113d700886afd15f6dd151210b206?context=explore
 # IA64
-#FROM tensorflow/tensorflow:2.14.0-gpu
+FROM tensorflow/tensorflow:2.14.0-gpu
 # ARM64
-FROM linaro/tensorflow-arm64-build:2.14-multipython
+#FROM linaro/tensorflow-arm64-build:2.14-multipython
 WORKDIR /src
 COPY /src/tflow.py .
 CMD ["python", "tflow.py"]

michael@test-deployment-vm:~/machine-learning$ sudo ./build.sh 
[+] Building 195.3s (8/8) FINISHED                                                                              docker:default
 => [internal] load build definition from Dockerfile                                                                      0.0s
 => => transferring dockerfile: 544B                                                                                      0.0s
 => [internal] load metadata for docker.io/tensorflow/tensorflow:2.14.0-gpu                                               0.6s
 => [internal] load .dockerignore                                                                                         0.0s
 => => transferring context: 2B                                                                                           0.0s
 => [1/3] FROM docker.io/tensorflow/tensorflow:2.14.0-gpu@sha256:64602abcd8cc4f4bdd6268ca0abc39e6d37113d700886afd15f6d  189.4s
 => => resolve docker.io/tensorflow/tensorflow:2.14.0-gpu@sha256:64602abcd8cc4f4bdd6268ca0abc39e6d37113d700886afd15f6dd1  0.0s
 => => sha256:4586c00479c64413175c7b90a66073cfd574ca0a9ee099495ba8a58a0819d8b9 4.62MB / 4.62MB                            0.2s
 => => sha256:6b851dcae6ca1461dde247915abc5048061f34332929ca8fb37d9dc18f2e2f44 29.53MB / 29.53MB                          0.5s
 => => sha256:4304fa233a80418a848565c712c6ebcb670c1c2d076c419f65f8bf3e49526b08 56.23MB / 56.23MB                          0.8s
 => => sha256:64602abcd8cc4f4bdd6268ca0abc39e6d37113d700886afd15f6dd151210b206 4.08kB / 4.08kB                            0.0s
 => => sha256:68d96d40670e06f0c7b926edf49528a881cf2fca37714de065eacfc75e831192 11.70kB / 11.70kB                          0.0s
 => => sha256:afa3f70b397f64c9ae4c406c33113bc1929d64b744b21f934978e7e4def3c7b6 181B / 181B                                0.3s
 => => sha256:01d85bc616acc99e7ec43f08799b125f582592bb3477af59065440d3e5e5fdf3 859B / 859B                                0.7s
 => => sha256:d963a42bc71212aa841b9e02f08b835b9c54a833eb2ace0733666774ca570cef 6.88kB / 6.88kB                            0.4s
 => => sha256:7d32cc707b9d271f8e29238607e96d4cd00fae7c8f5977f378960183cbf185de 767B / 767B                                0.7s
 => => extracting sha256:6b851dcae6ca1461dde247915abc5048061f34332929ca8fb37d9dc18f2e2f44                                 2.1s
 => => sha256:076ff2aa313a47d66f7bfea6ed89191e0446155f254ed7fb6fffc141dfb6a987 32.50MB / 32.50MB                          1.3s
 => => sha256:5a520d59fe5eceb751c85220f614e93eaf70b1abfcdd1e574a0fe344954f23d3 3.03GB / 3.03GB                           65.4s
 => => sha256:4cea12640dc80732267166e470945eb95384c30d60a28a1f11e9304d27c6cb1b 391B / 391B                                1.0s
 => => sha256:22d5103f16e8c41e440e21525f9411b09f5be4f4535d692d4b7607ad4aa90861 971B / 971B                                1.2s
 => => sha256:74669e8faa15cce692f221340fcd5d84a14321389ad859137e26f9cef887efba 127B / 127B                                1.4s
 => => sha256:400aa4b0a962d65aa89842bea7dee30cd774cec116ff32a1ca66633f66351710 37.16MB / 37.16MB                          2.4s
 => => sha256:81960391dc73ef9552118af3ef1670c5b873fbd4f676126087945a730d1ec821 580.32MB / 580.32MB                       16.6s
 => => sha256:45d81655ae234588cb3d08c95df981d9ae6fa1a15b73c8e14660fdb18f9b06fa 808B / 808B                                2.6s
 => => extracting sha256:4586c00479c64413175c7b90a66073cfd574ca0a9ee099495ba8a58a0819d8b9                                 0.3s
 => => sha256:4db7c9771a1984de8d2d83f9de6cd62ef14af7737a0d8733e75a0923ef8ddb3a 6.90kB / 6.90kB                            2.8s
 => => sha256:91c6784a9bf6ec3f4bd6f1562e09ae227761f1d4a3b44f8a2d9d27a436cebb4e 1.13kB / 1.13kB                            3.0s
 => => sha256:2ef3e831f8658e728821077ea54594b5bc43898fd70b3b55851d43c3ebaba8f6 1.13kB / 1.13kB                            3.2s
 => => extracting sha256:4304fa233a80418a848565c712c6ebcb670c1c2d076c419f65f8bf3e49526b08                                 2.3s
 => => extracting sha256:afa3f70b397f64c9ae4c406c33113bc1929d64b744b21f934978e7e4def3c7b6                                 0.0s
 => => extracting sha256:d963a42bc71212aa841b9e02f08b835b9c54a833eb2ace0733666774ca570cef                                 0.0s
 => => extracting sha256:01d85bc616acc99e7ec43f08799b125f582592bb3477af59065440d3e5e5fdf3                                 0.0s
 => => extracting sha256:7d32cc707b9d271f8e29238607e96d4cd00fae7c8f5977f378960183cbf185de                                 0.0s
 => => extracting sha256:4cea12640dc80732267166e470945eb95384c30d60a28a1f11e9304d27c6cb1b                                 0.0s
 => => extracting sha256:076ff2aa313a47d66f7bfea6ed89191e0446155f254ed7fb6fffc141dfb6a987                                 1.1s
 => => extracting sha256:5a520d59fe5eceb751c85220f614e93eaf70b1abfcdd1e574a0fe344954f23d3                                92.0s
 => => extracting sha256:22d5103f16e8c41e440e21525f9411b09f5be4f4535d692d4b7607ad4aa90861                                 0.0s
 => => extracting sha256:74669e8faa15cce692f221340fcd5d84a14321389ad859137e26f9cef887efba                                 0.0s
 => => extracting sha256:400aa4b0a962d65aa89842bea7dee30cd774cec116ff32a1ca66633f66351710                                 2.3s
 => => extracting sha256:81960391dc73ef9552118af3ef1670c5b873fbd4f676126087945a730d1ec821                                26.8s
 => => extracting sha256:45d81655ae234588cb3d08c95df981d9ae6fa1a15b73c8e14660fdb18f9b06fa                                 0.0s
 => => extracting sha256:4db7c9771a1984de8d2d83f9de6cd62ef14af7737a0d8733e75a0923ef8ddb3a                                 0.0s
 => => extracting sha256:91c6784a9bf6ec3f4bd6f1562e09ae227761f1d4a3b44f8a2d9d27a436cebb4e                                 0.0s
 => => extracting sha256:2ef3e831f8658e728821077ea54594b5bc43898fd70b3b55851d43c3ebaba8f6                                 0.0s
 => [internal] load build context                                                                                         0.0s
 => => transferring context: 4.47kB                                                                                       0.0s
 => [2/3] WORKDIR /src                                                                                                    5.1s
 => [3/3] COPY /src/tflow.py .                                                                                            0.0s
 => exporting to image                                                                                                    0.0s
 => => exporting layers                                                                                                   0.0s
 => => writing image sha256:ea3f894854c2f6368bfdc5ca8e56f37080b5a2c8e56a631f9797eddcf0d9b59e                              0.0s
 => => naming to docker.io/library/ml-tensorflow                                                                          0.0s
2026-01-08 02:58:40.371849: I tensorflow/core/util/port.cc:111] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2026-01-08 02:58:40.419271: E tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:9342] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2026-01-08 02:58:40.419309: E tensorflow/compiler/xla/stream_executor/cuda/cuda_fft.cc:609] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2026-01-08 02:58:40.419345: E tensorflow/compiler/xla/stream_executor/cuda/cuda_blas.cc:1518] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2026-01-08 02:58:40.428033: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2026-01-08 02:58:42.599362: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1886] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 20846 MB memory:  -> device: 0, name: NVIDIA L4, pci bus id: 0000:00:03.0, compute capability: 8.9
Downloading data from https://www.cs.toronto.edu/~kriz/cifar-100-python.tar.gz
169001437/169001437 [==============================] - 2s 0us/step
Batch: 2048 Epochs: 25
Epoch 1/25
2026-01-08 02:59:03.299688: I tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:442] Loaded cuDNN version 8600
2026-01-08 02:59:05.100965: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x13ee6380 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2026-01-08 02:59:05.101000: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): NVIDIA L4, Compute Capability 8.9
2026-01-08 02:59:05.106567: I tensorflow/compiler/mlir/tensorflow/utils/dump_mlir_util.cc:269] disabling MLIR crash reproducer, set env var `MLIR_CRASH_REPRODUCER_DIRECTORY` to enable.
2026-01-08 02:59:05.213421: I ./tensorflow/compiler/jit/device_compiler.h:186] Compiled cluster using XLA!  This line is logged at most once for the lifetime of the process.
25/25 [==============================] - 44s 397ms/step - loss: 4.9990 - accuracy: 0.0419
Epoch 2/25
25/25 [==============================] - 6s 231ms/step - loss: 3.8434 - accuracy: 0.1203
Epoch 3/25
25/25 [==============================] - 6s 232ms/step - loss: 3.4611 - accuracy: 0.1873
Epoch 4/25
25/25 [==============================] - 6s 232ms/step - loss: 3.0285 - accuracy: 0.2637
Epoch 5/25
25/25 [==============================] - 6s 233ms/step - loss: 2.6462 - accuracy: 0.3382
Epoch 6/25
25/25 [==============================] - 6s 234ms/step - loss: 2.2977 - accuracy: 0.4122
Epoch 7/25
25/25 [==============================] - 6s 234ms/step - loss: 1.9810 - accuracy: 0.4824
Epoch 8/25
25/25 [==============================] - 6s 234ms/step - loss: 1.7665 - accuracy: 0.5308
Epoch 9/25
25/25 [==============================] - 6s 234ms/step - loss: 1.4652 - accuracy: 0.5980
Epoch 10/25
25/25 [==============================] - 6s 234ms/step - loss: 1.1459 - accuracy: 0.6774
Epoch 11/25
25/25 [==============================] - 6s 235ms/step - loss: 1.0143 - accuracy: 0.7124
Epoch 12/25
25/25 [==============================] - 6s 236ms/step - loss: 1.2815 - accuracy: 0.6425
Epoch 13/25
25/25 [==============================] - 6s 236ms/step - loss: 1.0813 - accuracy: 0.6928
Epoch 14/25
25/25 [==============================] - 6s 237ms/step - loss: 2.1757 - accuracy: 0.5111
Epoch 15/25
25/25 [==============================] - 6s 237ms/step - loss: 3.7190 - accuracy: 0.1510
Epoch 16/25
25/25 [==============================] - 6s 238ms/step - loss: 3.2380 - accuracy: 0.2233
Epoch 17/25
25/25 [==============================] - 6s 239ms/step - loss: 2.9183 - accuracy: 0.2792
Epoch 18/25
25/25 [==============================] - 6s 238ms/step - loss: 2.6438 - accuracy: 0.3348
Epoch 19/25
25/25 [==============================] - 6s 239ms/step - loss: 2.3790 - accuracy: 0.3856
Epoch 20/25
25/25 [==============================] - 6s 238ms/step - loss: 2.1184 - accuracy: 0.4455
Epoch 21/25
25/25 [==============================] - 6s 238ms/step - loss: 1.8656 - accuracy: 0.5019
Epoch 22/25
25/25 [==============================] - 6s 238ms/step - loss: 1.6300 - accuracy: 0.5535
Epoch 23/25
25/25 [==============================] - 6s 237ms/step - loss: 1.3575 - accuracy: 0.6217
Epoch 24/25
25/25 [==============================] - 6s 237ms/step - loss: 1.0887 - accuracy: 0.6934
Epoch 25/25
25/25 [==============================] - 6s 237ms/step - loss: 0.8350 - accuracy: 0.7590

michael@test-deployment-vm:~$ nvidia-smi
Thu Jan  8 03:00:06 2026       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.172.08             Driver Version: 570.172.08     CUDA Version: 12.8     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA L4                      On  |   00000000:00:03.0 Off |                    0 |
| N/A   68C    P0             71W /   72W |   21078MiB /  23034MiB |    100%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A           11650      C   python                                21070MiB |
+-----------------------------------------------------------------------------------------+

michael@test-deployment-vm:~$ nvidia-smi
Thu Jan  8 03:01:43 2026       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.172.08             Driver Version: 570.172.08     CUDA Version: 12.8     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA L4                      On  |   00000000:00:03.0 Off |                    0 |
| N/A   76C    P0             71W /   72W |   21078MiB /  23034MiB |     90%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A           11650      C   python                                21070MiB |
+-----------------------------------------------------------------------------------------+

[obriensystems]

delete

michaelobrien@mbp7 m11-nvidia-gce-terraform-ubuntu % terraform destroy -var-file="marketplace_test.tfvars" --auto-approve
google_compute_instance.instance: Refreshing state... [id=projects/gce-nvidia-olx/zones/us-east1-c/instances/test-deployment-vm]

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  - destroy

Terraform will perform the following actions:

  # google_compute_instance.instance will be destroyed
  - resource "google_compute_instance" "instance" {
      - can_ip_forward             = false -> null
      - cpu_platform               = "Intel Cascade Lake" -> null
      - creation_timestamp         = "2026-01-07T18:43:01.741-08:00" -> null
      - current_status             = "RUNNING" -> null
      - deletion_protection        = false -> null
      - effective_labels           = {
          - "goog-partner-solution"      = "isol_plb32_0014m00001h36sfqaq_ldcwv2kqy3qx5iqd2clmass7maq6qmfu"
          - "goog-terraform-provisioned" = "true"
        } -> null
      - enable_display             = false -> null
      - id                         = "projects/gce-nvidia-olx/zones/us-east1-c/instances/test-deployment-vm" -> null
      - instance_id                = "8056210171832215386" -> null
      - label_fingerprint          = "3JkS9TMVDSc=" -> null
      - labels                     = {} -> null
      - machine_type               = "g2-standard-4" -> null
      - metadata                   = {
          - "google-logging-enable"    = "0"
          - "google-monitoring-enable" = "0"
          - "ssh-keys"                 = <<-EOT
                michael:ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPS7tM80zDGg3wjUoWjE2pk8Za+PMyBTZenD4wzj2ek/Saw9+TLbrWWdMQJvNBwSK19iyJ7k7SpsY+pq3Ak+UcA= google-ssh {"userName":"michael@obrienlabs.xyz","expireOn":"2026-01-08T03:02:50+0000"}
                michael:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAG9QIGQ7JFnoYnl9tJ7F35l/TO8g0mxW4R6u6TpiRLoo4rwkTdCy5M70WaQLNmUPK1hQ3flCzK1FGezgBbiWnifLclKhovV4GoPq/wWF5wdc7AWhu2ynvcxJwJiRAAdQAt86XMS2yPm7zRfVSA5OdNWnK1EEiP+iFdHmmELXDNvX8wF8vLHjwA/x5lzteOawvoM9moRyhctejSaT3tbLvRc8m31dZcBieNbSFDDAN9WjJ4ss5v8Vi/D9CUDck5jqP7WQ3rVqGrTtXXioodJ+XLgt+3m92LrgxYvq5EwtdjgJK5xldnqFlRkNc1LtHC+nIJmbJybtSN5IYdfMyFzWZz0= google-ssh {"userName":"michael@obrienlabs.xyz","expireOn":"2026-01-08T03:02:56+0000"}
            EOT
        } -> null
      - metadata_fingerprint       = "IRZ1g6zwPwo=" -> null
      - name                       = "test-deployment-vm" -> null
      - project                    = "gce-nvidia-olx" -> null
      - resource_policies          = [] -> null
      - self_link                  = "https://www.googleapis.com/compute/v1/projects/gce-nvidia-olx/zones/us-east1-c/instances/test-deployment-vm" -> null
      - tags                       = [
          - "test-deployment-deployment",
        ] -> null
      - tags_fingerprint           = "VyCqthjp4bo=" -> null
      - terraform_labels           = {
          - "goog-partner-solution"      = "isol_plb32_0014m00001h36sfqaq_ldcwv2kqy3qx5iqd2clmass7maq6qmfu"
          - "goog-terraform-provisioned" = "true"
        } -> null
      - zone                       = "us-east1-c" -> null
        # (4 unchanged attributes hidden)

      - boot_disk {
          - auto_delete                     = true -> null
          - device_name                     = "autogen-vm-tmpl-boot-disk" -> null
          - force_attach                    = false -> null
          - guest_os_features               = [
              - "VIRTIO_SCSI_MULTIQUEUE",
              - "SEV_CAPABLE",
              - "SEV_SNP_CAPABLE",
              - "SEV_LIVE_MIGRATABLE",
              - "SEV_LIVE_MIGRATABLE_V2",
              - "SNP_SVSM_CAPABLE",
              - "IDPF",
              - "TDX_CAPABLE",
              - "UEFI_COMPATIBLE",
              - "GVNIC",
            ] -> null
          - mode                            = "READ_WRITE" -> null
          - source                          = "https://www.googleapis.com/compute/v1/projects/gce-nvidia-olx/zones/us-east1-c/disks/test-deployment-vm" -> null
            # (6 unchanged attributes hidden)

          - initialize_params {
              - architecture                = "X86_64" -> null
              - enable_confidential_compute = false -> null
              - image                       = "https://www.googleapis.com/compute/v1/projects/mpi-nvidia-ngc-public/global/images/nvidia-gpu-cloud-vmi-base-2025-9-1-x86-64" -> null
              - labels                      = {} -> null
              - provisioned_iops            = 0 -> null
              - provisioned_throughput      = 0 -> null
              - resource_manager_tags       = {} -> null
              - resource_policies           = [] -> null
              - size                        = 128 -> null
              - type                        = "pd-ssd" -> null
                # (2 unchanged attributes hidden)
            }
        }

      - guest_accelerator {
          - count = 1 -> null
          - type  = "https://www.googleapis.com/compute/v1/projects/gce-nvidia-olx/zones/us-east1-c/acceleratorTypes/nvidia-l4" -> null
        }

      - network_interface {
          - internal_ipv6_prefix_length = 0 -> null
          - name                        = "nic0" -> null
          - network                     = "https://www.googleapis.com/compute/v1/projects/gce-nvidia-olx/global/networks/default" -> null
          - network_ip                  = "10.142.0.3" -> null
          - queue_count                 = 0 -> null
          - stack_type                  = "IPV4_ONLY" -> null
          - subnetwork                  = "https://www.googleapis.com/compute/v1/projects/gce-nvidia-olx/regions/us-east1/subnetworks/default" -> null
          - subnetwork_project          = "gce-nvidia-olx" -> null
            # (4 unchanged attributes hidden)

          - access_config {
              - nat_ip                 = "34.26.133.35" -> null
              - network_tier           = "PREMIUM" -> null
                # (1 unchanged attribute hidden)
            }
        }

      - scheduling {
          - automatic_restart           = true -> null
          - availability_domain         = 0 -> null
          - min_node_cpus               = 0 -> null
          - on_host_maintenance         = "TERMINATE" -> null
          - preemptible                 = false -> null
          - provisioning_model          = "STANDARD" -> null
            # (2 unchanged attributes hidden)
        }

      - service_account {
          - email  = "213593325912-compute@developer.gserviceaccount.com" -> null
          - scopes = [
              - "https://www.googleapis.com/auth/cloud-platform",
              - "https://www.googleapis.com/auth/cloud.useraccounts.readonly",
              - "https://www.googleapis.com/auth/devstorage.read_only",
              - "https://www.googleapis.com/auth/logging.write",
              - "https://www.googleapis.com/auth/monitoring.write",
            ] -> null
        }

      - shielded_instance_config {
          - enable_integrity_monitoring = true -> null
          - enable_secure_boot          = false -> null
          - enable_vtpm                 = true -> null
        }
    }

Plan: 0 to add, 0 to change, 1 to destroy.

Changes to Outputs:
  - instance_machine_type = "g2-standard-4" -> null
  - instance_nat_ip       = "34.26.133.35" -> null
  - instance_network      = "default" -> null
  - instance_self_link    = "https://www.googleapis.com/compute/v1/projects/gce-nvidia-olx/zones/us-east1-c/instances/test-deployment-vm" -> null
  - instance_zone         = "us-east1-c" -> null
google_compute_instance.instance: Destroying... [id=projects/gce-nvidia-olx/zones/us-east1-c/instances/test-deployment-vm]
google_compute_instance.instance: Still destroying... [id=projects/gce-nvidia-olx/zones/us-east1-c/instances/test-deployment-vm, 00m10s elapsed]
google_compute_instance.instance: Still destroying... [id=projects/gce-nvidia-olx/zones/us-east1-c/instances/test-deployment-vm, 00m20s elapsed]
google_compute_instance.instance: Still destroying... [id=projects/gce-nvidia-olx/zones/us-east1-c/instances/test-deployment-vm, 00m30s elapsed]
google_compute_instance.instance: Still destroying... [id=projects/gce-nvidia-olx/zones/us-east1-c/instances/test-deployment-vm, 00m40s elapsed]
google_compute_instance.instance: Still destroying... [id=projects/gce-nvidia-olx/zones/us-east1-c/instances/test-deployment-vm, 00m50s elapsed]
google_compute_instance.instance: Still destroying... [id=projects/gce-nvidia-olx/zones/us-east1-c/instances/test-deployment-vm, 01m00s elapsed]
google_compute_instance.instance: Destruction complete after 1m2s

Destroy complete! Resources: 1 destroyed.

[obriensystems]

Add gpt-oss:20b running via ollama - see ObrienlabsDev/blog#160

https://docs.ollama.com/linux

ichael@test-deployment-vm:~$ curl -fsSL https://ollama.com/install.sh | sh
>>> Installing ollama to /usr/local
>>> Downloading ollama-linux-amd64.tar.zst
######################################################################## 100.0%
>>> Creating ollama user...
>>> Adding ollama user to render group...
>>> Adding ollama user to video group...
>>> Adding current user to ollama group...
>>> Creating ollama systemd service...
>>> Enabling and starting ollama service...
Created symlink /etc/systemd/system/default.target.wants/ollama.service → /etc/systemd/system/ollama.service.
>>> NVIDIA GPU installed.


ichael@test-deployment-vm:~$ ollama run gpt-oss:20b --verbose
pulling manifest 
pulling e7b273f96360:  75% ▕██████████████████████████████████████████████████████████████████████████████████████████                               ▏  10 GB/ 13 GB  235 MB/s     14

[obriensystems]

michael@test-deployment-vm:~$ nvidia-smi
Sun Jan 18 23:43:40 2026       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.172.08             Driver Version: 570.172.08     CUDA Version: 12.8     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA L4                      On  |   00000000:00:03.0 Off |                    0 |
| N/A   80C    P0             71W /   72W |   12748MiB /  23034MiB |     90%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A           12770      C   /usr/local/bin/ollama                 12740MiB |
+-----------------------------------------------------------------------------------------+

total duration:       2m16.686959385s
load duration:        323.982983ms
prompt eval count:    593 token(s)
prompt eval duration: 344.916285ms
prompt eval rate:     1719.26 tokens/s
eval count:           7239 token(s)
eval duration:        2m9.919606758s
eval rate:            55.72 tokens/s


top - 23:48:57 up 38 min,  2 users,  load average: 0.01, 0.67, 0.70
Tasks: 151 total,   1 running, 150 sleeping,   0 stopped,   0 zombie
%Cpu0  :   0.0/0.0     0[                                                                                                    ] 
%Cpu1  :   0.0/0.0     0[                                                                                                    ] 
%Cpu2  :   0.0/0.0     0[                                                                                                    ] 
%Cpu3  :   0.0/0.0     0[                                                                                                    ] 
MiB Mem :  7.6/15990.5  [|||||||                                                                                             ] 
MiB Swap:  0.0/0.0      [                                                                                                    ] 

michael@test-deployment-vm:~$ df
Filesystem      1K-blocks     Used Available Use% Mounted on
/dev/root       128917488 25487052 103414052  20% /
tmpfs             8187120        0   8187120   0% /dev/shm
tmpfs             3274852     1056   3273796   1% /run
tmpfs                5120        0      5120   0% /run/lock
efivarfs               56       24        27  48% /sys/firmware/efi/efivars
/dev/nvme0n1p16    901520   136248    702144  17% /boot
/dev/nvme0n1p15    106832     6250    100582   6% /boot/efi
tmpfs             1637424       20   1637404   1% /run/user/1001
michael@test-deployment-vm:~$ free
               total        used        free      shared  buff/cache   available
Mem:        16374244     1244336     2213504        1080    13268252    15129908
Swap:              0           0           0


michael@test-deployment-vm:~$ ollama run gpt-oss:120b --verbose
pulling manifest 
pulling 6be6d66a3f54:   7% ▕████████                                                                                                                 ▏ 4.9 GB/ 65 GB  528 MB/s   1m5

[obriensystems]