Quantify copy time from the processing pod to NFS Vs BUCKET

[subash-taranga]

[subash-taranga]

Logs can be find here processed file copy to bucket logs.txt

subash@MSI:~/cloud-processing/standard-gke-cluster-gcs/argo$ argo logs @latest -n argo | grep 1342814727

We can see file copy starts as soon as Closed file take place and copy end with "sub-process exited"

subash@MSI:~/cloud-processing/standard-gke-cluster-nfs/argo$ argo logs pfnano-process-ww642 -n argo | grep 1995231051

**Can get full logs here
processed file copy to nfs logs 1.txt
processed file copy to nfs logs 2.txt
**

[tomcordruw]

@subash-taranga
I can confirm your observation from my end.
When testing a dataset processing 3 million events with both gcs bucket and nfs, output files were around 800MB in size and they took around 40 seconds to copy.

Testing a whole dataset, 29 million events, output files were ~2.0-2.5GB files and it took around 2 minutes.

[subash-taranga]

@tomcordruw Yes it is correct....Thanks a lot