empty files

[drewkenjo]

For certain runs and certain detectors the HIPO files are created empty without the set of histograms inside them.
According to Mohammad, the rerun for these runs produces files with histograms correctly.

[drewkenjo]

Zhiwen or any other chefs,
Could you provide the log file from the job with empty files?

[Hattawy]

I run clas12_monitoring on daily basis, I confirm that the second try of any empty-hipo-files run passes it during the second try. This is confirmed 100%.

[zhaozhiwen]

here is one example

cooked data /volatile/clas12/rg-b/production/recon/pass0/v28.13
log of clas12_monitoring /farm_out/clas12-1 (file created after 9/27 10PM)
output of clas12_monitoring /work/clas12/rg-b/offline_monitoring/submit_scripts_pass0v28.13
error message of "detectors.sh" /group/clas/www/clas12mon/html/hipo/rgb/pass0/log_v28.13
output of "detectors.sh" /group/clas/www/clas12mon/html/hipo/rgb/pass0/v28.13_org
output of "qa.sh" /group/clas/www/clas12mon/html/hipo/rgb/pass0/v28.13

[drewkenjo]

@zhaozhiwen could you please pinpoint the logfile with one run that has empty files?
Thank you!

[zhaozhiwen]

log_v28.13 shows band is not good for band 6227
the log files are /farm_out/clas12-1/timelines-6227-65069723-farm13013.err and /farm_out/clas12-1/timelines-6227-65069723-farm13013.out

It also shows htcc_npheAll is not good for all runs and I don't know what it means

[drewkenjo]

@zhaozhiwen Could you please change their permissions to read for all? I don't have access to them under my account.

[zhaozhiwen]

they are set readable by anyone, but that dir has some strange control. anyhow
I copy them to /scratch/clas12-1/ on ifarm1901

[drewkenjo]

I think it's plotting that causes the issue. Don't know why second try works though.
Yordanka is disabling plotting in clas12_monitoring. So hopefully it will stop.

[drewkenjo]

I run clas12_monitoring on daily basis, I confirm that the second try of any empty-hipo-files run passes it during the second try. This is confirmed 100%.

@Hattawy ANy chance you change some conditions on second try? maybe more memory?

[Hattawy]

@drewkenjo I change nothing at all.

[Hattawy]

@drewkenjo I also confirm that htcc_npheAll from clas12-timelines fails for all the runs.
You can check the log file under /work/clas12/mhattawy/rgc-timelines/clas12-timeline/rgc_pass0/log/htcc_npheAll.log

[zhaozhiwen]

yeah, htcc_npheAll is a problem

[zhaozhiwen]

@drewkenjo can we disable plotting in clas12_monitoring quickly? there are many runs fails and make it a lot of manual work to do.

[drewkenjo]

@zhaozhiwen I've submitted pull request with disabling plotting.

[drewkenjo]

@zhaozhiwen @Hattawy npheAll is empty because there is a typo in clas12_monitoring, where histogram has a name npeAll instead of npheAll. I'll fix it.