Excessive rucio metadata lookup

[FaroutYLq]

We crashed rucio server, and one suspect is that outsource jobs are doing overwhelming rucio metadata lookup. One example from a peaklet job

[root@rucio-xenon rucio_httpd]# grep mwt2-c071.campuscluster.illinois.edu httpd_access_log | grep 11/Dec/2023:12
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:29:12 -0600] "GET /auth/x509_proxy HTTP/1.1" 200 -
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:29:12 -0600] "GET /accounts/whoami HTTP/1.1" 303 -
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:29:12 -0600] "GET /accounts/production HTTP/1.1" 200 202
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:27 -0600] "GET /dids/xnt_049794/aqmon_hits-gsmdrzd6gz HTTP/1.1" 404 128
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:27 -0600] "GET /dids/xnt_049794/peak_classification_bayes-gkrokfhp5z HTTP/1.1" 404 143
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:27 -0600] "GET /dids/xnt_049794/corrected_areas-o44rjrrdci HTTP/1.1" 404 133
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:27 -0600] "GET /dids/xnt_049794/raw_records-rfzvpzj4mf HTTP/1.1" 200 186
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:27 -0600] "GET /dids/xnt_049794/raw_records-rfzvpzj4mf/rules HTTP/1.1" 200 1630
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:27 -0600] "GET /dids/xnt_049794/raw_records_he-rfzvpzj4mf HTTP/1.1" 200 189
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:27 -0600] "GET /dids/xnt_049794/raw_records_he-rfzvpzj4mf/rules HTTP/1.1" 200 1632
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:27 -0600] "GET /dids/xnt_049794/raw_records_aqmon-rfzvpzj4mf HTTP/1.1" 200 192
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:27 -0600] "GET /dids/xnt_049794/raw_records_aqmon-rfzvpzj4mf/rules HTTP/1.1" 200 1638
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:27 -0600] "GET /dids/xnt_049794/raw_records_nv-rfzvpzj4mf HTTP/1.1" 200 189
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:27 -0600] "GET /dids/xnt_049794/raw_records_nv-rfzvpzj4mf/rules HTTP/1.1" 200 1632
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:27 -0600] "GET /dids/xnt_049794/raw_records_aqmon_nv-rfzvpzj4mf HTTP/1.1" 200 195
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:27 -0600] "GET /dids/xnt_049794/raw_records_aqmon_nv-rfzvpzj4mf/rules HTTP/1.1" 200 1643
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:27 -0600] "GET /dids/xnt_049794/raw_records_aux_mv-rfzvpzj4mf HTTP/1.1" 200 193
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/raw_records_aux_mv-rfzvpzj4mf/rules HTTP/1.1" 200 1639
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/raw_records_mv-rfzvpzj4mf HTTP/1.1" 200 189
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/raw_records_mv-rfzvpzj4mf/rules HTTP/1.1" 200 1632
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/detector_time_offsets-xyl6d4sclw HTTP/1.1" 404 139
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/distinct_channels-qzbcespqp3 HTTP/1.1" 404 135
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/energy_estimates-c2fdcw5wcr HTTP/1.1" 404 134
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/event_ambience-b7qndgom42 HTTP/1.1" 404 132
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/event_area_per_channel-ywkbseoogl HTTP/1.1" 404 140
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/event_n_channel-ywkbseoogl HTTP/1.1" 404 133
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/event_basics-o5culyytu5 HTTP/1.1" 404 130
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/event_info-5ovx4l2iul HTTP/1.1" 404 128
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/event_info_double-22sfnn5kim HTTP/1.1" 404 135
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/event_ms_naive-epbbboxkuj HTTP/1.1" 404 132
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/event_pattern_fit-cahobtcvru HTTP/1.1" 404 135
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/peak_per_event-klsx7noczh HTTP/1.1" 404 132
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/event_positions-dayaj7r4yj HTTP/1.1" 404 133
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/event_s1_positions_cnn-hvvmp65uq2 HTTP/1.1" 404 140
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/event_s2_positions_cnn-bh27t2hzlw HTTP/1.1" 404 140
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/event_s2_positions_gcn-d5gu4vewsv HTTP/1.1" 404 140
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/event_s2_positions_mlp-f6ytzu6qqn HTTP/1.1" 404 140
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/event_shadow-zum2aprycl HTTP/1.1" 404 130
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/event_top_bottom_params-urabkrxdqc HTTP/1.1" 404 141
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/event_waveform-loszf2t5d6 HTTP/1.1" 404 132
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/event_w_bayes_class-yhok6eq7b3 HTTP/1.1" 404 137
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/raw_records_diagnostic-5jnmouazik HTTP/1.1" 404 140
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/individual_peak_monitor-spjp2pl4hl HTTP/1.1" 404 141
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/afterpulses-uf63o427z7 HTTP/1.1" 404 129
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/led_calibration-lsdigsccxn HTTP/1.1" 404 133
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/event_local_min_info-hazyatbm5u HTTP/1.1" 404 138
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/merged_s2s-wpzg6lsm2m HTTP/1.1" 404 128
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/merged_s2s_he-hohrh5yh6o HTTP/1.1" 404 131
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/online_monitor_mv-if3qqljiby HTTP/1.1" 404 135
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/online_monitor_nv-z2cushqctj HTTP/1.1" 404 135
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/online_peak_monitor-etr2pjxerr HTTP/1.1" 404 137
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/peak_ambience-aglg7ad3jl HTTP/1.1" 404 131
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/peak_basics-6bdyxhzzfz HTTP/1.1" 404 129
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/peak_basics_he-4bjhf4bqfd HTTP/1.1" 404 132
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/peak_corrections-tqortpgfmp HTTP/1.1" 404 134
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/peak_positions_cnn-iongm54rho HTTP/1.1" 404 136
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/peak_positions_gcn-sgi5hgfujv HTTP/1.1" 404 136
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/peak_positions_mlp-zfge4gdktl HTTP/1.1" 404 136
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/peak_proximity-abfkn7nmwv HTTP/1.1" 404 132
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/peak_s1_positions_cnn-vfvtclcw3u HTTP/1.1" 404 139
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/peak_shadow-pctjku7ovb HTTP/1.1" 404 129
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/peak_top_bottom_params-366morvvx5 HTTP/1.1" 404 140
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/peaklet_classification-p3m6pr2fhz HTTP/1.1" 404 140
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/peaklet_classification_he-jc5epv3vtq HTTP/1.1" 404 143
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/peaklets-euocvpkv3y HTTP/1.1" 200 183
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/peaklets-euocvpkv3y/rules HTTP/1.1" 200 -
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/lone_hits-euocvpkv3y HTTP/1.1" 200 184
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/lone_hits-euocvpkv3y/rules HTTP/1.1" 200 -
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/peaklets_he-qsiaztqf6c HTTP/1.1" 404 129
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/peaks-5i3zhnt5vx HTTP/1.1" 404 123
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/peaks_he-iakptlspfu HTTP/1.1" 404 126
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/records-hwcemk7dbt HTTP/1.1" 404 125
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/veto_regions-hwcemk7dbt HTTP/1.1" 200 187
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:29 -0600] "GET /dids/xnt_049794/veto_regions-hwcemk7dbt/rules HTTP/1.1" 200 803
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:29 -0600] "GET /dids/xnt_049794/pulse_counts-hwcemk7dbt HTTP/1.1" 200 187
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:29 -0600] "GET /dids/xnt_049794/pulse_counts-hwcemk7dbt/rules HTTP/1.1" 200 803
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:29 -0600] "GET /dids/xnt_049794/records_he-cbkz5nmyc4 HTTP/1.1" 404 128
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:29 -0600] "GET /dids/xnt_049794/pulse_counts_he-cbkz5nmyc4 HTTP/1.1" 404 133
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:29 -0600] "GET /dids/xnt_049794/s2_recon_pos_diff-5srxt25yui HTTP/1.1" 404 135
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:29 -0600] "GET /dids/xnt_049794/veto_intervals-hyu26wo3bv HTTP/1.1" 404 132
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:29 -0600] "GET /dids/xnt_049794/veto_proximity-c2u2t7vf5z HTTP/1.1" 404 132
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:29 -0600] "GET /dids/xnt_049794/events_sync_mv-e3gwcxlgee HTTP/1.1" 404 132
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:29 -0600] "GET /dids/xnt_049794/events_mv-5wxwp4k6sz HTTP/1.1" 404 127
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:29 -0600] "GET /dids/xnt_049794/hitlets_mv-yf3ezxotyn HTTP/1.1" 404 128
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:29 -0600] "GET /dids/xnt_049794/records_mv-gsrdzepgyt HTTP/1.1" 404 128
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:29 -0600] "GET /dids/xnt_049794/event_positions_nv-l6qg7r6k5v HTTP/1.1" 404 136
mwt2-c071.campuscluster.illinois.edu...

It is suspected to be what strax's rucio backend is doing. The question here:

• Can we bypass this in strax?
• If not, can we download manually what we need and then not using rucio remote?