Jobsub ID 47605.3@dunegpschedd01.fnal.gov
| Jobsub ID | 47605.3@dunegpschedd01.fnal.gov | |
| Workflow ID | 2501 | |
| Stage ID | 1 | |
| User name | pgranger@fnal.gov | |
| HTCondor Group | group_dune | |
| Requested | Processors | 1 |
| GPU | No | |
| RSS bytes | 4194304000 (4000 MiB) | |
| Wall seconds limit | 18000 (5 hours) | |
| Submitted time | 2025-09-17 15:34:21 | |
| Site | US_FNAL-FermiGrid | |
| Entry | FNAL_GPGrid_ce03_mcore_op_duneonly | |
| Last heartbeat | 2025-09-17 17:15:07 | |
| From worker node | Hostname | dunegli-6904182-0-fnpc19141.fnal.gov |
| cpuinfo | AMD EPYC 7502 32-Core Processor | |
| OS release | Scientific Linux release 7.9 (Nitrogen) | |
| Processors | 1 | |
| RSS bytes | 4194304000 (4000 MiB) | |
| Wall seconds limit | 172800 (48 hours) | |
| GPU | ||
| Inner Apptainer? | True | |
| Job state | stalled | |
| Started | 2025-09-17 15:35:06 | |
| Input files | fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66020489_406_20231205T025708Z_gen_g4_detsim_hitreco__20240510T020510Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6220954_149_20231122T081113Z_gen_g4_detsim_hitreco__20240503T064902Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50601160_140_20231204T232440Z_gen_g4_detsim_hitreco__20240509T222405Z_reco2.root | |
| Outputting started | 2025-09-17 16:13:02 | |
| Output files | ||
| Finished | 2025-09-17 18:10:49 | |
| List job events Cached HTCondor job logs | ||
Jobscript log (last 10,000 characters)
------------------------------ source:RootInput(read) 0.105981 0.175933 1.08759 0.144661 0.0754305 300 reco:pandora:StandardPandora 0.655626 2.95205 136.815 1.52855 9.09752 300 reco:pandoraTrack:LArPandoraTrackCreation 9.9941e-05 0.00677748 0.23849 0.000923315 0.0224634 300 reco:pandoraShower:LArPandoraModularShowerCreation 0.000140941 0.00871336 0.117004 0.00183656 0.0177364 300 reco:pandoracalo:Calorimetry 8.3461e-05 0.0241601 1.2187 0.000620002 0.107926 300 reco:pandorapid:Chi2ParticleID 3.1931e-05 0.000114078 0.000808518 8.5151e-05 0.000102246 300 reco:cvnmap:CVNMapper 1.573e-05 0.00691266 0.0334591 0.00801612 0.00539302 300 reco:cvneva:CVNEvaluator 1.409e-05 0.201821 1.52522 0.274662 0.146776 300 reco:energyrecnumu:EnergyReco 0.000207892 0.177036 3.12162 0.0068383 0.307413 300 reco:energyrecnue:EnergyReco 0.000111411 0.00235753 0.0857234 0.000382234 0.00801695 300 reco:energyrecnc:EnergyReco 0.00171827 0.402953 1.37608 0.441347 0.28879 300 reco:energyrecnumurange:EnergyReco 0.000133901 0.0394221 2.93483 0.00177641 0.215462 300 reco:energyrecnumumcs:EnergyReco 0.000370674 0.0400827 2.93523 0.00209416 0.215489 300 reco:energyrecnumumcsllhd:EnergyReco 0.000235982 0.0401641 2.93589 0.00209754 0.215871 300 reco:opdec:Deconvolution 0.333302 0.530406 1.12054 0.530893 0.107158 300 reco:ophitspe:OpHitFinderDeco 0.947565 0.966434 0.989692 0.970519 0.0116137 300 reco:opflash:OpFlashFinder 0.000216362 0.000893854 0.00252718 0.000831178 0.000407502 300 reco:opslicer:OpSlicer 0.000189102 0.172986 0.781702 0.125284 0.150302 300 reco:rns:RandomNumberSaver 1.384e-05 2.00295e-05 0.000205682 1.7445e-05 1.16849e-05 300 reco:anglereconue:NuAngularReco 8.3111e-05 0.00164526 0.0681024 0.000222777 0.00635654 300 reco:anglereconumu:NuAngularReco 5.6281e-05 0.00154697 0.0679225 0.000136271 0.00633465 300 reco:anglereconuepfps:NuAngularReco 8.8101e-05 0.00162899 0.0677729 0.000207432 0.00633506 300 reco:anglereconumupfps:NuAngularReco 8.1131e-05 0.00161178 0.0677819 0.000183056 0.00639769 300 reco:anglerecohits:NuAngularReco 6.5101e-05 0.00206418 0.0732689 0.00044001 0.00688892 300 [art]:TriggerResults:TriggerResultInserter 7.08e-06 8.58413e-06 2.815e-05 8.15e-06 1.62945e-06 300 end_path:cafmaker:CAFMaker 0.00097411 0.0213997 1.02333 0.00277227 0.0836099 300 ======================================================================================================================================== ==================================================================================================== MemoryTracker summary (base-10 MB units used) Peak virtual memory usage (VmPeak) : 3879.19 MB Peak resident set size usage (VmHWM): 2215.14 MB ==================================================================================================== lar exit code 0 === Start last 100 lines of lar log file === Boundary wire vector sizes: 346, 186, 118 minwire 0: 156 minwire 1: 2572 minwire 2: 0 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Classifier summary: Output 0: 0.994759, 0.000344806, 0.00489575, Output 1: 0.956604, 0.0417364, 0.00134636, 0.000312847, Output 2: 0.99249, 0.00732021, 0.00017034, 1.97878e-05, Running Ophitfinder with InputDigiType = 'recob' Found hits: 793! Begin processing the 295th record. run: 50601160 subRun: 1 event: 14095 at 17-Sep-2025 16:12:29 UTC Boundary wire vector sizes: 295, 159, 93 minwire 0: 1955 minwire 1: 1308 minwire 2: 1815 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Classifier summary: Output 0: 0.972259, 0.000662477, 0.0270789, Output 1: 0.992075, 0.00738106, 0.000390899, 0.000153096, Output 2: 0.979805, 0.0197459, 0.000386432, 6.32548e-05, Running Ophitfinder with InputDigiType = 'recob' Found hits: 1544! Begin processing the 296th record. run: 50601160 subRun: 1 event: 14096 at 17-Sep-2025 16:12:34 UTC Boundary wire vector sizes: 79, 76, 70 minwire 0: 900 minwire 1: 2756 minwire 2: 205 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Classifier summary: Output 0: 0.000271072, 0.998129, 0.00159967, Output 1: 0.280513, 0.713316, 0.0058936, 0.000277872, Output 2: 0.998926, 0.00104825, 2.47582e-05, 9.30839e-07, Running Ophitfinder with InputDigiType = 'recob' Found hits: 990! Begin processing the 297th record. run: 50601160 subRun: 1 event: 14097 at 17-Sep-2025 16:12:40 UTC Running Ophitfinder with InputDigiType = 'recob' Found hits: 591! Begin processing the 298th record. run: 50601160 subRun: 1 event: 14098 at 17-Sep-2025 16:12:45 UTC Boundary wire vector sizes: 181, 148, 139 minwire 0: 1871 minwire 1: 1125 minwire 2: 1862 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Classifier summary: Output 0: 0.00272543, 0.00273414, 0.99454, Output 1: 0.991486, 0.00818928, 0.000201928, 0.000123236, Output 2: 0.997407, 0.00254658, 3.73137e-05, 8.67955e-06, Running Ophitfinder with InputDigiType = 'recob' Found hits: 1534! Begin processing the 299th record. run: 50601160 subRun: 1 event: 14099 at 17-Sep-2025 16:12:50 UTC Boundary wire vector sizes: 141, 109, 110 minwire 0: 1412 minwire 1: 1772 minwire 2: 1089 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Classifier summary: Output 0: 0.0046854, 0.0513222, 0.943992, Output 1: 0.992671, 0.00681417, 0.000364683, 0.000149957, Output 2: 0.994472, 0.00542495, 8.32555e-05, 2.03348e-05, Running Ophitfinder with InputDigiType = 'recob' Found hits: 1700! Begin processing the 300th record. run: 50601160 subRun: 1 event: 14100 at 17-Sep-2025 16:12:56 UTC Boundary wire vector sizes: 469, 312, 293 minwire 0: 244 minwire 1: 2055 minwire 2: 285 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Classifier summary: Output 0: 0.0432232, 0.79204, 0.164737, Output 1: 0.0111043, 0.242148, 0.701996, 0.0447521, Output 2: 0.87697, 0.118695, 0.00426392, 7.03142e-05, Running Ophitfinder with InputDigiType = 'recob' Found hits: 1269! Art has completed and will exit with status 0. === End last 100 lines of lar log file === processed files root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/86/78/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66020489_406_20231205T025708Z_gen_g4_detsim_hitreco__20240510T020510Z_reco2.root root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/31/1f/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6220954_149_20231122T081113Z_gen_g4_detsim_hitreco__20240503T064902Z_reco2.root root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/a1/fd/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50601160_140_20231204T232440Z_gen_g4_detsim_hitreco__20240509T222405Z_reco2.root .: total 17420 -rw-r--r-- 1 dunegli fnalgrid 6587518 Sep 17 16:13 flatcaf.root -rw-r--r-- 1 dunegli fnalgrid 5439936 Sep 17 16:13 caf.root -rw-r--r-- 1 dunegli fnalgrid 5439936 Sep 17 16:13 caf_fd_hd_atmo_2501_20250917T153510Z.root -rw-r--r-- 1 dunegli fnalgrid 185363 Sep 17 16:13 caf_20250917T153510Z.log -rw-r--r-- 1 dunegli fnalgrid 76106 Sep 17 16:13 jobscript.log -rw-r--r-- 1 dunegli fnalgrid 560 Sep 17 16:13 caf_20250917T153510Z.file -rw-r--r-- 1 dunegli fnalgrid 560 Sep 17 16:13 caf_20250917T153510Z.pfns -rw-r--r-- 1 dunegli fnalgrid 560 Sep 17 15:35 file.list -rw-r--r-- 1 dunegli fnalgrid 560 Sep 17 16:13 justin-processed-pfns.txt -rw-r--r-- 1 dunegli fnalgrid 413 Sep 17 15:35 all-input-dids.txt -rw-r--r-- 1 dunegli fnalgrid 413 Sep 17 16:13 caf_20250917T153510Z.did -rw-r--r-- 1 dunegli fnalgrid 413 Sep 17 15:35 did.list -rw-r--r-- 1 dunegli fnalgrid 0 Sep 17 15:35 debugprod.log