Jobsub ID 44225.1@dunegpschedd02.fnal.gov
| Jobsub ID | 44225.1@dunegpschedd02.fnal.gov | |
| Workflow ID | 2501 | |
| Stage ID | 1 | |
| User name | pgranger@fnal.gov | |
| HTCondor Group | group_dune | |
| Requested | Processors | 1 |
| GPU | No | |
| RSS bytes | 4194304000 (4000 MiB) | |
| Wall seconds limit | 18000 (5 hours) | |
| Submitted time | 2025-09-20 12:37:00 | |
| Site | NL_NIKHEF | |
| Entry | VIRGO_NL_NIKHEF_klomp | |
| Last heartbeat | 2025-09-20 12:53:04 | |
| From worker node | Hostname | wn-snel-032.farm.nikhef.nl |
| cpuinfo | AMD EPYC 7H12 64-Core Processor | |
| OS release | Scientific Linux release 7.9 (Nitrogen) | |
| Processors | 1 | |
| RSS bytes | 4194304000 (4000 MiB) | |
| Wall seconds limit | 129600 (36 hours) | |
| GPU | ||
| Inner Apptainer? | True | |
| Job state | outputting_failed | |
| Started | 2025-09-20 12:38:27 | |
| Input files | fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50579427_746_20231204T045457Z_gen_g4_detsim_hitreco__20240509T211006Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50577113_198_20231204T005852Z_gen_g4_detsim_hitreco__20240509T213421Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50568178_992_20231202T173341Z_gen_g4_detsim_hitreco__20240507T210931Z_reco2.root | |
| Jobscript | Exit code | 0 |
| Real time | 14m (858s) | |
| CPU time | 13m (827s = 96%) | |
| Max RSS bytes | 1991405568 (1899 MiB) | |
| Outputting started | 2025-09-20 12:52:46 | |
| Output files | ||
| Finished | 2025-09-20 12:53:04 | |
| List job events Cached HTCondor job logs | ||
Jobscript log (last 10,000 characters)
303 0.0537279 0.00143477 0.00816886 300 reco:pandora:StandardPandora 0.0072383 1.837 27.2226 1.11428 3.34627 300 reco:pandoraTrack:LArPandoraTrackCreation 0.000102201 0.00512785 0.122627 0.00100927 0.0131344 300 reco:pandoraShower:LArPandoraModularShowerCreation 0.000143108 0.0124814 0.419798 0.00227819 0.0357869 300 reco:pandoracalo:Calorimetry 8.0962e-05 0.0187501 0.579147 0.000685634 0.0718417 300 reco:pandorapid:Chi2ParticleID 3.0858e-05 0.000116976 0.00107846 8.9422e-05 9.96164e-05 300 reco:cvnmap:CVNMapper 1.606e-05 0.00866624 0.0311257 0.00896993 0.00732711 300 reco:cvneva:CVNEvaluator 1.4186e-05 0.22247 1.94751 0.319798 0.1811 300 reco:energyrecnumu:EnergyReco 0.000215393 0.0286161 1.23904 0.00911947 0.0903029 300 reco:energyrecnue:EnergyReco 0.000123101 0.00177059 0.0417212 0.000439098 0.00431495 300 reco:energyrecnc:EnergyReco 0.00183652 0.0131316 0.105399 0.00980936 0.0135276 300 reco:energyrecnumurange:EnergyReco 0.000164307 0.0230034 1.29962 0.00207589 0.0909534 300 reco:energyrecnumumcs:EnergyReco 0.000420137 0.023159 1.15385 0.00229093 0.084262 300 reco:energyrecnumumcsllhd:EnergyReco 0.000278892 0.0232964 1.15679 0.00227745 0.0848658 300 reco:opdec:Deconvolution 0.00340838 0.14622 0.35334 0.140342 0.065894 300 reco:ophitspe:OpHitFinderDeco 0.00178738 0.00692865 0.0252688 0.00673946 0.00236491 300 reco:opflash:OpFlashFinder 0.000120846 0.0009349 0.00328121 0.000863587 0.00044199 300 reco:opslicer:OpSlicer 2.9375e-05 0.182151 0.611908 0.132053 0.158511 300 reco:rns:RandomNumberSaver 1.5159e-05 2.57708e-05 0.000364713 2.433e-05 1.99521e-05 300 reco:anglereconue:NuAngularReco 0.000111469 0.00116816 0.0308259 0.000291932 0.00313798 300 reco:anglereconumu:NuAngularReco 6.2628e-05 0.00102708 0.0310264 0.000156147 0.00313278 300 reco:anglereconuepfps:NuAngularReco 0.000101039 0.00110919 0.0296375 0.000238066 0.00309191 300 reco:anglereconumupfps:NuAngularReco 9.0509e-05 0.00107271 0.029849 0.000205159 0.00311172 300 reco:anglerecohits:NuAngularReco 7.1854e-05 0.00155733 0.0335089 0.000553907 0.00354929 300 [art]:TriggerResults:TriggerResultInserter 8.405e-06 1.04152e-05 5.8459e-05 1.0064e-05 3.01981e-06 300 end_path:cafmaker:CAFMaker 0.00116244 0.0151665 0.392779 0.00327086 0.0402061 300 ======================================================================================================================================== ==================================================================================================== MemoryTracker summary (base-10 MB units used) Peak virtual memory usage (VmPeak) : 3653.76 MB Peak resident set size usage (VmHWM): 1991.41 MB ==================================================================================================== lar exit code 0 === Start last 100 lines of lar log file === Running Ophitfinder with InputDigiType = 'recob' Found hits: 1748! Begin processing the 292nd record. run: 50568178 subRun: 1 event: 99292 at 20-Sep-2025 14:52:24 CEST Running Ophitfinder with InputDigiType = 'recob' Found hits: 287! Begin processing the 293rd record. run: 50568178 subRun: 1 event: 99293 at 20-Sep-2025 14:52:25 CEST Boundary wire vector sizes: 84, 113, 78 minwire 0: 887 minwire 1: 1404 minwire 2: 1014 Used alternate method to get min and max wires due to vertex determination failure: 0, 299 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Classifier summary: Output 0: 0.00638547, 0.00772509, 0.985889, Output 1: 0.0915644, 0.891212, 0.0166865, 0.000536571, Output 2: 0.980999, 0.0185059, 0.000481943, 1.31462e-05, Running Ophitfinder with InputDigiType = 'recob' Found hits: 1196! Begin processing the 294th record. run: 50568178 subRun: 1 event: 99294 at 20-Sep-2025 14:52:27 CEST Running Ophitfinder with InputDigiType = 'recob' Found hits: 820! Begin processing the 295th record. run: 50568178 subRun: 1 event: 99295 at 20-Sep-2025 14:52:29 CEST Running Ophitfinder with InputDigiType = 'recob' Found hits: 1054! Begin processing the 296th record. run: 50568178 subRun: 1 event: 99296 at 20-Sep-2025 14:52:30 CEST Boundary wire vector sizes: 1244, 1402, 442 minwire 0: 1655 minwire 1: 763 minwire 2: 1455 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Classifier summary: Output 0: 0.995789, 0.00061824, 0.00359243, Output 1: 0.00568292, 0.0217519, 0.191194, 0.781371, Output 2: 0.991524, 0.00800556, 0.000396898, 7.34233e-05, Running Ophitfinder with InputDigiType = 'recob' Found hits: 3097! Begin processing the 297th record. run: 50568178 subRun: 1 event: 99297 at 20-Sep-2025 14:52:33 CEST Boundary wire vector sizes: 1755, 3104, 2526 minwire 0: 1202 minwire 1: 219 minwire 2: 774 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Classifier summary: Output 0: 0.985438, 0.0055065, 0.00905553, Output 1: 0.919392, 0.0771384, 0.00254101, 0.000928394, Output 2: 0.922847, 0.0750647, 0.0018755, 0.000212839, Running Ophitfinder with InputDigiType = 'recob' Found hits: 3044! Begin processing the 298th record. run: 50568178 subRun: 1 event: 99298 at 20-Sep-2025 14:52:36 CEST Boundary wire vector sizes: 152, 332, 132 minwire 0: 1392 minwire 1: 1087 minwire 2: 1478 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Classifier summary: Output 0: 0.989171, 0.000479397, 0.0103495, Output 1: 0.97649, 0.0224665, 0.000930119, 0.00011354, Output 2: 0.981791, 0.0178548, 0.000328657, 2.57998e-05, Running Ophitfinder with InputDigiType = 'recob' Found hits: 1514! Begin processing the 299th record. run: 50568178 subRun: 1 event: 99299 at 20-Sep-2025 14:52:38 CEST Running Ophitfinder with InputDigiType = 'recob' Found hits: 226! Begin processing the 300th record. run: 50568178 subRun: 1 event: 99300 at 20-Sep-2025 14:52:39 CEST Boundary wire vector sizes: 3273, 2327, 1987 minwire 0: 223 minwire 1: 1595 minwire 2: 0 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Classifier summary: Output 0: 0.0673395, 0.876406, 0.0562542, Output 1: 0.481672, 0.359168, 0.105883, 0.0532764, Output 2: 0.049262, 0.190414, 0.365224, 0.3951, Running Ophitfinder with InputDigiType = 'recob' Found hits: 2667! Art has completed and will exit with status 0. === End last 100 lines of lar log file === processed files root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/5e/f2/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50579427_746_20231204T045457Z_gen_g4_detsim_hitreco__20240509T211006Z_reco2.root root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/b3/f0/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50577113_198_20231204T005852Z_gen_g4_detsim_hitreco__20240509T213421Z_reco2.root root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/e3/12/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50568178_992_20231202T173341Z_gen_g4_detsim_hitreco__20240507T210931Z_reco2.root .: total 15052 -rw-r--r--. 1 dune012 dune 5719538 Sep 20 14:52 flatcaf.root -rw-r--r--. 1 dune012 dune 4693301 Sep 20 14:52 caf.root -rw-r--r--. 1 dune012 dune 4693301 Sep 20 14:52 caf_fd_hd_atmo_2501_20250920T123836Z.root -rw-r--r--. 1 dune012 dune 182623 Sep 20 14:52 caf_20250920T123836Z.log -rw-r--r--. 1 dune012 dune 75168 Sep 20 14:52 jobscript.log -rw-r--r--. 1 dune012 dune 651 Sep 20 14:52 caf_20250920T123836Z.file -rw-r--r--. 1 dune012 dune 651 Sep 20 14:52 caf_20250920T123836Z.pfns -rw-r--r--. 1 dune012 dune 651 Sep 20 14:38 file.list -rw-r--r--. 1 dune012 dune 651 Sep 20 14:52 justin-processed-pfns.txt -rw-r--r--. 1 dune012 dune 414 Sep 20 14:38 all-input-dids.txt -rw-r--r--. 1 dune012 dune 414 Sep 20 14:52 caf_20250920T123836Z.did -rw-r--r--. 1 dune012 dune 414 Sep 20 14:38 did.list -rw-r--r--. 1 dune012 dune 0 Sep 20 14:38 debugprod.log