Jobsub ID 43720.0@dunegpschedd02.fnal.gov
| Jobsub ID | 43720.0@dunegpschedd02.fnal.gov | |
| Workflow ID | 2501 | |
| Stage ID | 1 | |
| User name | pgranger@fnal.gov | |
| HTCondor Group | group_dune | |
| Requested | Processors | 1 |
| GPU | No | |
| RSS bytes | 4194304000 (4000 MiB) | |
| Wall seconds limit | 18000 (5 hours) | |
| Submitted time | 2025-09-19 18:28:59 | |
| Site | NL_NIKHEF | |
| Entry | VIRGO_NL_NIKHEF_juk | |
| Last heartbeat | 2025-09-19 18:56:08 | |
| From worker node | Hostname | wn-sate-037.farm.nikhef.nl |
| cpuinfo | AMD EPYC 7551P 32-Core Processor | |
| OS release | Scientific Linux release 7.9 (Nitrogen) | |
| Processors | 1 | |
| RSS bytes | 4194304000 (4000 MiB) | |
| Wall seconds limit | 129600 (36 hours) | |
| GPU | ||
| Inner Apptainer? | True | |
| Job state | outputting_failed | |
| Started | 2025-09-19 18:29:29 | |
| Input files | fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6420351_959_20231204T044859Z_gen_g4_detsim_hitreco__20240509T201821Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6171714_372_20231126T014800Z_gen_g4_detsim_hitreco__20240507T183254Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74496296_693_20231201T212829Z_gen_g4_detsim_hitreco__20240508T045940Z_reco2.root | |
| Jobscript | Exit code | 0 |
| Real time | 26m (1580s) | |
| CPU time | 24m (1499s = 94%) | |
| Max RSS bytes | 2141032448 (2041 MiB) | |
| Outputting started | 2025-09-19 18:55:50 | |
| Output files | ||
| Finished | 2025-09-19 18:56:08 | |
| List job events Cached HTCondor job logs | ||
Jobscript log (last 10,000 characters)
0.0102058 2.54566 74.1463 1.63167 5.12656 400 reco:pandoraTrack:LArPandoraTrackCreation 0.0001327 0.00628969 0.139589 0.00127336 0.0164195 400 reco:pandoraShower:LArPandoraModularShowerCreation 0.00018073 0.0118647 0.484091 0.00269817 0.0317614 400 reco:pandoracalo:Calorimetry 0.00011156 0.0184037 1.06639 0.000875336 0.0831804 400 reco:pandorapid:Chi2ParticleID 4.0587e-05 0.000142488 0.00152838 0.000108975 0.000126117 400 reco:cvnmap:CVNMapper 2.0689e-05 0.00850114 0.0371018 0.0110105 0.00625698 400 reco:cvneva:CVNEvaluator 1.8766e-05 0.42483 2.26173 0.602809 0.298845 400 reco:energyrecnumu:EnergyReco 0.000282953 0.0319808 1.08928 0.0114588 0.0982329 400 reco:energyrecnue:EnergyReco 0.000180921 0.00236359 0.0507417 0.00057402 0.00565131 400 reco:energyrecnc:EnergyReco 0.00269258 0.0145662 0.101541 0.0127309 0.0133131 400 reco:energyrecnumurange:EnergyReco 0.000257644 0.0246091 1.0305 0.00289414 0.0918733 400 reco:energyrecnumumcs:EnergyReco 0.00055784 0.0254389 1.0023 0.00327619 0.0906631 400 reco:energyrecnumumcsllhd:EnergyReco 0.000429869 0.0254333 1.0042 0.00330813 0.0906764 400 reco:opdec:Deconvolution 0.0047548 0.168368 0.454052 0.16857 0.0744327 400 reco:ophitspe:OpHitFinderDeco 0.00237804 0.0084706 0.0309421 0.00848922 0.00258673 400 reco:opflash:OpFlashFinder 0.000118012 0.00116972 0.00329399 0.00113777 0.000559039 400 reco:opslicer:OpSlicer 4.1458e-05 0.22184 0.941588 0.172822 0.192661 400 reco:rns:RandomNumberSaver 1.8054e-05 2.76648e-05 0.000309211 2.68605e-05 1.44274e-05 400 reco:anglereconue:NuAngularReco 0.000147497 0.00146262 0.034898 0.000334294 0.00410443 400 reco:anglereconumu:NuAngularReco 9.2754e-05 0.0013513 0.0348001 0.000214493 0.00411784 400 reco:anglereconuepfps:NuAngularReco 0.000135575 0.0014651 0.0352878 0.000303652 0.00414397 400 reco:anglereconumupfps:NuAngularReco 0.000133481 0.00143718 0.0353059 0.00028186 0.00418428 400 reco:anglerecohits:NuAngularReco 0.000102964 0.00198514 0.0396811 0.000668402 0.00467987 400 [art]:TriggerResults:TriggerResultInserter 1.0821e-05 1.306e-05 6.2838e-05 1.2193e-05 3.49064e-06 400 end_path:cafmaker:CAFMaker 0.00103322 0.0185402 0.451713 0.00363598 0.0527918 400 ======================================================================================================================================== ==================================================================================================== MemoryTracker summary (base-10 MB units used) Peak virtual memory usage (VmPeak) : 3809.04 MB Peak resident set size usage (VmHWM): 2141.03 MB ==================================================================================================== lar exit code 0 === Start last 100 lines of lar log file === Boundary wire vector sizes: 49, 40, 44 minwire 0: 2125 minwire 1: 423 minwire 2: 2272 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Used alternate method to get min and max wires due to vertex determination failure: 0, 299 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Used alternate method to get min and max wires due to vertex determination failure: 104, 403 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Classifier summary: Output 0: 0.00326338, 0.00152735, 0.995209, Output 1: 0.154466, 0.840532, 0.00488792, 0.000113568, Output 2: 0.997712, 0.00218153, 0.000101616, 4.40257e-06, Running Ophitfinder with InputDigiType = 'recob' Found hits: 798! Begin processing the 393rd record. run: 74496296 subRun: 1 event: 69393 at 19-Sep-2025 20:55:21 CEST Boundary wire vector sizes: 232, 140, 153 minwire 0: 1193 minwire 1: 1736 minwire 2: 971 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Classifier summary: Output 0: 0.00124474, 0.99563, 0.00312499, Output 1: 0.234245, 0.760334, 0.00515393, 0.000267592, Output 2: 0.922881, 0.0756316, 0.00145315, 3.43146e-05, Running Ophitfinder with InputDigiType = 'recob' Found hits: 2032! Begin processing the 394th record. run: 74496296 subRun: 1 event: 69394 at 19-Sep-2025 20:55:24 CEST Running Ophitfinder with InputDigiType = 'recob' Found hits: 270! Begin processing the 395th record. run: 74496296 subRun: 1 event: 69395 at 19-Sep-2025 20:55:26 CEST Running Ophitfinder with InputDigiType = 'recob' Found hits: 540! Begin processing the 396th record. run: 74496296 subRun: 1 event: 69396 at 19-Sep-2025 20:55:28 CEST Boundary wire vector sizes: 41, 36, 35 minwire 0: 2199 minwire 1: 1282 minwire 2: 1807 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Used alternate method to get min and max wires due to vertex determination failure: 27, 326 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Used alternate method to get min and max wires due to vertex determination failure: 20, 319 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Classifier summary: Output 0: 0.197518, 0.0156228, 0.786859, Output 1: 0.135726, 0.463652, 0.354748, 0.0458744, Output 2: 0.833464, 0.161551, 0.00485709, 0.000128254, Running Ophitfinder with InputDigiType = 'recob' Found hits: 1423! Begin processing the 397th record. run: 74496296 subRun: 1 event: 69397 at 19-Sep-2025 20:55:31 CEST Running Ophitfinder with InputDigiType = 'recob' Found hits: 137! Begin processing the 398th record. run: 74496296 subRun: 1 event: 69398 at 19-Sep-2025 20:55:33 CEST Running Ophitfinder with InputDigiType = 'recob' Found hits: 617! Begin processing the 399th record. run: 74496296 subRun: 1 event: 69399 at 19-Sep-2025 20:55:34 CEST Boundary wire vector sizes: 4151, 3502, 2109 minwire 0: 2049 minwire 1: 503 minwire 2: 1715 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Classifier summary: Output 0: 0.0433205, 0.924046, 0.0326331, Output 1: 0.405869, 0.501377, 0.0687369, 0.0240171, Output 2: 0.0558826, 0.264429, 0.40476, 0.274929, Running Ophitfinder with InputDigiType = 'recob' Found hits: 2946! Begin processing the 400th record. run: 74496296 subRun: 1 event: 69400 at 19-Sep-2025 20:55:46 CEST Boundary wire vector sizes: 156, 75, 95 minwire 0: 1979 minwire 1: 150 minwire 2: 2365 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Classifier summary: Output 0: 0.939696, 0.00115273, 0.059151, Output 1: 0.775607, 0.195346, 0.0250803, 0.00396761, Output 2: 0.834705, 0.158209, 0.00661192, 0.000474047, Running Ophitfinder with InputDigiType = 'recob' Found hits: 818! Art has completed and will exit with status 0. === End last 100 lines of lar log file === processed files root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/c0/a3/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6420351_959_20231204T044859Z_gen_g4_detsim_hitreco__20240509T201821Z_reco2.root root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/00/fe/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6171714_372_20231126T014800Z_gen_g4_detsim_hitreco__20240507T183254Z_reco2.root root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/b4/e7/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74496296_693_20231201T212829Z_gen_g4_detsim_hitreco__20240508T045940Z_reco2.root .: total 18332 -rw-r--r--. 1 dune012 dune 6929322 Sep 19 20:55 flatcaf.root -rw-r--r--. 1 dune012 dune 5725552 Sep 19 20:55 caf.root -rw-r--r--. 1 dune012 dune 5725552 Sep 19 20:55 caf_fd_hd_atmo_2501_20250919T182938Z.root -rw-r--r--. 1 dune012 dune 242970 Sep 19 20:55 caf_20250919T182938Z.log -rw-r--r--. 1 dune012 dune 96289 Sep 19 20:55 jobscript.log -rw-r--r--. 1 dune012 dune 651 Sep 19 20:55 caf_20250919T182938Z.file -rw-r--r--. 1 dune012 dune 651 Sep 19 20:55 caf_20250919T182938Z.pfns -rw-r--r--. 1 dune012 dune 651 Sep 19 20:29 file.list -rw-r--r--. 1 dune012 dune 651 Sep 19 20:55 justin-processed-pfns.txt -rw-r--r--. 1 dune012 dune 412 Sep 19 20:29 all-input-dids.txt -rw-r--r--. 1 dune012 dune 412 Sep 19 20:55 caf_20250919T182938Z.did -rw-r--r--. 1 dune012 dune 412 Sep 19 20:29 did.list -rw-r--r--. 1 dune012 dune 0 Sep 19 20:29 debugprod.log