Jobsub ID 45491.39@dunegpschedd02.fnal.gov
| Jobsub ID | 45491.39@dunegpschedd02.fnal.gov | |
| Workflow ID | 2501 | |
| Stage ID | 1 | |
| User name | pgranger@fnal.gov | |
| HTCondor Group | group_dune | |
| Requested | Processors | 1 |
| GPU | No | |
| RSS bytes | 4194304000 (4000 MiB) | |
| Wall seconds limit | 18000 (5 hours) | |
| Submitted time | 2025-09-22 03:15:00 | |
| Site | UK_Durham | |
| Entry | DUNE_UK_SGridDurham_ce4 | |
| Last heartbeat | 2025-09-22 03:43:04 | |
| From worker node | Hostname | n256.dur.scotgrid.ac.uk |
| cpuinfo | AMD EPYC 7763 64-Core Processor | |
| OS release | Scientific Linux release 7.9 (Nitrogen) | |
| Processors | 1 | |
| RSS bytes | 4194304000 (4000 MiB) | |
| Wall seconds limit | 171000 (47 hours) | |
| GPU | ||
| Inner Apptainer? | True | |
| Job state | outputting_failed | |
| Started | 2025-09-22 03:15:44 | |
| Input files | fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6220762_414_20231122T074939Z_gen_g4_detsim_hitreco__20240503T064938Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6406500_973_20231202T131420Z_gen_g4_detsim_hitreco__20240507T222328Z_reco2.root | |
| Jobscript | Exit code | 0 |
| Real time | 26m (1611s) | |
| CPU time | 16m (1005s = 62%) | |
| Max RSS bytes | 2156851200 (2056 MiB) | |
| Outputting started | 2025-09-22 03:42:36 | |
| Output files | ||
| Finished | 2025-09-22 03:43:04 | |
| List job events Cached HTCondor job logs | ||
Jobscript log (last 10,000 characters)
----------------------------------------------------------------------------------------------------------- source:RootInput(read) 0.0213415 0.0501021 0.103596 0.0532753 0.0155096 200 reco:pandora:StandardPandora 0.132142 4.65444 288.385 1.32198 25.6698 200 reco:pandoraTrack:LArPandoraTrackCreation 9.7053e-05 0.00711131 0.281326 0.00110553 0.0245113 200 reco:pandoraShower:LArPandoraModularShowerCreation 0.000128813 0.0111909 0.215112 0.00248299 0.0275105 200 reco:pandoracalo:Calorimetry 8.3928e-05 0.0333734 1.62356 0.000817954 0.156127 200 reco:pandorapid:Chi2ParticleID 3.1179e-05 0.000114603 0.00145215 8.52205e-05 0.000129637 200 reco:cvnmap:CVNMapper 1.5529e-05 0.0112134 0.0785689 0.00689251 0.0112105 200 reco:cvneva:CVNEvaluator 1.4838e-05 0.30698 2.93403 0.320208 0.269799 200 reco:energyrecnumu:EnergyReco 0.000199185 0.0681652 1.81236 0.0110305 0.148722 200 reco:energyrecnue:EnergyReco 0.000112371 0.00252316 0.115237 0.000385195 0.00953064 200 reco:energyrecnc:EnergyReco 0.00143655 0.147406 1.20564 0.139472 0.139239 200 reco:energyrecnumurange:EnergyReco 0.000166193 0.0338243 1.61051 0.00162831 0.133099 200 reco:energyrecnumumcs:EnergyReco 0.000403449 0.0346569 1.86631 0.00217867 0.147151 200 reco:energyrecnumumcsllhd:EnergyReco 0.000295696 0.0337176 1.96039 0.00324909 0.150667 200 reco:opdec:Deconvolution 0.090944 0.306687 0.785702 0.2949 0.113879 200 reco:ophitspe:OpHitFinderDeco 0.18858 0.200253 0.235127 0.195648 0.0102197 200 reco:opflash:OpFlashFinder 9.05e-05 0.00103275 0.00976063 0.000838779 0.0011398 200 reco:opslicer:OpSlicer 3.0978e-05 0.247388 1.44608 0.194139 0.205824 200 reco:rns:RandomNumberSaver 1.5478e-05 4.42828e-05 0.00406061 2.2567e-05 0.000285456 200 reco:anglereconue:NuAngularReco 0.000117882 0.00178789 0.109626 0.000281509 0.00835029 200 reco:anglereconumu:NuAngularReco 6.2487e-05 0.00172499 0.0817793 0.000169423 0.00679025 200 reco:anglereconuepfps:NuAngularReco 9.4538e-05 0.00166575 0.0725984 0.00024161 0.0061907 200 reco:anglereconumupfps:NuAngularReco 8.5952e-05 0.00154895 0.0726005 0.00021508 0.00616604 200 reco:anglerecohits:NuAngularReco 7.5973e-05 0.00229421 0.0792098 0.00049476 0.00730237 200 [art]:TriggerResults:TriggerResultInserter 8.276e-06 9.68326e-06 4.1318e-05 9.2625e-06 2.74273e-06 200 end_path:cafmaker:CAFMaker 0.000931974 0.0219487 0.894815 0.00326542 0.0810582 200 ======================================================================================================================================== ==================================================================================================== MemoryTracker summary (base-10 MB units used) Peak virtual memory usage (VmPeak) : 3821.1 MB Peak resident set size usage (VmHWM): 2156.85 MB ==================================================================================================== lar exit code 0 === Start last 100 lines of lar log file === Found hits: 1646! Begin processing the 195th record. run: 6406500 subRun: 1 event: 97395 at 22-Sep-2025 04:42:12 BST Boundary wire vector sizes: 386, 437, 253 minwire 0: 780 minwire 1: 1868 minwire 2: 684 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Classifier summary: Output 0: 0.015913, 0.97136, 0.0127273, Output 1: 0.0138608, 0.910796, 0.0721256, 0.00321732, Output 2: 0.128974, 0.835864, 0.0348387, 0.000323715, Running Ophitfinder with InputDigiType = 'recob' Found hits: 1884! Begin processing the 196th record. run: 6406500 subRun: 1 event: 97396 at 22-Sep-2025 04:42:16 BST Boundary wire vector sizes: 93, 131, 116 minwire 0: 988 minwire 1: 1148 minwire 2: 716 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Classifier summary: Output 0: 0.963538, 0.000707044, 0.035755, Output 1: 0.00234927, 0.995939, 0.00164764, 6.42869e-05, Output 2: 0.939412, 0.0592542, 0.00131369, 2.01598e-05, Running Ophitfinder with InputDigiType = 'recob' Found hits: 1976! Begin processing the 197th record. run: 6406500 subRun: 1 event: 97397 at 22-Sep-2025 04:42:19 BST Boundary wire vector sizes: 76, 107, 76 minwire 0: 2239 minwire 1: 506 minwire 2: 2339 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Classifier summary: Output 0: 0.252503, 0.041666, 0.705831, Output 1: 0.0316262, 0.293239, 0.554232, 0.120903, Output 2: 0.372119, 0.550938, 0.0741982, 0.00274439, Running Ophitfinder with InputDigiType = 'recob' Found hits: 1224! Begin processing the 198th record. run: 6406500 subRun: 1 event: 97398 at 22-Sep-2025 04:42:22 BST Boundary wire vector sizes: 1915, 1522, 959 minwire 0: 998 minwire 1: 1952 minwire 2: 485 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Classifier summary: Output 0: 0.0295903, 0.942438, 0.0279721, Output 1: 0.0682611, 0.406576, 0.339185, 0.185978, Output 2: 0.104261, 0.344821, 0.385247, 0.16567, Running Ophitfinder with InputDigiType = 'recob' Found hits: 2967! Begin processing the 199th record. run: 6406500 subRun: 1 event: 97399 at 22-Sep-2025 04:42:29 BST Boundary wire vector sizes: 47, 37, 36 minwire 0: 1905 minwire 1: 561 minwire 2: 1930 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Used alternate method to get min and max wires due to vertex determination failure: 0, 299 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Used alternate method to get min and max wires due to vertex determination failure: 267, 566 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Classifier summary: Output 0: 0.134628, 0.0154062, 0.849966, Output 1: 0.25881, 0.735101, 0.00579744, 0.000290759, Output 2: 0.681846, 0.311516, 0.00652152, 0.000116097, Running Ophitfinder with InputDigiType = 'recob' Found hits: 1050! Begin processing the 200th record. run: 6406500 subRun: 1 event: 97400 at 22-Sep-2025 04:42:31 BST Boundary wire vector sizes: 189, 107, 94 minwire 0: 1203 minwire 1: 1793 minwire 2: 972 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Classifier summary: Output 0: 0.994178, 0.000458455, 0.00536315, Output 1: 0.97921, 0.020554, 0.000167042, 6.85623e-05, Output 2: 0.996894, 0.00302794, 6.55356e-05, 1.27194e-05, Running Ophitfinder with InputDigiType = 'recob' Found hits: 1754! Art has completed and will exit with status 0. === End last 100 lines of lar log file === processed files root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/9c/c6/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6220762_414_20231122T074939Z_gen_g4_detsim_hitreco__20240503T064938Z_reco2.root root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/5b/ae/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6406500_973_20231202T131420Z_gen_g4_detsim_hitreco__20240507T222328Z_reco2.root .: total 11100 -rw-r--r--. 1 dune001 dune 4246416 Sep 22 04:42 flatcaf.root -rw-r--r--. 1 dune001 dune 3441470 Sep 22 04:42 caf.root -rw-r--r--. 1 dune001 dune 3441470 Sep 22 04:42 caf_fd_hd_atmo_2501_20250922T031554Z.root -rw-r--r--. 1 dune001 dune 133044 Sep 22 04:42 caf_20250922T031554Z.log -rw-r--r--. 1 dune001 dune 56325 Sep 22 04:42 jobscript.log -rw-r--r--. 1 dune001 dune 432 Sep 22 04:42 caf_20250922T031554Z.file -rw-r--r--. 1 dune001 dune 432 Sep 22 04:42 caf_20250922T031554Z.pfns -rw-r--r--. 1 dune001 dune 432 Sep 22 04:15 file.list -rw-r--r--. 1 dune001 dune 432 Sep 22 04:42 justin-processed-pfns.txt -rw-r--r--. 1 dune001 dune 274 Sep 22 04:15 all-input-dids.txt -rw-r--r--. 1 dune001 dune 274 Sep 22 04:42 caf_20250922T031554Z.did -rw-r--r--. 1 dune001 dune 274 Sep 22 04:15 did.list -rw-r--r--. 1 dune001 dune 0 Sep 22 04:16 debugprod.log