Jobsub ID 45047.0@dunegpschedd02.fnal.gov
| Jobsub ID | 45047.0@dunegpschedd02.fnal.gov | |
| Workflow ID | 2501 | |
| Stage ID | 1 | |
| User name | pgranger@fnal.gov | |
| HTCondor Group | group_dune | |
| Requested | Processors | 1 |
| GPU | No | |
| RSS bytes | 4194304000 (4000 MiB) | |
| Wall seconds limit | 18000 (5 hours) | |
| Submitted time | 2025-09-21 15:58:23 | |
| Site | CZ_FZU | |
| Entry | Nova_CZ_FZU_osgce1 | |
| Last heartbeat | 2025-09-21 16:36:56 | |
| From worker node | Hostname | lupin32.farm.particle.cz |
| cpuinfo | AMD EPYC 7551 32-Core Processor | |
| OS release | Scientific Linux release 7.9 (Nitrogen) | |
| Processors | 1 | |
| RSS bytes | 4194304000 (4000 MiB) | |
| Wall seconds limit | 172800 (48 hours) | |
| GPU | ||
| Inner Apptainer? | True | |
| Job state | outputting_failed | |
| Started | 2025-09-21 15:59:24 | |
| Input files | fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50542963_672_20231202T061509Z_gen_g4_detsim_hitreco__20240507T193856Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50602933_622_20231205T104854Z_gen_g4_detsim_hitreco__20240510T040509Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74553259_39_20231207T130457Z_gen_g4_detsim_hitreco__20240510T063233Z_reco2.root | |
| Jobscript | Exit code | 0 |
| Real time | 37m (2232s) | |
| CPU time | 35m (2154s = 96%) | |
| Max RSS bytes | 2135687168 (2036 MiB) | |
| Outputting started | 2025-09-21 16:36:37 | |
| Output files | ||
| Finished | 2025-09-21 16:36:56 | |
| List job events Cached HTCondor job logs | ||
Jobscript log (last 10,000 characters)
-- source:RootInput(read) 0.00109099 0.0166357 0.0495006 0.0156226 0.00897221 300 reco:pandora:StandardPandora 0.0127947 5.78533 916.14 1.77525 52.8763 300 reco:pandoraTrack:LArPandoraTrackCreation 0.000162721 0.00906526 0.60567 0.00161887 0.0398265 300 reco:pandoraShower:LArPandoraModularShowerCreation 0.000217761 0.0148925 0.405746 0.00336295 0.0396823 300 reco:pandoracalo:Calorimetry 0.000123321 0.0281427 2.16986 0.00113506 0.151617 300 reco:pandorapid:Chi2ParticleID 4.942e-05 0.000182162 0.00159749 0.00013944 0.000160243 300 reco:cvnmap:CVNMapper 2.3621e-05 0.0139376 0.0888986 0.013947 0.0138479 300 reco:cvneva:CVNEvaluator 2.294e-05 0.480225 2.74919 0.627349 0.328674 300 reco:energyrecnumu:EnergyReco 0.000340661 0.0582355 4.94946 0.0114126 0.318742 300 reco:energyrecnue:EnergyReco 0.000191781 0.00318102 0.243284 0.000653298 0.0150051 300 reco:energyrecnc:EnergyReco 0.00274338 0.0236477 0.393237 0.0162336 0.0308237 300 reco:energyrecnumurange:EnergyReco 0.000307071 0.045898 4.39749 0.00304145 0.280758 300 reco:energyrecnumumcs:EnergyReco 0.000528292 0.0469451 4.3751 0.00351492 0.280614 300 reco:energyrecnumumcsllhd:EnergyReco 0.000376982 0.047011 4.38686 0.0035107 0.281071 300 reco:opdec:Deconvolution 0.00430307 0.179444 0.384818 0.181333 0.0761053 300 reco:ophitspe:OpHitFinderDeco 0.00376679 0.0127527 0.040654 0.0113812 0.00565068 300 reco:opflash:OpFlashFinder 0.000176811 0.00138596 0.00322309 0.00133185 0.000582151 300 reco:opslicer:OpSlicer 5.329e-05 0.233312 0.834508 0.190177 0.188328 300 reco:rns:RandomNumberSaver 2.478e-05 6.00208e-05 0.00344979 4.631e-05 0.000197481 300 reco:anglereconue:NuAngularReco 0.00017761 0.00215934 0.168667 0.000463362 0.0104626 300 reco:anglereconumu:NuAngularReco 0.00010143 0.00189635 0.168478 0.000246321 0.0104112 300 reco:anglereconuepfps:NuAngularReco 0.000154371 0.00204993 0.166558 0.000367877 0.0103242 300 reco:anglereconumupfps:NuAngularReco 0.000141081 0.00196782 0.165708 0.000315602 0.0102757 300 reco:anglerecohits:NuAngularReco 0.00011347 0.00264305 0.177957 0.000717229 0.0111283 300 [art]:TriggerResults:TriggerResultInserter 1.128e-05 1.57568e-05 6.298e-05 1.4355e-05 4.8169e-06 300 end_path:cafmaker:CAFMaker 0.00164303 0.0282777 1.80387 0.00900123 0.113972 300 ======================================================================================================================================== ==================================================================================================== MemoryTracker summary (base-10 MB units used) Peak virtual memory usage (VmPeak) : 3806.92 MB Peak resident set size usage (VmHWM): 2135.69 MB ==================================================================================================== lar exit code 0 === Start last 100 lines of lar log file === Running Ophitfinder with InputDigiType = 'recob' Found hits: 521! Begin processing the 295th record. run: 74553259 subRun: 1 event: 3995 at 21-Sep-2025 18:35:21 CEST Boundary wire vector sizes: 8804, 6772, 5873 minwire 0: 337 minwire 1: 685 minwire 2: 250 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Classifier summary: Output 0: 0.0354474, 0.901691, 0.0628614, Output 1: 0.232579, 0.542351, 0.17412, 0.0509492, Output 2: 0.0309222, 0.166098, 0.399041, 0.403939, Running Ophitfinder with InputDigiType = 'recob' Found hits: 3347! Begin processing the 296th record. run: 74553259 subRun: 1 event: 3996 at 21-Sep-2025 18:36:16 CEST Boundary wire vector sizes: 167, 104, 152 minwire 0: 2602 minwire 1: 1056 minwire 2: 2289 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Classifier summary: Output 0: 0.0566831, 0.0316705, 0.911646, Output 1: 0.0427791, 0.936902, 0.0194608, 0.000857637, Output 2: 0.73562, 0.258243, 0.00605002, 8.72643e-05, Running Ophitfinder with InputDigiType = 'recob' Found hits: 1695! Begin processing the 297th record. run: 74553259 subRun: 1 event: 3997 at 21-Sep-2025 18:36:20 CEST Boundary wire vector sizes: 337, 300, 317 minwire 0: 980 minwire 1: 836 minwire 2: 625 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Classifier summary: Output 0: 0.951577, 0.00261064, 0.0458123, Output 1: 0.00729682, 0.0292322, 0.0683686, 0.895102, Output 2: 0.430758, 0.499074, 0.0601771, 0.00999095, Running Ophitfinder with InputDigiType = 'recob' Found hits: 2399! Begin processing the 298th record. run: 74553259 subRun: 1 event: 3998 at 21-Sep-2025 18:36:24 CEST Boundary wire vector sizes: 49, 45, 43 minwire 0: 1817 minwire 1: 879 minwire 2: 1648 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Used alternate method to get min and max wires due to vertex determination failure: 429, 728 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Classifier summary: Output 0: 0.0487592, 0.00451382, 0.946727, Output 1: 0.0314282, 0.114528, 0.4664, 0.387644, Output 2: 0.931954, 0.0650228, 0.00285332, 0.000170361, Running Ophitfinder with InputDigiType = 'recob' Found hits: 2130! Begin processing the 299th record. run: 74553259 subRun: 1 event: 3999 at 21-Sep-2025 18:36:27 CEST Boundary wire vector sizes: 1017, 1340, 1102 minwire 0: 1170 minwire 1: 1159 minwire 2: 848 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Classifier summary: Output 0: 0.0520666, 0.286317, 0.661616, Output 1: 0.122363, 0.519785, 0.303437, 0.0544148, Output 2: 0.0223131, 0.117398, 0.481828, 0.37846, Running Ophitfinder with InputDigiType = 'recob' Found hits: 3166! Begin processing the 300th record. run: 74553259 subRun: 1 event: 4000 at 21-Sep-2025 18:36:32 CEST Boundary wire vector sizes: 162, 222, 146 minwire 0: 639 minwire 1: 1983 minwire 2: 367 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Classifier summary: Output 0: 0.000830424, 0.993102, 0.00606763, Output 1: 0.988319, 0.011469, 0.000124604, 8.69971e-05, Output 2: 0.99842, 0.0015308, 3.73507e-05, 1.20683e-05, Running Ophitfinder with InputDigiType = 'recob' Found hits: 1338! Art has completed and will exit with status 0. === End last 100 lines of lar log file === processed files root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/cc/67/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50542963_672_20231202T061509Z_gen_g4_detsim_hitreco__20240507T193856Z_reco2.root root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/15/35/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50602933_622_20231205T104854Z_gen_g4_detsim_hitreco__20240510T040509Z_reco2.root root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/d6/7c/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74553259_39_20231207T130457Z_gen_g4_detsim_hitreco__20240510T063233Z_reco2.root .: total 13536 -rw-r--r-- 1 fermilab users 5144282 Sep 21 18:36 flatcaf.root -rw-r--r-- 1 fermilab users 4200059 Sep 21 18:36 caf.root -rw-r--r-- 1 fermilab users 4200059 Sep 21 18:36 caf_fd_hd_atmo_2501_20250921T155934Z.root -rw-r--r-- 1 fermilab users 190691 Sep 21 18:36 caf_20250921T155934Z.log -rw-r--r-- 1 fermilab users 73893 Sep 21 18:36 jobscript.log -rw-r--r-- 1 fermilab users 560 Sep 21 18:36 caf_20250921T155934Z.file -rw-r--r-- 1 fermilab users 560 Sep 21 18:36 caf_20250921T155934Z.pfns -rw-r--r-- 1 fermilab users 560 Sep 21 17:59 file.list -rw-r--r-- 1 fermilab users 560 Sep 21 18:36 justin-processed-pfns.txt -rw-r--r-- 1 fermilab users 413 Sep 21 17:59 all-input-dids.txt -rw-r--r-- 1 fermilab users 413 Sep 21 18:36 caf_20250921T155934Z.did -rw-r--r-- 1 fermilab users 413 Sep 21 17:59 did.list -rw-r--r-- 1 fermilab users 0 Sep 21 17:59 debugprod.log