Jobsub ID 43481.44@dunegpschedd02.fnal.gov
| Jobsub ID | 43481.44@dunegpschedd02.fnal.gov | |
| Workflow ID | 2501 | |
| Stage ID | 1 | |
| User name | pgranger@fnal.gov | |
| HTCondor Group | group_dune | |
| Requested | Processors | 1 |
| GPU | No | |
| RSS bytes | 4194304000 (4000 MiB) | |
| Wall seconds limit | 18000 (5 hours) | |
| Submitted time | 2025-09-19 14:54:46 | |
| Site | US_FNAL-T1 | |
| Entry | CMSHTPC_T1_US_FNAL_condce_opp1_whole | |
| Last heartbeat | 2025-09-19 15:50:42 | |
| From worker node | Hostname | dunegli-46117-0-cmswn7012.fnal.gov |
| cpuinfo | AMD EPYC 9534 64-Core Processor | |
| OS release | Scientific Linux release 7.9 (Nitrogen) | |
| Processors | 1 | |
| RSS bytes | 4194304000 (4000 MiB) | |
| Wall seconds limit | 171000 (47 hours) | |
| GPU | ||
| Inner Apptainer? | True | |
| Job state | outputting_failed | |
| Started | 2025-09-19 14:55:57 | |
| Input files | fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6422149_1179_20231204T120528Z_gen_g4_detsim_hitreco__20240509T215213Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50261805_121_20231122T093319Z_gen_g4_detsim_hitreco__20240503T055927Z_reco2.root | |
| Jobscript | Exit code | 0 |
| Real time | 50m (3008s) | |
| CPU time | 12m (727s = 24%) | |
| Max RSS bytes | 2100064256 (2002 MiB) | |
| Outputting started | 2025-09-19 15:46:06 | |
| Output files | ||
| Finished | 2025-09-19 15:50:42 | |
| List job events Cached HTCondor job logs | ||
Jobscript log (last 10,000 characters)
10.5553 190.12 9.1492 12.9726 200 ---------------------------------------------------------------------------------------------------------------------------------------- source:RootInput(read) 0.292011 0.453654 0.638086 0.341695 0.146956 200 reco:pandora:StandardPandora 1.75933 4.50896 178.838 3.24588 12.5103 200 reco:pandoraTrack:LArPandoraTrackCreation 0.000114863 0.00618389 0.172928 0.00123175 0.0172886 200 reco:pandoraShower:LArPandoraModularShowerCreation 0.000160913 0.0112185 0.451582 0.00238827 0.0379843 200 reco:pandoracalo:Calorimetry 8.9715e-05 0.0182739 1.2519 0.000858581 0.0960757 200 reco:pandorapid:Chi2ParticleID 3.3461e-05 0.00015685 0.00169664 0.000115189 0.000161492 200 reco:cvnmap:CVNMapper 2.2493e-05 0.0069798 0.0994157 0.00658836 0.00855837 200 reco:cvneva:CVNEvaluator 1.7497e-05 0.205825 1.1852 0.257734 0.141753 200 reco:energyrecnumu:EnergyReco 0.000232639 0.328996 1.81543 0.00968975 0.518822 200 reco:energyrecnue:EnergyReco 8.2353e-05 0.00237665 0.114106 0.00050522 0.00933121 200 reco:energyrecnc:EnergyReco 0.00151033 0.91787 2.25445 1.18413 0.58218 200 reco:energyrecnumurange:EnergyReco 0.000254133 0.0150462 0.439343 0.00205103 0.0514501 200 reco:energyrecnumumcs:EnergyReco 0.000452932 0.0152274 0.466458 0.00241306 0.0494799 200 reco:energyrecnumumcsllhd:EnergyReco 0.000248324 0.0154367 0.493061 0.00240775 0.0513914 200 reco:opdec:Deconvolution 0.880575 1.16512 1.70755 1.25899 0.169591 200 reco:ophitspe:OpHitFinderDeco 2.62158 2.64438 2.78069 2.6504 0.0232537 200 reco:opflash:OpFlashFinder 0.000196605 0.00125633 0.00610125 0.00116415 0.000666029 200 reco:opslicer:OpSlicer 6.5279e-05 0.216133 0.703567 0.164307 0.173386 200 reco:rns:RandomNumberSaver 2.9134e-05 4.92801e-05 0.000291117 4.332e-05 2.37947e-05 200 reco:anglereconue:NuAngularReco 0.000177617 0.0010968 0.0173794 0.00043204 0.00210414 200 reco:anglereconumu:NuAngularReco 5.6915e-05 0.000798619 0.0146251 0.000178153 0.0018969 200 reco:anglereconuepfps:NuAngularReco 0.000110487 0.000920631 0.0154767 0.000302886 0.00197073 200 reco:anglereconumupfps:NuAngularReco 7.7426e-05 0.000846242 0.0169457 0.000233886 0.00198445 200 reco:anglerecohits:NuAngularReco 0.00010644 0.00152936 0.0226326 0.000685864 0.00270755 200 [art]:TriggerResults:TriggerResultInserter 9.644e-06 1.56611e-05 7.302e-05 1.4247e-05 6.40955e-06 200 end_path:cafmaker:CAFMaker 0.00186841 0.0155284 0.254632 0.00592669 0.0297192 200 ======================================================================================================================================== ==================================================================================================== MemoryTracker summary (base-10 MB units used) Peak virtual memory usage (VmPeak) : 3862.37 MB Peak resident set size usage (VmHWM): 2100.06 MB ==================================================================================================== lar exit code 0 === Start last 100 lines of lar log file === Boundary wire vector sizes: 149, 130, 133 minwire 0: 741 minwire 1: 1883 minwire 2: 412 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Classifier summary: Output 0: 0.9869, 0.000418069, 0.0126822, Output 1: 0.991165, 0.00848693, 0.000216502, 0.000131875, Output 2: 0.988529, 0.011162, 0.000253782, 5.53193e-05, Running Ophitfinder with InputDigiType = 'recob' Found hits: 1415! Begin processing the 195th record. run: 50261805 subRun: 1 event: 12195 at 19-Sep-2025 15:44:46 UTC Boundary wire vector sizes: 772, 521, 552 minwire 0: 1799 minwire 1: 628 minwire 2: 1961 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Classifier summary: Output 0: 0.000200457, 0.99898, 0.000819936, Output 1: 0.00296839, 0.99291, 0.00398644, 0.000135229, Output 2: 0.998298, 0.00166108, 4.0143e-05, 1.15733e-06, Running Ophitfinder with InputDigiType = 'recob' Found hits: 2103! Begin processing the 196th record. run: 50261805 subRun: 1 event: 12196 at 19-Sep-2025 15:45:00 UTC Boundary wire vector sizes: 348, 430, 386 minwire 0: 2088 minwire 1: 209 minwire 2: 2298 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Classifier summary: Output 0: 0.000381226, 0.997929, 0.00168987, Output 1: 0.00403363, 0.990869, 0.00494466, 0.000152424, Output 2: 0.997469, 0.00246639, 6.24716e-05, 1.81159e-06, Running Ophitfinder with InputDigiType = 'recob' Found hits: 1134! Begin processing the 197th record. run: 50261805 subRun: 1 event: 12197 at 19-Sep-2025 15:45:13 UTC Running Ophitfinder with InputDigiType = 'recob' Found hits: 282! Begin processing the 198th record. run: 50261805 subRun: 1 event: 12198 at 19-Sep-2025 15:45:25 UTC Boundary wire vector sizes: 68, 162, 127 minwire 0: 1514 minwire 1: 1980 minwire 2: 1031 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Classifier summary: Output 0: 0.988351, 0.000677095, 0.0109718, Output 1: 0.00392898, 0.993208, 0.00278686, 7.61931e-05, Output 2: 0.987808, 0.0118969, 0.000288904, 5.93126e-06, Running Ophitfinder with InputDigiType = 'recob' Found hits: 2048! Begin processing the 199th record. run: 50261805 subRun: 1 event: 12199 at 19-Sep-2025 15:45:38 UTC Boundary wire vector sizes: 297, 366, 188 minwire 0: 485 minwire 1: 1879 minwire 2: 488 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Classifier summary: Output 0: 0.00102219, 0.985305, 0.0136725, Output 1: 0.363232, 0.614673, 0.0208941, 0.00120134, Output 2: 0.995604, 0.00429122, 9.97887e-05, 4.83451e-06, Running Ophitfinder with InputDigiType = 'recob' Found hits: 1510! Begin processing the 200th record. run: 50261805 subRun: 1 event: 12200 at 19-Sep-2025 15:45:52 UTC Boundary wire vector sizes: 191, 182, 166 minwire 0: 202 minwire 1: 1839 minwire 2: 196 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Classifier summary: Output 0: 0.00355914, 0.00430539, 0.992135, Output 1: 0.966564, 0.032092, 0.000869228, 0.000475138, Output 2: 0.99728, 0.00263394, 6.9605e-05, 1.6088e-05, Running Ophitfinder with InputDigiType = 'recob' Found hits: 934! Art has completed and will exit with status 0. === End last 100 lines of lar log file === processed files root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/83/3f/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6422149_1179_20231204T120528Z_gen_g4_detsim_hitreco__20240509T215213Z_reco2.root root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/3f/09/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50261805_121_20231122T093319Z_gen_g4_detsim_hitreco__20240503T055927Z_reco2.root .: total 9264 -rw-r--r-- 1 dunegli fnalgrid 3552013 Sep 19 15:46 flatcaf.root -rw-r--r-- 1 dunegli fnalgrid 2856763 Sep 19 15:46 caf.root -rw-r--r-- 1 dunegli fnalgrid 2856763 Sep 19 15:46 caf_fd_hd_atmo_2501_20250919T145600Z.root -rw-r--r-- 1 dunegli fnalgrid 133406 Sep 19 15:46 caf_20250919T145600Z.log -rw-r--r-- 1 dunegli fnalgrid 49066 Sep 19 15:46 jobscript.log -rw-r--r-- 1 dunegli fnalgrid 374 Sep 19 15:46 caf_20250919T145600Z.file -rw-r--r-- 1 dunegli fnalgrid 374 Sep 19 15:46 caf_20250919T145600Z.pfns -rw-r--r-- 1 dunegli fnalgrid 374 Sep 19 14:55 file.list -rw-r--r-- 1 dunegli fnalgrid 374 Sep 19 15:46 justin-processed-pfns.txt -rw-r--r-- 1 dunegli fnalgrid 276 Sep 19 14:55 all-input-dids.txt -rw-r--r-- 1 dunegli fnalgrid 276 Sep 19 15:46 caf_20250919T145600Z.did -rw-r--r-- 1 dunegli fnalgrid 276 Sep 19 14:55 did.list -rw-r--r-- 1 dunegli fnalgrid 0 Sep 19 14:56 debugprod.log