justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 40051.132@dunegpschedd02.fnal.gov

Jobsub ID40051.132@dunegpschedd02.fnal.gov
Workflow ID2501
Stage ID1
User namepgranger@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-09-15 14:31:47
SiteUS_UChicago
EntryEngage_US_MWT2_uiuc_condce_mcore
Last heartbeat2025-09-15 16:12:40
From worker nodeHostnamemwt2-c070.campuscluster.illinois.edu
cpuinfoAMD EPYC 7443 24-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-09-15 14:33:37
Input filesfardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6492237_144_20231208T052441Z_gen_g4_detsim_hitreco__20240510T042645Z_reco2.root
fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6370891_237_20231201T105052Z_gen_g4_detsim_hitreco__20240507T202542Z_reco2.root
fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66004350_382_20231202T160820Z_gen_g4_detsim_hitreco__20240508T041454Z_reco2.root
fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74548636_673_20231207T073643Z_gen_g4_detsim_hitreco__20240510T063848Z_reco2.root
fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74521450_649_20231205T055144Z_gen_g4_detsim_hitreco__20240509T221954Z_reco2.root
fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6456061_385_20231207T123604Z_gen_g4_detsim_hitreco__20240510T055238Z_reco2.root
fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006610_965_20231203T025228Z_gen_g4_detsim_hitreco__20240508T071003Z_reco2.root
fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50571149_1035_20231202T192004Z_gen_g4_detsim_hitreco__20240508T061314Z_reco2.root
JobscriptExit code0
Real time1h (5927s)
CPU time41m (2469s = 41%)
Max RSS bytes2112364544 (2014 MiB)
Outputting started2025-09-15 16:12:24
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pgranger/cafs/fnal/02501/1/001/caf_fd_hd_atmo_2501_20250915T143350Z.root
Finished2025-09-15 16:12:40
Saved logsjustin-logs:40051.132-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

0.25078        800    
reco:energyrecnumurange:EnergyReco                    0.000269196    0.0366926      1.99764     0.00274294     0.142886        800    
reco:energyrecnumumcs:EnergyReco                      0.000515229    0.0367106      1.7198      0.00318128     0.135997        800    
reco:energyrecnumumcsllhd:EnergyReco                  0.000343396    0.0363584      1.6202       0.0031524     0.134664        800    
reco:opdec:Deconvolution                               0.316563      0.550109       1.15134      0.567518      0.109038        800    
reco:ophitspe:OpHitFinderDeco                          0.931393      0.974699       1.04445      0.965196      0.0305413       800    
reco:opflash:OpFlashFinder                            0.000191971   0.00149713     0.0050917    0.00145376    0.00060401       800    
reco:opslicer:OpSlicer                                4.9924e-05     0.250892       1.05723      0.209979      0.201586        800    
reco:rns:RandomNumberSaver                            2.4576e-05    6.5718e-05    0.000906544    5.882e-05    4.72667e-05      800    
reco:anglereconue:NuAngularReco                       0.000181251   0.00149108     0.0300066    0.000511101   0.00324827       800    
reco:anglereconumu:NuAngularReco                      6.9451e-05     0.001214      0.0295777    0.000270924   0.00313771       800    
reco:anglereconuepfps:NuAngularReco                   0.000112231   0.00134957     0.0297425    0.000405172   0.00319139       800    
reco:anglereconumupfps:NuAngularReco                  8.2686e-05    0.00125575     0.0297087    0.000324594   0.00313418       800    
reco:anglerecohits:NuAngularReco                      0.000107573   0.00193652     0.0329103    0.000795766   0.00372141       800    
[art]:TriggerResults:TriggerResultInserter             8.826e-06    1.65038e-05   0.000101631   1.5389e-05    6.10916e-06      800    
end_path:cafmaker:CAFMaker                            0.00138434     0.0207966      1.11798     0.00629294      0.0565         800    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3808.49 MB
  Peak resident set size usage (VmHWM): 2112.36 MB
====================================================================================================
lar exit code 0
=== Start last 100 lines of lar log file ===
Found hits: 1750!
Begin processing the 794th record. run: 50571149 subRun: 1 event: 103594 at 15-Sep-2025 11:11:41 CDT
Boundary wire vector sizes: 136, 105, 111
minwire 0: 1610
minwire 1: 1250
minwire 2: 1481
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.186128, 0.0351017, 0.778771, 
Output 1: 0.0262551, 0.0656885, 0.131123, 0.776934, 
Output 2: 0.612346, 0.360566, 0.0248365, 0.00225178, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2245!
Begin processing the 795th record. run: 50571149 subRun: 1 event: 103595 at 15-Sep-2025 11:11:47 CDT
Boundary wire vector sizes: 1022, 980, 894
minwire 0: 251
minwire 1: 2020
minwire 2: 154
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.001388, 0.992948, 0.00566372, 
Output 1: 0.0212234, 0.260797, 0.448287, 0.269692, 
Output 2: 0.91835, 0.0755623, 0.00570577, 0.000381841, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1738!
Begin processing the 796th record. run: 50571149 subRun: 1 event: 103596 at 15-Sep-2025 11:11:56 CDT
Boundary wire vector sizes: 140, 119, 53
minwire 0: 2229
minwire 1: 351
minwire 2: 2386
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000253298, 0.998712, 0.00103422, 
Output 1: 0.806922, 0.191619, 0.00126056, 0.000198917, 
Output 2: 0.999029, 0.00095524, 1.48496e-05, 1.11028e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 976!
Begin processing the 797th record. run: 50571149 subRun: 1 event: 103597 at 15-Sep-2025 11:12:02 CDT
Boundary wire vector sizes: 746, 620, 543
minwire 0: 2516
minwire 1: 502
minwire 2: 2482
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.974578, 0.00309181, 0.0223305, 
Output 1: 0.0870828, 0.497393, 0.364365, 0.051159, 
Output 2: 0.614719, 0.356464, 0.02785, 0.000967291, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1926!
Begin processing the 798th record. run: 50571149 subRun: 1 event: 103598 at 15-Sep-2025 11:12:08 CDT

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1048!
Begin processing the 799th record. run: 50571149 subRun: 1 event: 103599 at 15-Sep-2025 11:12:14 CDT
PandoraContentApi::GetList(*this, m_inputHitListName, pCaloHitList) return STATUS_CODE_NOT_INITIALIZED
    in function: GetVolumeIdToHitListMap
    in file:     /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/ALMA9/build/larpandoracontent/v04_16_00-buildFW/src/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 271
this->GetVolumeIdToHitListMap(volumeIdToHitListMap) return STATUS_CODE_NOT_INITIALIZED
    in function: Run
    in file:     /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/ALMA9/build/larpandoracontent/v04_16_00-buildFW/src/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 165
iter->second->Run() throw STATUS_CODE_NOT_INITIALIZED
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0003, LArDLMaster, STATUS_CODE_NOT_INITIALIZED

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 11!
Begin processing the 800th record. run: 50571149 subRun: 1 event: 103600 at 15-Sep-2025 11:12:17 CDT
Boundary wire vector sizes: 83, 131, 98
minwire 0: 1592
minwire 1: 430
minwire 2: 1902
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.189239, 0.0122274, 0.798533, 
Output 1: 0.0145885, 0.125439, 0.318695, 0.541277, 
Output 2: 0.197816, 0.61162, 0.172354, 0.0182099, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1150!
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
processed files
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/b9/e1/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6492237_144_20231208T052441Z_gen_g4_detsim_hitreco__20240510T042645Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/f4/b6/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6370891_237_20231201T105052Z_gen_g4_detsim_hitreco__20240507T202542Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/b2/bd/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66004350_382_20231202T160820Z_gen_g4_detsim_hitreco__20240508T041454Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/8c/d5/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74548636_673_20231207T073643Z_gen_g4_detsim_hitreco__20240510T063848Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/9a/68/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74521450_649_20231205T055144Z_gen_g4_detsim_hitreco__20240509T221954Z_reco2.root
root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/57/f5/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6456061_385_20231207T123604Z_gen_g4_detsim_hitreco__20240510T055238Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/d2/69/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006610_965_20231203T025228Z_gen_g4_detsim_hitreco__20240508T071003Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/56/67/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50571149_1035_20231202T192004Z_gen_g4_detsim_hitreco__20240508T061314Z_reco2.root
.:
total 37772
-rw-r--r-- 1 dune osgvo 14205856 Sep 15 11:12 flatcaf.root
-rw-r--r-- 1 dune osgvo 11882413 Sep 15 11:12 caf.root
-rw-r--r-- 1 dune osgvo 11882413 Sep 15 11:12 caf_fd_hd_atmo_2501_20250915T143350Z.root
-rw-r--r-- 1 dune osgvo   486747 Sep 15 11:12 caf_20250915T143350Z.log
-rw-r--r-- 1 dune osgvo   172144 Sep 15 11:12 jobscript.log
-rw-r--r-- 1 dune osgvo     1514 Sep 15 11:12 caf_20250915T143350Z.file
-rw-r--r-- 1 dune osgvo     1514 Sep 15 11:12 caf_20250915T143350Z.pfns
-rw-r--r-- 1 dune osgvo     1514 Sep 15 09:33 file.list
-rw-r--r-- 1 dune osgvo     1514 Sep 15 11:12 justin-processed-pfns.txt
-rw-r--r-- 1 dune osgvo     1102 Sep 15 09:33 all-input-dids.txt
-rw-r--r-- 1 dune osgvo     1102 Sep 15 11:12 caf_20250915T143350Z.did
-rw-r--r-- 1 dune osgvo     1102 Sep 15 09:33 did.list
-rw-r--r-- 1 dune osgvo        0 Sep 15 09:34 debugprod.log
justIN time: 2025-09-18 22:22:27 UTC       justIN version: 01.05.00