justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 40063.59@dunegpschedd02.fnal.gov

Jobsub ID40063.59@dunegpschedd02.fnal.gov
Workflow ID2501
Stage ID1
User namepgranger@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-09-15 14:50:48
SiteUK_Bristol
EntryCMSHTPC_T2_UK_SGrid_Bristol_lcgce02
Last heartbeat2025-09-15 16:31:02
From worker nodeHostnamehd87.dice.priv
cpuinfoAMD EPYC 7702P 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit259200 (72 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-09-15 14:52:25
Input filesfardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006610_664_20231202T203545Z_gen_g4_detsim_hitreco__20240508T070002Z_reco2.root
fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6413290_248_20231202T202528Z_gen_g4_detsim_hitreco__20240508T061715Z_reco2.root
fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6468412_786_20231207T081349Z_gen_g4_detsim_hitreco__20240510T021938Z_reco2.root
fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50581281_781_20231204T052622Z_gen_g4_detsim_hitreco__20240509T214033Z_reco2.root
fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74521450_1191_20231205T102122Z_gen_g4_detsim_hitreco__20240509T221136Z_reco2.root
fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50610358_60_20231207T075849Z_gen_g4_detsim_hitreco__20240510T034736Z_reco2.root
fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6409602_456_20231202T150521Z_gen_g4_detsim_hitreco__20240508T030546Z_reco2.root
fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6418365_227_20231203T105251Z_gen_g4_detsim_hitreco__20240508T042837Z_reco2.root
fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6453326_141_20231206T232619Z_gen_g4_detsim_hitreco__20240510T024815Z_reco2.root
fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50571149_828_20231202T191107Z_gen_g4_detsim_hitreco__20240508T064437Z_reco2.root
JobscriptExit code0
Real time1h (4667s)
CPU time58m (3513s = 75%)
Max RSS bytes2142228480 (2042 MiB)
Outputting started2025-09-15 16:10:13
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pgranger/cafs/fnal/02501/1/001/caf_fd_hd_atmo_2501_20250915T145234Z.root
Finished2025-09-15 16:31:02
Saved logsjustin-logs:40063.59-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

0.0331706     0.207342      0.588945      0.208095      0.0652582      1000    
reco:ophitspe:OpHitFinderDeco                          0.0960672     0.198879       3.27915      0.218063      0.108529       1000    
reco:opflash:OpFlashFinder                            0.000146701   0.000868416   0.00276766    0.00082513    0.000401068     1000    
reco:opslicer:OpSlicer                                3.0969e-05     0.171919      0.799505      0.129901      0.146792       1000    
reco:rns:RandomNumberSaver                            1.3375e-05    1.74459e-05   0.000205814   1.6081e-05    6.90675e-06     1000    
reco:anglereconue:NuAngularReco                       7.8139e-05    0.00135621     0.105048     0.000213989   0.00613723      1000    
reco:anglereconumu:NuAngularReco                      5.7349e-05    0.00126696     0.105366     0.000143354   0.00605148      1000    
reco:anglereconuepfps:NuAngularReco                   8.2237e-05    0.00134185     0.105577     0.000211039   0.00612732      1000    
reco:anglereconumupfps:NuAngularReco                  8.0143e-05    0.00131667     0.106036     0.000188341   0.00607282      1000    
reco:anglerecohits:NuAngularReco                       6.286e-05    0.00173736     0.109579     0.000464153   0.00650831      1000    
[art]:TriggerResults:TriggerResultInserter             7.134e-06    8.5284e-06    3.0819e-05     8.336e-06    1.5327e-06      1000    
end_path:cafmaker:CAFMaker                             0.0010473     0.0230297      5.63562     0.00295508     0.194931       1000    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3814.38 MB
  Peak resident set size usage (VmHWM): 2142.23 MB
====================================================================================================
lar exit code 0
=== Start last 100 lines of lar log file ===
Found hits: 1868!
Begin processing the 994th record. run: 50571149 subRun: 1 event: 82894 at 15-Sep-2025 16:09:56 UTC
PandoraContentApi::GetList(*this, m_inputHitListName, pCaloHitList) return STATUS_CODE_NOT_INITIALIZED
    in function: GetVolumeIdToHitListMap
    in file:     /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/ALMA9/build/larpandoracontent/v04_16_00-buildFW/src/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 271
this->GetVolumeIdToHitListMap(volumeIdToHitListMap) return STATUS_CODE_NOT_INITIALIZED
    in function: Run
    in file:     /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/ALMA9/build/larpandoracontent/v04_16_00-buildFW/src/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 165
iter->second->Run() throw STATUS_CODE_NOT_INITIALIZED
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0003, LArDLMaster, STATUS_CODE_NOT_INITIALIZED

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 0!
Begin processing the 995th record. run: 50571149 subRun: 1 event: 82895 at 15-Sep-2025 16:09:57 UTC
Boundary wire vector sizes: 116, 120, 95
minwire 0: 1162
minwire 1: 1735
minwire 2: 700
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.992658, 0.000782975, 0.00655929, 
Output 1: 0.0223361, 0.938224, 0.0385084, 0.00093108, 
Output 2: 0.990615, 0.00907218, 0.000304959, 7.48496e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2459!
Begin processing the 996th record. run: 50571149 subRun: 1 event: 82896 at 15-Sep-2025 16:09:59 UTC
Boundary wire vector sizes: 148, 128, 123
minwire 0: 774
minwire 1: 2015
minwire 2: 436
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0648663, 0.0918405, 0.843293, 
Output 1: 0.0240546, 0.631039, 0.333976, 0.0109313, 
Output 2: 0.983999, 0.015585, 0.000407802, 8.41026e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1437!
Begin processing the 997th record. run: 50571149 subRun: 1 event: 82897 at 15-Sep-2025 16:10:01 UTC
Boundary wire vector sizes: 197, 196, 177
minwire 0: 965
minwire 1: 2504
minwire 2: 383
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.908339, 0.0345586, 0.0571026, 
Output 1: 0.648488, 0.345506, 0.00578004, 0.000226268, 
Output 2: 0.276094, 0.699014, 0.0244612, 0.000431199, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1811!
Begin processing the 998th record. run: 50571149 subRun: 1 event: 82898 at 15-Sep-2025 16:10:04 UTC
Boundary wire vector sizes: 880, 1310, 1224
minwire 0: 614
minwire 1: 2050
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.993201, 0.00191299, 0.00488566, 
Output 1: 0.519481, 0.418295, 0.0578661, 0.00435871, 
Output 2: 0.131095, 0.816014, 0.0516228, 0.00126742, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2358!
Begin processing the 999th record. run: 50571149 subRun: 1 event: 82899 at 15-Sep-2025 16:10:07 UTC
Boundary wire vector sizes: 531, 427, 412
minwire 0: 1349
minwire 1: 1027
minwire 2: 1301
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.249449, 0.0375277, 0.713023, 
Output 1: 0.279274, 0.311948, 0.204109, 0.204669, 
Output 2: 0.158915, 0.317102, 0.311273, 0.212711, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2513!
Begin processing the 1000th record. run: 50571149 subRun: 1 event: 82900 at 15-Sep-2025 16:10:10 UTC

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1148!
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
processed files
root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/15/f0/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006610_664_20231202T203545Z_gen_g4_detsim_hitreco__20240508T070002Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/c0/f0/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6413290_248_20231202T202528Z_gen_g4_detsim_hitreco__20240508T061715Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/06/e8/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6468412_786_20231207T081349Z_gen_g4_detsim_hitreco__20240510T021938Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/fe/e1/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50581281_781_20231204T052622Z_gen_g4_detsim_hitreco__20240509T214033Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/c4/15/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74521450_1191_20231205T102122Z_gen_g4_detsim_hitreco__20240509T221136Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/8b/0f/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50610358_60_20231207T075849Z_gen_g4_detsim_hitreco__20240510T034736Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/58/5b/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6409602_456_20231202T150521Z_gen_g4_detsim_hitreco__20240508T030546Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/46/c7/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6418365_227_20231203T105251Z_gen_g4_detsim_hitreco__20240508T042837Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/81/cf/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6453326_141_20231206T232619Z_gen_g4_detsim_hitreco__20240510T024815Z_reco2.root
root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/e1/35/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50571149_828_20231202T191107Z_gen_g4_detsim_hitreco__20240508T064437Z_reco2.root
.:
total 47588
-rw-r--r-- 1 dnpil001 users 17901567 Sep 15 16:10 flatcaf.root
-rw-r--r-- 1 dnpil001 users 14988381 Sep 15 16:10 caf.root
-rw-r--r-- 1 dnpil001 users 14988381 Sep 15 16:10 caf_fd_hd_atmo_2501_20250915T145234Z.root
-rw-r--r-- 1 dnpil001 users   585587 Sep 15 16:10 caf_20250915T145234Z.log
-rw-r--r-- 1 dnpil001 users   252257 Sep 15 16:10 jobscript.log
-rw-r--r-- 1 dnpil001 users     1929 Sep 15 16:10 caf_20250915T145234Z.file
-rw-r--r-- 1 dnpil001 users     1929 Sep 15 16:10 caf_20250915T145234Z.pfns
-rw-r--r-- 1 dnpil001 users     1929 Sep 15 14:52 file.list
-rw-r--r-- 1 dnpil001 users     1929 Sep 15 16:10 justin-processed-pfns.txt
-rw-r--r-- 1 dnpil001 users     1375 Sep 15 14:52 all-input-dids.txt
-rw-r--r-- 1 dnpil001 users     1375 Sep 15 16:10 caf_20250915T145234Z.did
-rw-r--r-- 1 dnpil001 users     1375 Sep 15 14:52 did.list
-rw-r--r-- 1 dnpil001 users        0 Sep 15 14:52 debugprod.log
justIN time: 2025-09-18 19:13:34 UTC       justIN version: 01.05.00