justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 46597.62@dunegpschedd01.fnal.gov

Jobsub ID46597.62@dunegpschedd01.fnal.gov
Workflow ID2501
Stage ID1
User namepgranger@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-09-15 12:09:40
SiteUK_Oxford
EntryDUNE_UK_SGrid_Oxford_arc01
Last heartbeat2025-09-15 14:46:20
From worker nodeHostnamet2wn169.physics.ox.ac.uk
cpuinfoAMD EPYC 7452 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-09-15 12:10:49
Input filesfardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50620606_731_20231207T120858Z_gen_g4_detsim_hitreco__20240510T041948Z_reco2.root
fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66029763_432_20231207T032134Z_gen_g4_detsim_hitreco__20240510T024434Z_reco2.root
fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6455553_676_20231207T115512Z_gen_g4_detsim_hitreco__20240510T054414Z_reco2.root
fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006610_608_20231202T203536Z_gen_g4_detsim_hitreco__20240508T070043Z_reco2.root
fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6410210_685_20231203T082014Z_gen_g4_detsim_hitreco__20240508T060311Z_reco2.root
fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50642276_452_20231208T030621Z_gen_g4_detsim_hitreco__20240510T064245Z_reco2.root
fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74581430_697_20231208T082350Z_gen_g4_detsim_hitreco__20240510T062742Z_reco2.root
fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6413290_419_20231202T204334Z_gen_g4_detsim_hitreco__20240508T061326Z_reco2.root
fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50264716_83_20231122T133034Z_gen_g4_detsim_hitreco__20240507T180538Z_reco2.root
fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6207207_59_20231120T132634Z_gen_g4_detsim_hitreco__20240503T061513Z_reco2.root
JobscriptExit code0
Real time2h (7445s)
CPU time1h (5760s = 77%)
Max RSS bytes2062561280 (1967 MiB)
Outputting started2025-09-15 14:14:55
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pgranger/cafs/fnal/02501/1/001/caf_fd_hd_atmo_2501_20250915T121104Z.root
Finished2025-09-15 14:46:20
Saved logsjustin-logs:46597.62-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

51      0.0204341      1100    
reco:opflash:OpFlashFinder                            0.000203493   0.00188291    0.00748013     0.001818     0.00079353      1100    
reco:opslicer:OpSlicer                                4.6337e-05     0.318186       1.28077      0.251704      0.258354       1100    
reco:rns:RandomNumberSaver                            2.3714e-05    7.34631e-05   0.00386816    6.4306e-05    0.000121572     1100    
reco:anglereconue:NuAngularReco                       0.00021769     0.0020553     0.132541     0.000628534   0.00646842      1100    
reco:anglereconumu:NuAngularReco                      9.2785e-05    0.00171841     0.125438     0.000294009   0.00634267      1100    
reco:anglereconuepfps:NuAngularReco                   0.000139583   0.00187191      0.11801     0.00045653    0.00623527      1100    
reco:anglereconumupfps:NuAngularReco                  0.000110588   0.00179841     0.114345     0.000363334   0.00614747      1100    
reco:anglerecohits:NuAngularReco                      0.000118543   0.00268359     0.123548     0.000983974   0.00689737      1100    
[art]:TriggerResults:TriggerResultInserter            1.0891e-05    1.91565e-05   0.000156816   1.75085e-05   8.79277e-06     1100    
end_path:cafmaker:CAFMaker                             0.0021234     0.0288829      1.54731     0.00975552     0.0825248      1100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3777.93 MB
  Peak resident set size usage (VmHWM): 2062.56 MB
====================================================================================================
lar exit code 0
=== Start last 100 lines of lar log file ===

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 659!
Begin processing the 1093rd record. run: 6207207 subRun: 1 event: 11993 at 15-Sep-2025 15:14:22 BST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 655!
Begin processing the 1094th record. run: 6207207 subRun: 1 event: 11994 at 15-Sep-2025 15:14:25 BST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 495!
Begin processing the 1095th record. run: 6207207 subRun: 1 event: 11995 at 15-Sep-2025 15:14:29 BST
Boundary wire vector sizes: 52, 39, 10
minwire 0: 833
minwire 1: 1637
minwire 2: 1180
Used alternate method to get min and max wires due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max wires due to vertex determination failure: 68, 367
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max wires due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0112399, 0.0038445, 0.984916, 
Output 1: 0.0146006, 0.975319, 0.00971568, 0.000364295, 
Output 2: 0.997446, 0.00247504, 7.59512e-05, 2.87716e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1290!
Begin processing the 1096th record. run: 6207207 subRun: 1 event: 11996 at 15-Sep-2025 15:14:33 BST
Boundary wire vector sizes: 101, 179, 161
minwire 0: 2080
minwire 1: 1125
minwire 2: 1920
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.996611, 0.000693569, 0.00269519, 
Output 1: 0.00276385, 0.993491, 0.00366909, 7.60612e-05, 
Output 2: 0.996127, 0.00379423, 7.73968e-05, 1.42034e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1690!
Begin processing the 1097th record. run: 6207207 subRun: 1 event: 11997 at 15-Sep-2025 15:14:38 BST
PandoraContentApi::GetList(*this, m_inputHitListName, pCaloHitList) return STATUS_CODE_NOT_INITIALIZED
    in function: GetVolumeIdToHitListMap
    in file:     /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/ALMA9/build/larpandoracontent/v04_16_00-buildFW/src/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 271
this->GetVolumeIdToHitListMap(volumeIdToHitListMap) return STATUS_CODE_NOT_INITIALIZED
    in function: Run
    in file:     /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/ALMA9/build/larpandoracontent/v04_16_00-buildFW/src/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 165
iter->second->Run() throw STATUS_CODE_NOT_INITIALIZED
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0003, LArDLMaster, STATUS_CODE_NOT_INITIALIZED

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 0!
Begin processing the 1098th record. run: 6207207 subRun: 1 event: 11998 at 15-Sep-2025 15:14:40 BST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 936!
Begin processing the 1099th record. run: 6207207 subRun: 1 event: 11999 at 15-Sep-2025 15:14:43 BST
Boundary wire vector sizes: 208, 257, 188
minwire 0: 0
minwire 1: 2111
minwire 2: 99
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000684628, 0.995999, 0.00331642, 
Output 1: 0.0372022, 0.676637, 0.271592, 0.0145692, 
Output 2: 0.991685, 0.00801579, 0.000290701, 8.49376e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1166!
Begin processing the 1100th record. run: 6207207 subRun: 1 event: 12000 at 15-Sep-2025 15:14:48 BST
Boundary wire vector sizes: 43, 43, 35
minwire 0: 1730
minwire 1: 1422
minwire 2: 1219
Used alternate method to get min and max wires due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max wires due to vertex determination failure: 199, 498
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max wires due to vertex determination failure: 105, 404
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00329321, 0.00157248, 0.995134, 
Output 1: 0.128801, 0.860247, 0.0107373, 0.000214831, 
Output 2: 0.998372, 0.00153838, 8.60132e-05, 3.6078e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1848!
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
processed files
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/c8/e0/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50620606_731_20231207T120858Z_gen_g4_detsim_hitreco__20240510T041948Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/27/4d/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66029763_432_20231207T032134Z_gen_g4_detsim_hitreco__20240510T024434Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/83/11/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6455553_676_20231207T115512Z_gen_g4_detsim_hitreco__20240510T054414Z_reco2.root
root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/33/b8/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006610_608_20231202T203536Z_gen_g4_detsim_hitreco__20240508T070043Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/55/d1/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6410210_685_20231203T082014Z_gen_g4_detsim_hitreco__20240508T060311Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/f1/3f/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50642276_452_20231208T030621Z_gen_g4_detsim_hitreco__20240510T064245Z_reco2.root
root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/6e/d0/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74581430_697_20231208T082350Z_gen_g4_detsim_hitreco__20240510T062742Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/c0/56/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6413290_419_20231202T204334Z_gen_g4_detsim_hitreco__20240508T061326Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/3e/8a/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50264716_83_20231122T133034Z_gen_g4_detsim_hitreco__20240507T180538Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/82/8f/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6207207_59_20231120T132634Z_gen_g4_detsim_hitreco__20240503T061513Z_reco2.root
.:
total 50380
-rw-r--r-- 1 dune020 dune 18927349 Sep 15 15:14 flatcaf.root
-rw-r--r-- 1 dune020 dune 15840814 Sep 15 15:14 caf.root
-rw-r--r-- 1 dune020 dune 15840814 Sep 15 15:14 caf_fd_hd_atmo_2501_20250915T121104Z.root
-rw-r--r-- 1 dune020 dune   665574 Sep 15 15:14 caf_20250915T121104Z.log
-rw-r--r-- 1 dune020 dune   228248 Sep 15 15:14 jobscript.log
-rw-r--r-- 1 dune020 dune     1904 Sep 15 15:14 caf_20250915T121104Z.file
-rw-r--r-- 1 dune020 dune     1904 Sep 15 15:14 caf_20250915T121104Z.pfns
-rw-r--r-- 1 dune020 dune     1904 Sep 15 13:11 file.list
-rw-r--r-- 1 dune020 dune     1904 Sep 15 15:14 justin-processed-pfns.txt
-rw-r--r-- 1 dune020 dune     1374 Sep 15 13:11 all-input-dids.txt
-rw-r--r-- 1 dune020 dune     1374 Sep 15 15:14 caf_20250915T121104Z.did
-rw-r--r-- 1 dune020 dune     1374 Sep 15 13:11 did.list
-rw-r--r-- 1 dune020 dune        0 Sep 15 13:12 debugprod.log
justIN time: 2025-09-19 07:38:51 UTC       justIN version: 01.05.00