justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 41798.26@dunegpschedd02.fnal.gov

Jobsub ID41798.26@dunegpschedd02.fnal.gov
Workflow ID2501
Stage ID1
User namepgranger@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-09-18 09:32:17
SiteUK_Manchester
EntryUBoone_T2_UK_Manchester_ce01
Last heartbeat2025-09-18 11:17:11
From worker nodeHostnamewn1209291.tier2.hep.manchester.ac.uk
cpuinfoIntel(R) Xeon(R) Gold 5118 CPU @ 2.30GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-09-18 09:32:44
Input filesfardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1498_71_20231027T230010Z_gen_g4_detsim_hitreco__20240503T045603Z_reco2.root
fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6470038_600_20231208T004816Z_gen_g4_detsim_hitreco__20240510T031617Z_reco2.root
fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74511302_896_20231203T154247Z_gen_g4_detsim_hitreco__20240508T025412Z_reco2.root
fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74531892_359_20231207T033427Z_gen_g4_detsim_hitreco__20240510T034014Z_reco2.root
fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50624901_740_20231207T140623Z_gen_g4_detsim_hitreco__20240510T045523Z_reco2.root
JobscriptExit code0
Real time1h (6243s)
CPU time1h (5608s = 89%)
Max RSS bytes2101997568 (2004 MiB)
Outputting started2025-09-18 11:16:47
Output files
Finished2025-09-18 11:17:11
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

600    
reco:energyrecnumu:EnergyReco                         0.000374091    0.141838       5.59133      0.0328536     0.432438        600    
reco:energyrecnue:EnergyReco                          0.000182885   0.00699087     0.559675     0.00110755     0.0345952       600    
reco:energyrecnc:EnergyReco                           0.00222297      0.11127      0.754343      0.0924283     0.0980818       600    
reco:energyrecnumurange:EnergyReco                    0.000406894    0.0932352      3.74126     0.00405458      0.38368        600    
reco:energyrecnumumcs:EnergyReco                      0.000848422    0.0896148      3.50056     0.00511472     0.357029        600    
reco:energyrecnumumcsllhd:EnergyReco                  0.000773019    0.0883264      3.75571     0.00510753     0.355704        600    
reco:opdec:Deconvolution                               0.0388561     0.368255       1.00201       0.35742      0.144497        600    
reco:ophitspe:OpHitFinderDeco                          0.0861372     0.177537      0.342647      0.197879      0.0760718       600    
reco:opflash:OpFlashFinder                            0.000215078   0.00229785    0.00769686    0.00214598    0.00110839       600    
reco:opslicer:OpSlicer                                6.5521e-05      0.40674       2.46509      0.302035      0.359923        600    
reco:rns:RandomNumberSaver                            2.1652e-05    5.39956e-05   0.000535326   4.76245e-05   2.91954e-05      600    
reco:anglereconue:NuAngularReco                       0.000191512   0.00458777     0.272559     0.000653337    0.0208094       600    
reco:anglereconumu:NuAngularReco                      9.5736e-05    0.00443525     0.293156     0.000426363    0.0222468       600    
reco:anglereconuepfps:NuAngularReco                   0.000134572   0.00459709     0.290185     0.000562123    0.0221049       600    
reco:anglereconumupfps:NuAngularReco                  0.000114518   0.00453288     0.287904     0.000529651    0.0215541       600    
reco:anglerecohits:NuAngularReco                      0.000136087   0.00556002     0.303611     0.00116726     0.0224377       600    
[art]:TriggerResults:TriggerResultInserter            1.1281e-05    1.93274e-05   7.1522e-05    1.79585e-05   5.19483e-06      600    
end_path:cafmaker:CAFMaker                            0.00158922     0.0571969      3.1171       0.0103614     0.245392        600    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3762.61 MB
  Peak resident set size usage (VmHWM): 2102 MB
====================================================================================================
lar exit code 0
=== Start last 100 lines of lar log file ===
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1477!
Begin processing the 595th record. run: 50624901 subRun: 1 event: 74095 at 18-Sep-2025 12:16:03 BST
Boundary wire vector sizes: 277, 338, 257
minwire 0: 2482
minwire 1: 408
minwire 2: 2568
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00152824, 0.982637, 0.0158348, 
Output 1: 0.0110307, 0.868845, 0.116383, 0.00374101, 
Output 2: 0.984324, 0.0152354, 0.000430116, 1.02984e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1350!
Begin processing the 596th record. run: 50624901 subRun: 1 event: 74096 at 18-Sep-2025 12:16:09 BST
Boundary wire vector sizes: 4252, 4476, 1758
minwire 0: 0
minwire 1: 1616
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.15752, 0.759395, 0.083085, 
Output 1: 0.211367, 0.363167, 0.237427, 0.188038, 
Output 2: 0.0211079, 0.0764757, 0.264565, 0.637852, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 3015!
Begin processing the 597th record. run: 50624901 subRun: 1 event: 74097 at 18-Sep-2025 12:16:27 BST
Boundary wire vector sizes: 315, 322, 304
minwire 0: 1280
minwire 1: 959
minwire 2: 1125
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0330451, 0.0948067, 0.872148, 
Output 1: 0.0797973, 0.333409, 0.350536, 0.236258, 
Output 2: 0.234147, 0.574605, 0.169156, 0.0220924, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2751!
Begin processing the 598th record. run: 50624901 subRun: 1 event: 74098 at 18-Sep-2025 12:16:34 BST
Boundary wire vector sizes: 118, 116, 125
minwire 0: 656
minwire 1: 2942
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000336002, 0.997018, 0.00264605, 
Output 1: 0.0460591, 0.948679, 0.00510972, 0.000152313, 
Output 2: 0.99902, 0.000952737, 2.68855e-05, 8.70026e-07, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 934!
Begin processing the 599th record. run: 50624901 subRun: 1 event: 74099 at 18-Sep-2025 12:16:39 BST
Boundary wire vector sizes: 33, 36, 37
minwire 0: 2120
minwire 1: 243
minwire 2: 2141
Used alternate method to get min and max wires due to vertex determination failure: 274, 573
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max wires due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max wires due to vertex determination failure: 450, 749
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0105265, 0.00336431, 0.986109, 
Output 1: 0.87472, 0.121308, 0.00347081, 0.000501284, 
Output 2: 0.994901, 0.00491547, 0.000166437, 1.7343e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 814!
Begin processing the 600th record. run: 50624901 subRun: 1 event: 74100 at 18-Sep-2025 12:16:44 BST
PandoraContentApi::GetList(*this, m_inputHitListName, pCaloHitList) return STATUS_CODE_NOT_INITIALIZED
    in function: GetVolumeIdToHitListMap
    in file:     /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/ALMA9/build/larpandoracontent/v04_16_00-buildFW/src/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 271
this->GetVolumeIdToHitListMap(volumeIdToHitListMap) return STATUS_CODE_NOT_INITIALIZED
    in function: Run
    in file:     /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/ALMA9/build/larpandoracontent/v04_16_00-buildFW/src/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 165
iter->second->Run() throw STATUS_CODE_NOT_INITIALIZED
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0003, LArDLMaster, STATUS_CODE_NOT_INITIALIZED

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 83!
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
processed files
root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/ea/d6/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1498_71_20231027T230010Z_gen_g4_detsim_hitreco__20240503T045603Z_reco2.root
root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/31/ef/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6470038_600_20231208T004816Z_gen_g4_detsim_hitreco__20240510T031617Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/61/05/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74511302_896_20231203T154247Z_gen_g4_detsim_hitreco__20240508T025412Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/d1/c3/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74531892_359_20231207T033427Z_gen_g4_detsim_hitreco__20240510T034014Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/85/30/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50624901_740_20231207T140623Z_gen_g4_detsim_hitreco__20240510T045523Z_reco2.root
.:
total 30548
-rw-r--r-- 1 dune01 dune01 11516919 Sep 18 12:16 flatcaf.root
-rw-r--r-- 1 dune01 dune01  9595236 Sep 18 12:16 caf.root
-rw-r--r-- 1 dune01 dune01  9595236 Sep 18 12:16 caf_fd_hd_atmo_2501_20250918T093255Z.root
-rw-r--r-- 1 dune01 dune01   368969 Sep 18 12:16 caf_20250918T093255Z.log
-rw-r--r-- 1 dune01 dune01   152326 Sep 18 12:16 jobscript.log
-rw-r--r-- 1 dune01 dune01      933 Sep 18 12:16 caf_20250918T093255Z.file
-rw-r--r-- 1 dune01 dune01      933 Sep 18 12:16 caf_20250918T093255Z.pfns
-rw-r--r-- 1 dune01 dune01      933 Sep 18 10:32 file.list
-rw-r--r-- 1 dune01 dune01      933 Sep 18 12:16 justin-processed-pfns.txt
-rw-r--r-- 1 dune01 dune01      684 Sep 18 10:32 all-input-dids.txt
-rw-r--r-- 1 dune01 dune01      684 Sep 18 12:16 caf_20250918T093255Z.did
-rw-r--r-- 1 dune01 dune01      684 Sep 18 10:32 did.list
-rw-r--r-- 1 dune01 dune01        0 Sep 18 10:33 debugprod.log
justIN time: 2025-09-19 03:35:12 UTC       justIN version: 01.05.00