justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 48412.2@dunegpschedd01.fnal.gov

Jobsub ID48412.2@dunegpschedd01.fnal.gov
Workflow ID2501
Stage ID1
User namepgranger@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-09-18 11:18:22
SiteUK_Manchester
EntryUBoone_T2_UK_Manchester_ce02
Last heartbeat2025-09-18 13:16:38
From worker nodeHostnamewn1909240.tier2.hep.manchester.ac.uk
cpuinfoIntel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-09-18 11:19:51
Input filesfardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1498_71_20231027T230010Z_gen_g4_detsim_hitreco__20240503T045603Z_reco2.root
fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6470038_600_20231208T004816Z_gen_g4_detsim_hitreco__20240510T031617Z_reco2.root
fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74511302_896_20231203T154247Z_gen_g4_detsim_hitreco__20240508T025412Z_reco2.root
fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74531892_359_20231207T033427Z_gen_g4_detsim_hitreco__20240510T034014Z_reco2.root
fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50624901_740_20231207T140623Z_gen_g4_detsim_hitreco__20240510T045523Z_reco2.root
JobscriptExit code0
Real time1h (6913s)
CPU time1h (5979s = 86%)
Max RSS bytes2089426944 (1992 MiB)
Outputting started2025-09-18 13:15:06
Output files
Finished2025-09-18 13:16:38
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

600    
reco:energyrecnumu:EnergyReco                         0.000418505    0.174274       6.74827      0.039275      0.595198        600    
reco:energyrecnue:EnergyReco                          0.000181194    0.0089997     0.732545     0.00121539     0.0450466       600    
reco:energyrecnc:EnergyReco                           0.00239285     0.120963      0.838345      0.100748      0.110029        600    
reco:energyrecnumurange:EnergyReco                    0.000384969    0.108678       5.46577     0.00485889      0.48537        600    
reco:energyrecnumumcs:EnergyReco                      0.000834592    0.109698       5.19432     0.00644846     0.489406        600    
reco:energyrecnumumcsllhd:EnergyReco                  0.00070565     0.106353       4.94841     0.00655987     0.471633        600    
reco:opdec:Deconvolution                               0.0367525     0.408968      0.976168       0.39948      0.162037        600    
reco:ophitspe:OpHitFinderDeco                          0.0908117     0.187682      0.456925      0.240784      0.0749721       600    
reco:opflash:OpFlashFinder                            0.000195352   0.00319092     0.0308284    0.00242396    0.00306321       600    
reco:opslicer:OpSlicer                                6.0559e-05     0.452733       2.09763       0.35691       0.3834         600    
reco:rns:RandomNumberSaver                            2.6703e-05    8.47658e-05   0.000553637   7.4918e-05    4.2681e-05       600    
reco:anglereconue:NuAngularReco                       0.000204924   0.00559692     0.361582     0.000813987    0.0246998       600    
reco:anglereconumu:NuAngularReco                      9.1663e-05     0.0047162     0.304119     0.00048237     0.0220382       600    
reco:anglereconuepfps:NuAngularReco                   0.000159582   0.00499676     0.303086     0.000666742    0.0227464       600    
reco:anglereconumupfps:NuAngularReco                  0.000140797   0.00497844     0.324465     0.000611022    0.0233573       600    
reco:anglerecohits:NuAngularReco                      0.000138557   0.00653273      0.31905     0.00133697     0.0247389       600    
[art]:TriggerResults:TriggerResultInserter            1.0666e-05    2.54281e-05   7.7743e-05    2.2674e-05    1.00049e-05      600    
end_path:cafmaker:CAFMaker                            0.00200991     0.0669456      3.36456      0.0152528     0.250923        600    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3761.95 MB
  Peak resident set size usage (VmHWM): 2089.43 MB
====================================================================================================
lar exit code 0
=== Start last 100 lines of lar log file ===
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1477!
Begin processing the 595th record. run: 50624901 subRun: 1 event: 74095 at 18-Sep-2025 14:14:14 BST
Boundary wire vector sizes: 277, 338, 257
minwire 0: 2482
minwire 1: 408
minwire 2: 2568
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00152824, 0.982637, 0.0158348, 
Output 1: 0.0110307, 0.868845, 0.116383, 0.00374101, 
Output 2: 0.984324, 0.0152354, 0.000430116, 1.02984e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1350!
Begin processing the 596th record. run: 50624901 subRun: 1 event: 74096 at 18-Sep-2025 14:14:20 BST
Boundary wire vector sizes: 4252, 4476, 1758
minwire 0: 0
minwire 1: 1616
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.15752, 0.759395, 0.083085, 
Output 1: 0.211367, 0.363167, 0.237427, 0.188038, 
Output 2: 0.0211079, 0.0764757, 0.264565, 0.637852, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 3015!
Begin processing the 597th record. run: 50624901 subRun: 1 event: 74097 at 18-Sep-2025 14:14:42 BST
Boundary wire vector sizes: 315, 322, 304
minwire 0: 1280
minwire 1: 959
minwire 2: 1125
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0330451, 0.0948067, 0.872148, 
Output 1: 0.0797973, 0.333409, 0.350536, 0.236258, 
Output 2: 0.234147, 0.574605, 0.169156, 0.0220924, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2751!
Begin processing the 598th record. run: 50624901 subRun: 1 event: 74098 at 18-Sep-2025 14:14:50 BST
Boundary wire vector sizes: 118, 116, 125
minwire 0: 656
minwire 1: 2942
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000336002, 0.997018, 0.00264605, 
Output 1: 0.0460591, 0.948679, 0.00510972, 0.000152313, 
Output 2: 0.99902, 0.000952737, 2.68855e-05, 8.70026e-07, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 934!
Begin processing the 599th record. run: 50624901 subRun: 1 event: 74099 at 18-Sep-2025 14:14:56 BST
Boundary wire vector sizes: 33, 36, 37
minwire 0: 2120
minwire 1: 243
minwire 2: 2141
Used alternate method to get min and max wires due to vertex determination failure: 274, 573
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max wires due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max wires due to vertex determination failure: 450, 749
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0105265, 0.00336431, 0.986109, 
Output 1: 0.87472, 0.121308, 0.00347081, 0.000501284, 
Output 2: 0.994901, 0.00491547, 0.000166437, 1.7343e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 814!
Begin processing the 600th record. run: 50624901 subRun: 1 event: 74100 at 18-Sep-2025 14:15:01 BST
PandoraContentApi::GetList(*this, m_inputHitListName, pCaloHitList) return STATUS_CODE_NOT_INITIALIZED
    in function: GetVolumeIdToHitListMap
    in file:     /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/ALMA9/build/larpandoracontent/v04_16_00-buildFW/src/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 271
this->GetVolumeIdToHitListMap(volumeIdToHitListMap) return STATUS_CODE_NOT_INITIALIZED
    in function: Run
    in file:     /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/ALMA9/build/larpandoracontent/v04_16_00-buildFW/src/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 165
iter->second->Run() throw STATUS_CODE_NOT_INITIALIZED
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0003, LArDLMaster, STATUS_CODE_NOT_INITIALIZED

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 83!
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
processed files
root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/ea/d6/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1498_71_20231027T230010Z_gen_g4_detsim_hitreco__20240503T045603Z_reco2.root
root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/31/ef/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6470038_600_20231208T004816Z_gen_g4_detsim_hitreco__20240510T031617Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/61/05/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74511302_896_20231203T154247Z_gen_g4_detsim_hitreco__20240508T025412Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/d1/c3/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74531892_359_20231207T033427Z_gen_g4_detsim_hitreco__20240510T034014Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/85/30/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50624901_740_20231207T140623Z_gen_g4_detsim_hitreco__20240510T045523Z_reco2.root
.:
total 30548
-rw-r--r-- 1 dune01 dune01 11516919 Sep 18 14:15 flatcaf.root
-rw-r--r-- 1 dune01 dune01  9595243 Sep 18 14:15 caf.root
-rw-r--r-- 1 dune01 dune01  9595243 Sep 18 14:15 caf_fd_hd_atmo_2501_20250918T112003Z.root
-rw-r--r-- 1 dune01 dune01   368968 Sep 18 14:15 caf_20250918T112003Z.log
-rw-r--r-- 1 dune01 dune01   152329 Sep 18 14:15 jobscript.log
-rw-r--r-- 1 dune01 dune01      933 Sep 18 14:15 caf_20250918T112003Z.file
-rw-r--r-- 1 dune01 dune01      933 Sep 18 14:15 caf_20250918T112003Z.pfns
-rw-r--r-- 1 dune01 dune01      933 Sep 18 12:20 file.list
-rw-r--r-- 1 dune01 dune01      933 Sep 18 14:15 justin-processed-pfns.txt
-rw-r--r-- 1 dune01 dune01      684 Sep 18 12:20 all-input-dids.txt
-rw-r--r-- 1 dune01 dune01      684 Sep 18 14:15 caf_20250918T112003Z.did
-rw-r--r-- 1 dune01 dune01      684 Sep 18 12:20 did.list
-rw-r--r-- 1 dune01 dune01        0 Sep 18 12:20 debugprod.log
justIN time: 2025-09-18 19:07:24 UTC       justIN version: 01.05.00