Jobsub ID 45070.1@dunegpschedd02.fnal.gov

Jobsub ID	45070.1@dunegpschedd02.fnal.gov
Workflow ID	2501
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-21 16:30:25
Site	NL_NIKHEF
Entry	VIRGO_NL_NIKHEF_juk
Last heartbeat	2025-09-21 16:38:57
From worker node	Hostname	wn-pep-014.farm.nikhef.nl
	cpuinfo	Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	129600 (36 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-09-21 16:32:22
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74574126_605_20231208T025510Z_gen_g4_detsim_hitreco__20240510T041252Z_reco2.root
Jobscript	Exit code	0
	Real time	6m (377s)
	CPU time	4m (295s = 78%)
	Max RSS bytes	1992216576 (1899 MiB)
Outputting started	2025-09-21 16:38:41
Output files
Finished	2025-09-21 16:38:57
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

1      1.85622       21.229        1.3849        2.65291        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000121456   0.00412131     0.0742963    0.00103416     0.0103563       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000156214   0.00887683     0.0829008    0.00243097     0.0150179       100    
reco:pandoracalo:Calorimetry                          9.5794e-05     0.0132446     0.566313     0.000740926    0.0650143       100    
reco:pandorapid:Chi2ParticleID                        3.5487e-05    0.00012235    0.00123499    9.1794e-05     0.0001325       100    
reco:cvnmap:CVNMapper                                 1.7415e-05     0.0186913     0.0587895     0.0196812     0.0147705       100    
reco:cvneva:CVNEvaluator                              1.5963e-05     0.242506       3.23846      0.292988      0.329086        100    
reco:energyrecnumu:EnergyReco                         0.000250535    0.0407573     0.181511      0.015764      0.0466269       100    
reco:energyrecnue:EnergyReco                          0.000144972   0.00155729     0.0282997    0.000513732   0.00362348       100    
reco:energyrecnc:EnergyReco                           0.00194156     0.0730956     0.206818      0.0925117     0.0501725       100    
reco:energyrecnumurange:EnergyReco                    0.000233295    0.011469      0.180231     0.00206992     0.0272019       100    
reco:energyrecnumumcs:EnergyReco                      0.000874014    0.011932      0.159708     0.00257898     0.0253144       100    
reco:energyrecnumumcsllhd:EnergyReco                  0.000795174    0.0116376     0.163486     0.00247493     0.0259209       100    
reco:opdec:Deconvolution                               0.0495931     0.224394      0.639963      0.221369      0.0931485       100    
reco:ophitspe:OpHitFinderDeco                          0.126784      0.132664      0.149226      0.131934     0.00402819       100    
reco:opflash:OpFlashFinder                            0.000128323   0.00112494    0.00836146    0.00102374    0.000944034      100    
reco:opslicer:OpSlicer                                4.1733e-05     0.216973      0.938893      0.150907      0.213922        100    
reco:rns:RandomNumberSaver                            1.7448e-05    2.69464e-05   0.000339328   2.41125e-05   3.15153e-05      100    
reco:anglereconue:NuAngularReco                       0.000140659   0.000999063    0.0186219    0.000305653   0.00252433       100    
reco:anglereconumu:NuAngularReco                      7.2125e-05    0.000856764    0.018079     0.000173705   0.00247641       100    
reco:anglereconuepfps:NuAngularReco                   0.000111309   0.00097078     0.0182662    0.000255989   0.00249171       100    
reco:anglereconumupfps:NuAngularReco                  0.000100783   0.000920036    0.0182246    0.000231087   0.00249603       100    
reco:anglerecohits:NuAngularReco                      8.0852e-05    0.00139533     0.0216584    0.000542621   0.00291927       100    
[art]:TriggerResults:TriggerResultInserter             9.861e-06    1.17187e-05   4.4135e-05    1.1266e-05    3.4022e-06       100    
end_path:cafmaker:CAFMaker                            0.00121691     0.0132155     0.277252     0.00352992     0.0353346       100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3655.9 MB
  Peak resident set size usage (VmHWM): 1992.22 MB
====================================================================================================
lar exit code 0
=== Start last 100 lines of lar log file ===

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2368!
Begin processing the 92nd record. run: 74574126 subRun: 1 event: 60592 at 21-Sep-2025 18:38:19 CEST
Boundary wire vector sizes: 791, 878, 617
minwire 0: 699
minwire 1: 1767
minwire 2: 612
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00296022, 0.0567587, 0.940281, 
Output 1: 0.969965, 0.0297826, 0.000214025, 3.86819e-05, 
Output 2: 0.9986, 0.00137182, 2.28479e-05, 4.99729e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2553!
Begin processing the 93rd record. run: 74574126 subRun: 1 event: 60593 at 21-Sep-2025 18:38:22 CEST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 986!
Begin processing the 94th record. run: 74574126 subRun: 1 event: 60594 at 21-Sep-2025 18:38:24 CEST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 175!
Begin processing the 95th record. run: 74574126 subRun: 1 event: 60595 at 21-Sep-2025 18:38:26 CEST
Boundary wire vector sizes: 160, 73, 106
minwire 0: 935
minwire 1: 1314
minwire 2: 1026
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max wires due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.354384, 0.517881, 0.127736, 
Output 1: 0.864319, 0.131696, 0.00380019, 0.000185145, 
Output 2: 0.0737505, 0.908978, 0.017111, 0.000160962, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1339!
Begin processing the 96th record. run: 74574126 subRun: 1 event: 60596 at 21-Sep-2025 18:38:29 CEST
Boundary wire vector sizes: 54, 62, 50
minwire 0: 2191
minwire 1: 1044
minwire 2: 2034
Used alternate method to get min and max wires due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.274715, 0.258555, 0.46673, 
Output 1: 0.421545, 0.552728, 0.0249778, 0.000748649, 
Output 2: 0.575997, 0.418118, 0.00580883, 7.64277e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1600!
Begin processing the 97th record. run: 74574126 subRun: 1 event: 60597 at 21-Sep-2025 18:38:32 CEST
PandoraContentApi::GetList(*this, m_inputHitListName, pCaloHitList) return STATUS_CODE_NOT_INITIALIZED
    in function: GetVolumeIdToHitListMap
    in file:     /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/ALMA9/build/larpandoracontent/v04_16_00-buildFW/src/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 271
this->GetVolumeIdToHitListMap(volumeIdToHitListMap) return STATUS_CODE_NOT_INITIALIZED
    in function: Run
    in file:     /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/ALMA9/build/larpandoracontent/v04_16_00-buildFW/src/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 165
iter->second->Run() throw STATUS_CODE_NOT_INITIALIZED
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0003, LArDLMaster, STATUS_CODE_NOT_INITIALIZED

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 11!
Begin processing the 98th record. run: 74574126 subRun: 1 event: 60598 at 21-Sep-2025 18:38:32 CEST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 993!
Begin processing the 99th record. run: 74574126 subRun: 1 event: 60599 at 21-Sep-2025 18:38:34 CEST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 54!
Begin processing the 100th record. run: 74574126 subRun: 1 event: 60600 at 21-Sep-2025 18:38:36 CEST
Boundary wire vector sizes: 1006, 571, 926
minwire 0: 922
minwire 1: 713
minwire 2: 1374
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.996676, 0.000576289, 0.00274756, 
Output 1: 0.013111, 0.983292, 0.00350412, 9.30839e-05, 
Output 2: 0.996225, 0.00367174, 0.000100356, 2.60885e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2187!
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
processed files
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/46/55/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74574126_605_20231208T025510Z_gen_g4_detsim_hitreco__20240510T041252Z_reco2.root
.:
total 3548
-rw-r--r--. 1 dune012 dune 1417028 Sep 21 18:38 flatcaf.root
-rw-r--r--. 1 dune012 dune 1031972 Sep 21 18:38 caf.root
-rw-r--r--. 1 dune012 dune 1031972 Sep 21 18:38 caf_fd_hd_atmo_2501_20250921T163231Z.root
-rw-r--r--. 1 dune012 dune   75406 Sep 21 18:38 caf_20250921T163231Z.log
-rw-r--r--. 1 dune012 dune   32716 Sep 21 18:38 jobscript.log
-rw-r--r--. 1 dune012 dune     187 Sep 21 18:38 caf_20250921T163231Z.file
-rw-r--r--. 1 dune012 dune     187 Sep 21 18:38 caf_20250921T163231Z.pfns
-rw-r--r--. 1 dune012 dune     187 Sep 21 18:32 file.list
-rw-r--r--. 1 dune012 dune     187 Sep 21 18:38 justin-processed-pfns.txt
-rw-r--r--. 1 dune012 dune     138 Sep 21 18:32 all-input-dids.txt
-rw-r--r--. 1 dune012 dune     138 Sep 21 18:38 caf_20250921T163231Z.did
-rw-r--r--. 1 dune012 dune     138 Sep 21 18:32 did.list
-rw-r--r--. 1 dune012 dune       0 Sep 21 18:32 debugprod.log