Jobsub ID 11089.133@dunegpschedd02.fnal.gov

Jobsub ID	11089.133@dunegpschedd02.fnal.gov
Workflow ID	59
Stage ID	1
User name	epennacc@fnal.gov
HTCondor Group	group_dune.prod.mcsim
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-07-25 00:20:41
Site	UK_Imperial
Entry	DUNE_T2_UK_London_IC_ceprod02
Last heartbeat	2025-07-25 17:13:27
From worker node	Hostname	wk01.grid.hep.ph.ic.ac.uk
	cpuinfo	Intel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-07-25 00:22:22
Input files	fardet-hd:prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_lateralAPA_20250209T194812Z_gen_004468_supernova_g4_halfActiveVol_detsim_reco.root
Jobscript	Exit code	0
	Real time	4h (15838s)
	CPU time	3h (13549s = 85%)
	Max RSS bytes	1369059328 (1305 MiB)
Outputting started	2025-07-25 04:46:21
Output files
Finished	2025-07-25 17:13:27
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

ires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  25-Jul-2025 05:35:02 BST run: 5100 subRun: 0 event: 223398
Comparing two wires in the same plane: return failure
%MSG
33094
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 40391!
Genrating OpFlash 0 with 3 hits
Genrating OpFlash 1 with 3 hits
Genrating OpFlash 2 with 6 hits
Genrating OpFlash 3 with 3 hits
Genrating OpFlash 4 with 3 hits
Genrating OpFlash 5 with 7 hits
Genrating OpFlash 6 with 3 hits
Genrating OpFlash 7 with 8 hits
Genrating OpFlash 8 with 3 hits
Genrating OpFlash 9 with 4 hits
...
Generated 1156 OpFlashes
Begin processing the 49th record. run: 5100 subRun: 0 event: 223399 at 25-Jul-2025 05:37:49 BST
Failure in algorithm Alg0002, LArDLSNSignal, unknown exception
Failure in algorithm Alg0003, LArDLSNSignal, unknown exception
PandoraContentApi::GetList(*this, listname, pCaloHitList) return STATUS_CODE_NOT_INITIALIZED
    in function: Infer
    in file:     /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/AlmaLinux-9.4/build/larpandoracontent/v04_12_00-buildFW/src/larpandoradlcontent/LArVertex/DlVertexingAlgorithm.cc line#: 194
iter->second->Run() throw STATUS_CODE_NOT_INITIALIZED
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0004, LArDLVertexing, STATUS_CODE_NOT_INITIALIZED
DLVertexing: Input vertex list is empty! Can't perform pass 2
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  25-Jul-2025 05:39:09 BST run: 5100 subRun: 0 event: 223399
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  25-Jul-2025 05:39:09 BST run: 5100 subRun: 0 event: 223399
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  25-Jul-2025 05:39:09 BST run: 5100 subRun: 0 event: 223399
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  25-Jul-2025 05:39:09 BST run: 5100 subRun: 0 event: 223399
Comparing two wires in the same plane: return failure
%MSG
33258
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 40909!
Genrating OpFlash 0 with 10 hits
Genrating OpFlash 1 with 3 hits
Genrating OpFlash 2 with 4 hits
Genrating OpFlash 3 with 3 hits
Genrating OpFlash 4 with 14 hits
Genrating OpFlash 5 with 3 hits
Genrating OpFlash 6 with 3 hits
Genrating OpFlash 7 with 4 hits
Genrating OpFlash 8 with 3 hits
Genrating OpFlash 9 with 5 hits
...
Generated 1211 OpFlashes
Begin processing the 50th record. run: 5100 subRun: 0 event: 223400 at 25-Jul-2025 05:42:00 BST
Failure in algorithm Alg0002, LArDLSNSignal, unknown exception
Failure in algorithm Alg0003, LArDLSNSignal, unknown exception
PandoraContentApi::GetList(*this, listname, pCaloHitList) return STATUS_CODE_NOT_INITIALIZED
    in function: Infer
    in file:     /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/AlmaLinux-9.4/build/larpandoracontent/v04_12_00-buildFW/src/larpandoradlcontent/LArVertex/DlVertexingAlgorithm.cc line#: 194
iter->second->Run() throw STATUS_CODE_NOT_INITIALIZED
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0004, LArDLVertexing, STATUS_CODE_NOT_INITIALIZED
DLVertexing: Input vertex list is empty! Can't perform pass 2
33680
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 41060!
Genrating OpFlash 0 with 13 hits
Genrating OpFlash 1 with 3 hits
Genrating OpFlash 2 with 3 hits
Genrating OpFlash 3 with 6 hits
Genrating OpFlash 4 with 4 hits
Genrating OpFlash 5 with 6 hits
Genrating OpFlash 6 with 3 hits
Genrating OpFlash 7 with 3 hits
Genrating OpFlash 8 with 3 hits
Genrating OpFlash 9 with 7 hits
...
Generated 1197 OpFlashes
25-Jul-2025 05:46:16 BST  Closed output file "prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_lateralAPA_20250209T194812Z_gen_004468_supernova_g4_halfActiveVol_detsim_reco_20250725T002226Z_reco2.root"
25-Jul-2025 05:46:16 BST  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/69/e8/prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_lateralAPA_20250209T194812Z_gen_004468_supernova_g4_halfActiveVol_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              244.255       300.385       1170.3        278.165       127.732        50     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0932552     0.146766      0.390675      0.120705      0.060767        50     
reco:linecluster:LineCluster                            0.84264       1.02032       1.42993       1.10094      0.141421        50     
reco:trajcluster:TrajCluster                           0.280077      0.330044      0.396915      0.361177      0.0418102       50     
reco:pandora:StandardPandora                            76.9867       108.233       972.013       88.3501       123.843        50     
reco:pandoraTrack:LArPandoraTrackCreation             0.000161206   0.000298304   0.00424035    0.000215577   0.000564686      50     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000178791   0.000272011   0.00184833    0.000210608    0.000232        50     
reco:pandoracalo:Calorimetry                          0.000138023   0.000179823   0.000870436   0.00015786    0.000103056      50     
reco:pandorapid:Chi2ParticleID                        3.5355e-05    5.49891e-05   0.00048835    4.32735e-05   6.28205e-05      50     
reco:pmtracktc:PMAlgTrackMaker                        0.00425755     0.0353577     0.304309      0.0193259     0.0483006       50     
reco:pmtracktccalo:Calorimetry                        0.000109686   0.000426544    0.0019374    0.000346695   0.000276921      50     
reco:pmtracktcpid:Chi2ParticleID                      4.9056e-05    0.000100684   0.000541742   8.47105e-05   7.03298e-05      50     
reco:emtrkmichelid:EmTrackMichelId                      5.01552       6.66946       10.9762       6.32612       1.2504         50     
reco:opdec:Deconvolution                                15.8668       18.0432       22.8973       17.4039       1.80862        50     
reco:ophitspe:OpHitFinderDeco                           134.755       159.607       212.596       152.425       20.7208        50     
reco:opflash:OpFlashFinder                              5.42782       5.9921        6.97356       5.91629      0.351426        50     
reco:solarflash:SolarOpFlash                           0.143012      0.187973      0.311965       0.15552      0.0558912       50     
reco:rns:RandomNumberSaver                            4.9859e-05    7.38829e-05   0.00060632    5.6141e-05    8.17961e-05      50     
[art]:TriggerResults:TriggerResultInserter            1.5948e-05    2.43608e-05   6.5578e-05    2.2882e-05    7.31785e-06      50     
end_path:out1:RootOutput                               5.877e-06    6.88582e-06   1.7151e-05     6.415e-06    1.63346e-06      50     
end_path:out1:RootOutput(write)                        0.0958861      0.11721      0.177639       0.10471      0.0244337       50     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 2859.11 MB
  Peak resident set size usage (VmHWM): 1369.06 MB
====================================================================================================
Art has completed and will exit with status 0.
Moving on to metadata extractor 
============OUTPUT FILE:  prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_lateralAPA_20250209T194812Z_gen_004468_supernova_g4_halfActiveVol_detsim_reco_20250725T002226Z_reco2.root
Extracted metadata
checking file====== fardet-hd:prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_lateralAPA_20250209T194812Z_gen_004468_supernova_g4_halfActiveVol_detsim_reco.root
MD conversion to MetaCat OK
all-input-dids.txt
debugprod.log
jobscript.log
justin-processed-pfns.txt
prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_lateralAPA_20250209T194812Z_gen_004468_supernova_g4_halfActiveVol_detsim_reco_20250725T002226Z_reco2.root
prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_lateralAPA_20250209T194812Z_gen_004468_supernova_g4_halfActiveVol_detsim_reco_20250725T002226Z_reco2.root.ext.json
prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_lateralAPA_20250209T194812Z_gen_004468_supernova_g4_halfActiveVol_detsim_reco_20250725T002226Z_reco2.root.json
prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_lateralAPA_20250209T194812Z_gen_004468_supernova_g4_halfActiveVol_detsim_reco_20250725T002226Z_reco2.root.temp.json
reco2_hist.root