Jobsub ID 17672.66@dunegpschedd01.fnal.gov

Jobsub ID	17672.66@dunegpschedd01.fnal.gov
Workflow ID	58
Stage ID	1
User name	epennacc@fnal.gov
HTCondor Group	group_dune.prod.mcsim
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-07-24 12:32:04
Site	NL_NIKHEF
Entry	VIRGO_NL_NIKHEF_brug_multicore
Last heartbeat	2025-07-24 16:27:00
From worker node	Hostname	wn-snel-007.farm.nikhef.nl
	cpuinfo	AMD EPYC 7H12 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	129600 (36 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-07-24 13:19:10
Input files	fardet-hd:prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250207T084850Z_gen_001239_supernova_g4_detsim_reco.root
Jobscript	Exit code	0
	Real time	3h (11254s)
	CPU time	2h (10110s = 89%)
	Max RSS bytes	1781927936 (1699 MiB)
Outputting started	2025-07-24 16:26:45
Output files
Finished	2025-07-24 16:27:00
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

ting U view candidate length: 15
Printing V view candidate length: 13
Printing W view candidate length: 11
Printing 2D view candidate length: 39
Printing background CaloHitList length: 10
Printing New CaloHitList Names: 
CaloHitListU | CaloHitListV | CaloHitListW | CaloHitList2D | BCaloHitList
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  24-Jul-2025 18:20:40 CEST run: 5089 subRun: 0 event: 61949
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  24-Jul-2025 18:20:40 CEST run: 5089 subRun: 0 event: 61949
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  24-Jul-2025 18:20:40 CEST run: 5089 subRun: 0 event: 61949
Comparing two wires in the same plane: return failure
%MSG
%MSG-e pma::Track3D:  PMAlgTrackMaker:pmtracktc@BeginModule  24-Jul-2025 18:20:40 CEST run: 5089 subRun: 0 event: 61949
Flip, endpoint closer to vStart.
%MSG
54268
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 69987!
Genrating OpFlash 0 with 3 hits
Genrating OpFlash 1 with 6 hits
Genrating OpFlash 2 with 5 hits
Genrating OpFlash 3 with 3 hits
Genrating OpFlash 4 with 3 hits
Genrating OpFlash 5 with 3 hits
Genrating OpFlash 6 with 3 hits
Genrating OpFlash 7 with 4 hits
Genrating OpFlash 8 with 4 hits
Genrating OpFlash 9 with 4 hits
...
Generated 2054 OpFlashes
Begin processing the 50th record. run: 5089 subRun: 0 event: 61950 at 24-Jul-2025 18:23:05 CEST
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
Printing U view candidate length: 9
Printing V view candidate length: 13
Printing W view candidate length: 10
Printing 2D view candidate length: 32
Printing background CaloHitList length: 664
Printing New CaloHitList Names: 
SCaloHitListU | SCaloHitListV | SCaloHitListW | SCaloHitList2D | BBCaloHitList
*Photon identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Photon identified*
Printing U view candidate length: 8
Printing V view candidate length: 13
Printing W view candidate length: 8
Printing 2D view candidate length: 29
Printing background CaloHitList length: 1
Printing New CaloHitList Names: 
CaloHitListU | CaloHitListV | CaloHitListW | CaloHitList2D | BCaloHitList
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  24-Jul-2025 18:24:17 CEST run: 5089 subRun: 0 event: 61950
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  24-Jul-2025 18:24:17 CEST run: 5089 subRun: 0 event: 61950
Comparing two wires in the same plane: return failure
%MSG
54232
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 69798!
Genrating OpFlash 0 with 3 hits
Genrating OpFlash 1 with 3 hits
Genrating OpFlash 2 with 12 hits
Genrating OpFlash 3 with 5 hits
Genrating OpFlash 4 with 9 hits
Genrating OpFlash 5 with 4 hits
Genrating OpFlash 6 with 3 hits
Genrating OpFlash 7 with 11 hits
Genrating OpFlash 8 with 6 hits
Genrating OpFlash 9 with 11 hits
...
Generated 2114 OpFlashes
24-Jul-2025 18:26:42 CEST  Closed output file "prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250207T084850Z_gen_001239_supernova_g4_detsim_reco_20250724T131914Z_reco2.root"
24-Jul-2025 18:26:42 CEST  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/c1/b0/prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250207T084850Z_gen_001239_supernova_g4_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              202.802       211.715       221.108       211.998       4.4697         50     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0982391     0.130285      0.223771      0.111834      0.043153        50     
reco:linecluster:LineCluster                           0.895803      0.913949      0.951213      0.912018      0.0159076       50     
reco:trajcluster:TrajCluster                            0.29644      0.300689      0.330311      0.298582     0.00679489       50     
reco:pandora:StandardPandora                            60.7664       64.2067       68.0804       63.8638       1.87771        50     
reco:pandoraTrack:LArPandoraTrackCreation             0.000116087   0.00140853    0.00414069    0.00138162    0.000705829      50     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000146305   0.000769538   0.00578228    0.00066137    0.000761201      50     
reco:pandoracalo:Calorimetry                          8.6162e-05    0.000247367   0.00164843    0.000209518   0.000215375      50     
reco:pandorapid:Chi2ParticleID                        2.7261e-05    6.99467e-05   0.000653535   5.82445e-05   8.44611e-05      50     
reco:pmtracktc:PMAlgTrackMaker                        0.00171774     0.0297382     0.228615      0.0114537     0.0438953       50     
reco:pmtracktccalo:Calorimetry                         5.861e-05    0.00020886    0.000523712   0.000193037   9.15479e-05      50     
reco:pmtracktcpid:Chi2ParticleID                      2.2573e-05    5.42857e-05   0.000119494   5.1446e-05    1.7139e-05       50     
reco:emtrkmichelid:EmTrackMichelId                      5.39083       6.3651        7.94252       6.39139      0.506163        50     
reco:opdec:Deconvolution                                19.4731       20.5972       23.8799       20.285       0.987593        50     
reco:ophitspe:OpHitFinderDeco                           102.51        108.422       115.622       107.783       3.12887        50     
reco:opflash:OpFlashFinder                              9.5115        10.3758       12.1287       10.2294      0.547723        50     
reco:solarflash:SolarOpFlash                            0.22224      0.243141      0.403077      0.240415      0.0246114       50     
reco:rns:RandomNumberSaver                            2.0619e-05    2.76165e-05   0.000217287   2.30385e-05   2.72218e-05      50     
[art]:TriggerResults:TriggerResultInserter             9.518e-06    1.1504e-05    3.3292e-05    1.07105e-05   3.33671e-06      50     
end_path:out1:RootOutput                               2.876e-06    3.54694e-06   1.3996e-05     3.181e-06    1.61914e-06      50     
end_path:out1:RootOutput(write)                        0.117695      0.127145      0.229991      0.122893      0.0206645       50     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3255.59 MB
  Peak resident set size usage (VmHWM): 1781.93 MB
====================================================================================================
Art has completed and will exit with status 0.
Moving on to metadata extractor 
============OUTPUT FILE:  prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250207T084850Z_gen_001239_supernova_g4_detsim_reco_20250724T131914Z_reco2.root
Extracted metadata
checking file====== fardet-hd:prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250207T084850Z_gen_001239_supernova_g4_detsim_reco.root
MD conversion to MetaCat OK
all-input-dids.txt
debugprod.log
jobscript.log
justin-processed-pfns.txt
prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250207T084850Z_gen_001239_supernova_g4_detsim_reco_20250724T131914Z_reco2.root
prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250207T084850Z_gen_001239_supernova_g4_detsim_reco_20250724T131914Z_reco2.root.ext.json
prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250207T084850Z_gen_001239_supernova_g4_detsim_reco_20250724T131914Z_reco2.root.json
prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250207T084850Z_gen_001239_supernova_g4_detsim_reco_20250724T131914Z_reco2.root.temp.json
reco2_hist.root