Jobsub ID 17654.31@dunegpschedd01.fnal.gov

Jobsub ID	17654.31@dunegpschedd01.fnal.gov
Workflow ID	58
Stage ID	1
User name	epennacc@fnal.gov
HTCondor Group	group_dune.prod.mcsim
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-07-24 11:28:02
Site	NL_NIKHEF
Entry	VIRGO_NL_NIKHEF_dissel
Last heartbeat	2025-07-24 16:26:59
From worker node	Hostname	wn-sate-038.farm.nikhef.nl
	cpuinfo	AMD EPYC 7551P 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	129600 (36 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-07-24 12:00:43
Input files	fardet-hd:prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250206T135905Z_gen_003722_supernova_g4_detsim_reco.root
Jobscript	Exit code	0
	Real time	4h (15958s)
	CPU time	4h (14735s = 92%)
	Max RSS bytes	1781018624 (1698 MiB)
Outputting started	2025-07-24 16:26:42
Output files
Finished	2025-07-24 16:26:59
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

l identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
Printing U view candidate length: 39
Printing V view candidate length: 34
Printing W view candidate length: 40
Printing 2D view candidate length: 113
Printing background CaloHitList length: 617
Printing New CaloHitList Names: 
SCaloHitListU | SCaloHitListV | SCaloHitListW | SCaloHitList2D | BBCaloHitList
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Photon identified*
*Photon identified*
*Photon identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Photon identified*
*Photon identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Photon identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
Printing U view candidate length: 29
Printing V view candidate length: 28
Printing W view candidate length: 31
Printing 2D view candidate length: 88
Printing background CaloHitList length: 6
Printing New CaloHitList Names: 
CaloHitListU | CaloHitListV | CaloHitListW | CaloHitList2D | BCaloHitList
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  24-Jul-2025 18:23:31 CEST run: 5082 subRun: 0 event: 186100
Comparing two wires in the same plane: return failure
%MSG
53958
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 69126!
Genrating OpFlash 0 with 3 hits
Genrating OpFlash 1 with 8 hits
Genrating OpFlash 2 with 5 hits
Genrating OpFlash 3 with 3 hits
Genrating OpFlash 4 with 3 hits
Genrating OpFlash 5 with 9 hits
Genrating OpFlash 6 with 3 hits
Genrating OpFlash 7 with 3 hits
Genrating OpFlash 8 with 3 hits
Genrating OpFlash 9 with 3 hits
...
Generated 2077 OpFlashes
24-Jul-2025 18:26:37 CEST  Closed output file "prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250206T135905Z_gen_003722_supernova_g4_detsim_reco_20250724T120048Z_reco2.root"
24-Jul-2025 18:26:37 CEST  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/a6/a1/prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250206T135905Z_gen_003722_supernova_g4_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              295.262       306.185       317.437       306.183       5.26614        50     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.095952      0.128931      0.223377       0.10437      0.0436442       50     
reco:linecluster:LineCluster                           0.840072      0.882984       1.28093       0.87257      0.0609851       50     
reco:trajcluster:TrajCluster                           0.278909      0.282155      0.310301      0.280285     0.00596856       50     
reco:pandora:StandardPandora                            107.473       114.996       123.112       115.158       4.0506         50     
reco:pandoraTrack:LArPandoraTrackCreation             0.000230715   0.00474781    0.00973779    0.00457644    0.00259627       50     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000235564    0.0011913    0.00931958    0.00104057    0.00121497       50     
reco:pandoracalo:Calorimetry                          0.000165752   0.000398651   0.00293985    0.000345611   0.000375546      50     
reco:pandorapid:Chi2ParticleID                        4.7419e-05    0.000111625   0.00113193    9.12825e-05   0.000146772      50     
reco:pmtracktc:PMAlgTrackMaker                         0.0050746     0.0312375     0.157416      0.0214329     0.0304934       50     
reco:pmtracktccalo:Calorimetry                        0.00015421    0.000388314   0.00074873    0.000363219   0.000130363      50     
reco:pmtracktcpid:Chi2ParticleID                      4.5125e-05    9.50621e-05   0.000193354   9.06755e-05   2.68949e-05      50     
reco:emtrkmichelid:EmTrackMichelId                      7.63638       9.0116        10.4731       8.96039      0.668664        50     
reco:opdec:Deconvolution                                23.2078       23.782        24.5528       23.7775      0.303189        50     
reco:ophitspe:OpHitFinderDeco                           134.678       140.477       145.539       140.804       2.09687        50     
reco:opflash:OpFlashFinder                               14.37        16.0401       19.7183       15.6922       1.22894        50     
reco:solarflash:SolarOpFlash                           0.322228      0.361964      0.444149       0.36257      0.0209613       50     
reco:rns:RandomNumberSaver                            3.9725e-05    5.82582e-05   0.000465677   4.80755e-05   5.85991e-05      50     
[art]:TriggerResults:TriggerResultInserter            1.7503e-05    2.31646e-05   7.1786e-05    2.21615e-05   7.39634e-06      50     
end_path:out1:RootOutput                               6.052e-06    7.95236e-06   2.7211e-05     7.614e-06    2.86641e-06      50     
end_path:out1:RootOutput(write)                        0.171088      0.182528      0.198371      0.182171     0.00575004       50     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3253.11 MB
  Peak resident set size usage (VmHWM): 1781.02 MB
====================================================================================================
Art has completed and will exit with status 0.
Moving on to metadata extractor 
============OUTPUT FILE:  prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250206T135905Z_gen_003722_supernova_g4_detsim_reco_20250724T120048Z_reco2.root
Extracted metadata
checking file====== fardet-hd:prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250206T135905Z_gen_003722_supernova_g4_detsim_reco.root
MD conversion to MetaCat OK
all-input-dids.txt
debugprod.log
jobscript.log
justin-processed-pfns.txt
prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250206T135905Z_gen_003722_supernova_g4_detsim_reco_20250724T120048Z_reco2.root
prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250206T135905Z_gen_003722_supernova_g4_detsim_reco_20250724T120048Z_reco2.root.ext.json
prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250206T135905Z_gen_003722_supernova_g4_detsim_reco_20250724T120048Z_reco2.root.json
prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250206T135905Z_gen_003722_supernova_g4_detsim_reco_20250724T120048Z_reco2.root.temp.json
reco2_hist.root