Jobsub ID 17655.80@dunegpschedd01.fnal.gov

Jobsub ID	17655.80@dunegpschedd01.fnal.gov
Workflow ID	58
Stage ID	1
User name	epennacc@fnal.gov
HTCondor Group	group_dune.prod.mcsim
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-07-24 11:30:02
Site	NL_NIKHEF
Entry	VIRGO_NL_NIKHEF_juk_multicore
Last heartbeat	2025-07-24 16:27:03
From worker node	Hostname	wn-sate-029.farm.nikhef.nl
	cpuinfo	AMD EPYC 7551P 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	129600 (36 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-07-24 12:05:06
Input files	fardet-hd:prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250206T092448Z_gen_000480_supernova_g4_detsim_reco.root
Jobscript	Exit code	0
	Real time	4h (15687s)
	CPU time	4h (14570s = 92%)
	Max RSS bytes	1781813248 (1699 MiB)
Outputting started	2025-07-24 16:26:34
Output files
Finished	2025-07-24 16:27:03
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

entified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
Printing U view candidate length: 24
Printing V view candidate length: 36
Printing W view candidate length: 26
Printing 2D view candidate length: 86
Printing background CaloHitList length: 607
Printing New CaloHitList Names: 
SCaloHitListU | SCaloHitListV | SCaloHitListW | SCaloHitList2D | BBCaloHitList
*Photon identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Photon identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Photon identified*
*Photon identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Photon identified*
*Electron identified*
*Photon identified*
*Photon identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Photon identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
Printing U view candidate length: 21
Printing V view candidate length: 33
Printing W view candidate length: 25
Printing 2D view candidate length: 79
Printing background CaloHitList length: 1
Printing New CaloHitList Names: 
CaloHitListU | CaloHitListV | CaloHitListW | CaloHitList2D | BCaloHitList
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  24-Jul-2025 18:23:23 CEST run: 5082 subRun: 0 event: 24000
Comparing two wires in the same plane: return failure
%MSG
54829
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 71472!
Genrating OpFlash 0 with 3 hits
Genrating OpFlash 1 with 3 hits
Genrating OpFlash 2 with 4 hits
Genrating OpFlash 3 with 3 hits
Genrating OpFlash 4 with 4 hits
Genrating OpFlash 5 with 4 hits
Genrating OpFlash 6 with 3 hits
Genrating OpFlash 7 with 3 hits
Genrating OpFlash 8 with 3 hits
Genrating OpFlash 9 with 4 hits
...
Generated 2240 OpFlashes
24-Jul-2025 18:26:30 CEST  Closed output file "prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250206T092448Z_gen_000480_supernova_g4_detsim_reco_20250724T120510Z_reco2.root"
24-Jul-2025 18:26:30 CEST  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/82/33/prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250206T092448Z_gen_000480_supernova_g4_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              283.12        300.989       329.428       299.42        11.0413        50     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0925008     0.126489      0.218864      0.106127      0.0440603       50     
reco:linecluster:LineCluster                            0.83807      0.862313      0.905665      0.859872      0.0180488       50     
reco:trajcluster:TrajCluster                           0.277323      0.283258      0.312878      0.280514     0.00860348       50     
reco:pandora:StandardPandora                            96.0878       110.454       135.823       110.215       9.25281        50     
reco:pandoraTrack:LArPandoraTrackCreation             0.000203232   0.00424638    0.00818282    0.00477055    0.00243211       50     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000222999   0.00134083     0.0066769    0.00109826     0.0011588       50     
reco:pandoracalo:Calorimetry                          0.000160723   0.000389478   0.00285814    0.000348386   0.000365247      50     
reco:pandorapid:Chi2ParticleID                        4.4775e-05    0.000109752   0.000918598   9.9437e-05    0.000117373      50     
reco:pmtracktc:PMAlgTrackMaker                        0.00333732     0.0274719     0.160644      0.0181492     0.0311202       50     
reco:pmtracktccalo:Calorimetry                        0.000128131   0.000389517   0.000754629   0.000385765   0.000143308      50     
reco:pmtracktcpid:Chi2ParticleID                       4.271e-05    9.69556e-05   0.000164029   9.58005e-05   2.58223e-05      50     
reco:emtrkmichelid:EmTrackMichelId                       6.661        9.05313       11.9122       8.8095        1.21692        50     
reco:opdec:Deconvolution                                23.6906       24.492        25.4661       24.3994      0.446821        50     
reco:ophitspe:OpHitFinderDeco                           135.763       140.122       143.991       140.226       1.62758        50     
reco:opflash:OpFlashFinder                              13.4273       15.0138       16.8725       14.7639      0.888683        50     
reco:solarflash:SolarOpFlash                           0.333352       0.36013      0.404634      0.358791      0.0144986       50     
reco:rns:RandomNumberSaver                            4.4614e-05    7.08092e-05   0.00037794    5.6581e-05    5.60051e-05      50     
[art]:TriggerResults:TriggerResultInserter            1.8214e-05    3.14614e-05   0.000351841   2.3665e-05    4.61341e-05      50     
end_path:out1:RootOutput                               6.442e-06    9.69794e-06   2.2763e-05    9.1425e-06    2.77652e-06      50     
end_path:out1:RootOutput(write)                        0.175793      0.186571       0.19931      0.185635     0.00590828       50     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3255.51 MB
  Peak resident set size usage (VmHWM): 1781.81 MB
====================================================================================================
Art has completed and will exit with status 0.
Moving on to metadata extractor 
============OUTPUT FILE:  prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250206T092448Z_gen_000480_supernova_g4_detsim_reco_20250724T120510Z_reco2.root
Extracted metadata
checking file====== fardet-hd:prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250206T092448Z_gen_000480_supernova_g4_detsim_reco.root
MD conversion to MetaCat OK
all-input-dids.txt
debugprod.log
jobscript.log
justin-processed-pfns.txt
prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250206T092448Z_gen_000480_supernova_g4_detsim_reco_20250724T120510Z_reco2.root
prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250206T092448Z_gen_000480_supernova_g4_detsim_reco_20250724T120510Z_reco2.root.ext.json
prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250206T092448Z_gen_000480_supernova_g4_detsim_reco_20250724T120510Z_reco2.root.json
prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250206T092448Z_gen_000480_supernova_g4_detsim_reco_20250724T120510Z_reco2.root.temp.json
reco2_hist.root