Jobsub ID 17668.30@dunegpschedd01.fnal.gov

Jobsub ID	17668.30@dunegpschedd01.fnal.gov
Workflow ID	58
Stage ID	1
User name	epennacc@fnal.gov
HTCondor Group	group_dune.prod.mcsim
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-07-24 12:14:04
Site	NL_NIKHEF
Entry	VIRGO_NL_NIKHEF_brug
Last heartbeat	2025-07-24 16:27:01
From worker node	Hostname	wn-snel-026.farm.nikhef.nl
	cpuinfo	AMD EPYC 7H12 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	129600 (36 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-07-24 12:57:53
Input files	fardet-hd:prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250207T100220Z_gen_002218_supernova_g4_detsim_reco.root
Jobscript	Exit code	0
	Real time	3h (12530s)
	CPU time	3h (11354s = 90%)
	Max RSS bytes	1779974144 (1697 MiB)
Outputting started	2025-07-24 16:26:43
Output files
Finished	2025-07-24 16:27:01
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
Printing U view candidate length: 23
Printing V view candidate length: 24
Printing W view candidate length: 19
Printing 2D view candidate length: 66
Printing background CaloHitList length: 625
Printing New CaloHitList Names: 
SCaloHitListU | SCaloHitListV | SCaloHitListW | SCaloHitList2D | BBCaloHitList
*Electron identified*
*Electron identified*
*Electron identified*
*Photon identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Photon identified*
*Photon identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Photon identified*
*Photon identified*
*Electron identified*
*Electron identified*
*Photon identified*
*Photon identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
Printing U view candidate length: 19
Printing V view candidate length: 23
Printing W view candidate length: 15
Printing 2D view candidate length: 57
Printing background CaloHitList length: 5
Printing New CaloHitList Names: 
CaloHitListU | CaloHitListV | CaloHitListW | CaloHitList2D | BCaloHitList
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  24-Jul-2025 18:24:15 CEST run: 5089 subRun: 0 event: 110900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  24-Jul-2025 18:24:15 CEST run: 5089 subRun: 0 event: 110900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  24-Jul-2025 18:24:15 CEST run: 5089 subRun: 0 event: 110900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  24-Jul-2025 18:24:15 CEST run: 5089 subRun: 0 event: 110900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  24-Jul-2025 18:24:15 CEST run: 5089 subRun: 0 event: 110900
Comparing two wires in the same plane: return failure
%MSG
54970
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 71596!
Genrating OpFlash 0 with 7 hits
Genrating OpFlash 1 with 3 hits
Genrating OpFlash 2 with 3 hits
Genrating OpFlash 3 with 5 hits
Genrating OpFlash 4 with 5 hits
Genrating OpFlash 5 with 3 hits
Genrating OpFlash 6 with 4 hits
Genrating OpFlash 7 with 3 hits
Genrating OpFlash 8 with 3 hits
Genrating OpFlash 9 with 5 hits
...
Generated 2155 OpFlashes
24-Jul-2025 18:26:40 CEST  Closed output file "prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250207T100220Z_gen_002218_supernova_g4_detsim_reco_20250724T125756Z_reco2.root"
24-Jul-2025 18:26:40 CEST  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/ac/9a/prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250207T100220Z_gen_002218_supernova_g4_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              217.915       237.444       259.432       237.813       11.6893        50     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.095968      0.126745      0.219618      0.105564      0.0437161       50     
reco:linecluster:LineCluster                           0.879275      0.901424      0.945809      0.898605      0.014777        50     
reco:trajcluster:TrajCluster                           0.285137      0.294691      0.311371      0.293399     0.00504307       50     
reco:pandora:StandardPandora                            67.5388       82.3953       98.3969       84.3147       10.8932        50     
reco:pandoraTrack:LArPandoraTrackCreation             0.000127258   0.00269417    0.00943765    0.00206111    0.00200039       50     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000161703   0.000972344    0.0083821    0.000741919   0.00124884       50     
reco:pandoracalo:Calorimetry                          8.4979e-05    0.000274116   0.00271213    0.000217482   0.000358367      50     
reco:pandorapid:Chi2ParticleID                        3.0147e-05    7.7545e-05    0.00100041    6.10295e-05   0.000132405      50     
reco:pmtracktc:PMAlgTrackMaker                        0.00164694     0.0285438      0.12991      0.0160876     0.0288138       50     
reco:pmtracktccalo:Calorimetry                        6.1184e-05    0.000275839   0.000590846   0.000263047   0.000102877      50     
reco:pmtracktcpid:Chi2ParticleID                      2.3895e-05    6.6361e-05    9.3996e-05    6.55985e-05   1.57818e-05      50     
reco:emtrkmichelid:EmTrackMichelId                      5.86853       6.95059       8.2175        7.01297      0.531351        50     
reco:opdec:Deconvolution                                20.5087       21.7796       22.7172       21.9228      0.526498        50     
reco:ophitspe:OpHitFinderDeco                           106.167       112.852       119.436       113.62        3.35513        50     
reco:opflash:OpFlashFinder                              10.6448       11.7144       13.2773       11.7307      0.594841        50     
reco:solarflash:SolarOpFlash                           0.237581      0.258447      0.285046      0.257267      0.0100044       50     
reco:rns:RandomNumberSaver                             2.151e-05    3.02727e-05   0.000245069   2.5758e-05    3.07398e-05      50     
[art]:TriggerResults:TriggerResultInserter             1.09e-05     1.24908e-05   3.5958e-05    1.19225e-05   3.45666e-06      50     
end_path:out1:RootOutput                               3.046e-06    3.83498e-06   1.5258e-05    3.6015e-06    1.67228e-06      50     
end_path:out1:RootOutput(write)                        0.124435      0.137731      0.234917      0.134394      0.0202738       50     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3252.93 MB
  Peak resident set size usage (VmHWM): 1779.97 MB
====================================================================================================
Art has completed and will exit with status 0.
Moving on to metadata extractor 
============OUTPUT FILE:  prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250207T100220Z_gen_002218_supernova_g4_detsim_reco_20250724T125756Z_reco2.root
Extracted metadata
checking file====== fardet-hd:prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250207T100220Z_gen_002218_supernova_g4_detsim_reco.root
MD conversion to MetaCat OK
all-input-dids.txt
debugprod.log
jobscript.log
justin-processed-pfns.txt
prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250207T100220Z_gen_002218_supernova_g4_detsim_reco_20250724T125756Z_reco2.root
prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250207T100220Z_gen_002218_supernova_g4_detsim_reco_20250724T125756Z_reco2.root.ext.json
prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250207T100220Z_gen_002218_supernova_g4_detsim_reco_20250724T125756Z_reco2.root.json
prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250207T100220Z_gen_002218_supernova_g4_detsim_reco_20250724T125756Z_reco2.root.temp.json
reco2_hist.root