Jobsub ID 17654.135@dunegpschedd01.fnal.gov

Jobsub ID	17654.135@dunegpschedd01.fnal.gov
Workflow ID	58
Stage ID	1
User name	epennacc@fnal.gov
HTCondor Group	group_dune.prod.mcsim
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-07-24 11:28:02
Site	UK_RAL-Tier1
Entry	LIGO_UK_RAL_arc_ce02
Last heartbeat	2025-07-24 16:27:02
From worker node	Hostname	dune001-7387822.0-lcg2636.gridpp.rl.ac.uk
	cpuinfo	AMD EPYC 7763 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	216000 (60 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-07-24 12:03:16
Input files	fardet-hd:prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250208T114617Z_gen_001700_supernova_g4_detsim_reco.root
Jobscript	Exit code	0
	Real time	4h (15805s)
	CPU time	3h (14250s = 90%)
	Max RSS bytes	1693118464 (1614 MiB)
Outputting started	2025-07-24 16:26:42
Output files
Finished	2025-07-24 16:27:02
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

nal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
Printing U view candidate length: 18
Printing V view candidate length: 34
Printing W view candidate length: 26
Printing 2D view candidate length: 78
Printing background CaloHitList length: 688
Printing New CaloHitList Names: 
SCaloHitListU | SCaloHitListV | SCaloHitListW | SCaloHitList2D | BBCaloHitList
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Photon identified*
*Photon identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Photon identified*
*Photon identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
Printing U view candidate length: 13
Printing V view candidate length: 33
Printing W view candidate length: 26
Printing 2D view candidate length: 72
Printing background CaloHitList length: 2
Printing New CaloHitList Names: 
CaloHitListU | CaloHitListV | CaloHitListW | CaloHitList2D | BCaloHitList
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  24-Jul-2025 16:22:38 UTC run: 5097 subRun: 0 event: 85000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  24-Jul-2025 16:22:38 UTC run: 5097 subRun: 0 event: 85000
Comparing two wires in the same plane: return failure
%MSG
55204
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 71413!
Genrating OpFlash 0 with 3 hits
Genrating OpFlash 1 with 3 hits
Genrating OpFlash 2 with 5 hits
Genrating OpFlash 3 with 3 hits
Genrating OpFlash 4 with 3 hits
Genrating OpFlash 5 with 3 hits
Genrating OpFlash 6 with 3 hits
Genrating OpFlash 7 with 4 hits
Genrating OpFlash 8 with 6 hits
Genrating OpFlash 9 with 3 hits
...
Generated 2186 OpFlashes
24-Jul-2025 16:26:35 UTC  Closed output file "prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250208T114617Z_gen_001700_supernova_g4_detsim_reco_20250724T120324Z_reco2.root"
24-Jul-2025 16:26:36 UTC  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/ad/8c/prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250208T114617Z_gen_001700_supernova_g4_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              250.571       299.949       377.242       297.293       22.4672        50     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.137955      0.191442      0.878694      0.150227      0.114283        50     
reco:linecluster:LineCluster                            1.25487       1.28281       1.43803       1.27649      0.0302412       50     
reco:trajcluster:TrajCluster                           0.414055      0.421759      0.461621      0.416464      0.011411        50     
reco:pandora:StandardPandora                            64.2956       77.7561       100.124       76.9236       7.86362        50     
reco:pandoraTrack:LArPandoraTrackCreation             0.000209905   0.00254402     0.0119002    0.00224529    0.00182018       50     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000220206   0.00117769     0.0120865    0.000881078   0.00158976       50     
reco:pandoracalo:Calorimetry                          0.000137573   0.000412141    0.0043345    0.000315008   0.000569692      50     
reco:pandorapid:Chi2ParticleID                        4.2171e-05    0.000118551   0.00160705    8.4447e-05    0.000213709      50     
reco:pmtracktc:PMAlgTrackMaker                        0.00290447     0.0291759     0.130161      0.0227316     0.0272744       50     
reco:pmtracktccalo:Calorimetry                        8.0052e-05    0.000373664   0.000881192   0.000356999   0.000151629      50     
reco:pmtracktcpid:Chi2ParticleID                      3.3841e-05    8.9427e-05    0.000181464   8.4527e-05    2.78823e-05      50     
reco:emtrkmichelid:EmTrackMichelId                      5.54761       7.57741       10.2425       7.04316       1.40849        50     
reco:opdec:Deconvolution                                18.3558       22.7335       25.1544       23.0714       1.83223        50     
reco:ophitspe:OpHitFinderDeco                           144.146       177.18        230.311       175.355       17.2866        50     
reco:opflash:OpFlashFinder                              11.2695       12.2959       13.7213       12.0845      0.598112        50     
reco:solarflash:SolarOpFlash                           0.197689      0.305094      0.441893      0.336352      0.0836494       50     
reco:rns:RandomNumberSaver                            3.7931e-05    6.13364e-05   0.000273407   5.7751e-05    3.18925e-05      50     
[art]:TriggerResults:TriggerResultInserter             1.597e-05    2.4019e-05    4.6211e-05    2.25455e-05   6.36961e-06      50     
end_path:out1:RootOutput                               2.79e-06     7.59684e-06   2.0181e-05     6.95e-06     3.25326e-06      50     
end_path:out1:RootOutput(write)                        0.130251      0.170483      0.217762      0.187128      0.0311413       50     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3287.22 MB
  Peak resident set size usage (VmHWM): 1693.12 MB
====================================================================================================
Art has completed and will exit with status 0.
Moving on to metadata extractor 
============OUTPUT FILE:  prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250208T114617Z_gen_001700_supernova_g4_detsim_reco_20250724T120324Z_reco2.root
Extracted metadata
checking file====== fardet-hd:prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250208T114617Z_gen_001700_supernova_g4_detsim_reco.root
MD conversion to MetaCat OK
all-input-dids.txt
debugprod.log
jobscript.log
justin-processed-pfns.txt
prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250208T114617Z_gen_001700_supernova_g4_detsim_reco_20250724T120324Z_reco2.root
prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250208T114617Z_gen_001700_supernova_g4_detsim_reco_20250724T120324Z_reco2.root.ext.json
prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250208T114617Z_gen_001700_supernova_g4_detsim_reco_20250724T120324Z_reco2.root.json
prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250208T114617Z_gen_001700_supernova_g4_detsim_reco_20250724T120324Z_reco2.root.temp.json
reco2_hist.root