Jobsub ID 17678.133@dunegpschedd01.fnal.gov

Jobsub ID	17678.133@dunegpschedd01.fnal.gov
Workflow ID	58
Stage ID	1
User name	epennacc@fnal.gov
HTCondor Group	group_dune.prod.mcsim
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-07-24 12:58:06
Site	UK_RAL-Tier1
Entry	LIGO_UK_RAL_arc_ce04
Last heartbeat	2025-07-24 16:27:02
From worker node	Hostname	dune001-6934862.0-lcg2695.gridpp.rl.ac.uk
	cpuinfo	AMD EPYC 9654 96-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	216000 (60 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-07-24 13:44:38
Input files	fardet-hd:prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250206T114103Z_gen_001028_supernova_g4_detsim_reco.root
Jobscript	Exit code	0
	Real time	2h (9726s)
	CPU time	2h (8427s = 86%)
	Max RSS bytes	1724186624 (1644 MiB)
Outputting started	2025-07-24 16:26:46
Output files
Finished	2025-07-24 16:27:02
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

OpFlash 5 with 3 hits
Genrating OpFlash 6 with 4 hits
Genrating OpFlash 7 with 3 hits
Genrating OpFlash 8 with 3 hits
Genrating OpFlash 9 with 3 hits
...
Generated 2140 OpFlashes
Begin processing the 50th record. run: 5082 subRun: 0 event: 51400 at 24-Jul-2025 16:23:50 UTC
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
Printing U view candidate length: 19
Printing V view candidate length: 20
Printing W view candidate length: 20
Printing 2D view candidate length: 59
Printing background CaloHitList length: 638
Printing New CaloHitList Names: 
SCaloHitListU | SCaloHitListV | SCaloHitListW | SCaloHitList2D | BBCaloHitList
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Photon identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Photon identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Photon identified*
*Electron identified*
*Photon identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
Printing U view candidate length: 19
Printing V view candidate length: 19
Printing W view candidate length: 19
Printing 2D view candidate length: 57
Printing background CaloHitList length: 3
Printing New CaloHitList Names: 
CaloHitListU | CaloHitListV | CaloHitListW | CaloHitList2D | BCaloHitList
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  24-Jul-2025 16:24:42 UTC run: 5082 subRun: 0 event: 51400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  24-Jul-2025 16:24:42 UTC run: 5082 subRun: 0 event: 51400
Comparing two wires in the same plane: return failure
%MSG
54437
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 70414!
Genrating OpFlash 0 with 4 hits
Genrating OpFlash 1 with 4 hits
Genrating OpFlash 2 with 3 hits
Genrating OpFlash 3 with 5 hits
Genrating OpFlash 4 with 3 hits
Genrating OpFlash 5 with 3 hits
Genrating OpFlash 6 with 4 hits
Genrating OpFlash 7 with 3 hits
Genrating OpFlash 8 with 6 hits
Genrating OpFlash 9 with 6 hits
...
Generated 2133 OpFlashes
24-Jul-2025 16:26:41 UTC  Closed output file "prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250206T114103Z_gen_001028_supernova_g4_detsim_reco_20250724T134442Z_reco2.root"
24-Jul-2025 16:26:41 UTC  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/bc/93/prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250206T114103Z_gen_001028_supernova_g4_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              160.793        181.2        200.935       180.351       8.97356        50     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.124773      0.166244      0.279325      0.136297      0.0573193       50     
reco:linecluster:LineCluster                            1.11397       1.15537       1.60193       1.14393      0.0750936       50     
reco:trajcluster:TrajCluster                           0.370935      0.377182      0.519546       0.37407      0.0204311       50     
reco:pandora:StandardPandora                            46.0059       51.7262       60.8142       51.5114       3.41109        50     
reco:pandoraTrack:LArPandoraTrackCreation             0.000121823   0.00166948    0.00816529    0.00152085    0.00132331       50     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000147231   0.000773535   0.00706896    0.00065412    0.000934388      50     
reco:pandoracalo:Calorimetry                          0.000103616   0.000264795   0.00222879    0.000220882   0.000289813      50     
reco:pandorapid:Chi2ParticleID                         2.667e-05    8.16689e-05   0.000871732   6.6766e-05    0.000114914      50     
reco:pmtracktc:PMAlgTrackMaker                        0.00158204     0.0173037     0.152718     0.00797346     0.0253711       50     
reco:pmtracktccalo:Calorimetry                        5.0216e-05    0.000225013   0.000465071   0.000221003   0.000102221      50     
reco:pmtracktcpid:Chi2ParticleID                      1.6755e-05    5.31871e-05   0.000101433   5.18735e-05   1.82703e-05      50     
reco:emtrkmichelid:EmTrackMichelId                      3.74644       4.75969       6.49849       4.71878      0.674504        50     
reco:opdec:Deconvolution                                13.5462       15.1244       17.7095       14.8882       1.04878        50     
reco:ophitspe:OpHitFinderDeco                           81.417        97.8405       119.984       98.1759       8.16332        50     
reco:opflash:OpFlashFinder                              8.79056       9.73815       12.0398       9.46028      0.692469        50     
reco:solarflash:SolarOpFlash                           0.150374      0.179884      0.278573      0.163151      0.0344304       50     
reco:rns:RandomNumberSaver                            4.1753e-05    5.50132e-05   0.000262475   4.90785e-05   3.02227e-05      50     
[art]:TriggerResults:TriggerResultInserter             1.334e-05    1.88475e-05    3.385e-05    1.86275e-05   2.93353e-06      50     
end_path:out1:RootOutput                               5.178e-06    8.85774e-06   2.3946e-05     8.157e-06    2.79962e-06      50     
end_path:out1:RootOutput(write)                        0.0972351      0.11081      0.155783      0.103065      0.017058        50     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3285.25 MB
  Peak resident set size usage (VmHWM): 1724.19 MB
====================================================================================================
Art has completed and will exit with status 0.
Moving on to metadata extractor 
============OUTPUT FILE:  prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250206T114103Z_gen_001028_supernova_g4_detsim_reco_20250724T134442Z_reco2.root
Extracted metadata
checking file====== fardet-hd:prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250206T114103Z_gen_001028_supernova_g4_detsim_reco.root
MD conversion to MetaCat OK
all-input-dids.txt
debugprod.log
jobscript.log
justin-processed-pfns.txt
prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250206T114103Z_gen_001028_supernova_g4_detsim_reco_20250724T134442Z_reco2.root
prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250206T114103Z_gen_001028_supernova_g4_detsim_reco_20250724T134442Z_reco2.root.ext.json
prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250206T114103Z_gen_001028_supernova_g4_detsim_reco_20250724T134442Z_reco2.root.json
prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250206T114103Z_gen_001028_supernova_g4_detsim_reco_20250724T134442Z_reco2.root.temp.json
reco2_hist.root