Jobsub ID 17791.107@dunegpschedd01.fnal.gov

Jobsub ID	17791.107@dunegpschedd01.fnal.gov
Workflow ID	62
Stage ID	1
User name	higuera@fnal.gov
HTCondor Group	group_dune.prod.mcsim
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-07-24 23:34:38
Site	US_FNAL-FermiGrid
Entry	FNAL_GPGrid_ce03_mcore_op_duneonly
Last heartbeat	2025-07-25 03:44:22
From worker node	Hostname	dunegli-6171140-0-fnpc23046.fnal.gov
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	172800 (48 hours)
	GPU
	Inner Apptainer?	True
Job state	finished
Started	2025-07-25 01:30:28
Input files	fardet-hd:prodmarley_nue_es_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250217T074339Z_gen_001023_supernova_g4_detsim_reco.root
Jobscript	Exit code	0
	Real time	2h (7984s)
	CPU time	2h (7899s = 98%)
	Max RSS bytes	1781604352 (1699 MiB)
Outputting started	2025-07-25 03:43:33
Output files	fardet-hd:prodmarley_nue_es_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250217T074339Z_gen_001023_supernova_g4_detsim_reco_20250725T013031Z_reco2.root
Finished	2025-07-25 03:44:22
Saved logs	justin-logs:17791.107-dunegpschedd01.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

put vertex list is empty! Can't perform pass 2
54548
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 70351!
Genrating OpFlash 0 with 5 hits
Genrating OpFlash 1 with 5 hits
Genrating OpFlash 2 with 3 hits
Genrating OpFlash 3 with 3 hits
Genrating OpFlash 4 with 3 hits
Genrating OpFlash 5 with 4 hits
Genrating OpFlash 6 with 4 hits
Genrating OpFlash 7 with 6 hits
Genrating OpFlash 8 with 4 hits
Genrating OpFlash 9 with 4 hits
...
Generated 2163 OpFlashes
Begin processing the 50th record. run: 5187 subRun: 0 event: 51150 at 25-Jul-2025 03:40:53 UTC
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
Printing U view candidate length: 12
Printing V view candidate length: 21
Printing W view candidate length: 18
Printing 2D view candidate length: 51
Printing background CaloHitList length: 649
Printing New CaloHitList Names: 
SCaloHitListU | SCaloHitListV | SCaloHitListW | SCaloHitList2D | BBCaloHitList
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
Printing U view candidate length: 12
Printing V view candidate length: 21
Printing W view candidate length: 18
Printing 2D view candidate length: 51
Printing background CaloHitList length: 0
Printing New CaloHitList Names: 
CaloHitListU | CaloHitListV | CaloHitListW | CaloHitList2D | BCaloHitList
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  25-Jul-2025 03:41:31 UTC run: 5187 subRun: 0 event: 51150
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  25-Jul-2025 03:41:31 UTC run: 5187 subRun: 0 event: 51150
Comparing two wires in the same plane: return failure
%MSG
54754
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 70761!
Genrating OpFlash 0 with 3 hits
Genrating OpFlash 1 with 3 hits
Genrating OpFlash 2 with 5 hits
Genrating OpFlash 3 with 4 hits
Genrating OpFlash 4 with 3 hits
Genrating OpFlash 5 with 3 hits
Genrating OpFlash 6 with 3 hits
Genrating OpFlash 7 with 3 hits
Genrating OpFlash 8 with 3 hits
Genrating OpFlash 9 with 6 hits
...
Generated 2178 OpFlashes
25-Jul-2025 03:43:30 UTC  Closed output file "prodmarley_nue_es_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250217T074339Z_gen_001023_supernova_g4_detsim_reco_20250725T013031Z_reco2.root"
25-Jul-2025 03:43:30 UTC  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/b9/cf/prodmarley_nue_es_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250217T074339Z_gen_001023_supernova_g4_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              153.656       158.045       162.113       157.977       2.21288        50     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00159983     0.0385601     0.676772      0.019866      0.0980329       50     
reco:linecluster:LineCluster                           0.0123561     0.0503314     0.283135      0.0438412     0.0404385       50     
reco:trajcluster:TrajCluster                          0.00330688    0.00590197     0.0317635    0.00469121    0.00522437       50     
reco:pandora:StandardPandora                            35.9866       37.3905       39.1649       37.4014      0.652198        50     
reco:pandoraTrack:LArPandoraTrackCreation             0.000108679   0.00105731    0.00760638    0.00118179    0.00108832       50     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000130972   0.000634964   0.00808697    0.000477597   0.00110111       50     
reco:pandoracalo:Calorimetry                          8.8129e-05    0.000208833   0.00240108    0.000169415   0.000318456      50     
reco:pandorapid:Chi2ParticleID                        2.6701e-05    6.90485e-05   0.000939212   5.1243e-05    0.000128402      50     
reco:pmtracktc:PMAlgTrackMaker                         0.001578      0.014884      0.122648     0.00408241     0.025756        50     
reco:pmtracktccalo:Calorimetry                        4.8093e-05    0.000146603   0.000288824   0.00014121    5.83404e-05      50     
reco:pmtracktcpid:Chi2ParticleID                      1.8365e-05    4.2255e-05    6.8201e-05    4.16045e-05   1.13956e-05      50     
reco:emtrkmichelid:EmTrackMichelId                      3.94115       4.55169       5.39912       4.55852      0.274315        50     
reco:opdec:Deconvolution                                12.0473       12.3859       13.1128       12.3648      0.201994        50     
reco:ophitspe:OpHitFinderDeco                           91.9539       94.8718       97.525        94.7333       1.48578        50     
reco:opflash:OpFlashFinder                              7.99189       8.47324       9.43955       8.45177      0.309327        50     
reco:solarflash:SolarOpFlash                           0.143962       0.15887      0.174838      0.159006     0.00640448       50     
reco:rns:RandomNumberSaver                            3.9897e-05    5.24234e-05   0.000292411   4.72255e-05   3.45805e-05      50     
[art]:TriggerResults:TriggerResultInserter            1.3495e-05    1.75984e-05    3.654e-05    1.6717e-05    3.76894e-06      50     
end_path:out1:RootOutput                               5.681e-06    7.4564e-06    2.1301e-05    7.0435e-06    2.33923e-06      50     
end_path:out1:RootOutput(write)                        0.094573      0.0997393     0.107416      0.0995967    0.00286004       50     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3250.42 MB
  Peak resident set size usage (VmHWM): 1781.6 MB
====================================================================================================
Art has completed and will exit with status 0.
Moving on to metadata extractor 
============OUTPUT FILE:  prodmarley_nue_es_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250217T074339Z_gen_001023_supernova_g4_detsim_reco_20250725T013031Z_reco2.root
Extracted metadata
checking file====== fardet-hd:prodmarley_nue_es_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250217T074339Z_gen_001023_supernova_g4_detsim_reco.root
MD conversion to MetaCat OK
all-input-dids.txt
debugprod.log
jobscript.log
justin-processed-pfns.txt
prodmarley_nue_es_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250217T074339Z_gen_001023_supernova_g4_detsim_reco_20250725T013031Z_reco2.root
prodmarley_nue_es_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250217T074339Z_gen_001023_supernova_g4_detsim_reco_20250725T013031Z_reco2.root.ext.json
prodmarley_nue_es_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250217T074339Z_gen_001023_supernova_g4_detsim_reco_20250725T013031Z_reco2.root.json
prodmarley_nue_es_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250217T074339Z_gen_001023_supernova_g4_detsim_reco_20250725T013031Z_reco2.root.temp.json
reco2_hist.root