Jobsub ID 17624.41@dunegpschedd01.fnal.gov

Jobsub ID	17624.41@dunegpschedd01.fnal.gov
Workflow ID	55
Stage ID	1
User name	higuera@fnal.gov
HTCondor Group	group_dune.prod.mcsim
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-07-24 09:29:52
Site	US_FNAL-T1
Entry	CMSHTPC_T1_US_FNAL_condce_opp1_whole
Last heartbeat	2025-07-24 22:45:43
From worker node	Hostname	dunegli-44332-0-cmswn3007.fnal.gov
	cpuinfo	AMD EPYC 7542 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	GPU
	Inner Apptainer?	True
Job state	finished
Started	2025-07-24 13:03:31
Input files	fardet-hd:prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250205T172038Z_gen_000985_supernova_g4_detsim_reco.root
Jobscript	Exit code	0
	Real time	9h (34797s)
	CPU time	9h (34444s = 98%)
	Max RSS bytes	1726521344 (1646 MiB)
Outputting started	2025-07-24 22:43:29
Output files	fardet-hd:prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250205T172038Z_gen_000985_supernova_g4_detsim_reco_20250724T130342Z_reco2.root
Finished	2025-07-24 22:45:43
Saved logs	justin-logs:17624.41-dunegpschedd01.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

MAlgTrackMaker:pmtracktc@BeginModule  24-Jul-2025 22:28:30 UTC run: 5064 subRun: 0 event: 49249
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  24-Jul-2025 22:28:30 UTC run: 5064 subRun: 0 event: 49249
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  24-Jul-2025 22:28:30 UTC run: 5064 subRun: 0 event: 49249
Comparing two wires in the same plane: return failure
%MSG
55062
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 71799!
Genrating OpFlash 0 with 13 hits
Genrating OpFlash 1 with 3 hits
Genrating OpFlash 2 with 3 hits
Genrating OpFlash 3 with 4 hits
Genrating OpFlash 4 with 5 hits
Genrating OpFlash 5 with 3 hits
Genrating OpFlash 6 with 4 hits
Genrating OpFlash 7 with 3 hits
Genrating OpFlash 8 with 5 hits
Genrating OpFlash 9 with 3 hits
...
Generated 2280 OpFlashes
Begin processing the 50th record. run: 5064 subRun: 0 event: 49250 at 24-Jul-2025 22:33:14 UTC
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
Printing U view candidate length: 21
Printing V view candidate length: 12
Printing W view candidate length: 9
Printing 2D view candidate length: 42
Printing background CaloHitList length: 608
Printing New CaloHitList Names: 
SCaloHitListU | SCaloHitListV | SCaloHitListW | SCaloHitList2D | BBCaloHitList
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
Printing U view candidate length: 20
Printing V view candidate length: 12
Printing W view candidate length: 7
Printing 2D view candidate length: 39
Printing background CaloHitList length: 5
Printing New CaloHitList Names: 
CaloHitListU | CaloHitListV | CaloHitListW | CaloHitList2D | BCaloHitList
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  24-Jul-2025 22:38:47 UTC run: 5064 subRun: 0 event: 49250
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  24-Jul-2025 22:38:47 UTC run: 5064 subRun: 0 event: 49250
Comparing two wires in the same plane: return failure
%MSG
54312
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 70514!
Genrating OpFlash 0 with 3 hits
Genrating OpFlash 1 with 5 hits
Genrating OpFlash 2 with 4 hits
Genrating OpFlash 3 with 3 hits
Genrating OpFlash 4 with 3 hits
Genrating OpFlash 5 with 3 hits
Genrating OpFlash 6 with 4 hits
Genrating OpFlash 7 with 3 hits
Genrating OpFlash 8 with 4 hits
Genrating OpFlash 9 with 3 hits
...
Generated 2168 OpFlashes
24-Jul-2025 22:43:19 UTC  Closed output file "prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250205T172038Z_gen_000985_supernova_g4_detsim_reco_20250724T130342Z_reco2.root"
24-Jul-2025 22:43:19 UTC  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/24/4b/prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250205T172038Z_gen_000985_supernova_g4_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              481.111       690.232       807.735       712.142       68.9372        50     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00395473     0.0174679     0.0376955     0.0170424    0.00857006       50     
reco:linecluster:LineCluster                           0.029094      0.0779766     0.593317      0.0630446     0.0835309       50     
reco:trajcluster:TrajCluster                          0.00936932     0.0174739     0.0493256     0.0162759    0.00686291       50     
reco:pandora:StandardPandora                            228.086       405.629       496.48        425.395       63.9536        50     
reco:pandoraTrack:LArPandoraTrackCreation             0.00095161     0.0199587     0.0390666     0.0217365    0.00940772       50     
reco:pandoraShower:LArPandoraModularShowerCreation    0.00128814    0.00374099     0.029479     0.00255483    0.00512578       50     
reco:pandoracalo:Calorimetry                          0.000268424   0.00103105    0.00291715    0.000892362   0.000555892      50     
reco:pandorapid:Chi2ParticleID                        9.8308e-05    0.000288701   0.000892918   0.000243536   0.000167751      50     
reco:pmtracktc:PMAlgTrackMaker                        0.00498919     0.0497451     0.216102      0.0314232     0.0516396       50     
reco:pmtracktccalo:Calorimetry                        0.000343386   0.000869371   0.00167055    0.000815294   0.000329094      50     
reco:pmtracktcpid:Chi2ParticleID                      6.4112e-05    0.000201728   0.000454138   0.000168352   9.15666e-05      50     
reco:emtrkmichelid:EmTrackMichelId                      8.62911       29.621        43.034        28.7981       6.21459        50     
reco:opdec:Deconvolution                                19.5596       26.9574       37.5794       26.8389       4.10609        50     
reco:ophitspe:OpHitFinderDeco                           155.826       178.216       209.266       175.894       13.6862        50     
reco:opflash:OpFlashFinder                              23.4676       48.7955       68.5028       50.1516       11.7489        50     
reco:solarflash:SolarOpFlash                           0.346772       0.54643       1.01578      0.524535      0.151312        50     
reco:rns:RandomNumberSaver                            8.1265e-05    0.000186189    0.0019778    0.000137147   0.00026178       50     
[art]:TriggerResults:TriggerResultInserter            2.9036e-05    6.05398e-05   0.000363606   4.6409e-05    4.8385e-05       50     
end_path:out1:RootOutput                              1.1031e-05    1.99152e-05   8.7387e-05    1.6947e-05    1.18724e-05      50     
end_path:out1:RootOutput(write)                        0.198766       0.27351      0.385666      0.268846      0.0490741       50     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3254.49 MB
  Peak resident set size usage (VmHWM): 1726.52 MB
====================================================================================================
Art has completed and will exit with status 0.
Moving on to metadata extractor 
============OUTPUT FILE:  prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250205T172038Z_gen_000985_supernova_g4_detsim_reco_20250724T130342Z_reco2.root
Extracted metadata
checking file====== fardet-hd:prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250205T172038Z_gen_000985_supernova_g4_detsim_reco.root
MD conversion to MetaCat OK
all-input-dids.txt
debugprod.log
jobscript.log
justin-processed-pfns.txt
old_md.json
prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250205T172038Z_gen_000985_supernova_g4_detsim_reco_20250724T130342Z_reco2.root
prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250205T172038Z_gen_000985_supernova_g4_detsim_reco_20250724T130342Z_reco2.root.ext.json
prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250205T172038Z_gen_000985_supernova_g4_detsim_reco_20250724T130342Z_reco2.root.json
prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250205T172038Z_gen_000985_supernova_g4_detsim_reco_20250724T130342Z_reco2.root.temp.json
reco2_hist.root