Jobsub ID 17733.1@dunegpschedd01.fnal.gov

Jobsub ID	17733.1@dunegpschedd01.fnal.gov
Workflow ID	58
Stage ID	1
User name	epennacc@fnal.gov
HTCondor Group	group_dune.prod.mcsim
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-07-24 17:48:18
Site	US_UChicago
Entry	Engage_US_MWT2_uiuc_condce_mcore
Last heartbeat	2025-07-25 03:14:26
From worker node	Hostname	mwt2-c141.campuscluster.illinois.edu
	cpuinfo	Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	86400 (24 hours)
	GPU
	Inner Apptainer?	True
Job state	finished
Started	2025-07-24 17:53:09
Input files	fardet-hd:prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250206T122900Z_gen_002634_supernova_g4_detsim_reco.root
Jobscript	Exit code	0
	Real time	9h (33555s)
	CPU time	9h (32792s = 97%)
	Max RSS bytes	1750331392 (1669 MiB)
Outputting started	2025-07-25 03:12:25
Output files	fardet-hd:prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250206T122900Z_gen_002634_supernova_g4_detsim_reco_20250724T175313Z_reco2.root
Finished	2025-07-25 03:14:26
Saved logs	justin-logs:17733.1-dunegpschedd01.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

inting U view candidate length: 19
Printing V view candidate length: 15
Printing W view candidate length: 14
Printing 2D view candidate length: 48
Printing background CaloHitList length: 2
Printing New CaloHitList Names: 
CaloHitListU | CaloHitListV | CaloHitListW | CaloHitList2D | BCaloHitList
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  24-Jul-2025 21:53:18 CDT run: 5082 subRun: 0 event: 131699
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  24-Jul-2025 21:53:18 CDT run: 5082 subRun: 0 event: 131699
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  24-Jul-2025 21:53:18 CDT run: 5082 subRun: 0 event: 131699
Comparing two wires in the same plane: return failure
%MSG
55266
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 72460!
Genrating OpFlash 0 with 3 hits
Genrating OpFlash 1 with 5 hits
Genrating OpFlash 2 with 4 hits
Genrating OpFlash 3 with 3 hits
Genrating OpFlash 4 with 12 hits
Genrating OpFlash 5 with 28 hits
Genrating OpFlash 6 with 14 hits
Genrating OpFlash 7 with 6 hits
Genrating OpFlash 8 with 3 hits
Genrating OpFlash 9 with 5 hits
...
Generated 2184 OpFlashes
Begin processing the 50th record. run: 5082 subRun: 0 event: 131700 at 24-Jul-2025 22:01:07 CDT
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
Printing U view candidate length: 7
Printing V view candidate length: 10
Printing W view candidate length: 13
Printing 2D view candidate length: 30
Printing background CaloHitList length: 640
Printing New CaloHitList Names: 
SCaloHitListU | SCaloHitListV | SCaloHitListW | SCaloHitList2D | BBCaloHitList
*Photon identified*
*Photon identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Photon identified*
*Photon identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Photon identified*
*Photon identified*
Printing U view candidate length: 6
Printing V view candidate length: 10
Printing W view candidate length: 9
Printing 2D view candidate length: 25
Printing background CaloHitList length: 2
Printing New CaloHitList Names: 
CaloHitListU | CaloHitListV | CaloHitListW | CaloHitList2D | BCaloHitList
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  24-Jul-2025 22:03:36 CDT run: 5082 subRun: 0 event: 131700
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  24-Jul-2025 22:03:36 CDT run: 5082 subRun: 0 event: 131700
Trying to add data product: ShowerPCA. This element does not exist in the element holder
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  24-Jul-2025 22:03:36 CDT run: 5082 subRun: 0 event: 131700
Comparing two wires in the same plane: return failure
%MSG
54861
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 71733!
Genrating OpFlash 0 with 3 hits
Genrating OpFlash 1 with 4 hits
Genrating OpFlash 2 with 4 hits
Genrating OpFlash 3 with 6 hits
Genrating OpFlash 4 with 3 hits
Genrating OpFlash 5 with 4 hits
Genrating OpFlash 6 with 3 hits
Genrating OpFlash 7 with 3 hits
Genrating OpFlash 8 with 3 hits
Genrating OpFlash 9 with 3 hits
...
Generated 2186 OpFlashes
24-Jul-2025 22:12:07 CDT  Closed output file "prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250206T122900Z_gen_002634_supernova_g4_detsim_reco_20250724T175313Z_reco2.root"
24-Jul-2025 22:12:07 CDT  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/c1/52/prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250206T122900Z_gen_002634_supernova_g4_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              589.876       666.326       736.879       656.001       39.8126        50     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00741238     0.0473653     0.489712      0.0276415     0.0701257       50     
reco:linecluster:LineCluster                           0.0732486     0.132331      0.346708      0.117013      0.0498887       50     
reco:trajcluster:TrajCluster                           0.0232208     0.0648743      1.49984      0.0272278     0.206659        50     
reco:pandora:StandardPandora                            121.462       148.881       168.922       148.222       9.49693        50     
reco:pandoraTrack:LArPandoraTrackCreation             0.000454367   0.00764666     0.016213     0.00902127    0.00406823       50     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000418075   0.00537315     0.151279     0.00178423     0.0210318       50     
reco:pandoracalo:Calorimetry                          0.000292064   0.000935239   0.00908497    0.000659553   0.00132942       50     
reco:pandorapid:Chi2ParticleID                        7.0558e-05    0.00097488     0.0384456    0.000178551   0.00535715       50     
reco:pmtracktc:PMAlgTrackMaker                        0.00450072     0.0486951      0.31453      0.0230092     0.0555544       50     
reco:pmtracktccalo:Calorimetry                        0.000168545   0.000638295   0.00112091    0.000623602   0.000232386      50     
reco:pmtracktcpid:Chi2ParticleID                      3.9818e-05    0.000182163   0.000307861   0.000186356   5.46138e-05      50     
reco:emtrkmichelid:EmTrackMichelId                      11.0729       16.8278       20.9933       16.8571       2.11278        50     
reco:opdec:Deconvolution                                28.1139       33.393        37.3959       33.3458       2.13129        50     
reco:ophitspe:OpHitFinderDeco                           376.762       444.755       504.975       438.08        34.1482        50     
reco:opflash:OpFlashFinder                              17.9257       21.2898       28.5007       21.2137       1.81552        50     
reco:solarflash:SolarOpFlash                           0.354863      0.563382      0.949965      0.549645      0.101326        50     
reco:rns:RandomNumberSaver                            9.0773e-05    0.000608693    0.0222284    0.000126501   0.00309262       50     
[art]:TriggerResults:TriggerResultInserter            4.0472e-05    5.57102e-05   0.000101305   5.2565e-05    1.11704e-05      50     
end_path:out1:RootOutput                              1.0633e-05    0.000196153   0.00913897    1.2853e-05    0.00127755       50     
end_path:out1:RootOutput(write)                        0.200898      0.299866      0.738456      0.289175      0.0723931       50     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3251.99 MB
  Peak resident set size usage (VmHWM): 1750.33 MB
====================================================================================================
Art has completed and will exit with status 0.
Moving on to metadata extractor 
============OUTPUT FILE:  prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250206T122900Z_gen_002634_supernova_g4_detsim_reco_20250724T175313Z_reco2.root
Extracted metadata
checking file====== fardet-hd:prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250206T122900Z_gen_002634_supernova_g4_detsim_reco.root
MD conversion to MetaCat OK
all-input-dids.txt
debugprod.log
jobscript.log
justin-processed-pfns.txt
prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250206T122900Z_gen_002634_supernova_g4_detsim_reco_20250724T175313Z_reco2.root
prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250206T122900Z_gen_002634_supernova_g4_detsim_reco_20250724T175313Z_reco2.root.ext.json
prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250206T122900Z_gen_002634_supernova_g4_detsim_reco_20250724T175313Z_reco2.root.json
prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250206T122900Z_gen_002634_supernova_g4_detsim_reco_20250724T175313Z_reco2.root.temp.json
reco2_hist.root