Jobsub ID 11128.44@dunegpschedd02.fnal.gov

Jobsub ID	11128.44@dunegpschedd02.fnal.gov
Workflow ID	60
Stage ID	1
User name	higuera@fnal.gov
HTCondor Group	group_dune.prod.mcsim
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-07-25 02:42:48
Site	US_UChicago
Entry	Engage_US_MWT2_uiuc_condce_mcore
Last heartbeat	2025-07-25 10:35:40
From worker node	Hostname	iut2-c298.iu.edu
	cpuinfo	Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	86400 (24 hours)
	GPU
	Inner Apptainer?	True
Job state	finished
Started	2025-07-25 04:25:08
Input files	fardet-hd:prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_lateralAPA_20250209T173822Z_gen_001638_supernova_g4_halfActiveVol_detsim_reco.root
Jobscript	Exit code	0
	Real time	6h (22165s)
	CPU time	6h (21673s = 97%)
	Max RSS bytes	1642258432 (1566 MiB)
Outputting started	2025-07-25 10:34:34
Output files	fardet-hd:prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_lateralAPA_20250209T173822Z_gen_001638_supernova_g4_halfActiveVol_detsim_reco_20250725T042513Z_reco2.root
Finished	2025-07-25 10:35:40
Saved logs	justin-logs:11128.44-dunegpschedd02.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

D | BCaloHitList
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  25-Jul-2025 06:24:28 EDT run: 5100 subRun: 0 event: 81899
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  25-Jul-2025 06:24:28 EDT run: 5100 subRun: 0 event: 81899
Comparing two wires in the same plane: return failure
%MSG
32875
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 40405!
Genrating OpFlash 0 with 3 hits
Genrating OpFlash 1 with 3 hits
Genrating OpFlash 2 with 3 hits
Genrating OpFlash 3 with 4 hits
Genrating OpFlash 4 with 4 hits
Genrating OpFlash 5 with 3 hits
Genrating OpFlash 6 with 4 hits
Genrating OpFlash 7 with 3 hits
Genrating OpFlash 8 with 3 hits
Genrating OpFlash 9 with 6 hits
...
Generated 1157 OpFlashes
Begin processing the 50th record. run: 5100 subRun: 0 event: 81900 at 25-Jul-2025 06:28:31 EDT
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
Printing U view candidate length: 18
Printing V view candidate length: 15
Printing W view candidate length: 9
Printing 2D view candidate length: 42
Printing background CaloHitList length: 445
Printing New CaloHitList Names: 
SCaloHitListU | SCaloHitListV | SCaloHitListW | SCaloHitList2D | BBCaloHitList
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Photon identified*
*Photon identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Photon identified*
Printing U view candidate length: 17
Printing V view candidate length: 15
Printing W view candidate length: 9
Printing 2D view candidate length: 41
Printing background CaloHitList length: 1
Printing New CaloHitList Names: 
CaloHitListU | CaloHitListV | CaloHitListW | CaloHitList2D | BCaloHitList
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  25-Jul-2025 06:30:50 EDT run: 5100 subRun: 0 event: 81900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  25-Jul-2025 06:30:50 EDT run: 5100 subRun: 0 event: 81900
Comparing two wires in the same plane: return failure
%MSG
32990
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 40400!
Genrating OpFlash 0 with 4 hits
Genrating OpFlash 1 with 3 hits
Genrating OpFlash 2 with 3 hits
Genrating OpFlash 3 with 4 hits
Genrating OpFlash 4 with 4 hits
Genrating OpFlash 5 with 3 hits
Genrating OpFlash 6 with 3 hits
Genrating OpFlash 7 with 4 hits
Genrating OpFlash 8 with 11 hits
Genrating OpFlash 9 with 4 hits
...
Generated 1183 OpFlashes
25-Jul-2025 06:34:28 EDT  Closed output file "prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_lateralAPA_20250209T173822Z_gen_001638_supernova_g4_halfActiveVol_detsim_reco_20250725T042513Z_reco2.root"
25-Jul-2025 06:34:28 EDT  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/b2/6b/prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_lateralAPA_20250209T173822Z_gen_001638_supernova_g4_halfActiveVol_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              328.476       439.635       503.302       442.976       39.5913        50     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00780108     0.0202971     0.0422188     0.0179878    0.00862573       50     
reco:linecluster:LineCluster                           0.0766984     0.136766      0.361735      0.115657      0.0552982       50     
reco:trajcluster:TrajCluster                           0.023294      0.030895      0.120473      0.0257369     0.0158034       50     
reco:pandora:StandardPandora                            121.106       162.043       192.495       165.447       14.2358        50     
reco:pandoraTrack:LArPandoraTrackCreation             0.000370055   0.00792383     0.0190048    0.00901578    0.00396605       50     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000378119   0.00211863     0.0228119     0.0016924    0.00303722       50     
reco:pandoracalo:Calorimetry                          0.000251506   0.00078562    0.00688363    0.000644961   0.000900837      50     
reco:pandorapid:Chi2ParticleID                        6.8039e-05    0.000201022   0.00251642    0.000153834   0.000333866      50     
reco:pmtracktc:PMAlgTrackMaker                        0.00294735     0.0514749     0.163682      0.0256364     0.0528851       50     
reco:pmtracktccalo:Calorimetry                        0.000195049   0.000715584    0.0036529    0.000629826   0.000504669      50     
reco:pmtracktcpid:Chi2ParticleID                       6.006e-05    0.000156967   0.000308492   0.000146652   5.59859e-05      50     
reco:emtrkmichelid:EmTrackMichelId                      7.34709       11.1855       14.349        11.0296       1.63329        50     
reco:opdec:Deconvolution                                18.8421       21.1727       23.2548       21.3348       1.4154         50     
reco:ophitspe:OpHitFinderDeco                           152.235       235.923       288.742       236.332       30.084         50     
reco:opflash:OpFlashFinder                              6.89293       8.55535       10.5261       8.48079      0.944153        50     
reco:solarflash:SolarOpFlash                           0.237683      0.331259      0.496263      0.311656      0.0687929       50     
reco:rns:RandomNumberSaver                             6.23e-05     0.00012182    0.000732681   9.47545e-05   0.00012225       50     
[art]:TriggerResults:TriggerResultInserter            2.3162e-05    5.98699e-05   0.000811331   4.27125e-05   0.000107919      50     
end_path:out1:RootOutput                               6.922e-06    1.00935e-05   3.4794e-05     9.207e-06    3.96275e-06      50     
end_path:out1:RootOutput(write)                        0.121277      0.171505      0.231179      0.162603      0.0270492       50     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3132.57 MB
  Peak resident set size usage (VmHWM): 1642.26 MB
====================================================================================================
Art has completed and will exit with status 0.
Moving on to metadata extractor 
============OUTPUT FILE:  prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_lateralAPA_20250209T173822Z_gen_001638_supernova_g4_halfActiveVol_detsim_reco_20250725T042513Z_reco2.root
Extracted metadata
checking file====== fardet-hd:prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_lateralAPA_20250209T173822Z_gen_001638_supernova_g4_halfActiveVol_detsim_reco.root
MD conversion to MetaCat OK
all-input-dids.txt
debugprod.log
jobscript.log
justin-processed-pfns.txt
prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_lateralAPA_20250209T173822Z_gen_001638_supernova_g4_halfActiveVol_detsim_reco_20250725T042513Z_reco2.root
prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_lateralAPA_20250209T173822Z_gen_001638_supernova_g4_halfActiveVol_detsim_reco_20250725T042513Z_reco2.root.ext.json
prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_lateralAPA_20250209T173822Z_gen_001638_supernova_g4_halfActiveVol_detsim_reco_20250725T042513Z_reco2.root.json
prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_lateralAPA_20250209T173822Z_gen_001638_supernova_g4_halfActiveVol_detsim_reco_20250725T042513Z_reco2.root.temp.json
reco2_hist.root