Jobsub ID 17437.59@dunegpschedd01.fnal.gov

Jobsub ID	17437.59@dunegpschedd01.fnal.gov
Workflow ID	54
Stage ID	1
User name	epennacc@fnal.gov
HTCondor Group	group_dune.prod.mcsim
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-07-23 16:04:55
Site	CERN
Entry	CMSHTPC_T2_CH_CERN_ce508
Last heartbeat	2025-07-24 00:48:50
From worker node	Hostname	b9p04p1288.cern.ch
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	343800 (95 hours)
	GPU
	Inner Apptainer?	True
Job state	finished
Started	2025-07-23 16:13:18
Input files	fardet-hd:fardet-hd_mc_le_mc_2024a_list-4375_input.list_out1_reco1_supernova_dune10kt_1x2x6_hit-reconstructed_merged_skip000200_lim000025_mergeLar_20241218T163135.root
Jobscript	Exit code	0
	Real time	8h (30815s)
	CPU time	2h (8118s = 26%)
	Max RSS bytes	1448136704 (1381 MiB)
Outputting started	2025-07-24 00:46:54
Output files	fardet-hd:fardet-hd_mc_le_mc_2024a_list-4375_input.list_out1_reco1_supernova_dune10kt_1x2x6_hit-reconstructed_merged_skip000200_lim000025_mergeLar_20241218T163135_20250723T161325Z_reco2.root
Finished	2025-07-24 00:48:50
Saved logs	justin-logs:17437.59-dunegpschedd01.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

itListW | CaloHitList2D | BCaloHitList
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  24-Jul-2025 02:46:40 CEST run: 4375 subRun: 0 event: 22499
Comparing two wires in the same plane: return failure
%MSG
301
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 661!
Genrating OpFlash 0 with 62 hits
Genrating OpFlash 1 with 9 hits
Genrating OpFlash 2 with 8 hits
Genrating OpFlash 3 with 9 hits
Genrating OpFlash 4 with 3 hits
Genrating OpFlash 5 with 4 hits
Genrating OpFlash 6 with 27 hits
Genrating OpFlash 7 with 6 hits
Genrating OpFlash 8 with 5 hits
Genrating OpFlash 9 with 4 hits
...
Generated 36 OpFlashes
Begin processing the 2500th record. run: 4375 subRun: 0 event: 22500 at 24-Jul-2025 02:46:43 CEST
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
Printing U view candidate length: 15
Printing V view candidate length: 16
Printing W view candidate length: 10
Printing 2D view candidate length: 41
Printing background CaloHitList length: 2
Printing New CaloHitList Names: 
SCaloHitListU | SCaloHitListV | SCaloHitListW | SCaloHitList2D | BBCaloHitList
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Photon identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Photon identified*
*Electron identified*
*Electron identified*
*Photon identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
Printing U view candidate length: 16
Printing V view candidate length: 17
Printing W view candidate length: 10
Printing 2D view candidate length: 43
Printing background CaloHitList length: 0
Printing New CaloHitList Names: 
CaloHitListU | CaloHitListV | CaloHitListW | CaloHitList2D | BCaloHitList
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  24-Jul-2025 02:46:48 CEST run: 4375 subRun: 0 event: 22500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  24-Jul-2025 02:46:48 CEST run: 4375 subRun: 0 event: 22500
Comparing two wires in the same plane: return failure
%MSG
236
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 424!
Genrating OpFlash 0 with 5 hits
Genrating OpFlash 1 with 22 hits
Genrating OpFlash 2 with 10 hits
Genrating OpFlash 3 with 3 hits
Genrating OpFlash 4 with 3 hits
Genrating OpFlash 5 with 3 hits
Genrating OpFlash 6 with 10 hits
Genrating OpFlash 7 with 3 hits
Genrating OpFlash 8 with 12 hits
Genrating OpFlash 9 with 5 hits
...
Generated 11 OpFlashes
24-Jul-2025 02:46:50 CEST  Closed output file "fardet-hd_mc_le_mc_2024a_list-4375_input.list_out1_reco1_supernova_dune10kt_1x2x6_hit-reconstructed_merged_skip000200_lim000025_mergeLar_20241218T163135_20250723T161325Z_reco2.root"
24-Jul-2025 02:46:50 CEST  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/f3/14/fardet-hd_mc_le_mc_2024a_list-4375_input.list_out1_reco1_supernova_dune10kt_1x2x6_hit-reconstructed_merged_skip000200_lim000025_mergeLar_20241218T163135.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1.88856       5.49997       192.915       5.50474       3.84583       2500    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.000294554    0.0133693      0.79687     0.000534587    0.0399017      2500    
reco:linecluster:LineCluster                           0.612082      0.987605       1.61294       1.0288       0.0650537      2500    
reco:trajcluster:TrajCluster                           0.306441      0.308465      0.376456       0.30821     0.00220165      2500    
reco:pandora:StandardPandora                           0.312852       2.31317       189.91        2.36362       3.77306       2500    
reco:pandoraTrack:LArPandoraTrackCreation             0.000104131   0.00042153    0.00253457    0.000420786   0.000158612     2500    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000148123   0.00085479    0.00583431    0.000840002   0.000362438     2500    
reco:pandoracalo:Calorimetry                          8.0141e-05    0.000292308   0.00190281    0.000287615   0.000105873     2500    
reco:pandorapid:Chi2ParticleID                         2.69e-05     8.59525e-05   0.000782021   8.6951e-05    2.85745e-05     2500    
reco:pmtracktc:PMAlgTrackMaker                        0.000987675    0.0339834     0.495517      0.0157992     0.0551196      2500    
reco:pmtracktccalo:Calorimetry                        5.5961e-05    0.000282827   0.00457064    0.000258709   0.00017048      2500    
reco:pmtracktcpid:Chi2ParticleID                       2.053e-05    7.48795e-05   0.000884223   7.1906e-05    3.18064e-05     2500    
reco:emtrkmichelid:EmTrackMichelId                    0.000110552    0.458545       1.25785      0.430426      0.218304       2500    
reco:opdec:Deconvolution                               0.308563      0.627186       1.19288      0.677162      0.100635       2500    
reco:ophitspe:OpHitFinderDeco                         0.000215413    0.738647       2.16619      0.702081      0.353613       2500    
reco:opflash:OpFlashFinder                            0.000112912   0.000634711   0.00447104    0.000587279   0.000306895     2500    
reco:solarflash:SolarOpFlash                          4.0501e-05    0.00446153     0.0198049    0.00388716    0.00304196      2500    
reco:rns:RandomNumberSaver                             1.682e-05    3.76381e-05   0.000576089    3.474e-05    2.89846e-05     2500    
[art]:TriggerResults:TriggerResultInserter             8.11e-06     1.66579e-05   0.000304694   1.4555e-05    1.6675e-05      2500    
end_path:out1:RootOutput                               2.68e-06     7.1105e-06    0.000202513    6.93e-06     5.12992e-06     2500    
end_path:out1:RootOutput(write)                        0.0045878     0.0109078     0.0289207     0.0107985    0.00262885      2500    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 2959.37 MB
  Peak resident set size usage (VmHWM): 1448.14 MB
====================================================================================================
Art has completed and will exit with status 0.
Moving on to metadata extractor 
============OUTPUT FILE:  fardet-hd_mc_le_mc_2024a_list-4375_input.list_out1_reco1_supernova_dune10kt_1x2x6_hit-reconstructed_merged_skip000200_lim000025_mergeLar_20241218T163135_20250723T161325Z_reco2.root
Extracted metadata
checking file====== fardet-hd:fardet-hd_mc_le_mc_2024a_list-4375_input.list_out1_reco1_supernova_dune10kt_1x2x6_hit-reconstructed_merged_skip000200_lim000025_mergeLar_20241218T163135.root
MD conversion to MetaCat OK
all-input-dids.txt
debugprod.log
fardet-hd_mc_le_mc_2024a_list-4375_input.list_out1_reco1_supernova_dune10kt_1x2x6_hit-reconstructed_merged_skip000200_lim000025_mergeLar_20241218T163135_20250723T161325Z_reco2.root
fardet-hd_mc_le_mc_2024a_list-4375_input.list_out1_reco1_supernova_dune10kt_1x2x6_hit-reconstructed_merged_skip000200_lim000025_mergeLar_20241218T163135_20250723T161325Z_reco2.root.ext.json
fardet-hd_mc_le_mc_2024a_list-4375_input.list_out1_reco1_supernova_dune10kt_1x2x6_hit-reconstructed_merged_skip000200_lim000025_mergeLar_20241218T163135_20250723T161325Z_reco2.root.json
fardet-hd_mc_le_mc_2024a_list-4375_input.list_out1_reco1_supernova_dune10kt_1x2x6_hit-reconstructed_merged_skip000200_lim000025_mergeLar_20241218T163135_20250723T161325Z_reco2.root.temp.json
jobscript.log
justin-processed-pfns.txt
old_md.json
reco2_hist.root