Jobsub ID 17437.41@dunegpschedd01.fnal.gov

Jobsub ID	17437.41@dunegpschedd01.fnal.gov
Workflow ID	54
Stage ID	1
User name	epennacc@fnal.gov
HTCondor Group	group_dune.prod.mcsim
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-07-23 16:04:55
Site	CERN
Entry	CMSHTPC_T2_CH_CERN_ce513
Last heartbeat	2025-07-24 04:15:31
From worker node	Hostname	b9p13p1202.cern.ch
	cpuinfo	Intel(R) Xeon(R) Silver 4216 CPU @ 2.10GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	343800 (95 hours)
	GPU
	Inner Apptainer?	True
Job state	finished
Started	2025-07-23 16:13:28
Input files	fardet-hd:fardet-hd_mc_le_mc_2024a_workflow0000003786_out1_reco1_supernova_dune10kt_1x2x6_hit-reconstructed_merged_skip000250_lim000025_mergeLar_20241217T145251.root
Jobscript	Exit code	0
	Real time	11h (43191s)
	CPU time	4h (16040s = 37%)
	Max RSS bytes	1477488640 (1409 MiB)
Outputting started	2025-07-24 04:13:21
Output files	fardet-hd:fardet-hd_mc_le_mc_2024a_workflow0000003786_out1_reco1_supernova_dune10kt_1x2x6_hit-reconstructed_merged_skip000250_lim000025_mergeLar_20241217T145251_20250723T161336Z_reco2.root
Finished	2025-07-24 04:15:31
Saved logs	justin-logs:17437.41-dunegpschedd01.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

omparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  24-Jul-2025 06:12:48 CEST run: 3786 subRun: 0 event: 48699
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  24-Jul-2025 06:12:48 CEST run: 3786 subRun: 0 event: 48699
Comparing two wires in the same plane: return failure
%MSG
152
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 230!
Genrating OpFlash 0 with 25 hits
Genrating OpFlash 1 with 9 hits
Genrating OpFlash 2 with 5 hits
Genrating OpFlash 3 with 3 hits
Genrating OpFlash 4 with 3 hits
Genrating OpFlash 5 with 5 hits
Genrating OpFlash 6 with 3 hits
...
Generated 7 OpFlashes
Begin processing the 2500th record. run: 3786 subRun: 0 event: 48700 at 24-Jul-2025 06:12:58 CEST
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
Printing U view candidate length: 14
Printing V view candidate length: 8
Printing W view candidate length: 12
Printing 2D view candidate length: 34
Printing background CaloHitList length: 5
Printing New CaloHitList Names: 
SCaloHitListU | SCaloHitListV | SCaloHitListW | SCaloHitList2D | BBCaloHitList
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Photon identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Photon identified*
*Electron identified*
*Photon identified*
*Electron identified*
*Photon identified*
*Electron identified*
*Photon identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Photon identified*
*Electron identified*
*Photon identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Photon identified*
Printing U view candidate length: 17
Printing V view candidate length: 9
Printing W view candidate length: 13
Printing 2D view candidate length: 39
Printing background CaloHitList length: 0
Printing New CaloHitList Names: 
CaloHitListU | CaloHitListV | CaloHitListW | CaloHitList2D | BCaloHitList
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  24-Jul-2025 06:13:04 CEST run: 3786 subRun: 0 event: 48700
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  24-Jul-2025 06:13:04 CEST run: 3786 subRun: 0 event: 48700
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  24-Jul-2025 06:13:04 CEST run: 3786 subRun: 0 event: 48700
Comparing two wires in the same plane: return failure
%MSG
131
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 294!
Genrating OpFlash 0 with 44 hits
Genrating OpFlash 1 with 3 hits
Genrating OpFlash 2 with 4 hits
Genrating OpFlash 3 with 3 hits
Genrating OpFlash 4 with 3 hits
Genrating OpFlash 5 with 7 hits
Genrating OpFlash 6 with 4 hits
Genrating OpFlash 7 with 5 hits
Genrating OpFlash 8 with 4 hits
Genrating OpFlash 9 with 3 hits
...
Generated 17 OpFlashes
24-Jul-2025 06:13:16 CEST  Closed output file "fardet-hd_mc_le_mc_2024a_workflow0000003786_out1_reco1_supernova_dune10kt_1x2x6_hit-reconstructed_merged_skip000250_lim000025_mergeLar_20241217T145251_20250723T161336Z_reco2.root"
24-Jul-2025 06:13:16 CEST  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/3e/34/fardet-hd_mc_le_mc_2024a_workflow0000003786_out1_reco1_supernova_dune10kt_1x2x6_hit-reconstructed_merged_skip000250_lim000025_mergeLar_20241217T145251.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              8.99679       15.9598       315.504       15.934        6.20937       2500    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.000490168    0.0133471     0.232584     0.000934895    0.0360638      2500    
reco:linecluster:LineCluster                           0.611586      0.932795       1.30812      0.933961      0.0291922      2500    
reco:trajcluster:TrajCluster                           0.306125      0.308564      0.337832      0.308379     0.00146706      2500    
reco:pandora:StandardPandora                            0.31474        4.077        303.743       4.10564       6.03472       2500    
reco:pandoraTrack:LArPandoraTrackCreation             0.000121419   0.000581666   0.00883989    0.000572944   0.000283207     2500    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000179192   0.00114061     0.0162802    0.00109689    0.000583833     2500    
reco:pandoracalo:Calorimetry                          9.5157e-05    0.00037188     0.0054902    0.000359202   0.000177103     2500    
reco:pandorapid:Chi2ParticleID                        3.3186e-05    9.58459e-05   0.00210621    9.7018e-05    4.88573e-05     2500    
reco:pmtracktc:PMAlgTrackMaker                        0.00134805     0.0509628     0.918602      0.025127      0.0791157      2500    
reco:pmtracktccalo:Calorimetry                        7.9162e-05    0.000398935   0.00141937    0.000390317   0.000171325     2500    
reco:pmtracktcpid:Chi2ParticleID                      2.8101e-05    9.64223e-05   0.00314876    9.4355e-05    6.82315e-05     2500    
reco:emtrkmichelid:EmTrackMichelId                    0.000136556    0.871922       2.64061      0.810427      0.432886       2500    
reco:opdec:Deconvolution                               0.307885      0.677479       1.08772      0.727366      0.115699       2500    
reco:ophitspe:OpHitFinderDeco                         0.000236486     1.57382       4.45587       1.5171       0.724757       2500    
reco:opflash:OpFlashFinder                            0.000149924   0.000782091   0.00451173    0.000733721   0.000341439     2500    
reco:solarflash:SolarOpFlash                           5.083e-05    0.00598052     0.0260644    0.00516271    0.00414769      2500    
reco:rns:RandomNumberSaver                            1.8016e-05    4.56527e-05    0.0017473    3.42355e-05   8.03232e-05     2500    
[art]:TriggerResults:TriggerResultInserter            1.0693e-05    1.85285e-05   0.000608595   1.63325e-05   2.2696e-05      2500    
end_path:out1:RootOutput                               2.609e-06    4.87249e-06   0.000159675    3.996e-06    3.72939e-06     2500    
end_path:out1:RootOutput(write)                         7.12748       7.44325       7.6339        7.46658      0.0580531      2500    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 2970.83 MB
  Peak resident set size usage (VmHWM): 1477.49 MB
====================================================================================================
Art has completed and will exit with status 0.
Moving on to metadata extractor 
============OUTPUT FILE:  fardet-hd_mc_le_mc_2024a_workflow0000003786_out1_reco1_supernova_dune10kt_1x2x6_hit-reconstructed_merged_skip000250_lim000025_mergeLar_20241217T145251_20250723T161336Z_reco2.root
Extracted metadata
checking file====== fardet-hd:fardet-hd_mc_le_mc_2024a_workflow0000003786_out1_reco1_supernova_dune10kt_1x2x6_hit-reconstructed_merged_skip000250_lim000025_mergeLar_20241217T145251.root
MD conversion to MetaCat OK
all-input-dids.txt
debugprod.log
fardet-hd_mc_le_mc_2024a_workflow0000003786_out1_reco1_supernova_dune10kt_1x2x6_hit-reconstructed_merged_skip000250_lim000025_mergeLar_20241217T145251_20250723T161336Z_reco2.root
fardet-hd_mc_le_mc_2024a_workflow0000003786_out1_reco1_supernova_dune10kt_1x2x6_hit-reconstructed_merged_skip000250_lim000025_mergeLar_20241217T145251_20250723T161336Z_reco2.root.ext.json
fardet-hd_mc_le_mc_2024a_workflow0000003786_out1_reco1_supernova_dune10kt_1x2x6_hit-reconstructed_merged_skip000250_lim000025_mergeLar_20241217T145251_20250723T161336Z_reco2.root.json
fardet-hd_mc_le_mc_2024a_workflow0000003786_out1_reco1_supernova_dune10kt_1x2x6_hit-reconstructed_merged_skip000250_lim000025_mergeLar_20241217T145251_20250723T161336Z_reco2.root.temp.json
jobscript.log
justin-processed-pfns.txt
old_md.json
reco2_hist.root