Jobsub ID 17436.166@dunegpschedd01.fnal.gov

Jobsub ID	17436.166@dunegpschedd01.fnal.gov
Workflow ID	54
Stage ID	1
User name	epennacc@fnal.gov
HTCondor Group	group_dune.prod.mcsim
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-07-23 16:02:55
Site	UK_Manchester
Entry	UBoone_T2_UK_Manchester_ce02
Last heartbeat	2025-07-24 05:36:40
From worker node	Hostname	wn1906250.tier2.hep.manchester.ac.uk
	cpuinfo	Intel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	GPU
	Inner Apptainer?	True
Job state	finished
Started	2025-07-23 16:12:01
Input files	fardet-hd:fardet-hd_mc_le_mc_2024a_list-4374_input.list_out1_reco1_supernova_dune10kt_1x2x6_hit-reconstructed_merged_skip000125_lim000025_mergeLar_20241218T162502.root
Jobscript	Exit code	0
	Real time	13h (48122s)
	CPU time	4h (17758s = 36%)
	Max RSS bytes	1439203328 (1372 MiB)
Outputting started	2025-07-24 05:34:05
Output files	fardet-hd:fardet-hd_mc_le_mc_2024a_list-4374_input.list_out1_reco1_supernova_dune10kt_1x2x6_hit-reconstructed_merged_skip000125_lim000025_mergeLar_20241218T162502_20250723T161208Z_reco2.root
Finished	2025-07-24 05:36:40
Saved logs	justin-logs:17436.166-dunegpschedd01.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ron identified*
*Electron identified*
*Electron identified*
Printing U view candidate length: 26
Printing V view candidate length: 25
Printing W view candidate length: 13
Printing 2D view candidate length: 64
Printing background CaloHitList length: 4
Printing New CaloHitList Names: 
CaloHitListU | CaloHitListV | CaloHitListW | CaloHitList2D | BCaloHitList
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  24-Jul-2025 06:33:07 BST run: 4374 subRun: 0 event: 14999
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  24-Jul-2025 06:33:07 BST run: 4374 subRun: 0 event: 14999
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  24-Jul-2025 06:33:07 BST run: 4374 subRun: 0 event: 14999
Comparing two wires in the same plane: return failure
%MSG
%MSG-e pma::Track3D:  PMAlgTrackMaker:pmtracktc@BeginModule  24-Jul-2025 06:33:07 BST run: 4374 subRun: 0 event: 14999
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrackMaker:pmtracktc@BeginModule  24-Jul-2025 06:33:07 BST run: 4374 subRun: 0 event: 14999
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrackMaker:pmtracktc@BeginModule  24-Jul-2025 06:33:07 BST run: 4374 subRun: 0 event: 14999
TuneFullTree failed.
%MSG
%MSG-e pma::Track3D:  PMAlgTrackMaker:pmtracktc@BeginModule  24-Jul-2025 06:33:07 BST run: 4374 subRun: 0 event: 14999
Flip, endpoint closer to vStart.
%MSG
313
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 768!
Genrating OpFlash 0 with 31 hits
Genrating OpFlash 1 with 67 hits
Genrating OpFlash 2 with 16 hits
Genrating OpFlash 3 with 6 hits
Genrating OpFlash 4 with 3 hits
Genrating OpFlash 5 with 6 hits
Genrating OpFlash 6 with 3 hits
Genrating OpFlash 7 with 7 hits
Genrating OpFlash 8 with 6 hits
Genrating OpFlash 9 with 9 hits
...
Generated 41 OpFlashes
Begin processing the 2500th record. run: 4374 subRun: 0 event: 15000 at 24-Jul-2025 06:33:12 BST
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
Printing U view candidate length: 5
Printing V view candidate length: 4
Printing W view candidate length: 3
Printing 2D view candidate length: 12
Printing background CaloHitList length: 6
Printing New CaloHitList Names: 
SCaloHitListU | SCaloHitListV | SCaloHitListW | SCaloHitList2D | BBCaloHitList
*Electron identified*
*Photon identified*
*Electron identified*
*Photon identified*
*Photon identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Photon identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
Printing U view candidate length: 5
Printing V view candidate length: 5
Printing W view candidate length: 5
Printing 2D view candidate length: 15
Printing background CaloHitList length: 2
Printing New CaloHitList Names: 
CaloHitListU | CaloHitListV | CaloHitListW | CaloHitList2D | BCaloHitList
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  24-Jul-2025 06:33:20 BST run: 4374 subRun: 0 event: 15000
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  24-Jul-2025 06:33:20 BST run: 4374 subRun: 0 event: 15000
Trying to add data product: ShowerPCA. This element does not exist in the element holder
%MSG
98
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 205!
Genrating OpFlash 0 with 28 hits
Genrating OpFlash 1 with 4 hits
Genrating OpFlash 2 with 3 hits
Genrating OpFlash 3 with 3 hits
Genrating OpFlash 4 with 3 hits
Genrating OpFlash 5 with 4 hits
Genrating OpFlash 6 with 5 hits
Genrating OpFlash 7 with 9 hits
Genrating OpFlash 8 with 3 hits
...
Generated 9 OpFlashes
24-Jul-2025 06:33:28 BST  Closed output file "fardet-hd_mc_le_mc_2024a_list-4374_input.list_out1_reco1_supernova_dune10kt_1x2x6_hit-reconstructed_merged_skip000125_lim000025_mergeLar_20241218T162502_20250723T161208Z_reco2.root"
24-Jul-2025 06:33:28 BST  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/a3/c2/fardet-hd_mc_le_mc_2024a_list-4374_input.list_out1_reco1_supernova_dune10kt_1x2x6_hit-reconstructed_merged_skip000125_lim000025_mergeLar_20241218T162502.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2.61591       10.1467       22.1543       10.3117       1.82171       2500    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.000549789    0.0176887     0.353599     0.00121024     0.0474151      2500    
reco:linecluster:LineCluster                           0.812009       1.32076       1.83977       1.36938      0.0845643      2500    
reco:trajcluster:TrajCluster                           0.406156      0.410243      0.841547       0.4091       0.0126529      2500    
reco:pandora:StandardPandora                           0.416934       4.79198       16.9404       4.94136      0.917538       2500    
reco:pandoraTrack:LArPandoraTrackCreation             0.000143786   0.000770109   0.00819245    0.000762281   0.000333082     2500    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000213125   0.00159926     0.474666     0.00136335    0.00948691      2500    
reco:pandoracalo:Calorimetry                          0.000113429   0.000625399    0.306082     0.000489665   0.00611492      2500    
reco:pandorapid:Chi2ParticleID                        3.7133e-05    0.000137666   0.00123367    0.000139774   5.06562e-05     2500    
reco:pmtracktc:PMAlgTrackMaker                        0.00145856     0.0573357      1.02841      0.0277419     0.0883423      2500    
reco:pmtracktccalo:Calorimetry                        9.5235e-05    0.000489173   0.00618944    0.000484686   0.000244193     2500    
reco:pmtracktcpid:Chi2ParticleID                      3.4448e-05    0.000129953    0.0052591    0.00012811    0.000111969     2500    
reco:emtrkmichelid:EmTrackMichelId                    0.000178566     0.97885       2.73116      0.924375      0.455931       2500    
reco:opdec:Deconvolution                               0.407589      0.871006       1.44386      0.930218      0.144753       2500    
reco:ophitspe:OpHitFinderDeco                         0.000270988     1.66505       4.86213       1.61128      0.764656       2500    
reco:opflash:OpFlashFinder                            0.000174187   0.00112021     0.0124529     0.0010087    0.000605984     2500    
reco:solarflash:SolarOpFlash                          6.5975e-05    0.00726981     0.205859      0.0062412    0.00648951      2500    
reco:rns:RandomNumberSaver                            1.9569e-05    6.58251e-05   0.00102032    6.5029e-05    3.33491e-05     2500    
[art]:TriggerResults:TriggerResultInserter            1.1676e-05    2.90806e-05   0.000177231   2.4529e-05    1.4226e-05      2500    
end_path:out1:RootOutput                               2.819e-06    7.90733e-06   4.3574e-05     7.485e-06    3.90502e-06     2500    
end_path:out1:RootOutput(write)                       0.00720362     0.0200831     0.135976      0.0193881    0.00624012      2500    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 2978.98 MB
  Peak resident set size usage (VmHWM): 1439.2 MB
====================================================================================================
Art has completed and will exit with status 0.
Moving on to metadata extractor 
============OUTPUT FILE:  fardet-hd_mc_le_mc_2024a_list-4374_input.list_out1_reco1_supernova_dune10kt_1x2x6_hit-reconstructed_merged_skip000125_lim000025_mergeLar_20241218T162502_20250723T161208Z_reco2.root
Extracted metadata
checking file====== fardet-hd:fardet-hd_mc_le_mc_2024a_list-4374_input.list_out1_reco1_supernova_dune10kt_1x2x6_hit-reconstructed_merged_skip000125_lim000025_mergeLar_20241218T162502.root
MD conversion to MetaCat OK
all-input-dids.txt
debugprod.log
fardet-hd_mc_le_mc_2024a_list-4374_input.list_out1_reco1_supernova_dune10kt_1x2x6_hit-reconstructed_merged_skip000125_lim000025_mergeLar_20241218T162502_20250723T161208Z_reco2.root
fardet-hd_mc_le_mc_2024a_list-4374_input.list_out1_reco1_supernova_dune10kt_1x2x6_hit-reconstructed_merged_skip000125_lim000025_mergeLar_20241218T162502_20250723T161208Z_reco2.root.ext.json
fardet-hd_mc_le_mc_2024a_list-4374_input.list_out1_reco1_supernova_dune10kt_1x2x6_hit-reconstructed_merged_skip000125_lim000025_mergeLar_20241218T162502_20250723T161208Z_reco2.root.json
fardet-hd_mc_le_mc_2024a_list-4374_input.list_out1_reco1_supernova_dune10kt_1x2x6_hit-reconstructed_merged_skip000125_lim000025_mergeLar_20241218T162502_20250723T161208Z_reco2.root.temp.json
jobscript.log
justin-processed-pfns.txt
old_md.json
reco2_hist.root