Jobsub ID 13208.51@dunegpschedd02.fnal.gov

Jobsub ID	13208.51@dunegpschedd02.fnal.gov
Workflow ID	163
Stage ID	1
User name	higuera@fnal.gov
HTCondor Group	group_dune.prod.mcsim
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-07-31 05:35:28
Site	UK_Manchester
Entry	UBoone_T2_UK_Manchester_ce02
Last heartbeat	2025-07-31 19:28:06
From worker node	Hostname	wn1208081.tier2.hep.manchester.ac.uk
	cpuinfo	AMD EPYC 7702P 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-07-31 05:36:38
Input files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T062611Z_gen_002083_supernova_g4stage1_g4stage2_detsim_reco.root
Jobscript	Exit code	0
	Real time	13h (47518s)
	CPU time	12h (43621s = 91%)
	Max RSS bytes	2502049792 (2386 MiB)
Outputting started	2025-07-31 18:48:37
Output files
Finished	2025-07-31 19:28:06
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:43:45 BST run: 5031 subRun: 0 event: 20830
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:43:46 BST run: 5031 subRun: 0 event: 20830
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:43:46 BST run: 5031 subRun: 0 event: 20830
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:43:46 BST run: 5031 subRun: 0 event: 20830
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:43:46 BST run: 5031 subRun: 0 event: 20830
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:43:46 BST run: 5031 subRun: 0 event: 20830
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:43:46 BST run: 5031 subRun: 0 event: 20830
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:43:46 BST run: 5031 subRun: 0 event: 20830
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:43:46 BST run: 5031 subRun: 0 event: 20830
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:43:46 BST run: 5031 subRun: 0 event: 20830
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:43:46 BST run: 5031 subRun: 0 event: 20830
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:43:46 BST run: 5031 subRun: 0 event: 20830
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:43:46 BST run: 5031 subRun: 0 event: 20830
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:43:46 BST run: 5031 subRun: 0 event: 20830
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:43:46 BST run: 5031 subRun: 0 event: 20830
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:43:46 BST run: 5031 subRun: 0 event: 20830
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:43:46 BST run: 5031 subRun: 0 event: 20830
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:43:46 BST run: 5031 subRun: 0 event: 20830
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:43:47 BST run: 5031 subRun: 0 event: 20830
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:43:47 BST run: 5031 subRun: 0 event: 20830
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:43:47 BST run: 5031 subRun: 0 event: 20830
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:43:47 BST run: 5031 subRun: 0 event: 20830
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:43:47 BST run: 5031 subRun: 0 event: 20830
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:43:47 BST run: 5031 subRun: 0 event: 20830
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:43:47 BST run: 5031 subRun: 0 event: 20830
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:43:47 BST run: 5031 subRun: 0 event: 20830
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:43:47 BST run: 5031 subRun: 0 event: 20830
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:43:47 BST run: 5031 subRun: 0 event: 20830
Comparing two wires in the same plane: return failure
%MSG
31-Jul-2025 19:48:29 BST  Closed output file "prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T062611Z_gen_002083_supernova_g4stage1_g4stage2_detsim_reco_20250731T053651Z_reco2.root"
31-Jul-2025 19:48:29 BST  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/10/8f/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T062611Z_gen_002083_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              4426.95       4698.82       5171.17       4637.39       233.387        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00545486     0.0347869     0.136267      0.013485      0.0459955       10     
reco:pandora:StandardPandora                            3964.2        4263.95       4729.33       4198.94       224.048        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000296829    0.0143275     0.0884371    0.00686331     0.0253662       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000325664   0.00260052     0.0154895    0.00123765    0.00437927       10     
reco:pandoracalo:Calorimetry                          0.000260692   0.00287137    0.00827942    0.00146075    0.00277333       10     
reco:pandorapid:Chi2ParticleID                         6.93e-05     0.000297764   0.00203512    0.000102979   0.000580148      10     
reco:linecluster:LineCluster                            69.9009       76.2617       83.5966       74.9767       4.24407        10     
reco:trajcluster:TrajCluster                           0.509459      0.619233      0.686967      0.633761      0.0508389       10     
reco:pmtracktc:PMAlgTrackMaker                          12.0029       13.1629       14.3828       13.3115      0.636709        10     
reco:emtrkmichelid:EmTrackMichelId                      276.945       342.611       367.801        353.2        25.6235        10     
reco:solarflash:SolarOpFlash                            1.56291       1.72211       1.9534        1.69739      0.114225        10     
[art]:TriggerResults:TriggerResultInserter            2.5087e-05    3.73832e-05   8.5771e-05    3.1264e-05    1.7342e-05       10     
end_path:out1:RootOutput                               4.158e-06    9.7192e-06    4.6107e-05    5.6105e-06    1.2157e-05       10     
end_path:out1:RootOutput(write)                        0.379917      0.440205      0.529714      0.434974      0.042116        10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4046.99 MB
  Peak resident set size usage (VmHWM): 2502.05 MB
====================================================================================================
Art has completed and will exit with status 0.
Moving on to metadata extractor 
============OUTPUT FILE:  prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T062611Z_gen_002083_supernova_g4stage1_g4stage2_detsim_reco_20250731T053651Z_reco2.root
Extracted metadata
checking file====== fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T062611Z_gen_002083_supernova_g4stage1_g4stage2_detsim_reco.root
MD conversion to MetaCat OK
all-input-dids.txt
debugprod.log
jobscript.log
justin-processed-pfns.txt
prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T062611Z_gen_002083_supernova_g4stage1_g4stage2_detsim_reco_20250731T053651Z_reco2.root
prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T062611Z_gen_002083_supernova_g4stage1_g4stage2_detsim_reco_20250731T053651Z_reco2.root.ext.json
prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T062611Z_gen_002083_supernova_g4stage1_g4stage2_detsim_reco_20250731T053651Z_reco2.root.json
reco2_hist.root