Jobsub ID 19842.80@dunegpschedd01.fnal.gov

Jobsub ID	19842.80@dunegpschedd01.fnal.gov
Workflow ID	166
Stage ID	1
User name	higuera@fnal.gov
HTCondor Group	group_dune.prod.mcsim
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-07-31 14:56:57
Site	UK_Lancaster
Entry	UBoone_UK_Lancaster_HEC_grendel_ce02
Last heartbeat	2025-07-31 20:06:14
From worker node	Hostname	comp22-07
	cpuinfo	Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-07-31 14:58:28
Input files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250129T083038Z_gen_001973_supernova_g4stage1_g4stage2_detsim_reco.root
Jobscript	Exit code	0
	Real time	4h (15227s)
	CPU time	4h (14830s = 97%)
	Max RSS bytes	2566455296 (2447 MiB)
Outputting started	2025-07-31 19:12:16
Output files
Finished	2025-07-31 20:06:14
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:09:04 BST run: 4980 subRun: 0 event: 19730
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:09:04 BST run: 4980 subRun: 0 event: 19730
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:09:04 BST run: 4980 subRun: 0 event: 19730
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:09:04 BST run: 4980 subRun: 0 event: 19730
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:09:04 BST run: 4980 subRun: 0 event: 19730
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:09:04 BST run: 4980 subRun: 0 event: 19730
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:09:04 BST run: 4980 subRun: 0 event: 19730
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:09:04 BST run: 4980 subRun: 0 event: 19730
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:09:04 BST run: 4980 subRun: 0 event: 19730
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:09:04 BST run: 4980 subRun: 0 event: 19730
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:09:04 BST run: 4980 subRun: 0 event: 19730
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:09:04 BST run: 4980 subRun: 0 event: 19730
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:09:04 BST run: 4980 subRun: 0 event: 19730
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:09:04 BST run: 4980 subRun: 0 event: 19730
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:09:04 BST run: 4980 subRun: 0 event: 19730
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:09:04 BST run: 4980 subRun: 0 event: 19730
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:09:04 BST run: 4980 subRun: 0 event: 19730
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:09:04 BST run: 4980 subRun: 0 event: 19730
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:09:04 BST run: 4980 subRun: 0 event: 19730
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:09:04 BST run: 4980 subRun: 0 event: 19730
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:09:04 BST run: 4980 subRun: 0 event: 19730
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:09:04 BST run: 4980 subRun: 0 event: 19730
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:09:04 BST run: 4980 subRun: 0 event: 19730
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:09:04 BST run: 4980 subRun: 0 event: 19730
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:09:05 BST run: 4980 subRun: 0 event: 19730
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:09:05 BST run: 4980 subRun: 0 event: 19730
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:09:05 BST run: 4980 subRun: 0 event: 19730
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 20:09:05 BST run: 4980 subRun: 0 event: 19730
Comparing two wires in the same plane: return failure
%MSG
31-Jul-2025 20:12:11 BST  Closed output file "prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250129T083038Z_gen_001973_supernova_g4stage1_g4stage2_detsim_reco_20250731T145833Z_reco2.root"
31-Jul-2025 20:12:11 BST  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/38/e7/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250129T083038Z_gen_001973_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1429.63       1488.05       1530.35       1489.32       35.7038        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00264999     0.0288528     0.120265     0.00712596     0.0451816       10     
reco:pandora:StandardPandora                             1252         1301.09       1337.01       1300.31       30.0395        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000217162   0.00860763     0.0624087    0.00255929     0.018097        10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000214312   0.00182619     0.0040516    0.00177801     0.0014454       10     
reco:pandoracalo:Calorimetry                          0.000173854   0.00120509    0.00393089    0.000377177   0.00140467       10     
reco:pandorapid:Chi2ParticleID                         4.24e-05     0.000143476   0.000489618   5.5978e-05    0.000173466      10     
reco:linecluster:LineCluster                           0.944064       1.11085       1.55799       1.07268      0.176926        10     
reco:trajcluster:TrajCluster                           0.234905      0.271942      0.288615      0.276335      0.0154483       10     
reco:pmtracktc:PMAlgTrackMaker                          3.95202       4.45965       4.8613        4.53664      0.283504        10     
reco:emtrkmichelid:EmTrackMichelId                      166.188       179.715       190.589       181.029       6.46889        10     
reco:solarflash:SolarOpFlash                            1.06964       1.15405       1.39763       1.15947      0.091541        10     
[art]:TriggerResults:TriggerResultInserter            1.8377e-05    2.92271e-05   4.7921e-05    2.7891e-05    7.50994e-06      10     
end_path:out1:RootOutput                               7.091e-06    9.0019e-06    1.8878e-05     7.667e-06    3.39763e-06      10     
end_path:out1:RootOutput(write)                        0.185305      0.200294      0.211939       0.2009      0.00765374       10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4043.92 MB
  Peak resident set size usage (VmHWM): 2566.46 MB
====================================================================================================
Art has completed and will exit with status 0.
Moving on to metadata extractor 
============OUTPUT FILE:  prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250129T083038Z_gen_001973_supernova_g4stage1_g4stage2_detsim_reco_20250731T145833Z_reco2.root
Extracted metadata
checking file====== fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250129T083038Z_gen_001973_supernova_g4stage1_g4stage2_detsim_reco.root
MD conversion to MetaCat OK
all-input-dids.txt
debugprod.log
jobscript.log
justin-processed-pfns.txt
prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250129T083038Z_gen_001973_supernova_g4stage1_g4stage2_detsim_reco_20250731T145833Z_reco2.root
prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250129T083038Z_gen_001973_supernova_g4stage1_g4stage2_detsim_reco_20250731T145833Z_reco2.root.ext.json
prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250129T083038Z_gen_001973_supernova_g4stage1_g4stage2_detsim_reco_20250731T145833Z_reco2.root.json
reco2_hist.root