Jobsub ID 13316.5@dunegpschedd02.fnal.gov

Jobsub ID	13316.5@dunegpschedd02.fnal.gov
Workflow ID	111
Stage ID	1
User name	higuera@fnal.gov
HTCondor Group	group_dune.prod.mcsim
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-07-31 14:10:54
Site	UK_RAL-Tier1
Entry	LIGO_UK_RAL_arc_ce02
Last heartbeat	2025-07-31 19:07:41
From worker node	Hostname	dune001-7486933.0-lcg2703.gridpp.rl.ac.uk
	cpuinfo	AMD EPYC 9654 96-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	216000 (60 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-07-31 14:12:22
Input files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250126T011217Z_gen_004916_supernova_g4stage1_g4stage2_detsim_reco.root
Jobscript	Exit code	0
	Real time	4h (15989s)
	CPU time	4h (15157s = 94%)
	Max RSS bytes	2409639936 (2298 MiB)
Outputting started	2025-07-31 18:38:52
Output files
Finished	2025-07-31 19:07:41
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 18:36:08 UTC run: 4901 subRun: 0 event: 49160
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 18:36:08 UTC run: 4901 subRun: 0 event: 49160
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 18:36:08 UTC run: 4901 subRun: 0 event: 49160
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 18:36:08 UTC run: 4901 subRun: 0 event: 49160
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 18:36:08 UTC run: 4901 subRun: 0 event: 49160
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 18:36:08 UTC run: 4901 subRun: 0 event: 49160
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 18:36:08 UTC run: 4901 subRun: 0 event: 49160
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 18:36:08 UTC run: 4901 subRun: 0 event: 49160
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 18:36:08 UTC run: 4901 subRun: 0 event: 49160
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 18:36:08 UTC run: 4901 subRun: 0 event: 49160
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 18:36:08 UTC run: 4901 subRun: 0 event: 49160
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 18:36:08 UTC run: 4901 subRun: 0 event: 49160
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 18:36:08 UTC run: 4901 subRun: 0 event: 49160
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 18:36:08 UTC run: 4901 subRun: 0 event: 49160
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 18:36:08 UTC run: 4901 subRun: 0 event: 49160
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 18:36:08 UTC run: 4901 subRun: 0 event: 49160
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 18:36:08 UTC run: 4901 subRun: 0 event: 49160
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 18:36:08 UTC run: 4901 subRun: 0 event: 49160
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 18:36:08 UTC run: 4901 subRun: 0 event: 49160
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 18:36:08 UTC run: 4901 subRun: 0 event: 49160
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 18:36:08 UTC run: 4901 subRun: 0 event: 49160
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 18:36:08 UTC run: 4901 subRun: 0 event: 49160
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 18:36:09 UTC run: 4901 subRun: 0 event: 49160
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 18:36:09 UTC run: 4901 subRun: 0 event: 49160
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 18:36:09 UTC run: 4901 subRun: 0 event: 49160
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 18:36:09 UTC run: 4901 subRun: 0 event: 49160
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 18:36:09 UTC run: 4901 subRun: 0 event: 49160
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 18:36:09 UTC run: 4901 subRun: 0 event: 49160
Comparing two wires in the same plane: return failure
%MSG
31-Jul-2025 18:38:45 UTC  Closed output file "prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250126T011217Z_gen_004916_supernova_g4stage1_g4stage2_detsim_reco_20250731T141229Z_reco2.root"
31-Jul-2025 18:38:46 UTC  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/6b/af/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250126T011217Z_gen_004916_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1402.29       1544.41       1733.06       1525.25       88.7934        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00465553     0.0373687     0.155149      0.0125997     0.0531834       10     
reco:pandora:StandardPandora                            1230.52       1378.81       1545.26       1366.54       80.4155        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000323259    0.0145858     0.103926     0.00455683     0.0300966       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000354025   0.00293964     0.0205055    0.00121185    0.00586938       10     
reco:pandoracalo:Calorimetry                          0.000351722   0.00261831     0.010798     0.000532224   0.00321412       10     
reco:pandorapid:Chi2ParticleID                        6.5198e-05    0.000268119   0.00171771    0.000115915   0.000483518      10     
reco:linecluster:LineCluster                            1.12539       1.29338       1.79589       1.18017      0.211034        10     
reco:trajcluster:TrajCluster                           0.306504      0.379487      0.448016      0.379357      0.0426475       10     
reco:pmtracktc:PMAlgTrackMaker                          6.48676       8.17029       9.68156       8.35134      0.914842        10     
reco:emtrkmichelid:EmTrackMichelId                      134.989       153.935       174.485       154.167       13.724         10     
reco:solarflash:SolarOpFlash                            1.3522        1.49787       1.79305       1.45687      0.133562        10     
[art]:TriggerResults:TriggerResultInserter            2.8593e-05    4.45063e-05   6.2715e-05    4.4207e-05    8.71059e-06      10     
end_path:out1:RootOutput                               8.693e-06    1.28836e-05   3.1358e-05    1.0451e-05    6.28484e-06      10     
end_path:out1:RootOutput(write)                        0.247585      0.264751      0.277693      0.266376     0.00958204       10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4057.53 MB
  Peak resident set size usage (VmHWM): 2409.64 MB
====================================================================================================
Art has completed and will exit with status 0.
Moving on to metadata extractor 
============OUTPUT FILE:  prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250126T011217Z_gen_004916_supernova_g4stage1_g4stage2_detsim_reco_20250731T141229Z_reco2.root
Extracted metadata
checking file====== fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250126T011217Z_gen_004916_supernova_g4stage1_g4stage2_detsim_reco.root
MD conversion to MetaCat OK
all-input-dids.txt
debugprod.log
jobscript.log
justin-processed-pfns.txt
prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250126T011217Z_gen_004916_supernova_g4stage1_g4stage2_detsim_reco_20250731T141229Z_reco2.root
prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250126T011217Z_gen_004916_supernova_g4stage1_g4stage2_detsim_reco_20250731T141229Z_reco2.root.ext.json
prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250126T011217Z_gen_004916_supernova_g4stage1_g4stage2_detsim_reco_20250731T141229Z_reco2.root.json
reco2_hist.root