Jobsub ID 11295.36@dunegpschedd02.fnal.gov

Jobsub ID	11295.36@dunegpschedd02.fnal.gov
Workflow ID	75
Stage ID	1
User name	higuera@fnal.gov
HTCondor Group	group_dune.prod.mcsim
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-07-25 20:51:50
Site	UK_RAL-Tier1
Entry	LIGO_UK_RAL_arc_ce03
Last heartbeat	2025-07-26 01:54:53
From worker node	Hostname	dune001-7744254.0-lcg2714.gridpp.rl.ac.uk
	cpuinfo	AMD EPYC 9654 96-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	216000 (60 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-07-25 22:17:14
Input files	fardet-vd:prodmarley_nue_es_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250214T171627Z_gen_001543_supernova_g4stage1_g4stage2_detsim_reco.root
Jobscript	Exit code	0
	Real time	2h (8070s)
	CPU time	2h (7897s = 97%)
	Max RSS bytes	2504245248 (2388 MiB)
Outputting started	2025-07-26 00:31:44
Output files
Finished	2025-07-26 01:54:53
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

vent: 15430
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  26-Jul-2025 00:29:06 UTC run: 5177 subRun: 0 event: 15430
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  26-Jul-2025 00:29:06 UTC run: 5177 subRun: 0 event: 15430
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  26-Jul-2025 00:29:06 UTC run: 5177 subRun: 0 event: 15430
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  26-Jul-2025 00:29:06 UTC run: 5177 subRun: 0 event: 15430
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  26-Jul-2025 00:29:06 UTC run: 5177 subRun: 0 event: 15430
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  26-Jul-2025 00:29:06 UTC run: 5177 subRun: 0 event: 15430
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  26-Jul-2025 00:29:06 UTC run: 5177 subRun: 0 event: 15430
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  26-Jul-2025 00:29:06 UTC run: 5177 subRun: 0 event: 15430
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  26-Jul-2025 00:29:06 UTC run: 5177 subRun: 0 event: 15430
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  26-Jul-2025 00:29:06 UTC run: 5177 subRun: 0 event: 15430
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  26-Jul-2025 00:29:06 UTC run: 5177 subRun: 0 event: 15430
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  26-Jul-2025 00:29:06 UTC run: 5177 subRun: 0 event: 15430
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  26-Jul-2025 00:29:06 UTC run: 5177 subRun: 0 event: 15430
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  26-Jul-2025 00:29:06 UTC run: 5177 subRun: 0 event: 15430
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  26-Jul-2025 00:29:06 UTC run: 5177 subRun: 0 event: 15430
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  26-Jul-2025 00:29:06 UTC run: 5177 subRun: 0 event: 15430
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  26-Jul-2025 00:29:06 UTC run: 5177 subRun: 0 event: 15430
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  26-Jul-2025 00:29:06 UTC run: 5177 subRun: 0 event: 15430
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  26-Jul-2025 00:29:06 UTC run: 5177 subRun: 0 event: 15430
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  26-Jul-2025 00:29:06 UTC run: 5177 subRun: 0 event: 15430
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  26-Jul-2025 00:29:06 UTC run: 5177 subRun: 0 event: 15430
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  26-Jul-2025 00:29:06 UTC run: 5177 subRun: 0 event: 15430
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  26-Jul-2025 00:29:06 UTC run: 5177 subRun: 0 event: 15430
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  26-Jul-2025 00:29:06 UTC run: 5177 subRun: 0 event: 15430
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  26-Jul-2025 00:29:06 UTC run: 5177 subRun: 0 event: 15430
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  26-Jul-2025 00:29:06 UTC run: 5177 subRun: 0 event: 15430
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  26-Jul-2025 00:29:06 UTC run: 5177 subRun: 0 event: 15430
Comparing two wires in the same plane: return failure
%MSG
%MSG-e pma::Track3D:  PMAlgTrackMaker:pmtracktc@BeginModule  26-Jul-2025 00:29:07 UTC run: 5177 subRun: 0 event: 15430
Flip, endpoint closer to vStart.
%MSG
26-Jul-2025 00:31:40 UTC  Closed output file "prodmarley_nue_es_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250214T171627Z_gen_001543_supernova_g4stage1_g4stage2_detsim_reco_20250725T221717Z_reco2.root"
26-Jul-2025 00:31:40 UTC  Closed input file "root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/fardet-vd/fd/84/prodmarley_nue_es_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250214T171627Z_gen_001543_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              713.155       795.107       943.529       770.967       74.5343        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00398644     0.0200444     0.0703488    0.00877564     0.0249496       10     
reco:pandora:StandardPandora                            630.766       697.031       798.885        681.1        56.6808        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000211949   0.00573754     0.0374722    0.00173233     0.0110149       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000213882   0.00130635     0.0063409    0.000699664   0.00175204       10     
reco:pandoracalo:Calorimetry                          0.00021913    0.000887084   0.00181954    0.000693089   0.000631258      10     
reco:pandorapid:Chi2ParticleID                        3.8779e-05    0.000146946   0.000674847   8.32605e-05   0.000182394      10     
reco:linecluster:LineCluster                           0.261174       0.33239       0.47662      0.303898      0.0659513       10     
reco:trajcluster:TrajCluster                           0.185857      0.250453      0.356343      0.219655      0.0600521       10     
reco:pmtracktc:PMAlgTrackMaker                          3.7435        4.6851        8.09725       4.41347       1.18695        10     
reco:emtrkmichelid:EmTrackMichelId                      76.7173       92.1776       150.182       85.7911       20.6256        10     
reco:solarflash:SolarOpFlash                           0.335459      0.438153      0.716331      0.403037      0.107655        10     
[art]:TriggerResults:TriggerResultInserter            1.7676e-05    3.38129e-05   6.2514e-05    3.1347e-05    1.08314e-05      10     
end_path:out1:RootOutput                               9.154e-06    1.27302e-05    3.332e-05     9.86e-06     7.00627e-06      10     
end_path:out1:RootOutput(write)                        0.131029       0.16324      0.236035      0.148861      0.0316682       10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4073.19 MB
  Peak resident set size usage (VmHWM): 2504.25 MB
====================================================================================================
Art has completed and will exit with status 0.
Moving on to metadata extractor 
============OUTPUT FILE:  prodmarley_nue_es_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250214T171627Z_gen_001543_supernova_g4stage1_g4stage2_detsim_reco_20250725T221717Z_reco2.root
Extracted metadata
checking file====== fardet-vd:prodmarley_nue_es_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250214T171627Z_gen_001543_supernova_g4stage1_g4stage2_detsim_reco.root
MD conversion to MetaCat OK
all-input-dids.txt
debugprod.log
jobscript.log
justin-processed-pfns.txt
prodmarley_nue_es_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250214T171627Z_gen_001543_supernova_g4stage1_g4stage2_detsim_reco_20250725T221717Z_reco2.root
prodmarley_nue_es_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250214T171627Z_gen_001543_supernova_g4stage1_g4stage2_detsim_reco_20250725T221717Z_reco2.root.ext.json
prodmarley_nue_es_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250214T171627Z_gen_001543_supernova_g4stage1_g4stage2_detsim_reco_20250725T221717Z_reco2.root.json
reco2_hist.root