Jobsub ID 19829.48@dunegpschedd01.fnal.gov

Jobsub ID	19829.48@dunegpschedd01.fnal.gov
Workflow ID	163
Stage ID	1
User name	higuera@fnal.gov
HTCondor Group	group_dune.prod.mcsim
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-07-31 14:12:55
Site	CERN
Entry	CMSHTPC_T2_CH_CERN_ce507
Last heartbeat	2025-07-31 19:52:12
From worker node	Hostname	b9p18p4824.cern.ch
	cpuinfo	AMD EPYC 7302 16-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	343800 (95 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-07-31 14:14:44
Input files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250128T111936Z_gen_003768_supernova_g4stage1_g4stage2_detsim_reco.root
Jobscript	Exit code	0
	Real time	4h (17665s)
	CPU time	4h (16849s = 95%)
	Max RSS bytes	2552410112 (2434 MiB)
Outputting started	2025-07-31 19:09:11
Output files
Finished	2025-07-31 19:52:12
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

n failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:06:35 CEST run: 4961 subRun: 0 event: 37680
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:06:35 CEST run: 4961 subRun: 0 event: 37680
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:06:35 CEST run: 4961 subRun: 0 event: 37680
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:06:35 CEST run: 4961 subRun: 0 event: 37680
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:06:35 CEST run: 4961 subRun: 0 event: 37680
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:06:35 CEST run: 4961 subRun: 0 event: 37680
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:06:35 CEST run: 4961 subRun: 0 event: 37680
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:06:35 CEST run: 4961 subRun: 0 event: 37680
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:06:35 CEST run: 4961 subRun: 0 event: 37680
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:06:35 CEST run: 4961 subRun: 0 event: 37680
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:06:35 CEST run: 4961 subRun: 0 event: 37680
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:06:35 CEST run: 4961 subRun: 0 event: 37680
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:06:35 CEST run: 4961 subRun: 0 event: 37680
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:06:35 CEST run: 4961 subRun: 0 event: 37680
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:06:35 CEST run: 4961 subRun: 0 event: 37680
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:06:35 CEST run: 4961 subRun: 0 event: 37680
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:06:35 CEST run: 4961 subRun: 0 event: 37680
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:06:35 CEST run: 4961 subRun: 0 event: 37680
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:06:35 CEST run: 4961 subRun: 0 event: 37680
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:06:35 CEST run: 4961 subRun: 0 event: 37680
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:06:35 CEST run: 4961 subRun: 0 event: 37680
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:06:35 CEST run: 4961 subRun: 0 event: 37680
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:06:35 CEST run: 4961 subRun: 0 event: 37680
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:06:35 CEST run: 4961 subRun: 0 event: 37680
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:06:35 CEST run: 4961 subRun: 0 event: 37680
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:06:35 CEST run: 4961 subRun: 0 event: 37680
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:06:36 CEST run: 4961 subRun: 0 event: 37680
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 21:06:36 CEST run: 4961 subRun: 0 event: 37680
Comparing two wires in the same plane: return failure
%MSG
31-Jul-2025 21:09:05 CEST  Closed output file "prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250128T111936Z_gen_003768_supernova_g4stage1_g4stage2_detsim_reco_20250731T141451Z_reco2.root"
31-Jul-2025 21:09:05 CEST  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/4f/30/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250128T111936Z_gen_003768_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1498.32       1720.47       1969.69       1720.71       139.11         10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00308659     0.0294381     0.125026     0.00676593     0.0466158       10     
reco:pandora:StandardPandora                            1365.79       1568.18       1794.61       1564.11       125.163        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000211003   0.00521014     0.0337414    0.00212413    0.00957125       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000226593   0.00154954    0.00719185    0.00100011    0.00193318       10     
reco:pandoracalo:Calorimetry                          0.000221063   0.000951055    0.0030541    0.000395071   0.000903291      10     
reco:pandorapid:Chi2ParticleID                         4.739e-05    0.000169309   0.000768751   9.3611e-05    0.000204457      10     
reco:linecluster:LineCluster                            1.01103       1.26836       1.98441       1.07822      0.349559        10     
reco:trajcluster:TrajCluster                           0.238967      0.318785      0.386388      0.323752      0.0541646       10     
reco:pmtracktc:PMAlgTrackMaker                          4.38091       6.55677       8.92397       6.70456       1.63273        10     
reco:emtrkmichelid:EmTrackMichelId                      119.47        142.396       174.444       139.595       17.8377        10     
reco:solarflash:SolarOpFlash                            1.21565       1.45404       1.87872       1.41503      0.192023        10     
[art]:TriggerResults:TriggerResultInserter             2.629e-05    4.06085e-05   8.3261e-05    3.49805e-05   1.76217e-05      10     
end_path:out1:RootOutput                               5.69e-06     1.05801e-05   3.2181e-05     7.075e-06    7.7038e-06       10     
end_path:out1:RootOutput(write)                        0.187693      0.257602      0.330877      0.266621      0.0457276       10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4036.31 MB
  Peak resident set size usage (VmHWM): 2552.41 MB
====================================================================================================
Art has completed and will exit with status 0.
Moving on to metadata extractor 
============OUTPUT FILE:  prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250128T111936Z_gen_003768_supernova_g4stage1_g4stage2_detsim_reco_20250731T141451Z_reco2.root
Extracted metadata
checking file====== fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250128T111936Z_gen_003768_supernova_g4stage1_g4stage2_detsim_reco.root
MD conversion to MetaCat OK
all-input-dids.txt
debugprod.log
jobscript.log
justin-processed-pfns.txt
prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250128T111936Z_gen_003768_supernova_g4stage1_g4stage2_detsim_reco_20250731T141451Z_reco2.root
prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250128T111936Z_gen_003768_supernova_g4stage1_g4stage2_detsim_reco_20250731T141451Z_reco2.root.ext.json
prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250128T111936Z_gen_003768_supernova_g4stage1_g4stage2_detsim_reco_20250731T141451Z_reco2.root.json
reco2_hist.root