Jobsub ID 19873.73@dunegpschedd01.fnal.gov

Jobsub ID	19873.73@dunegpschedd01.fnal.gov
Workflow ID	163
Stage ID	1
User name	higuera@fnal.gov
HTCondor Group	group_dune.prod.mcsim
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-07-31 15:34:06
Site	CERN
Entry	CMSHTPC_T2_CH_CERN_ce507
Last heartbeat	2025-07-31 20:37:56
From worker node	Hostname	b9p24p7870.cern.ch
	cpuinfo	AMD EPYC 7302 16-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	343800 (95 hours)
	GPU
	Inner Apptainer?	True
Job state	stalled
Started	2025-07-31 15:47:37
Input files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T041345Z_gen_002690_supernova_g4stage1_g4stage2_detsim_reco.root
Outputting started	2025-07-31 20:23:27
Output files
Finished	2025-07-31 21:10:57
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

n failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 22:21:00 CEST run: 5019 subRun: 0 event: 26900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 22:21:00 CEST run: 5019 subRun: 0 event: 26900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 22:21:00 CEST run: 5019 subRun: 0 event: 26900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 22:21:00 CEST run: 5019 subRun: 0 event: 26900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 22:21:00 CEST run: 5019 subRun: 0 event: 26900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 22:21:00 CEST run: 5019 subRun: 0 event: 26900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 22:21:00 CEST run: 5019 subRun: 0 event: 26900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 22:21:00 CEST run: 5019 subRun: 0 event: 26900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 22:21:00 CEST run: 5019 subRun: 0 event: 26900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 22:21:00 CEST run: 5019 subRun: 0 event: 26900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 22:21:00 CEST run: 5019 subRun: 0 event: 26900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 22:21:00 CEST run: 5019 subRun: 0 event: 26900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 22:21:00 CEST run: 5019 subRun: 0 event: 26900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 22:21:00 CEST run: 5019 subRun: 0 event: 26900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 22:21:00 CEST run: 5019 subRun: 0 event: 26900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 22:21:00 CEST run: 5019 subRun: 0 event: 26900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 22:21:00 CEST run: 5019 subRun: 0 event: 26900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 22:21:00 CEST run: 5019 subRun: 0 event: 26900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 22:21:00 CEST run: 5019 subRun: 0 event: 26900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 22:21:00 CEST run: 5019 subRun: 0 event: 26900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 22:21:00 CEST run: 5019 subRun: 0 event: 26900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 22:21:00 CEST run: 5019 subRun: 0 event: 26900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 22:21:00 CEST run: 5019 subRun: 0 event: 26900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 22:21:00 CEST run: 5019 subRun: 0 event: 26900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 22:21:00 CEST run: 5019 subRun: 0 event: 26900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 22:21:00 CEST run: 5019 subRun: 0 event: 26900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 22:21:00 CEST run: 5019 subRun: 0 event: 26900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 22:21:00 CEST run: 5019 subRun: 0 event: 26900
Comparing two wires in the same plane: return failure
%MSG
31-Jul-2025 22:23:22 CEST  Closed output file "prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T041345Z_gen_002690_supernova_g4stage1_g4stage2_detsim_reco_20250731T154743Z_reco2.root"
31-Jul-2025 22:23:22 CEST  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/31/30/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T041345Z_gen_002690_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1528.86       1622.32       1706.35       1626.36       59.466         10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0029759     0.0286387     0.118405     0.00809629     0.0438407       10     
reco:pandora:StandardPandora                            1385.24       1479.74       1565.26        1486         59.4231        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000208623   0.00501512     0.0390406    0.00118561     0.011395        10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000230253   0.00128053     0.0071836    0.000636679   0.00200077       10     
reco:pandoracalo:Calorimetry                          0.000238083   0.00113194    0.00212637    0.00101795    0.000781982      10     
reco:pandorapid:Chi2ParticleID                         4.657e-05    0.000150658   0.000781402   8.2486e-05    0.000212048      10     
reco:linecluster:LineCluster                           0.955336       1.06985       1.43429       0.99464       0.16115        10     
reco:trajcluster:TrajCluster                            0.2332       0.277833      0.346124       0.25664      0.0415372       10     
reco:pmtracktc:PMAlgTrackMaker                          4.36259       4.91268       6.18542       4.79263      0.535784        10     
reco:emtrkmichelid:EmTrackMichelId                      130.932       134.825       142.415       134.038       3.37272        10     
reco:solarflash:SolarOpFlash                            1.14788       1.23725       1.58256       1.17438      0.129495        10     
[art]:TriggerResults:TriggerResultInserter             2.61e-05     3.47754e-05   7.8711e-05    2.97605e-05   1.51714e-05      10     
end_path:out1:RootOutput                               9.25e-06     1.57762e-05   4.1391e-05    1.0085e-05    1.19447e-05      10     
end_path:out1:RootOutput(write)                        0.191488       0.21726      0.286095      0.199562      0.035568        10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4045.55 MB
  Peak resident set size usage (VmHWM): 2558.23 MB
====================================================================================================
Art has completed and will exit with status 0.
Moving on to metadata extractor 
============OUTPUT FILE:  prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T041345Z_gen_002690_supernova_g4stage1_g4stage2_detsim_reco_20250731T154743Z_reco2.root
Extracted metadata
checking file====== fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T041345Z_gen_002690_supernova_g4stage1_g4stage2_detsim_reco.root
MD conversion to MetaCat OK
all-input-dids.txt
debugprod.log
jobscript.log
justin-processed-pfns.txt
prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T041345Z_gen_002690_supernova_g4stage1_g4stage2_detsim_reco_20250731T154743Z_reco2.root
prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T041345Z_gen_002690_supernova_g4stage1_g4stage2_detsim_reco_20250731T154743Z_reco2.root.ext.json
prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T041345Z_gen_002690_supernova_g4stage1_g4stage2_detsim_reco_20250731T154743Z_reco2.root.json
reco2_hist.root