Jobsub ID 13236.39@dunegpschedd02.fnal.gov

Jobsub ID	13236.39@dunegpschedd02.fnal.gov
Workflow ID	166
Stage ID	1
User name	higuera@fnal.gov
HTCondor Group	group_dune.prod.mcsim
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-07-31 08:24:40
Site	NL_NIKHEF
Entry	VIRGO_NL_NIKHEF_dissel
Last heartbeat	2025-07-31 14:10:12
From worker node	Hostname	wn-sate-040.farm.nikhef.nl
	cpuinfo	AMD EPYC 7551P 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	129600 (36 hours)
	GPU
	Inner Apptainer?	True
Job state	stalled
Started	2025-07-31 09:19:52
Input files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T072540Z_gen_004390_supernova_g4stage1_g4stage2_detsim_reco.root
Outputting started	2025-07-31 14:08:26
Output files
Finished	2025-07-31 15:30:23
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

n failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 16:05:48 CEST run: 5031 subRun: 0 event: 43900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 16:05:48 CEST run: 5031 subRun: 0 event: 43900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 16:05:48 CEST run: 5031 subRun: 0 event: 43900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 16:05:48 CEST run: 5031 subRun: 0 event: 43900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 16:05:48 CEST run: 5031 subRun: 0 event: 43900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 16:05:48 CEST run: 5031 subRun: 0 event: 43900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 16:05:48 CEST run: 5031 subRun: 0 event: 43900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 16:05:48 CEST run: 5031 subRun: 0 event: 43900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 16:05:48 CEST run: 5031 subRun: 0 event: 43900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 16:05:48 CEST run: 5031 subRun: 0 event: 43900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 16:05:48 CEST run: 5031 subRun: 0 event: 43900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 16:05:48 CEST run: 5031 subRun: 0 event: 43900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 16:05:48 CEST run: 5031 subRun: 0 event: 43900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 16:05:48 CEST run: 5031 subRun: 0 event: 43900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 16:05:48 CEST run: 5031 subRun: 0 event: 43900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 16:05:48 CEST run: 5031 subRun: 0 event: 43900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 16:05:48 CEST run: 5031 subRun: 0 event: 43900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 16:05:48 CEST run: 5031 subRun: 0 event: 43900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 16:05:48 CEST run: 5031 subRun: 0 event: 43900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 16:05:48 CEST run: 5031 subRun: 0 event: 43900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 16:05:48 CEST run: 5031 subRun: 0 event: 43900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 16:05:48 CEST run: 5031 subRun: 0 event: 43900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 16:05:48 CEST run: 5031 subRun: 0 event: 43900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 16:05:48 CEST run: 5031 subRun: 0 event: 43900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 16:05:48 CEST run: 5031 subRun: 0 event: 43900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 16:05:48 CEST run: 5031 subRun: 0 event: 43900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 16:05:48 CEST run: 5031 subRun: 0 event: 43900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 16:05:48 CEST run: 5031 subRun: 0 event: 43900
Comparing two wires in the same plane: return failure
%MSG
31-Jul-2025 16:08:20 CEST  Closed output file "prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T072540Z_gen_004390_supernova_g4stage1_g4stage2_detsim_reco_20250731T091956Z_reco2.root"
31-Jul-2025 16:08:21 CEST  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/ef/fb/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T072540Z_gen_004390_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1640.32       1694.16       1907.04       1666.39       77.4503        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00282678     0.0321866     0.144418     0.00714983     0.0524161       10     
reco:pandora:StandardPandora                            1491.39       1539.15       1725.07       1513.65       68.865         10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000232486   0.00645843     0.0520059    0.000264506    0.0153289       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000228278   0.00147279    0.00930028    0.000262613   0.00269698       10     
reco:pandoracalo:Calorimetry                          0.000175509   0.00265912    0.00408335    0.00342742    0.00144712       10     
reco:pandorapid:Chi2ParticleID                        4.7249e-05    0.000176275   0.000711706   5.9341e-05    0.000231046      10     
reco:linecluster:LineCluster                            1.15139       1.22919       1.74342       1.1711       0.171853        10     
reco:trajcluster:TrajCluster                           0.340121      0.363062       0.39017      0.361604      0.0132872       10     
reco:pmtracktc:PMAlgTrackMaker                          6.02311       6.39408       6.81466       6.37495      0.228835        10     
reco:emtrkmichelid:EmTrackMichelId                      136.803       145.24        171.246       142.341       9.15253        10     
reco:solarflash:SolarOpFlash                            1.42651       1.4742        1.73386       1.44952      0.0870668       10     
[art]:TriggerResults:TriggerResultInserter            1.8464e-05    2.63935e-05   6.0825e-05    2.25575e-05   1.17993e-05      10     
end_path:out1:RootOutput                               6.332e-06    8.8357e-06    2.7201e-05     6.868e-06    6.1295e-06       10     
end_path:out1:RootOutput(write)                        0.260674      0.269982      0.279198      0.269664     0.00546002       10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4024.97 MB
  Peak resident set size usage (VmHWM): 2557.37 MB
====================================================================================================
Art has completed and will exit with status 0.
Moving on to metadata extractor 
============OUTPUT FILE:  prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T072540Z_gen_004390_supernova_g4stage1_g4stage2_detsim_reco_20250731T091956Z_reco2.root
Extracted metadata
checking file====== fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T072540Z_gen_004390_supernova_g4stage1_g4stage2_detsim_reco.root
MD conversion to MetaCat OK
all-input-dids.txt
debugprod.log
jobscript.log
justin-processed-pfns.txt
prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T072540Z_gen_004390_supernova_g4stage1_g4stage2_detsim_reco_20250731T091956Z_reco2.root
prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T072540Z_gen_004390_supernova_g4stage1_g4stage2_detsim_reco_20250731T091956Z_reco2.root.ext.json
prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T072540Z_gen_004390_supernova_g4stage1_g4stage2_detsim_reco_20250731T091956Z_reco2.root.json
reco2_hist.root