Jobsub ID 13375.9@dunegpschedd02.fnal.gov

Jobsub ID	13375.9@dunegpschedd02.fnal.gov
Workflow ID	166
Stage ID	1
User name	higuera@fnal.gov
HTCondor Group	group_dune.prod.mcsim
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-07-31 15:37:59
Site	UK_RAL-Tier1
Entry	LIGO_UK_RAL_arc_ce02
Last heartbeat	2025-07-31 19:37:43
From worker node	Hostname	dune001-7488727.0-lcg2707.gridpp.rl.ac.uk
	cpuinfo	AMD EPYC 9654 96-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	216000 (60 hours)
	GPU
	Inner Apptainer?	True
Job state	stalled
Started	2025-07-31 16:16:12
Input files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250129T164523Z_gen_002679_supernova_g4stage1_g4stage2_detsim_reco.root
Outputting started	2025-07-31 19:37:43
Output files
Finished	2025-07-31 21:10:56
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:35:02 UTC run: 4984 subRun: 0 event: 26790
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:35:02 UTC run: 4984 subRun: 0 event: 26790
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:35:02 UTC run: 4984 subRun: 0 event: 26790
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:35:02 UTC run: 4984 subRun: 0 event: 26790
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:35:02 UTC run: 4984 subRun: 0 event: 26790
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:35:02 UTC run: 4984 subRun: 0 event: 26790
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:35:02 UTC run: 4984 subRun: 0 event: 26790
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:35:02 UTC run: 4984 subRun: 0 event: 26790
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:35:02 UTC run: 4984 subRun: 0 event: 26790
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:35:02 UTC run: 4984 subRun: 0 event: 26790
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:35:02 UTC run: 4984 subRun: 0 event: 26790
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:35:02 UTC run: 4984 subRun: 0 event: 26790
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:35:02 UTC run: 4984 subRun: 0 event: 26790
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:35:02 UTC run: 4984 subRun: 0 event: 26790
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:35:02 UTC run: 4984 subRun: 0 event: 26790
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:35:02 UTC run: 4984 subRun: 0 event: 26790
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:35:02 UTC run: 4984 subRun: 0 event: 26790
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:35:02 UTC run: 4984 subRun: 0 event: 26790
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:35:02 UTC run: 4984 subRun: 0 event: 26790
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:35:02 UTC run: 4984 subRun: 0 event: 26790
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:35:02 UTC run: 4984 subRun: 0 event: 26790
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:35:03 UTC run: 4984 subRun: 0 event: 26790
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:35:03 UTC run: 4984 subRun: 0 event: 26790
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:35:03 UTC run: 4984 subRun: 0 event: 26790
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:35:03 UTC run: 4984 subRun: 0 event: 26790
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:35:03 UTC run: 4984 subRun: 0 event: 26790
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:35:03 UTC run: 4984 subRun: 0 event: 26790
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:35:03 UTC run: 4984 subRun: 0 event: 26790
Comparing two wires in the same plane: return failure
%MSG
31-Jul-2025 19:37:38 UTC  Closed output file "prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250129T164523Z_gen_002679_supernova_g4stage1_g4stage2_detsim_reco_20250731T161617Z_reco2.root"
31-Jul-2025 19:37:39 UTC  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/3a/71/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250129T164523Z_gen_002679_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1033.89       1185.88       1331.63       1197.79       108.172        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00332047     0.0362511     0.144777      0.0124648     0.0490427       10     
reco:pandora:StandardPandora                            917.049       1054.17       1198.68       1061.68       105.273        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000334294   0.00803564     0.0489317    0.00370232     0.0138562       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000314074   0.00192302    0.00497142     0.0012845    0.00149257       10     
reco:pandoracalo:Calorimetry                          0.000279242   0.00111742    0.00535655    0.000542609   0.00146091       10     
reco:pandorapid:Chi2ParticleID                        7.7967e-05    0.000168841   0.000455597   0.000108238   0.000132227      10     
reco:linecluster:LineCluster                            1.06618       1.18537       1.55244       1.15417      0.127178        10     
reco:trajcluster:TrajCluster                           0.260663      0.342385       0.41325      0.351415      0.0430388       10     
reco:pmtracktc:PMAlgTrackMaker                          5.23376       6.66099       9.02508       6.56866       1.26445        10     
reco:emtrkmichelid:EmTrackMichelId                      98.4105       121.89        151.941       121.23        13.7588        10     
reco:solarflash:SolarOpFlash                            1.14853       1.35277       1.49839       1.35536      0.0957425       10     
[art]:TriggerResults:TriggerResultInserter            2.6991e-05    4.01397e-05   7.6967e-05    3.3851e-05    1.58945e-05      10     
end_path:out1:RootOutput                               7.441e-06    1.34914e-05   2.9966e-05    1.1257e-05    6.42734e-06      10     
end_path:out1:RootOutput(write)                        0.164586      0.225394      0.272522      0.238534      0.0349966       10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4106.8 MB
  Peak resident set size usage (VmHWM): 2496.73 MB
====================================================================================================
Art has completed and will exit with status 0.
Moving on to metadata extractor 
============OUTPUT FILE:  prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250129T164523Z_gen_002679_supernova_g4stage1_g4stage2_detsim_reco_20250731T161617Z_reco2.root
Extracted metadata
checking file====== fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250129T164523Z_gen_002679_supernova_g4stage1_g4stage2_detsim_reco.root
MD conversion to MetaCat OK
all-input-dids.txt
debugprod.log
jobscript.log
justin-processed-pfns.txt
prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250129T164523Z_gen_002679_supernova_g4stage1_g4stage2_detsim_reco_20250731T161617Z_reco2.root
prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250129T164523Z_gen_002679_supernova_g4stage1_g4stage2_detsim_reco_20250731T161617Z_reco2.root.ext.json
prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250129T164523Z_gen_002679_supernova_g4stage1_g4stage2_detsim_reco_20250731T161617Z_reco2.root.json
reco2_hist.root