Jobsub ID 19873.142@dunegpschedd01.fnal.gov

Jobsub ID	19873.142@dunegpschedd01.fnal.gov
Workflow ID	163
Stage ID	1
User name	higuera@fnal.gov
HTCondor Group	group_dune.prod.mcsim
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-07-31 15:34:06
Site	UK_RAL-Tier1
Entry	LIGO_UK_RAL_arc_ce01
Last heartbeat	2025-07-31 20:22:44
From worker node	Hostname	dune001-8380840.0-lcg2694.gridpp.rl.ac.uk
	cpuinfo	AMD EPYC 9654 96-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	216000 (60 hours)
	GPU
	Inner Apptainer?	True
Job state	stalled
Started	2025-07-31 15:52:24
Input files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T043937Z_gen_002660_supernova_g4stage1_g4stage2_detsim_reco.root
Outputting started	2025-07-31 20:02:42
Output files
Finished	2025-07-31 21:10:56
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:59:53 UTC run: 5030 subRun: 0 event: 26600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:59:53 UTC run: 5030 subRun: 0 event: 26600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:59:53 UTC run: 5030 subRun: 0 event: 26600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:59:53 UTC run: 5030 subRun: 0 event: 26600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:59:53 UTC run: 5030 subRun: 0 event: 26600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:59:53 UTC run: 5030 subRun: 0 event: 26600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:59:53 UTC run: 5030 subRun: 0 event: 26600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:59:53 UTC run: 5030 subRun: 0 event: 26600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:59:53 UTC run: 5030 subRun: 0 event: 26600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:59:53 UTC run: 5030 subRun: 0 event: 26600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:59:54 UTC run: 5030 subRun: 0 event: 26600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:59:54 UTC run: 5030 subRun: 0 event: 26600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:59:54 UTC run: 5030 subRun: 0 event: 26600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:59:54 UTC run: 5030 subRun: 0 event: 26600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:59:54 UTC run: 5030 subRun: 0 event: 26600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:59:54 UTC run: 5030 subRun: 0 event: 26600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:59:54 UTC run: 5030 subRun: 0 event: 26600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:59:54 UTC run: 5030 subRun: 0 event: 26600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:59:54 UTC run: 5030 subRun: 0 event: 26600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:59:54 UTC run: 5030 subRun: 0 event: 26600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:59:54 UTC run: 5030 subRun: 0 event: 26600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:59:54 UTC run: 5030 subRun: 0 event: 26600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:59:54 UTC run: 5030 subRun: 0 event: 26600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:59:54 UTC run: 5030 subRun: 0 event: 26600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:59:54 UTC run: 5030 subRun: 0 event: 26600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:59:54 UTC run: 5030 subRun: 0 event: 26600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:59:54 UTC run: 5030 subRun: 0 event: 26600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:59:54 UTC run: 5030 subRun: 0 event: 26600
Comparing two wires in the same plane: return failure
%MSG
31-Jul-2025 20:02:34 UTC  Closed output file "prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T043937Z_gen_002660_supernova_g4stage1_g4stage2_detsim_reco_20250731T155230Z_reco2.root"
31-Jul-2025 20:02:34 UTC  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/1f/e6/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T043937Z_gen_002660_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1266.2        1473.23       1653.33       1470.14       111.496        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00438372     0.0353308     0.152047      0.0103647     0.0521034       10     
reco:pandora:StandardPandora                            1133.21       1316.3        1483.65       1311.76       107.649        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000348496    0.0100611     0.0585381    0.00500831     0.0163738       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000315055   0.00210976    0.00992659    0.00130971    0.00263705       10     
reco:pandoracalo:Calorimetry                          0.000361886   0.000926808   0.00291806    0.000526093   0.000902439      10     
reco:pandorapid:Chi2ParticleID                         6.003e-05    0.000212198   0.00103073    0.000123992   0.000274092      10     
reco:linecluster:LineCluster                            1.09314       1.23728       1.63016       1.20734      0.140198        10     
reco:trajcluster:TrajCluster                           0.311312      0.384146      0.502731       0.3671       0.054576        10     
reco:pmtracktc:PMAlgTrackMaker                          6.38404       8.04004       9.66819       8.20961       1.03105        10     
reco:emtrkmichelid:EmTrackMichelId                      121.288       145.585       159.862       144.884       11.0448        10     
reco:solarflash:SolarOpFlash                            1.32471       1.4018        1.46706       1.40915      0.0465979       10     
[art]:TriggerResults:TriggerResultInserter            2.9364e-05    4.79333e-05   9.9701e-05    4.2259e-05    1.87755e-05      10     
end_path:out1:RootOutput                               9.054e-06    1.43536e-05   3.2038e-05    1.2454e-05    6.28513e-06      10     
end_path:out1:RootOutput(write)                        0.193013      0.238976      0.264469      0.252271      0.0273312       10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4080.08 MB
  Peak resident set size usage (VmHWM): 2460.68 MB
====================================================================================================
Art has completed and will exit with status 0.
Moving on to metadata extractor 
============OUTPUT FILE:  prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T043937Z_gen_002660_supernova_g4stage1_g4stage2_detsim_reco_20250731T155230Z_reco2.root
Extracted metadata
checking file====== fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T043937Z_gen_002660_supernova_g4stage1_g4stage2_detsim_reco.root
MD conversion to MetaCat OK
all-input-dids.txt
debugprod.log
jobscript.log
justin-processed-pfns.txt
prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T043937Z_gen_002660_supernova_g4stage1_g4stage2_detsim_reco_20250731T155230Z_reco2.root
prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T043937Z_gen_002660_supernova_g4stage1_g4stage2_detsim_reco_20250731T155230Z_reco2.root.ext.json
prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T043937Z_gen_002660_supernova_g4stage1_g4stage2_detsim_reco_20250731T155230Z_reco2.root.json
reco2_hist.root