Jobsub ID 13317.161@dunegpschedd02.fnal.gov

Jobsub ID	13317.161@dunegpschedd02.fnal.gov
Workflow ID	163
Stage ID	1
User name	higuera@fnal.gov
HTCondor Group	group_dune.prod.mcsim
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-07-31 14:10:56
Site	UK_Durham
Entry	DUNE_UK_SGridDurham_ce3
Last heartbeat	2025-07-31 19:33:24
From worker node	Hostname	n252.dur.scotgrid.ac.uk
	cpuinfo	AMD EPYC 7763 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	GPU
	Inner Apptainer?	True
Job state	stalled
Started	2025-07-31 14:12:58
Input files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T092916Z_gen_003498_supernova_g4stage1_g4stage2_detsim_reco.root
Outputting started	2025-07-31 19:00:54
Output files
Finished	2025-07-31 20:35:02
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:57:58 BST run: 5021 subRun: 0 event: 34980
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:57:59 BST run: 5021 subRun: 0 event: 34980
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:57:59 BST run: 5021 subRun: 0 event: 34980
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:57:59 BST run: 5021 subRun: 0 event: 34980
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:57:59 BST run: 5021 subRun: 0 event: 34980
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:57:59 BST run: 5021 subRun: 0 event: 34980
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:57:59 BST run: 5021 subRun: 0 event: 34980
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:57:59 BST run: 5021 subRun: 0 event: 34980
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:57:59 BST run: 5021 subRun: 0 event: 34980
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:57:59 BST run: 5021 subRun: 0 event: 34980
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:57:59 BST run: 5021 subRun: 0 event: 34980
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:57:59 BST run: 5021 subRun: 0 event: 34980
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:57:59 BST run: 5021 subRun: 0 event: 34980
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:57:59 BST run: 5021 subRun: 0 event: 34980
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:57:59 BST run: 5021 subRun: 0 event: 34980
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:57:59 BST run: 5021 subRun: 0 event: 34980
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:57:59 BST run: 5021 subRun: 0 event: 34980
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:57:59 BST run: 5021 subRun: 0 event: 34980
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:57:59 BST run: 5021 subRun: 0 event: 34980
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:57:59 BST run: 5021 subRun: 0 event: 34980
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:57:59 BST run: 5021 subRun: 0 event: 34980
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:57:59 BST run: 5021 subRun: 0 event: 34980
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:57:59 BST run: 5021 subRun: 0 event: 34980
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:57:59 BST run: 5021 subRun: 0 event: 34980
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:57:59 BST run: 5021 subRun: 0 event: 34980
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:57:59 BST run: 5021 subRun: 0 event: 34980
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:57:59 BST run: 5021 subRun: 0 event: 34980
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:57:59 BST run: 5021 subRun: 0 event: 34980
Comparing two wires in the same plane: return failure
%MSG
31-Jul-2025 20:00:46 BST  Closed output file "prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T092916Z_gen_003498_supernova_g4stage1_g4stage2_detsim_reco_20250731T141305Z_reco2.root"
31-Jul-2025 20:00:46 BST  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/9a/65/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T092916Z_gen_003498_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1007.06       1360.05       1879.5        1374.64       213.307        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00487981     0.0386614     0.158195      0.0127766     0.0552408       10     
reco:pandora:StandardPandora                            864.527       1184.57       1622.55       1198.14       186.537        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000281032    0.0162527     0.119061     0.00329011     0.0346118       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000282053   0.00293529     0.0209584    0.000992312   0.00601477       10     
reco:pandoracalo:Calorimetry                          0.000218362   0.00157061    0.00718245    0.000351155   0.00245307       10     
reco:pandorapid:Chi2ParticleID                         5.259e-05    0.000358897   0.00273543    8.76455e-05   0.000792903      10     
reco:linecluster:LineCluster                            2.37637       3.29244       4.75185       3.1629       0.662583        10     
reco:trajcluster:TrajCluster                           0.216658      0.293829       0.59814      0.251803       0.11356        10     
reco:pmtracktc:PMAlgTrackMaker                          3.79661       5.07294       10.0049       4.44968       1.74901        10     
reco:emtrkmichelid:EmTrackMichelId                      130.846       163.741       239.816       159.997       27.7511        10     
reco:solarflash:SolarOpFlash                            1.96849       2.82919       4.82704       2.5943       0.802698        10     
[art]:TriggerResults:TriggerResultInserter             2.661e-05    4.82623e-05   0.000163469    3.427e-05    3.90706e-05      10     
end_path:out1:RootOutput                               5.851e-06    1.43479e-05   5.0575e-05     1.066e-05    1.24183e-05      10     
end_path:out1:RootOutput(write)                        0.164329      0.185408      0.301252      0.172126      0.0392425       10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4041.17 MB
  Peak resident set size usage (VmHWM): 2565.66 MB
====================================================================================================
Art has completed and will exit with status 0.
Moving on to metadata extractor 
============OUTPUT FILE:  prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T092916Z_gen_003498_supernova_g4stage1_g4stage2_detsim_reco_20250731T141305Z_reco2.root
Extracted metadata
checking file====== fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T092916Z_gen_003498_supernova_g4stage1_g4stage2_detsim_reco.root
MD conversion to MetaCat OK
all-input-dids.txt
debugprod.log
jobscript.log
justin-processed-pfns.txt
prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T092916Z_gen_003498_supernova_g4stage1_g4stage2_detsim_reco_20250731T141305Z_reco2.root
prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T092916Z_gen_003498_supernova_g4stage1_g4stage2_detsim_reco_20250731T141305Z_reco2.root.ext.json
prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T092916Z_gen_003498_supernova_g4stage1_g4stage2_detsim_reco_20250731T141305Z_reco2.root.json
reco2_hist.root