Jobsub ID 19700.79@dunegpschedd01.fnal.gov

Jobsub ID	19700.79@dunegpschedd01.fnal.gov
Workflow ID	166
Stage ID	1
User name	higuera@fnal.gov
HTCondor Group	group_dune.prod.mcsim
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-07-31 03:49:23
Site	UK_Durham
Entry	DUNE_UK_SGridDurham_ce4
Last heartbeat	2025-07-31 14:12:42
From worker node	Hostname	n246.dur.scotgrid.ac.uk
	cpuinfo	AMD EPYC 7763 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	GPU
	Inner Apptainer?	True
Job state	stalled
Started	2025-07-31 03:51:56
Input files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T031150Z_gen_004740_supernova_g4stage1_g4stage2_detsim_reco.root
Outputting started	2025-07-31 14:05:21
Output files
Finished	2025-07-31 15:30:23
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

wo wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 14:59:52 BST run: 5027 subRun: 0 event: 47400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 14:59:52 BST run: 5027 subRun: 0 event: 47400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 14:59:52 BST run: 5027 subRun: 0 event: 47400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 14:59:52 BST run: 5027 subRun: 0 event: 47400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 14:59:52 BST run: 5027 subRun: 0 event: 47400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 14:59:52 BST run: 5027 subRun: 0 event: 47400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 14:59:52 BST run: 5027 subRun: 0 event: 47400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 14:59:52 BST run: 5027 subRun: 0 event: 47400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 14:59:52 BST run: 5027 subRun: 0 event: 47400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 14:59:52 BST run: 5027 subRun: 0 event: 47400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 14:59:52 BST run: 5027 subRun: 0 event: 47400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 14:59:52 BST run: 5027 subRun: 0 event: 47400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 14:59:53 BST run: 5027 subRun: 0 event: 47400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 14:59:53 BST run: 5027 subRun: 0 event: 47400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 14:59:53 BST run: 5027 subRun: 0 event: 47400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 14:59:53 BST run: 5027 subRun: 0 event: 47400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 14:59:53 BST run: 5027 subRun: 0 event: 47400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 14:59:53 BST run: 5027 subRun: 0 event: 47400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 14:59:53 BST run: 5027 subRun: 0 event: 47400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 14:59:53 BST run: 5027 subRun: 0 event: 47400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 14:59:53 BST run: 5027 subRun: 0 event: 47400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 14:59:53 BST run: 5027 subRun: 0 event: 47400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 14:59:53 BST run: 5027 subRun: 0 event: 47400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 14:59:53 BST run: 5027 subRun: 0 event: 47400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 14:59:53 BST run: 5027 subRun: 0 event: 47400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 14:59:53 BST run: 5027 subRun: 0 event: 47400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 14:59:53 BST run: 5027 subRun: 0 event: 47400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 14:59:53 BST run: 5027 subRun: 0 event: 47400
Comparing two wires in the same plane: return failure
%MSG
31-Jul-2025 15:05:13 BST  Closed output file "prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T031150Z_gen_004740_supernova_g4stage1_g4stage2_detsim_reco_20250731T035203Z_reco2.root"
31-Jul-2025 15:05:14 BST  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/9f/e9/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T031150Z_gen_004740_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              3204.01       3482.58       3801.63       3442.05       176.997        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0134409     0.0630697     0.184993      0.038606      0.056865        10     
reco:pandora:StandardPandora                            2803.36       3070.5        3385.61       3045.43       166.978        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000676069    0.0391405     0.245299      0.0218204     0.0696241       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000643558   0.00735933     0.0578071    0.00186219     0.0168349       10     
reco:pandoracalo:Calorimetry                          0.000635382   0.00873117     0.0233377    0.000970401   0.00988555       10     
reco:pandorapid:Chi2ParticleID                        0.000144924   0.000831455   0.00645114    0.000203485   0.00187393       10     
reco:linecluster:LineCluster                            1.43442       1.92879       3.0344        1.58503      0.586754        10     
reco:trajcluster:TrajCluster                           0.345706      0.548752      0.716467       0.57008      0.0961462       10     
reco:pmtracktc:PMAlgTrackMaker                          4.51355       8.04628       10.8474       8.55612       2.12869        10     
reco:emtrkmichelid:EmTrackMichelId                      315.789       399.161       474.463       402.154       45.8472        10     
reco:solarflash:SolarOpFlash                            1.41673       1.96535       3.29754       1.84635      0.546277        10     
[art]:TriggerResults:TriggerResultInserter            5.2329e-05    0.000127604   0.000382624   0.000105419   9.05368e-05      10     
end_path:out1:RootOutput                              1.8686e-05    3.86815e-05   0.000104848   3.07935e-05   2.44239e-05      10     
end_path:out1:RootOutput(write)                        0.186366      0.303419      0.383252      0.350917      0.0801042       10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4044.04 MB
  Peak resident set size usage (VmHWM): 2570 MB
====================================================================================================
Art has completed and will exit with status 0.
Moving on to metadata extractor 
============OUTPUT FILE:  prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T031150Z_gen_004740_supernova_g4stage1_g4stage2_detsim_reco_20250731T035203Z_reco2.root
Extracted metadata
checking file====== fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T031150Z_gen_004740_supernova_g4stage1_g4stage2_detsim_reco.root
MD conversion to MetaCat OK
all-input-dids.txt
debugprod.log
jobscript.log
justin-processed-pfns.txt
prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T031150Z_gen_004740_supernova_g4stage1_g4stage2_detsim_reco_20250731T035203Z_reco2.root
prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T031150Z_gen_004740_supernova_g4stage1_g4stage2_detsim_reco_20250731T035203Z_reco2.root.ext.json
prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T031150Z_gen_004740_supernova_g4stage1_g4stage2_detsim_reco_20250731T035203Z_reco2.root.json
reco2_hist.root