Jobsub ID 13336.59@dunegpschedd02.fnal.gov

Jobsub ID	13336.59@dunegpschedd02.fnal.gov
Workflow ID	166
Stage ID	1
User name	higuera@fnal.gov
HTCondor Group	group_dune.prod.mcsim
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-07-31 14:58:57
Site	UK_Lancaster
Entry	UBoone_UK_Lancaster_HEC_grendel_ce02
Last heartbeat	2025-07-31 19:29:14
From worker node	Hostname	comp19-07
	cpuinfo	Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-07-31 15:01:30
Input files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250204T134346Z_gen_004821_supernova_g4stage1_g4stage2_detsim_reco.root
Jobscript	Exit code	0
	Real time	3h (14191s)
	CPU time	3h (13786s = 97%)
	Max RSS bytes	2579296256 (2459 MiB)
Outputting started	2025-07-31 18:58:02
Output files
Finished	2025-07-31 19:29:14
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:55:17 BST run: 5043 subRun: 0 event: 48210
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:55:17 BST run: 5043 subRun: 0 event: 48210
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:55:17 BST run: 5043 subRun: 0 event: 48210
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:55:17 BST run: 5043 subRun: 0 event: 48210
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:55:17 BST run: 5043 subRun: 0 event: 48210
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:55:17 BST run: 5043 subRun: 0 event: 48210
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:55:17 BST run: 5043 subRun: 0 event: 48210
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:55:17 BST run: 5043 subRun: 0 event: 48210
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:55:17 BST run: 5043 subRun: 0 event: 48210
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:55:17 BST run: 5043 subRun: 0 event: 48210
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:55:17 BST run: 5043 subRun: 0 event: 48210
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:55:17 BST run: 5043 subRun: 0 event: 48210
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:55:17 BST run: 5043 subRun: 0 event: 48210
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:55:17 BST run: 5043 subRun: 0 event: 48210
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:55:17 BST run: 5043 subRun: 0 event: 48210
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:55:17 BST run: 5043 subRun: 0 event: 48210
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:55:17 BST run: 5043 subRun: 0 event: 48210
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:55:17 BST run: 5043 subRun: 0 event: 48210
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:55:17 BST run: 5043 subRun: 0 event: 48210
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:55:17 BST run: 5043 subRun: 0 event: 48210
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:55:17 BST run: 5043 subRun: 0 event: 48210
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:55:17 BST run: 5043 subRun: 0 event: 48210
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:55:17 BST run: 5043 subRun: 0 event: 48210
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:55:17 BST run: 5043 subRun: 0 event: 48210
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:55:17 BST run: 5043 subRun: 0 event: 48210
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:55:17 BST run: 5043 subRun: 0 event: 48210
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:55:17 BST run: 5043 subRun: 0 event: 48210
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 19:55:17 BST run: 5043 subRun: 0 event: 48210
Comparing two wires in the same plane: return failure
%MSG
31-Jul-2025 19:57:58 BST  Closed output file "prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250204T134346Z_gen_004821_supernova_g4stage1_g4stage2_detsim_reco_20250731T150133Z_reco2.root"
31-Jul-2025 19:57:59 BST  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/9a/38/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250204T134346Z_gen_004821_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1356.03       1382.57       1415.99       1376.16       19.9322        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00247572     0.0314031      0.13775     0.00669364     0.0499784       10     
reco:pandora:StandardPandora                            1181.6        1208.15       1239.48       1205.06       17.6432        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.00021309    0.00566613     0.0380052    0.00233331     0.0109213       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000223753   0.00131871    0.00685848    0.000746031   0.00186669       10     
reco:pandoracalo:Calorimetry                          0.000258797   0.00105689    0.00384663    0.00031019    0.00122423       10     
reco:pandorapid:Chi2ParticleID                        4.2503e-05    0.00013575    0.00072542    7.69005e-05   0.000196976      10     
reco:linecluster:LineCluster                           0.956778       1.06898       1.54108       1.04315      0.162584        10     
reco:trajcluster:TrajCluster                           0.255132      0.266228      0.285269      0.265012     0.00923282       10     
reco:pmtracktc:PMAlgTrackMaker                          4.28364       4.59324       4.98011       4.59065      0.270822        10     
reco:emtrkmichelid:EmTrackMichelId                      158.228       167.096       174.497       168.243       5.08355        10     
reco:solarflash:SolarOpFlash                            1.07829       1.15424       1.42413       1.15065      0.096145        10     
[art]:TriggerResults:TriggerResultInserter            2.3621e-05    2.81692e-05   3.9632e-05    2.7827e-05    4.27831e-06      10     
end_path:out1:RootOutput                               5.902e-06    8.0717e-06    1.6946e-05    6.8725e-06    3.3058e-06       10     
end_path:out1:RootOutput(write)                        0.186469      0.195788      0.207336       0.19451     0.00528976       10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4042.82 MB
  Peak resident set size usage (VmHWM): 2579.3 MB
====================================================================================================
Art has completed and will exit with status 0.
Moving on to metadata extractor 
============OUTPUT FILE:  prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250204T134346Z_gen_004821_supernova_g4stage1_g4stage2_detsim_reco_20250731T150133Z_reco2.root
Extracted metadata
checking file====== fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250204T134346Z_gen_004821_supernova_g4stage1_g4stage2_detsim_reco.root
MD conversion to MetaCat OK
all-input-dids.txt
debugprod.log
jobscript.log
justin-processed-pfns.txt
prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250204T134346Z_gen_004821_supernova_g4stage1_g4stage2_detsim_reco_20250731T150133Z_reco2.root
prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250204T134346Z_gen_004821_supernova_g4stage1_g4stage2_detsim_reco_20250731T150133Z_reco2.root.ext.json
prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250204T134346Z_gen_004821_supernova_g4stage1_g4stage2_detsim_reco_20250731T150133Z_reco2.root.json
reco2_hist.root