Jobsub ID 273497.45@dunegpschedd02.fnal.gov

Jobsub ID	273497.45@dunegpschedd02.fnal.gov
Workflow ID	11974
Stage ID	1
User name	imawby@fnal.gov
Requested	Processors	1
	GPU	No
	RSS bytes	1048576000 (1000 MiB)
	Wall seconds limit	7200 (2 hours)
Submitted time	2026-01-15 20:18:16
Site	FR_CCIN2P3
Entry	DUNE_FR_CCIN2P3_cccondorce03
Last heartbeat	2026-01-15 21:27:33
From worker node	Hostname	ccwcondor0032
	cpuinfo	AMD EPYC 9334 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	1048576000 (1000 MiB)
	Wall seconds limit	106200 (29 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2026-01-15 20:19:11
Input files	fardet-hd:nu_dune10kt_1x2x6_1109_709_20230827T003341Z_gen_g4_detsim_hitreco__20240229T163930Z_reco2.root
Jobscript	Exit code	0
	Real time	8m (534s)
	CPU time	5m (341s = 63%)
	Max RSS bytes	1160200192 (1106 MiB)
Outputting started	2026-01-15 20:28:07
Output files
Finished	2026-01-15 21:27:33
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

localProducts_larsoft_v10_13_00_e26_prof
----------- this block should be empty ------------------
---------------------------------------------------------
Input PFN = root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/2b/2c/nu_dune10kt_1x2x6_1109_709_20230827T003341Z_gen_g4_detsim_hitreco__20240229T163930Z_reco2.root
lar exit code 0
=== Start last 100 lines of lar log file ===
Begin processing the 29th record. run: 1109 subRun: 1 event: 70929 at 15-Jan-2026 21:23:52 CET
Begin processing the 30th record. run: 1109 subRun: 1 event: 70930 at 15-Jan-2026 21:23:58 CET
Begin processing the 31st record. run: 1109 subRun: 1 event: 70931 at 15-Jan-2026 21:24:01 CET
Begin processing the 32nd record. run: 1109 subRun: 1 event: 70932 at 15-Jan-2026 21:24:04 CET
Begin processing the 33rd record. run: 1109 subRun: 1 event: 70933 at 15-Jan-2026 21:24:07 CET
Begin processing the 34th record. run: 1109 subRun: 1 event: 70934 at 15-Jan-2026 21:24:10 CET
Begin processing the 35th record. run: 1109 subRun: 1 event: 70935 at 15-Jan-2026 21:24:13 CET
Begin processing the 36th record. run: 1109 subRun: 1 event: 70936 at 15-Jan-2026 21:24:18 CET
Begin processing the 37th record. run: 1109 subRun: 1 event: 70937 at 15-Jan-2026 21:24:20 CET
Begin processing the 38th record. run: 1109 subRun: 1 event: 70938 at 15-Jan-2026 21:24:23 CET
Begin processing the 39th record. run: 1109 subRun: 1 event: 70939 at 15-Jan-2026 21:24:26 CET
Begin processing the 40th record. run: 1109 subRun: 1 event: 70940 at 15-Jan-2026 21:24:29 CET
Begin processing the 41st record. run: 1109 subRun: 1 event: 70941 at 15-Jan-2026 21:24:32 CET
Begin processing the 42nd record. run: 1109 subRun: 1 event: 70942 at 15-Jan-2026 21:24:41 CET
Begin processing the 43rd record. run: 1109 subRun: 1 event: 70943 at 15-Jan-2026 21:24:44 CET
Begin processing the 44th record. run: 1109 subRun: 1 event: 70944 at 15-Jan-2026 21:24:48 CET
Begin processing the 45th record. run: 1109 subRun: 1 event: 70945 at 15-Jan-2026 21:24:51 CET
Begin processing the 46th record. run: 1109 subRun: 1 event: 70946 at 15-Jan-2026 21:24:54 CET
Begin processing the 47th record. run: 1109 subRun: 1 event: 70947 at 15-Jan-2026 21:24:57 CET
Begin processing the 48th record. run: 1109 subRun: 1 event: 70948 at 15-Jan-2026 21:25:00 CET
Begin processing the 49th record. run: 1109 subRun: 1 event: 70949 at 15-Jan-2026 21:25:06 CET
Begin processing the 50th record. run: 1109 subRun: 1 event: 70950 at 15-Jan-2026 21:25:10 CET
PandoraContentApi::GetList(*this, listName, pList) return STATUS_CODE_NOT_INITIALIZED
    in function: GetList
    in file:     /exp/dune/app/users/imawby/dunesw_validation/srcs/larpandoracontent/larpandoradlcontent/LArTwoDReco/DLThreeDClusterSplittingAlgorithm.cc line#: 194
Begin processing the 51st record. run: 1109 subRun: 1 event: 70951 at 15-Jan-2026 21:25:12 CET
Begin processing the 52nd record. run: 1109 subRun: 1 event: 70952 at 15-Jan-2026 21:25:16 CET
Begin processing the 53rd record. run: 1109 subRun: 1 event: 70953 at 15-Jan-2026 21:25:19 CET
Begin processing the 54th record. run: 1109 subRun: 1 event: 70954 at 15-Jan-2026 21:25:22 CET
Begin processing the 55th record. run: 1109 subRun: 1 event: 70955 at 15-Jan-2026 21:25:25 CET
Begin processing the 56th record. run: 1109 subRun: 1 event: 70956 at 15-Jan-2026 21:25:28 CET
Begin processing the 57th record. run: 1109 subRun: 1 event: 70957 at 15-Jan-2026 21:25:31 CET
Begin processing the 58th record. run: 1109 subRun: 1 event: 70958 at 15-Jan-2026 21:25:35 CET
Begin processing the 59th record. run: 1109 subRun: 1 event: 70959 at 15-Jan-2026 21:25:37 CET
Begin processing the 60th record. run: 1109 subRun: 1 event: 70960 at 15-Jan-2026 21:25:40 CET
Begin processing the 61st record. run: 1109 subRun: 1 event: 70961 at 15-Jan-2026 21:25:43 CET
Begin processing the 62nd record. run: 1109 subRun: 1 event: 70962 at 15-Jan-2026 21:25:46 CET
Begin processing the 63rd record. run: 1109 subRun: 1 event: 70963 at 15-Jan-2026 21:25:49 CET
Begin processing the 64th record. run: 1109 subRun: 1 event: 70964 at 15-Jan-2026 21:25:52 CET
Begin processing the 65th record. run: 1109 subRun: 1 event: 70965 at 15-Jan-2026 21:25:55 CET
Begin processing the 66th record. run: 1109 subRun: 1 event: 70966 at 15-Jan-2026 21:25:59 CET
Begin processing the 67th record. run: 1109 subRun: 1 event: 70967 at 15-Jan-2026 21:26:02 CET
Begin processing the 68th record. run: 1109 subRun: 1 event: 70968 at 15-Jan-2026 21:26:05 CET
Begin processing the 69th record. run: 1109 subRun: 1 event: 70969 at 15-Jan-2026 21:26:08 CET
Begin processing the 70th record. run: 1109 subRun: 1 event: 70970 at 15-Jan-2026 21:26:11 CET
Begin processing the 71st record. run: 1109 subRun: 1 event: 70971 at 15-Jan-2026 21:26:14 CET
Begin processing the 72nd record. run: 1109 subRun: 1 event: 70972 at 15-Jan-2026 21:26:17 CET
Begin processing the 73rd record. run: 1109 subRun: 1 event: 70973 at 15-Jan-2026 21:26:21 CET
Begin processing the 74th record. run: 1109 subRun: 1 event: 70974 at 15-Jan-2026 21:26:25 CET
Begin processing the 75th record. run: 1109 subRun: 1 event: 70975 at 15-Jan-2026 21:26:28 CET
Begin processing the 76th record. run: 1109 subRun: 1 event: 70976 at 15-Jan-2026 21:26:31 CET
Begin processing the 77th record. run: 1109 subRun: 1 event: 70977 at 15-Jan-2026 21:26:33 CET
Begin processing the 78th record. run: 1109 subRun: 1 event: 70978 at 15-Jan-2026 21:26:37 CET
Begin processing the 79th record. run: 1109 subRun: 1 event: 70979 at 15-Jan-2026 21:26:40 CET
Begin processing the 80th record. run: 1109 subRun: 1 event: 70980 at 15-Jan-2026 21:26:44 CET
Begin processing the 81st record. run: 1109 subRun: 1 event: 70981 at 15-Jan-2026 21:26:50 CET
Begin processing the 82nd record. run: 1109 subRun: 1 event: 70982 at 15-Jan-2026 21:26:58 CET
Begin processing the 83rd record. run: 1109 subRun: 1 event: 70983 at 15-Jan-2026 21:27:02 CET
PandoraContentApi::GetList(*this, listName, pList) return STATUS_CODE_NOT_INITIALIZED
    in function: GetList
    in file:     /exp/dune/app/users/imawby/dunesw_validation/srcs/larpandoracontent/larpandoradlcontent/LArTwoDReco/DLThreeDClusterSplittingAlgorithm.cc line#: 194
Begin processing the 84th record. run: 1109 subRun: 1 event: 70984 at 15-Jan-2026 21:27:05 CET
Begin processing the 85th record. run: 1109 subRun: 1 event: 70985 at 15-Jan-2026 21:27:08 CET
Begin processing the 86th record. run: 1109 subRun: 1 event: 70986 at 15-Jan-2026 21:27:11 CET
Begin processing the 87th record. run: 1109 subRun: 1 event: 70987 at 15-Jan-2026 21:27:15 CET
Begin processing the 88th record. run: 1109 subRun: 1 event: 70988 at 15-Jan-2026 21:27:18 CET
Begin processing the 89th record. run: 1109 subRun: 1 event: 70989 at 15-Jan-2026 21:27:21 CET
Begin processing the 90th record. run: 1109 subRun: 1 event: 70990 at 15-Jan-2026 21:27:24 CET
Begin processing the 91st record. run: 1109 subRun: 1 event: 70991 at 15-Jan-2026 21:27:27 CET
Begin processing the 92nd record. run: 1109 subRun: 1 event: 70992 at 15-Jan-2026 21:27:31 CET
Begin processing the 93rd record. run: 1109 subRun: 1 event: 70993 at 15-Jan-2026 21:27:33 CET
Begin processing the 94th record. run: 1109 subRun: 1 event: 70994 at 15-Jan-2026 21:27:42 CET
Begin processing the 95th record. run: 1109 subRun: 1 event: 70995 at 15-Jan-2026 21:27:46 CET
Begin processing the 96th record. run: 1109 subRun: 1 event: 70996 at 15-Jan-2026 21:27:50 CET
Begin processing the 97th record. run: 1109 subRun: 1 event: 70997 at 15-Jan-2026 21:27:53 CET
Begin processing the 98th record. run: 1109 subRun: 1 event: 70998 at 15-Jan-2026 21:27:55 CET
Begin processing the 99th record. run: 1109 subRun: 1 event: 70999 at 15-Jan-2026 21:27:58 CET
Begin processing the 100th record. run: 1109 subRun: 1 event: 71000 at 15-Jan-2026 21:28:02 CET
15-Jan-2026 21:28:05 CET  Closed output file "nu_dune10kt_1x2x6_1109_709_20230827T003341Z_gen_g4_detsim_hitreco__20240229T163930Z_reco2_reco2.root"
15-Jan-2026 21:28:05 CET  Closed input file "root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/2b/2c/nu_dune10kt_1x2x6_1109_709_20230827T003341Z_gen_g4_detsim_hitreco__20240229T163930Z_reco2.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                      2.22009       3.13198       8.37052       2.67353       1.33407        100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.0415023     0.056196      0.0872217     0.0567897    0.00933139       100    
reco:pandora:StandardPandora                    2.17431       3.06522       8.28956       2.60776       1.32761        100    
[art]:TriggerResults:TriggerResultInserter    1.2919e-05    2.89933e-05   0.000161703   2.4757e-05    1.90447e-05      100    
end_path:out1:RootOutput                       3.285e-06    6.01294e-06   1.8177e-05    6.2945e-06    1.84526e-06      100    
end_path:out1:RootOutput(write)               0.00143206     0.0102963     0.0382627     0.0083924     0.0083271       100    
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 2160.26 MB
  Peak resident set size usage (VmHWM): 1160.2 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===