Jobsub ID 273497.19@dunegpschedd02.fnal.gov

Jobsub ID	273497.19@dunegpschedd02.fnal.gov
Workflow ID	11974
Stage ID	1
User name	imawby@fnal.gov
Requested	Processors	1
	GPU	No
	RSS bytes	1048576000 (1000 MiB)
	Wall seconds limit	7200 (2 hours)
Submitted time	2026-01-15 20:18:16
Site	ES_CIEMAT
Entry	DUNE_T2_ES_CIEMAT_condorce2
Last heartbeat	2026-01-15 21:45:34
From worker node	Hostname	gaew0205.ciemat.es
	cpuinfo	Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	1048576000 (1000 MiB)
	Wall seconds limit	232200 (64 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2026-01-15 20:19:43
Input files	fardet-hd:nu_dune10kt_1x2x6_1111_489_20230827T033426Z_gen_g4_detsim_hitreco__20240229T172054Z_reco2.root
Jobscript	Exit code	0
	Real time	26m (1591s)
	CPU time	20m (1229s = 77%)
	Max RSS bytes	1418289152 (1352 MiB)
Outputting started	2026-01-15 20:46:15
Output files
Finished	2026-01-15 21:45:34
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

cts_larsoft_v10_13_00_e26_prof
----------- this block should be empty ------------------
---------------------------------------------------------
Input PFN = root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/d9/f5/nu_dune10kt_1x2x6_1111_489_20230827T033426Z_gen_g4_detsim_hitreco__20240229T172054Z_reco2.root
lar exit code 0
=== Start last 100 lines of lar log file ===
Begin processing the 26th record. run: 1111 subRun: 1 event: 48926 at 15-Jan-2026 21:28:13 CET
Begin processing the 27th record. run: 1111 subRun: 1 event: 48927 at 15-Jan-2026 21:28:18 CET
Begin processing the 28th record. run: 1111 subRun: 1 event: 48928 at 15-Jan-2026 21:28:22 CET
Begin processing the 29th record. run: 1111 subRun: 1 event: 48929 at 15-Jan-2026 21:28:28 CET
Begin processing the 30th record. run: 1111 subRun: 1 event: 48930 at 15-Jan-2026 21:28:34 CET
Begin processing the 31st record. run: 1111 subRun: 1 event: 48931 at 15-Jan-2026 21:28:38 CET
Begin processing the 32nd record. run: 1111 subRun: 1 event: 48932 at 15-Jan-2026 21:28:43 CET
Begin processing the 33rd record. run: 1111 subRun: 1 event: 48933 at 15-Jan-2026 21:28:48 CET
Begin processing the 34th record. run: 1111 subRun: 1 event: 48934 at 15-Jan-2026 21:28:53 CET
Begin processing the 35th record. run: 1111 subRun: 1 event: 48935 at 15-Jan-2026 21:28:58 CET
Begin processing the 36th record. run: 1111 subRun: 1 event: 48936 at 15-Jan-2026 21:29:02 CET
PandoraContentApi::GetList(*this, listName, pList) return STATUS_CODE_NOT_INITIALIZED
    in function: GetList
    in file:     /exp/dune/app/users/imawby/dunesw_validation/srcs/larpandoracontent/larpandoradlcontent/LArTwoDReco/DLThreeDClusterSplittingAlgorithm.cc line#: 194
Begin processing the 37th record. run: 1111 subRun: 1 event: 48937 at 15-Jan-2026 21:29:06 CET
Begin processing the 38th record. run: 1111 subRun: 1 event: 48938 at 15-Jan-2026 21:29:11 CET
Begin processing the 39th record. run: 1111 subRun: 1 event: 48939 at 15-Jan-2026 21:29:15 CET
Begin processing the 40th record. run: 1111 subRun: 1 event: 48940 at 15-Jan-2026 21:29:20 CET
Begin processing the 41st record. run: 1111 subRun: 1 event: 48941 at 15-Jan-2026 21:29:26 CET
Begin processing the 42nd record. run: 1111 subRun: 1 event: 48942 at 15-Jan-2026 21:29:30 CET
Begin processing the 43rd record. run: 1111 subRun: 1 event: 48943 at 15-Jan-2026 21:29:35 CET
Begin processing the 44th record. run: 1111 subRun: 1 event: 48944 at 15-Jan-2026 21:29:42 CET
Begin processing the 45th record. run: 1111 subRun: 1 event: 48945 at 15-Jan-2026 21:29:48 CET
Begin processing the 46th record. run: 1111 subRun: 1 event: 48946 at 15-Jan-2026 21:29:53 CET
Begin processing the 47th record. run: 1111 subRun: 1 event: 48947 at 15-Jan-2026 21:31:40 CET
Begin processing the 48th record. run: 1111 subRun: 1 event: 48948 at 15-Jan-2026 21:31:46 CET
Begin processing the 49th record. run: 1111 subRun: 1 event: 48949 at 15-Jan-2026 21:31:50 CET
Begin processing the 50th record. run: 1111 subRun: 1 event: 48950 at 15-Jan-2026 21:31:55 CET
Begin processing the 51st record. run: 1111 subRun: 1 event: 48951 at 15-Jan-2026 21:32:01 CET
Begin processing the 52nd record. run: 1111 subRun: 1 event: 48952 at 15-Jan-2026 21:32:08 CET
Begin processing the 53rd record. run: 1111 subRun: 1 event: 48953 at 15-Jan-2026 21:32:13 CET
Begin processing the 54th record. run: 1111 subRun: 1 event: 48954 at 15-Jan-2026 21:32:18 CET
Begin processing the 55th record. run: 1111 subRun: 1 event: 48955 at 15-Jan-2026 21:32:23 CET
Begin processing the 56th record. run: 1111 subRun: 1 event: 48956 at 15-Jan-2026 21:32:33 CET
Begin processing the 57th record. run: 1111 subRun: 1 event: 48957 at 15-Jan-2026 21:32:38 CET
Begin processing the 58th record. run: 1111 subRun: 1 event: 48958 at 15-Jan-2026 21:32:43 CET
Begin processing the 59th record. run: 1111 subRun: 1 event: 48959 at 15-Jan-2026 21:40:57 CET
Begin processing the 60th record. run: 1111 subRun: 1 event: 48960 at 15-Jan-2026 21:42:02 CET
Begin processing the 61st record. run: 1111 subRun: 1 event: 48961 at 15-Jan-2026 21:42:06 CET
Begin processing the 62nd record. run: 1111 subRun: 1 event: 48962 at 15-Jan-2026 21:42:12 CET
Begin processing the 63rd record. run: 1111 subRun: 1 event: 48963 at 15-Jan-2026 21:42:18 CET
Begin processing the 64th record. run: 1111 subRun: 1 event: 48964 at 15-Jan-2026 21:42:23 CET
Begin processing the 65th record. run: 1111 subRun: 1 event: 48965 at 15-Jan-2026 21:42:28 CET
Begin processing the 66th record. run: 1111 subRun: 1 event: 48966 at 15-Jan-2026 21:42:33 CET
Begin processing the 67th record. run: 1111 subRun: 1 event: 48967 at 15-Jan-2026 21:42:38 CET
Begin processing the 68th record. run: 1111 subRun: 1 event: 48968 at 15-Jan-2026 21:42:43 CET
Begin processing the 69th record. run: 1111 subRun: 1 event: 48969 at 15-Jan-2026 21:42:48 CET
Begin processing the 70th record. run: 1111 subRun: 1 event: 48970 at 15-Jan-2026 21:42:55 CET
Begin processing the 71st record. run: 1111 subRun: 1 event: 48971 at 15-Jan-2026 21:43:00 CET
Begin processing the 72nd record. run: 1111 subRun: 1 event: 48972 at 15-Jan-2026 21:43:06 CET
Begin processing the 73rd record. run: 1111 subRun: 1 event: 48973 at 15-Jan-2026 21:43:11 CET
Begin processing the 74th record. run: 1111 subRun: 1 event: 48974 at 15-Jan-2026 21:43:19 CET
Begin processing the 75th record. run: 1111 subRun: 1 event: 48975 at 15-Jan-2026 21:43:23 CET
Begin processing the 76th record. run: 1111 subRun: 1 event: 48976 at 15-Jan-2026 21:43:28 CET
Begin processing the 77th record. run: 1111 subRun: 1 event: 48977 at 15-Jan-2026 21:43:35 CET
Begin processing the 78th record. run: 1111 subRun: 1 event: 48978 at 15-Jan-2026 21:43:41 CET
Begin processing the 79th record. run: 1111 subRun: 1 event: 48979 at 15-Jan-2026 21:43:47 CET
Begin processing the 80th record. run: 1111 subRun: 1 event: 48980 at 15-Jan-2026 21:43:51 CET
Begin processing the 81st record. run: 1111 subRun: 1 event: 48981 at 15-Jan-2026 21:43:56 CET
Begin processing the 82nd record. run: 1111 subRun: 1 event: 48982 at 15-Jan-2026 21:44:01 CET
Begin processing the 83rd record. run: 1111 subRun: 1 event: 48983 at 15-Jan-2026 21:44:17 CET
Begin processing the 84th record. run: 1111 subRun: 1 event: 48984 at 15-Jan-2026 21:44:21 CET
Begin processing the 85th record. run: 1111 subRun: 1 event: 48985 at 15-Jan-2026 21:44:26 CET
Begin processing the 86th record. run: 1111 subRun: 1 event: 48986 at 15-Jan-2026 21:44:31 CET
Begin processing the 87th record. run: 1111 subRun: 1 event: 48987 at 15-Jan-2026 21:44:35 CET
Begin processing the 88th record. run: 1111 subRun: 1 event: 48988 at 15-Jan-2026 21:44:39 CET
Begin processing the 89th record. run: 1111 subRun: 1 event: 48989 at 15-Jan-2026 21:44:43 CET
Begin processing the 90th record. run: 1111 subRun: 1 event: 48990 at 15-Jan-2026 21:45:14 CET
Begin processing the 91st record. run: 1111 subRun: 1 event: 48991 at 15-Jan-2026 21:45:19 CET
Begin processing the 92nd record. run: 1111 subRun: 1 event: 48992 at 15-Jan-2026 21:45:23 CET
Begin processing the 93rd record. run: 1111 subRun: 1 event: 48993 at 15-Jan-2026 21:45:28 CET
Begin processing the 94th record. run: 1111 subRun: 1 event: 48994 at 15-Jan-2026 21:45:32 CET
Begin processing the 95th record. run: 1111 subRun: 1 event: 48995 at 15-Jan-2026 21:45:39 CET
Begin processing the 96th record. run: 1111 subRun: 1 event: 48996 at 15-Jan-2026 21:45:46 CET
Begin processing the 97th record. run: 1111 subRun: 1 event: 48997 at 15-Jan-2026 21:45:52 CET
Begin processing the 98th record. run: 1111 subRun: 1 event: 48998 at 15-Jan-2026 21:45:57 CET
Begin processing the 99th record. run: 1111 subRun: 1 event: 48999 at 15-Jan-2026 21:46:03 CET
Begin processing the 100th record. run: 1111 subRun: 1 event: 49000 at 15-Jan-2026 21:46:08 CET
15-Jan-2026 21:46:13 CET  Closed output file "nu_dune10kt_1x2x6_1111_489_20230827T033426Z_gen_g4_detsim_hitreco__20240229T172054Z_reco2_reco2.root"
15-Jan-2026 21:46:13 CET  Closed input file "root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/d9/f5/nu_dune10kt_1x2x6_1111_489_20230827T033426Z_gen_g4_detsim_hitreco__20240229T172054Z_reco2.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                      3.01049       11.9042       489.488       4.47523       49.5108        100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.0554431     0.0975877     0.907642      0.0813961     0.0861608       100    
reco:pandora:StandardPandora                    2.93036       11.7827       489.266       4.35925       49.496         100    
[art]:TriggerResults:TriggerResultInserter    1.3967e-05    2.96139e-05   0.000114288   2.62665e-05   1.5475e-05       100    
end_path:out1:RootOutput                       2.509e-06    3.99945e-06   1.9208e-05     3.689e-06    2.11185e-06      100    
end_path:out1:RootOutput(write)               0.00146544     0.0236779     0.181254      0.0147074     0.0310634       100    
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 2408.99 MB
  Peak resident set size usage (VmHWM): 1418.29 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===