Jobsub ID 257486.1@dunegpschedd02.fnal.gov

Jobsub ID	257486.1@dunegpschedd02.fnal.gov
Workflow ID	10929
Stage ID	1
User name	imawby@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	1048576000 (1000 MiB)
	Wall seconds limit	7200 (2 hours)
Submitted time	2025-12-04 07:33:15
Site	US_Michigan
Entry	HCC_US_Michigan_gate02
Last heartbeat	2025-12-04 07:56:44
From worker node	Hostname	c6-12-29-2.aglt2.org
	cpuinfo	Intel(R) Xeon(R) Gold 6240R CPU @ 2.40GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	1048576000 (1000 MiB)
	Wall seconds limit	36000 (10 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-12-04 07:34:27
Input files	fardet-hd:nu_dune10kt_1x2x6_1114_463_20230827T153337Z_gen_g4_detsim_hitreco__20240223T012654Z_reco2.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-12-04 07:56:44
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

Fetching extract service.
StandardRawDigitPrepService::ctor: Fetching channel status provider.
StandardRawDigitPrepService::ctor:   Channel status provider: @0xc567c50
StandardRawDigitPrepService::ctor:   Extract service: @0xde95e00
StandardRawDigitPrepService::ctor: Fetching deconvolution service.
StandardRawDigitPrepService::ctor:   Deconvolution service: @0xbfcc5b0
StandardRawDigitPrepService::ctor: Fetching ROI building service.
StandardRawDigitPrepService::ctor:   ROI building service: @0x4b93dd0
StandardRawDigitPrepService::ctor: Fetching wire building service.
StandardRawDigitPrepService::ctor:   Wire building service: @0x4b93da0
StandardRawDigitPrepService::ctor: StandardRawDigitPrepService:
StandardRawDigitPrepService::ctor:              LogLevel: 1
StandardRawDigitPrepService::ctor:               SkipBad: 1
StandardRawDigitPrepService::ctor:             SkipNoisy: 0
StandardRawDigitPrepService::ctor:   ChannelStatusOnline: 0
StandardRawDigitPrepService::ctor:          DoMitigation: 0
StandardRawDigitPrepService::ctor:  DoEarlySignalFinding: 0
StandardRawDigitPrepService::ctor:        DoNoiseRemoval: 0
StandardRawDigitPrepService::ctor:       DoDeconvolution: 1
StandardRawDigitPrepService::ctor:  DoPedestalAdjustment: 0
StandardRawDigitPrepService::ctor:                 DoROI: 1
StandardRawDigitPrepService::ctor:               DoWires: 1
StandardRawDigitPrepService::ctor:                DoDump: 0
StandardRawDigitPrepService::ctor:  DoIntermediateStates: 0
StandardRawDigitPrepService::ctor: No display tools.
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
04-Dec-2025 02:36:05 EST  Initiating request to open input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/35/ed/nu_dune10kt_1x2x6_1114_463_20230827T153337Z_gen_g4_detsim_hitreco__20240223T012654Z_reco2.root"
04-Dec-2025 02:36:11 EST  Opened input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/35/ed/nu_dune10kt_1x2x6_1114_463_20230827T153337Z_gen_g4_detsim_hitreco__20240223T012654Z_reco2.root"
Begin processing the 1st record. run: 1114 subRun: 1 event: 46301 at 04-Dec-2025 02:36:12 EST
Loaded the TorchScript model '/cvmfs/fifeuser1.opensciencegrid.org/sw/dune/2eb16b67837199f6df3fa6681d86eb3e0fa401cc/PandoraNet_Vertex_DUNEFD_HD_Accel_1_U_v04_06_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser1.opensciencegrid.org/sw/dune/2eb16b67837199f6df3fa6681d86eb3e0fa401cc/PandoraNet_Vertex_DUNEFD_HD_Accel_1_V_v04_06_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser1.opensciencegrid.org/sw/dune/2eb16b67837199f6df3fa6681d86eb3e0fa401cc/PandoraNet_Vertex_DUNEFD_HD_Accel_1_W_v04_06_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser1.opensciencegrid.org/sw/dune/2eb16b67837199f6df3fa6681d86eb3e0fa401cc/PandoraNet_Vertex_DUNEFD_HD_Accel_2_U_v04_06_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser1.opensciencegrid.org/sw/dune/2eb16b67837199f6df3fa6681d86eb3e0fa401cc/PandoraNet_Vertex_DUNEFD_HD_Accel_2_V_v04_06_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser1.opensciencegrid.org/sw/dune/2eb16b67837199f6df3fa6681d86eb3e0fa401cc/PandoraNet_Vertex_DUNEFD_HD_Accel_2_W_v04_06_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser1.opensciencegrid.org/sw/dune/2eb16b67837199f6df3fa6681d86eb3e0fa401cc/PandoraNet_SecVertex_DUNEFD_HD_Accel_1_U_v04_13_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser1.opensciencegrid.org/sw/dune/2eb16b67837199f6df3fa6681d86eb3e0fa401cc/PandoraNet_SecVertex_DUNEFD_HD_Accel_1_V_v04_13_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser1.opensciencegrid.org/sw/dune/2eb16b67837199f6df3fa6681d86eb3e0fa401cc/PandoraNet_SecVertex_DUNEFD_HD_Accel_1_W_v04_13_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser1.opensciencegrid.org/sw/dune/2eb16b67837199f6df3fa6681d86eb3e0fa401cc/WindowModel_U.pt'
Loaded the TorchScript model '/cvmfs/fifeuser1.opensciencegrid.org/sw/dune/2eb16b67837199f6df3fa6681d86eb3e0fa401cc/SplitPosModel_U.pt'
Loaded the TorchScript model '/cvmfs/fifeuser1.opensciencegrid.org/sw/dune/2eb16b67837199f6df3fa6681d86eb3e0fa401cc/PandoraNet_Hierarchy_DUNEFD_HD_T_Edge_v014_15_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser1.opensciencegrid.org/sw/dune/2eb16b67837199f6df3fa6681d86eb3e0fa401cc/PandoraNet_Hierarchy_DUNEFD_HD_T_Class_v014_15_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser1.opensciencegrid.org/sw/dune/2eb16b67837199f6df3fa6681d86eb3e0fa401cc/PandoraNet_Hierarchy_DUNEFD_HD_S_Class_v014_15_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser1.opensciencegrid.org/sw/dune/2eb16b67837199f6df3fa6681d86eb3e0fa401cc/PandoraNet_Hierarchy_DUNEFD_HD_TT_Edge_v014_15_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser1.opensciencegrid.org/sw/dune/2eb16b67837199f6df3fa6681d86eb3e0fa401cc/PandoraNet_Hierarchy_DUNEFD_HD_TT_Class_v014_15_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser1.opensciencegrid.org/sw/dune/2eb16b67837199f6df3fa6681d86eb3e0fa401cc/PandoraNet_Hierarchy_DUNEFD_HD_TS_Edge_v014_15_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser1.opensciencegrid.org/sw/dune/2eb16b67837199f6df3fa6681d86eb3e0fa401cc/PandoraNet_Hierarchy_DUNEFD_HD_TS_Class_v014_15_00.pt'
PandoraMonitoring, only able to use default TApplication (limited functionality).
04-Dec-2025 02:36:28 EST  Opened output file with pattern "%ifb_reco2.root"
Begin processing the 2nd record. run: 1114 subRun: 1 event: 46302 at 04-Dec-2025 02:37:25 EST
Begin processing the 3rd record. run: 1114 subRun: 1 event: 46303 at 04-Dec-2025 02:37:32 EST
Begin processing the 4th record. run: 1114 subRun: 1 event: 46304 at 04-Dec-2025 02:37:49 EST
Begin processing the 5th record. run: 1114 subRun: 1 event: 46305 at 04-Dec-2025 02:37:57 EST
Begin processing the 6th record. run: 1114 subRun: 1 event: 46306 at 04-Dec-2025 02:38:01 EST
Begin processing the 7th record. run: 1114 subRun: 1 event: 46307 at 04-Dec-2025 02:38:05 EST
Begin processing the 8th record. run: 1114 subRun: 1 event: 46308 at 04-Dec-2025 02:38:11 EST
Begin processing the 9th record. run: 1114 subRun: 1 event: 46309 at 04-Dec-2025 02:38:16 EST
Begin processing the 10th record. run: 1114 subRun: 1 event: 46310 at 04-Dec-2025 02:38:46 EST
Begin processing the 11th record. run: 1114 subRun: 1 event: 46311 at 04-Dec-2025 02:38:55 EST
Begin processing the 12th record. run: 1114 subRun: 1 event: 46312 at 04-Dec-2025 02:38:59 EST
Begin processing the 13th record. run: 1114 subRun: 1 event: 46313 at 04-Dec-2025 02:39:04 EST
Begin processing the 14th record. run: 1114 subRun: 1 event: 46314 at 04-Dec-2025 02:39:09 EST
Begin processing the 15th record. run: 1114 subRun: 1 event: 46315 at 04-Dec-2025 02:39:14 EST
Begin processing the 16th record. run: 1114 subRun: 1 event: 46316 at 04-Dec-2025 02:39:18 EST
Begin processing the 17th record. run: 1114 subRun: 1 event: 46317 at 04-Dec-2025 02:39:21 EST
Begin processing the 18th record. run: 1114 subRun: 1 event: 46318 at 04-Dec-2025 02:39:26 EST
Begin processing the 19th record. run: 1114 subRun: 1 event: 46319 at 04-Dec-2025 02:39:30 EST
Begin processing the 20th record. run: 1114 subRun: 1 event: 46320 at 04-Dec-2025 02:39:45 EST
Begin processing the 21st record. run: 1114 subRun: 1 event: 46321 at 04-Dec-2025 02:40:00 EST
Begin processing the 22nd record. run: 1114 subRun: 1 event: 46322 at 04-Dec-2025 02:40:04 EST
Begin processing the 23rd record. run: 1114 subRun: 1 event: 46323 at 04-Dec-2025 02:40:08 EST
Begin processing the 24th record. run: 1114 subRun: 1 event: 46324 at 04-Dec-2025 02:40:15 EST
Begin processing the 25th record. run: 1114 subRun: 1 event: 46325 at 04-Dec-2025 02:40:19 EST
Begin processing the 26th record. run: 1114 subRun: 1 event: 46326 at 04-Dec-2025 02:40:24 EST
Begin processing the 27th record. run: 1114 subRun: 1 event: 46327 at 04-Dec-2025 02:40:28 EST
Begin processing the 28th record. run: 1114 subRun: 1 event: 46328 at 04-Dec-2025 02:40:32 EST
Begin processing the 29th record. run: 1114 subRun: 1 event: 46329 at 04-Dec-2025 02:40:37 EST
Begin processing the 30th record. run: 1114 subRun: 1 event: 46330 at 04-Dec-2025 02:40:41 EST
Begin processing the 31st record. run: 1114 subRun: 1 event: 46331 at 04-Dec-2025 02:40:46 EST
Begin processing the 32nd record. run: 1114 subRun: 1 event: 46332 at 04-Dec-2025 02:40:50 EST
Begin processing the 33rd record. run: 1114 subRun: 1 event: 46333 at 04-Dec-2025 02:40:55 EST
Begin processing the 34th record. run: 1114 subRun: 1 event: 46334 at 04-Dec-2025 02:40:58 EST
Begin processing the 35th record. run: 1114 subRun: 1 event: 46335 at 04-Dec-2025 02:41:03 EST
Begin processing the 36th record. run: 1114 subRun: 1 event: 46336 at 04-Dec-2025 02:41:08 EST
Begin processing the 37th record. run: 1114 subRun: 1 event: 46337 at 04-Dec-2025 02:41:15 EST
Begin processing the 38th record. run: 1114 subRun: 1 event: 46338 at 04-Dec-2025 02:41:19 EST
Begin processing the 39th record. run: 1114 subRun: 1 event: 46339 at 04-Dec-2025 02:41:25 EST
Begin processing the 40th record. run: 1114 subRun: 1 event: 46340 at 04-Dec-2025 02:41:30 EST
Begin processing the 41st record. run: 1114 subRun: 1 event: 46341 at 04-Dec-2025 02:41:37 EST
Begin processing the 42nd record. run: 1114 subRun: 1 event: 46342 at 04-Dec-2025 02:41:41 EST
Begin processing the 43rd record. run: 1114 subRun: 1 event: 46343 at 04-Dec-2025 02:41:46 EST
Begin processing the 44th record. run: 1114 subRun: 1 event: 46344 at 04-Dec-2025 02:41:51 EST
Begin processing the 45th record. run: 1114 subRun: 1 event: 46345 at 04-Dec-2025 02:41:57 EST
Begin processing the 46th record. run: 1114 subRun: 1 event: 46346 at 04-Dec-2025 02:42:02 EST
=== End last 100 lines of lar log file ===
mv: cannot stat 'ClusterValidation_WithAlg.root': No such file or directory
mv: cannot stat 'ClusterValidation_WithoutAlg.root': No such file or directory