Jobsub ID 253022.1@dunegpschedd02.fnal.gov

Jobsub ID	253022.1@dunegpschedd02.fnal.gov
Workflow ID	10784
Stage ID	1
User name	imawby@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	1048576000 (1000 MiB)
	Wall seconds limit	7200 (2 hours)
Submitted time	2025-11-26 12:04:26
Site	US_UChicago
Entry	Engage_US_MWT2_iut2_gk02_condce_mcore
Last heartbeat	2025-11-26 12:21:04
From worker node	Hostname	iut2-c300.iu.edu
	cpuinfo	Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	1048576000 (1000 MiB)
	Wall seconds limit	86400 (24 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2025-11-26 12:04:59
Input files	fardet-hd:nu_dune10kt_1x2x6_1100_466_20230826T034405Z_gen_g4_detsim_hitreco__20240222T223837Z_reco2.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-11-26 12:21:04
Saved logs	justin-logs:253022.1-dunegpschedd02.fnal.gov.logs.tgz
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

tart last 100 lines of lar log file ===
Loaded the TorchScript model '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/94e4c2c7b24493105f6d9d9fe3544c7f8747e523/PandoraNet_Hierarchy_DUNEFD_HD_TT_Edge_v014_15_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/94e4c2c7b24493105f6d9d9fe3544c7f8747e523/PandoraNet_Hierarchy_DUNEFD_HD_TT_Class_v014_15_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/94e4c2c7b24493105f6d9d9fe3544c7f8747e523/PandoraNet_Hierarchy_DUNEFD_HD_TS_Edge_v014_15_00.pt'
Loaded the TorchScript model '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/94e4c2c7b24493105f6d9d9fe3544c7f8747e523/PandoraNet_Hierarchy_DUNEFD_HD_TS_Class_v014_15_00.pt'
PandoraMonitoring, only able to use default TApplication (limited functionality).
26-Nov-2025 07:06:18 EST  Opened output file with pattern "%ifb_reco2.root"
Begin processing the 2nd record. run: 1100 subRun: 1 event: 46602 at 26-Nov-2025 07:06:57 EST
Begin processing the 3rd record. run: 1100 subRun: 1 event: 46603 at 26-Nov-2025 07:07:03 EST
Begin processing the 4th record. run: 1100 subRun: 1 event: 46604 at 26-Nov-2025 07:07:11 EST
Begin processing the 5th record. run: 1100 subRun: 1 event: 46605 at 26-Nov-2025 07:07:14 EST
Begin processing the 6th record. run: 1100 subRun: 1 event: 46606 at 26-Nov-2025 07:07:21 EST
Begin processing the 7th record. run: 1100 subRun: 1 event: 46607 at 26-Nov-2025 07:07:28 EST
Begin processing the 8th record. run: 1100 subRun: 1 event: 46608 at 26-Nov-2025 07:07:32 EST
Begin processing the 9th record. run: 1100 subRun: 1 event: 46609 at 26-Nov-2025 07:07:39 EST
Begin processing the 10th record. run: 1100 subRun: 1 event: 46610 at 26-Nov-2025 07:07:44 EST
Begin processing the 11th record. run: 1100 subRun: 1 event: 46611 at 26-Nov-2025 07:07:47 EST
Begin processing the 12th record. run: 1100 subRun: 1 event: 46612 at 26-Nov-2025 07:07:52 EST
Begin processing the 13th record. run: 1100 subRun: 1 event: 46613 at 26-Nov-2025 07:07:56 EST
Begin processing the 14th record. run: 1100 subRun: 1 event: 46614 at 26-Nov-2025 07:08:00 EST
Begin processing the 15th record. run: 1100 subRun: 1 event: 46615 at 26-Nov-2025 07:08:04 EST
Begin processing the 16th record. run: 1100 subRun: 1 event: 46616 at 26-Nov-2025 07:08:12 EST
Begin processing the 17th record. run: 1100 subRun: 1 event: 46617 at 26-Nov-2025 07:08:18 EST
Begin processing the 18th record. run: 1100 subRun: 1 event: 46618 at 26-Nov-2025 07:08:26 EST
Begin processing the 19th record. run: 1100 subRun: 1 event: 46619 at 26-Nov-2025 07:08:29 EST
Begin processing the 20th record. run: 1100 subRun: 1 event: 46620 at 26-Nov-2025 07:08:34 EST
Begin processing the 21st record. run: 1100 subRun: 1 event: 46621 at 26-Nov-2025 07:08:38 EST
Begin processing the 22nd record. run: 1100 subRun: 1 event: 46622 at 26-Nov-2025 07:08:42 EST
Begin processing the 23rd record. run: 1100 subRun: 1 event: 46623 at 26-Nov-2025 07:08:49 EST
Begin processing the 24th record. run: 1100 subRun: 1 event: 46624 at 26-Nov-2025 07:08:53 EST
Begin processing the 25th record. run: 1100 subRun: 1 event: 46625 at 26-Nov-2025 07:11:15 EST
Begin processing the 26th record. run: 1100 subRun: 1 event: 46626 at 26-Nov-2025 07:11:20 EST
Begin processing the 27th record. run: 1100 subRun: 1 event: 46627 at 26-Nov-2025 07:11:32 EST
Begin processing the 28th record. run: 1100 subRun: 1 event: 46628 at 26-Nov-2025 07:11:37 EST
Begin processing the 29th record. run: 1100 subRun: 1 event: 46629 at 26-Nov-2025 07:11:42 EST
Begin processing the 30th record. run: 1100 subRun: 1 event: 46630 at 26-Nov-2025 07:11:51 EST
Begin processing the 31st record. run: 1100 subRun: 1 event: 46631 at 26-Nov-2025 07:11:55 EST
Begin processing the 32nd record. run: 1100 subRun: 1 event: 46632 at 26-Nov-2025 07:11:59 EST
Begin processing the 33rd record. run: 1100 subRun: 1 event: 46633 at 26-Nov-2025 07:12:03 EST
Begin processing the 34th record. run: 1100 subRun: 1 event: 46634 at 26-Nov-2025 07:12:07 EST
Begin processing the 35th record. run: 1100 subRun: 1 event: 46635 at 26-Nov-2025 07:12:10 EST
Begin processing the 36th record. run: 1100 subRun: 1 event: 46636 at 26-Nov-2025 07:12:15 EST
Begin processing the 37th record. run: 1100 subRun: 1 event: 46637 at 26-Nov-2025 07:12:20 EST
Begin processing the 38th record. run: 1100 subRun: 1 event: 46638 at 26-Nov-2025 07:12:25 EST
Begin processing the 39th record. run: 1100 subRun: 1 event: 46639 at 26-Nov-2025 07:12:29 EST
Begin processing the 40th record. run: 1100 subRun: 1 event: 46640 at 26-Nov-2025 07:12:35 EST
Begin processing the 41st record. run: 1100 subRun: 1 event: 46641 at 26-Nov-2025 07:12:53 EST
Begin processing the 42nd record. run: 1100 subRun: 1 event: 46642 at 26-Nov-2025 07:13:41 EST
Begin processing the 43rd record. run: 1100 subRun: 1 event: 46643 at 26-Nov-2025 07:13:51 EST
Begin processing the 44th record. run: 1100 subRun: 1 event: 46644 at 26-Nov-2025 07:14:39 EST
Begin processing the 45th record. run: 1100 subRun: 1 event: 46645 at 26-Nov-2025 07:14:46 EST
Begin processing the 46th record. run: 1100 subRun: 1 event: 46646 at 26-Nov-2025 07:14:51 EST
Begin processing the 47th record. run: 1100 subRun: 1 event: 46647 at 26-Nov-2025 07:15:04 EST
Begin processing the 48th record. run: 1100 subRun: 1 event: 46648 at 26-Nov-2025 07:15:09 EST
Begin processing the 49th record. run: 1100 subRun: 1 event: 46649 at 26-Nov-2025 07:15:24 EST
Begin processing the 50th record. run: 1100 subRun: 1 event: 46650 at 26-Nov-2025 07:15:36 EST
Begin processing the 51st record. run: 1100 subRun: 1 event: 46651 at 26-Nov-2025 07:15:40 EST
Begin processing the 52nd record. run: 1100 subRun: 1 event: 46652 at 26-Nov-2025 07:15:44 EST
Begin processing the 53rd record. run: 1100 subRun: 1 event: 46653 at 26-Nov-2025 07:15:55 EST
Begin processing the 54th record. run: 1100 subRun: 1 event: 46654 at 26-Nov-2025 07:15:58 EST
Begin processing the 55th record. run: 1100 subRun: 1 event: 46655 at 26-Nov-2025 07:16:09 EST
Begin processing the 56th record. run: 1100 subRun: 1 event: 46656 at 26-Nov-2025 07:16:13 EST
Begin processing the 57th record. run: 1100 subRun: 1 event: 46657 at 26-Nov-2025 07:16:22 EST
Begin processing the 58th record. run: 1100 subRun: 1 event: 46658 at 26-Nov-2025 07:16:28 EST
Begin processing the 59th record. run: 1100 subRun: 1 event: 46659 at 26-Nov-2025 07:16:35 EST
Begin processing the 60th record. run: 1100 subRun: 1 event: 46660 at 26-Nov-2025 07:16:41 EST
Begin processing the 61st record. run: 1100 subRun: 1 event: 46661 at 26-Nov-2025 07:16:45 EST
Begin processing the 62nd record. run: 1100 subRun: 1 event: 46662 at 26-Nov-2025 07:16:50 EST
Begin processing the 63rd record. run: 1100 subRun: 1 event: 46663 at 26-Nov-2025 07:16:53 EST
Begin processing the 64th record. run: 1100 subRun: 1 event: 46664 at 26-Nov-2025 07:16:59 EST
Begin processing the 65th record. run: 1100 subRun: 1 event: 46665 at 26-Nov-2025 07:17:13 EST
Begin processing the 66th record. run: 1100 subRun: 1 event: 46666 at 26-Nov-2025 07:17:20 EST
Begin processing the 67th record. run: 1100 subRun: 1 event: 46667 at 26-Nov-2025 07:17:23 EST
Begin processing the 68th record. run: 1100 subRun: 1 event: 46668 at 26-Nov-2025 07:17:27 EST
Begin processing the 69th record. run: 1100 subRun: 1 event: 46669 at 26-Nov-2025 07:17:31 EST
Begin processing the 70th record. run: 1100 subRun: 1 event: 46670 at 26-Nov-2025 07:17:36 EST
Begin processing the 71st record. run: 1100 subRun: 1 event: 46671 at 26-Nov-2025 07:17:43 EST
Begin processing the 72nd record. run: 1100 subRun: 1 event: 46672 at 26-Nov-2025 07:17:51 EST
Begin processing the 73rd record. run: 1100 subRun: 1 event: 46673 at 26-Nov-2025 07:17:59 EST
Begin processing the 74th record. run: 1100 subRun: 1 event: 46674 at 26-Nov-2025 07:18:04 EST
Begin processing the 75th record. run: 1100 subRun: 1 event: 46675 at 26-Nov-2025 07:18:08 EST
Begin processing the 76th record. run: 1100 subRun: 1 event: 46676 at 26-Nov-2025 07:18:12 EST
Begin processing the 77th record. run: 1100 subRun: 1 event: 46677 at 26-Nov-2025 07:18:16 EST
Begin processing the 78th record. run: 1100 subRun: 1 event: 46678 at 26-Nov-2025 07:18:20 EST
Begin processing the 79th record. run: 1100 subRun: 1 event: 46679 at 26-Nov-2025 07:18:23 EST
Begin processing the 80th record. run: 1100 subRun: 1 event: 46680 at 26-Nov-2025 07:18:39 EST
Begin processing the 81st record. run: 1100 subRun: 1 event: 46681 at 26-Nov-2025 07:18:43 EST
Begin processing the 82nd record. run: 1100 subRun: 1 event: 46682 at 26-Nov-2025 07:18:47 EST
Begin processing the 83rd record. run: 1100 subRun: 1 event: 46683 at 26-Nov-2025 07:18:54 EST
Begin processing the 84th record. run: 1100 subRun: 1 event: 46684 at 26-Nov-2025 07:19:10 EST
Begin processing the 85th record. run: 1100 subRun: 1 event: 46685 at 26-Nov-2025 07:19:17 EST
Begin processing the 86th record. run: 1100 subRun: 1 event: 46686 at 26-Nov-2025 07:19:24 EST
Begin processing the 87th record. run: 1100 subRun: 1 event: 46687 at 26-Nov-2025 07:19:38 EST
Begin processing the 88th record. run: 1100 subRun: 1 event: 46688 at 26-Nov-2025 07:19:44 EST
Begin processing the 89th record. run: 1100 subRun: 1 event: 46689 at 26-Nov-2025 07:19:47 EST
Begin processing the 90th record. run: 1100 subRun: 1 event: 46690 at 26-Nov-2025 07:20:02 EST
Begin processing the 91st record. run: 1100 subRun: 1 event: 46691 at 26-Nov-2025 07:20:05 EST
Begin processing the 92nd record. run: 1100 subRun: 1 event: 46692 at 26-Nov-2025 07:20:09 EST
Begin processing the 93rd record. run: 1100 subRun: 1 event: 46693 at 26-Nov-2025 07:20:27 EST
Begin processing the 94th record. run: 1100 subRun: 1 event: 46694 at 26-Nov-2025 07:20:35 EST
Begin processing the 95th record. run: 1100 subRun: 1 event: 46695 at 26-Nov-2025 07:20:38 EST
=== End last 100 lines of lar log file ===
mv: cannot stat 'ClusterValidation_WithAlg.root': No such file or directory
mv: cannot stat 'ClusterValidation_WithoutAlg.root': No such file or directory