Jobsub ID 268033.0@dunegpschedd02.fnal.gov

Jobsub ID	268033.0@dunegpschedd02.fnal.gov
Workflow ID	11699
Stage ID	1
User name	smanthey@fnal.gov
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2026-01-10 21:52:05
Site	CA_SFU
Entry	DUNE_CA_SFU_lcg-ce3
Last heartbeat	2026-01-10 22:33:23
From worker node	Hostname	fc30653.int.fir.alliancecan.ca
	cpuinfo	AMD EPYC 9655 96-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	84598 (23 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2026-01-10 22:28:40
Input files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250130T213643Z_gen_004765_supernova_g4stage1_g4stage2_detsim_reco_20250729T103117Z_reco2.root
Jobscript	Exit code	127
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2026-01-10 22:33:23
Saved logs	justin-logs:268033.0-dunegpschedd02.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

Input PFN = root://lcg-dunese1.sfu.computecanada.ca:1094//dune/RSE/fardet-vd/0b/4a/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250130T213643Z_gen_004765_supernova_g4stage1_g4stage2_detsim_reco_20250729T103117Z_reco2.root
Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org
Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/
Setting up larsoft
/cvmfs/fifeuser1.opensciencegrid.org/sw/dune/de1cb88ca23141dedcd38419bc7fddead0bcc7e5
../justin-jobscript: line 54: /cvmfs/fifeuser1.opensciencegrid.org/sw/dune/de1cb88ca23141dedcd38419bc7fddead0bcc7e5/larsoft_v10_08_01/localProducts_larsoft_v10_08_01_e26_prof/setup_grid: Input/output error
Error encountered when setting up product: dunesw
ERROR: Product 'dunesw' (with qualifiers 'e26:prof'), has no v10_08_01d00 version (or may not exist)
/tmp/upsOaZ4Ct: line 14: /cvmfs/dune.opensciencegrid.org/products/dune/ups/v6_0_7/Linux64bit+3.10-2.17/bin/setup_fail: Input/output error
../justin-jobscript: line 61: /cvmfs/larsoft.opensciencegrid.org/products/mrb/v6_09_11/bin/mrb: Input/output error
/tmp/upsQLiUIw: line 354: /cvmfs/larsoft.opensciencegrid.org/products/mrb/v6_09_11/libexec/mrbSetEnv: Input/output error
/tmp/upsQLiUIw: line 356: /cvmfs/larsoft.opensciencegrid.org/products/mrb/v6_09_11/libexec/setup_local_products: Input/output error

ERROR: MRB_INSTALL is not defined - please source localProducts_XXX/setup

/libXrdPosixPreload.so
lar exit code 127
=== Start last 1000 lines of lar log file ===
../justin-jobscript: line 81: lar: command not found
=== End last 100 lines of lar log file ===