Jobsub ID 302477.0@dunegpschedd01.fnal.gov

Jobsub ID	302477.0@dunegpschedd01.fnal.gov
Workflow ID	12587
Stage ID	1
User name	schellma@fnal.gov
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2026-01-30 17:37:22
Site	US_FNAL-FermiGrid
Entry	FNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat	2026-01-30 17:39:32
From worker node	Hostname	dunegli-8488654-0-fnpc23049.fnal.gov
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	172800 (48 hours)
	GPU
	Inner Apptainer?	True
Job state	none_processed
Started	2026-01-30 17:39:20
Input files	hd-protodune-det-reco:np04hd_raw_run032672_0225_dataflow0_datawriter_0_20241105T200250_reco_stage1_reco_stage2_20241105T231250_keepup.root
Outputting started	2026-01-30 17:39:26
Output files
Finished	2026-01-30 17:39:32
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

/fifeuser2.opensciencegrid.org/sw/dune/f0fd9778f987a22e820118ca8f96cbed59213fdb/test_larsoft/localProducts_larsoft_v09_91_02d01_e26_prof}/' is not a directory
WARNING:  '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/f0fd9778f987a22e820118ca8f96cbed59213fdb/test_larsoft/localProducts_larsoft_v09_91_02d01_e26_prof}/' is not a directory
WARNING:  '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/f0fd9778f987a22e820118ca8f96cbed59213fdb/test_larsoft/localProducts_larsoft_v09_91_02d01_e26_prof}/' is not a directory
WARNING:  '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/f0fd9778f987a22e820118ca8f96cbed59213fdb/test_larsoft/localProducts_larsoft_v09_91_02d01_e26_prof}/' is not a directory
WARNING:  '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/f0fd9778f987a22e820118ca8f96cbed59213fdb/test_larsoft/localProducts_larsoft_v09_91_02d01_e26_prof}/' is not a directory
WARNING:  '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/f0fd9778f987a22e820118ca8f96cbed59213fdb/test_larsoft/localProducts_larsoft_v09_91_02d01_e26_prof}/' is not a directory
WARNING:  '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/f0fd9778f987a22e820118ca8f96cbed59213fdb/test_larsoft/localProducts_larsoft_v09_91_02d01_e26_prof}/' is not a directory
WARNING:  '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/f0fd9778f987a22e820118ca8f96cbed59213fdb/test_larsoft/localProducts_larsoft_v09_91_02d01_e26_prof}/' is not a directory
WARNING:  '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/f0fd9778f987a22e820118ca8f96cbed59213fdb/test_larsoft/localProducts_larsoft_v09_91_02d01_e26_prof}/' is not a directory
WARNING:  '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/f0fd9778f987a22e820118ca8f96cbed59213fdb/test_larsoft/localProducts_larsoft_v09_91_02d01_e26_prof}/' is not a directory
WARNING:  '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/f0fd9778f987a22e820118ca8f96cbed59213fdb/test_larsoft/localProducts_larsoft_v09_91_02d01_e26_prof}/' is not a directory
WARNING:  '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/f0fd9778f987a22e820118ca8f96cbed59213fdb/test_larsoft/localProducts_larsoft_v09_91_02d01_e26_prof}/' is not a directory
WARNING:  '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/f0fd9778f987a22e820118ca8f96cbed59213fdb/test_larsoft/localProducts_larsoft_v09_91_02d01_e26_prof}/' is not a directory
WARNING:  '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/f0fd9778f987a22e820118ca8f96cbed59213fdb/test_larsoft/localProducts_larsoft_v09_91_02d01_e26_prof}/' is not a directory
WARNING:  '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/f0fd9778f987a22e820118ca8f96cbed59213fdb/test_larsoft/localProducts_larsoft_v09_91_02d01_e26_prof}/' is not a directory
WARNING:  '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/f0fd9778f987a22e820118ca8f96cbed59213fdb/test_larsoft/localProducts_larsoft_v09_91_02d01_e26_prof}/' is not a directory
WARNING:  '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/f0fd9778f987a22e820118ca8f96cbed59213fdb/test_larsoft/localProducts_larsoft_v09_91_02d01_e26_prof}/' is not a directory
WARNING:  '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/f0fd9778f987a22e820118ca8f96cbed59213fdb/test_larsoft/localProducts_larsoft_v09_91_02d01_e26_prof}/' is not a directory
WARNING:  '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/f0fd9778f987a22e820118ca8f96cbed59213fdb/test_larsoft/localProducts_larsoft_v09_91_02d01_e26_prof}/' is not a directory
WARNING:  '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/f0fd9778f987a22e820118ca8f96cbed59213fdb/test_larsoft/localProducts_larsoft_v09_91_02d01_e26_prof}/' is not a directory
WARNING:  '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/f0fd9778f987a22e820118ca8f96cbed59213fdb/test_larsoft/localProducts_larsoft_v09_91_02d01_e26_prof}/' is not a directory
WARNING:  '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/f0fd9778f987a22e820118ca8f96cbed59213fdb/test_larsoft/localProducts_larsoft_v09_91_02d01_e26_prof}/' is not a directory
WARNING:  '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/f0fd9778f987a22e820118ca8f96cbed59213fdb/test_larsoft/localProducts_larsoft_v09_91_02d01_e26_prof}/' is not a directory
WARNING:  '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/f0fd9778f987a22e820118ca8f96cbed59213fdb/test_larsoft/localProducts_larsoft_v09_91_02d01_e26_prof}/' is not a directory
WARNING:  '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/f0fd9778f987a22e820118ca8f96cbed59213fdb/test_larsoft/localProducts_larsoft_v09_91_02d01_e26_prof}/' is not a directory
WARNING:  '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/f0fd9778f987a22e820118ca8f96cbed59213fdb/test_larsoft/localProducts_larsoft_v09_91_02d01_e26_prof}/' is not a directory
WARNING:  '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/f0fd9778f987a22e820118ca8f96cbed59213fdb/test_larsoft/localProducts_larsoft_v09_91_02d01_e26_prof}/' is not a directory
WARNING:  '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/f0fd9778f987a22e820118ca8f96cbed59213fdb/test_larsoft/localProducts_larsoft_v09_91_02d01_e26_prof}/' is not a directory
WARNING:  '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/f0fd9778f987a22e820118ca8f96cbed59213fdb/test_larsoft/localProducts_larsoft_v09_91_02d01_e26_prof}/' is not a directory
WARNING:  '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/f0fd9778f987a22e820118ca8f96cbed59213fdb/test_larsoft/localProducts_larsoft_v09_91_02d01_e26_prof}/' is not a directory
WARNING:  '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/f0fd9778f987a22e820118ca8f96cbed59213fdb/test_larsoft/localProducts_larsoft_v09_91_02d01_e26_prof}/' is not a directory
WARNING:  '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/f0fd9778f987a22e820118ca8f96cbed59213fdb/test_larsoft/localProducts_larsoft_v09_91_02d01_e26_prof}/' is not a directory
WARNING:  '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/f0fd9778f987a22e820118ca8f96cbed59213fdb/test_larsoft/localProducts_larsoft_v09_91_02d01_e26_prof}/' is not a directory
WARNING:  '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/f0fd9778f987a22e820118ca8f96cbed59213fdb/test_larsoft/localProducts_larsoft_v09_91_02d01_e26_prof}/' is not a directory
WARNING:  '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/f0fd9778f987a22e820118ca8f96cbed59213fdb/test_larsoft/localProducts_larsoft_v09_91_02d01_e26_prof}/' is not a directory
WARNING:  '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/f0fd9778f987a22e820118ca8f96cbed59213fdb/test_larsoft/localProducts_larsoft_v09_91_02d01_e26_prof}/' is not a directory
MRB_INSTALL is /cvmfs/fifeuser2.opensciencegrid.org/sw/dune/f0fd9778f987a22e820118ca8f96cbed59213fdb/test_larsoft/localProducts_larsoft_v09_91_02d01_e26_prof

MRB_PROJECT=larsoft
MRB_PROJECT_VERSION=v09_91_02d01
MRB_QUALS=e26:prof
MRB_TOP=test_larsoft
MRB_SOURCE=test_larsoft/srcs
MRB_BUILDDIR=test_larsoft/build_slf7.x86_64
MRB_INSTALL=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/f0fd9778f987a22e820118ca8f96cbed59213fdb/test_larsoft/localProducts_larsoft_v09_91_02d01_e26_prof

PRODUCTS=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/f0fd9778f987a22e820118ca8f96cbed59213fdb/test_larsoft/localProducts_larsoft_v09_91_02d01_e26_prof:/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/f0fd9778f987a22e820118ca8f96cbed59213fdb/test_larsoft/localProducts_larsoft_v09_91_02d01_e26_prof}/:/cvmfs/dune.opensciencegrid.org/products/dune:/cvmfs/larsoft.opensciencegrid.org/products:/cvmfs/larsoft.opensciencegrid.org/packages:/cvmfs/fermilab.opensciencegrid.org/products/common/db/
CETPKG_INSTALL=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/f0fd9778f987a22e820118ca8f96cbed59213fdb/test_larsoft/localProducts_larsoft_v09_91_02d01_e26_prof
 got here
local product directory is /cvmfs/fifeuser2.opensciencegrid.org/sw/dune/f0fd9778f987a22e820118ca8f96cbed59213fdb/test_larsoft/localProducts_larsoft_v09_91_02d01_e26_prof
----------- this block should be empty ------------------
Error encountered when setting up product: dunesw
WARNING:  '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/f0fd9778f987a22e820118ca8f96cbed59213fdb/test_larsoft/localProducts_larsoft_v09_91_02d01_e26_prof}/' is not a directory
WARNING:  '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/f0fd9778f987a22e820118ca8f96cbed59213fdb/test_larsoft/localProducts_larsoft_v09_91_02d01_e26_prof}/' is not a directory
WARNING:  '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/f0fd9778f987a22e820118ca8f96cbed59213fdb/test_larsoft/localProducts_larsoft_v09_91_02d01_e26_prof}/' is not a directory
ERROR: Version conflict -- dependency tree requires versions conflicting with current setup of product duneana: version v09_91_02d01 vs v09_85_00d00
Error encountered when setting up product: protoduneana
WARNING:  '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/f0fd9778f987a22e820118ca8f96cbed59213fdb/test_larsoft/localProducts_larsoft_v09_91_02d01_e26_prof}/' is not a directory
WARNING:  '/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/f0fd9778f987a22e820118ca8f96cbed59213fdb/test_larsoft/localProducts_larsoft_v09_91_02d01_e26_prof}/' is not a directory
ERROR: Version conflict -- dependency tree requires versions conflicting with current setup of product duneprototypes: version v09_91_02d01 vs v09_85_00d00
---------------------------------------------------------
/cvmfs/larsoft.opensciencegrid.org/products/xrootd/v5_4_3b/Linux64bit+3.10-2.17-e26-p3913-prof/lib/libXrdPosixPreload.so
../justin-jobscript: line 95: np04hd_raw_run032672_0225_dataflow0_datawriter_0_20241105T200250_reco_stage1_reco_stage2_20241105T231250_keepup_analyze_2026-01-30T_173926Z.log: Read-only file system
=== Start last 1000 lines of lar log file ===
tail: cannot open 'np04hd_raw_run032672_0225_dataflow0_datawriter_0_20241105T200250_reco_stage1_reco_stage2_20241105T231250_keepup_ana_2026-01-30T_173926Z.log' for reading: No such file or directory
=== End last 1000 lines of lar log file ===
lar exit code 0
../justin-jobscript: line 108: justin-processed-pfns.txt: Read-only file system
tar: *.log: Cannot stat: No such file or directory
tar (child): 302477.0_dunegpschedd01.fnal.gov.logs.tgz: Cannot open: Read-only file system
tar (child): Error is not recoverable: exiting now
tar: Child returned status 2
tar: Error is not recoverable: exiting now