Jobsub ID 301451.0@dunegpschedd01.fnal.gov

Jobsub ID	301451.0@dunegpschedd01.fnal.gov
Workflow ID	12432
Stage ID	1
User name	lwhite86@fnal.gov
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	3600 (1 hours)
Submitted time	2026-01-27 10:09:15
Site	US_FNAL-FermiGrid
Entry	FNAL_GPGrid_ce03_mcore_op_duneonly
Last heartbeat	2026-01-27 10:13:19
From worker node	Hostname	dunegli-8294317-0-fnpc23043.fnal.gov
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	172800 (48 hours)
	GPU
	Inner Apptainer?	True
Job state	finished
Started	2026-01-27 10:10:50
Input files	fardet-hd:nue_dune10kt_1x2x6_1423_450_20230827T133953Z_gen_g4_detsim_hitreco__20240221T003651Z_reco2.root
Jobscript	Exit code	0
	Real time	2m (140s)
	CPU time	2m (125s = 89%)
	Max RSS bytes	7483224064 (7136 MiB)
Outputting started	2026-01-27 10:13:11
Output files
Finished	2026-01-27 10:13:19
Saved logs	justin-logs:301451.0-dunegpschedd01.fnal.gov.logs.tgz
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

10_00_13       -f Linux64bit+3.10-2.17 -q e26:prof        -z /cvmfs/larsoft.opensciencegrid.org/products
lardataalg        v10_01_04       -f Linux64bit+3.10-2.17 -q e26:prof        -z /cvmfs/larsoft.opensciencegrid.org/products
lardataobj        v10_03_01       -f Linux64bit+3.10-2.17 -q e26:prof        -z /cvmfs/larsoft.opensciencegrid.org/products
lareventdisplay   v10_00_29       -f Linux64bit+3.10-2.17 -q e26:prof        -z /cvmfs/larsoft.opensciencegrid.org/products
larevt            v10_00_15       -f Linux64bit+3.10-2.17 -q e26:prof        -z /cvmfs/larsoft.opensciencegrid.org/products
larexamples       v10_00_25       -f Linux64bit+3.10-2.17 -q e26:prof        -z /cvmfs/larsoft.opensciencegrid.org/products
larfinder         v09_00_04       -f NULL                                    -z /cvmfs/larsoft.opensciencegrid.org/products
larg4             v10_01_01       -f Linux64bit+3.10-2.17 -q e26:prof        -z /cvmfs/larsoft.opensciencegrid.org/products
larpandora        v10_01_04       -f Linux64bit+3.10-2.17 -q e26:prof        -z /cvmfs/larsoft.opensciencegrid.org/products
larpandoracontent  v04_17_03       -f Linux64bit+3.10-2.17 -q e26:prof        -z /cvmfs/larsoft.opensciencegrid.org/products
larreco           v10_02_09       -f Linux64bit+3.10-2.17 -q e26:prof        -z /cvmfs/larsoft.opensciencegrid.org/products
larrecodnn        v10_01_24       -f Linux64bit+3.10-2.17 -q e26:prof        -z /cvmfs/larsoft.opensciencegrid.org/products
larsim            v10_08_01       -f Linux64bit+3.10-2.17 -q e26:prof        -z /cvmfs/larsoft.opensciencegrid.org/products
larsimdnn         v10_01_03       -f Linux64bit+3.10-2.17 -q e26:prof        -z /cvmfs/larsoft.opensciencegrid.org/products
larsimrad         v10_00_25       -f Linux64bit+3.10-2.17 -q e26:prof        -z /cvmfs/larsoft.opensciencegrid.org/products
larsoft           v10_12_02       -f Linux64bit+3.10-2.17 -q e26:prof        -z /cvmfs/larsoft.opensciencegrid.org/products
larsoftobj        v10_04_01       -f Linux64bit+3.10-2.17 -q e26:prof        -z /cvmfs/larsoft.opensciencegrid.org/products
larsoft_data      v1_02_02        -f NULL                                    -z /cvmfs/larsoft.opensciencegrid.org/products
larvecutils       v09_04_02       -f Linux64bit+3.10-2.17 -q e26:prof        -z /cvmfs/larsoft.opensciencegrid.org/products
larwirecell       v10_01_24       -f Linux64bit+3.10-2.17 -q e26:prof        -z /cvmfs/larsoft.opensciencegrid.org/products
lhapdf            v6_5_5          -f Linux64bit+3.10-2.17 -q e26:p3915:prof  -z /cvmfs/larsoft.opensciencegrid.org/products
libjpeg_turbo     v2_1_0          -f Linux64bit+3.10-2.17                    -z /cvmfs/larsoft.opensciencegrid.org/products
libpng            v1_6_37         -f Linux64bit+3.10-2.17                    -z /cvmfs/larsoft.opensciencegrid.org/products
libtorch          v2_1_1b         -f Linux64bit+3.10-2.17 -q e26             -z /cvmfs/larsoft.opensciencegrid.org/products
libwda            v2_30_0a        -f Linux64bit+3.10-2.17                    -z /cvmfs/larsoft.opensciencegrid.org/products
libxml2           v2_9_12         -f Linux64bit+3.10-2.17                    -z /cvmfs/larsoft.opensciencegrid.org/products
log4cpp           v1_1_3e         -f Linux64bit+3.10-2.17 -q e26:prof        -z /cvmfs/larsoft.opensciencegrid.org/products
marley            v1_2_1d         -f Linux64bit+3.10-2.17 -q e26:prof        -z /cvmfs/larsoft.opensciencegrid.org/products
messagefacility   v2_10_05        -f Linux64bit+3.10-2.17 -q e26:prof        -z /cvmfs/larsoft.opensciencegrid.org/products
mrb               v6_09_11        -f Linux64bit+3.10-2.17                    -z /cvmfs/larsoft.opensciencegrid.org/products
ninja             v1_11_1         -f Linux64bit+3.10-2.17                    -z /cvmfs/larsoft.opensciencegrid.org/products
nlohmann_json     v3_11_2         -f NULL                                    -z /cvmfs/larsoft.opensciencegrid.org/products
nucondb           v2_6_3          -f Linux64bit+3.10-2.17 -q e26:p3915:prof  -z /cvmfs/larsoft.opensciencegrid.org/products
nuevdb            v1_09_11        -f Linux64bit+3.10-2.17 -q e26:prof        -z /cvmfs/larsoft.opensciencegrid.org/products
nufinder          v1_02_01        -f NULL                                    -z /cvmfs/larsoft.opensciencegrid.org/products
nug4              v1_16_10        -f Linux64bit+3.10-2.17 -q e26:prof        -z /cvmfs/larsoft.opensciencegrid.org/products
nugen             v1_23_00        -f Linux64bit+3.10-2.17 -q e26:prof        -z /cvmfs/larsoft.opensciencegrid.org/products
numpy             v1_24_3         -f Linux64bit+3.10-2.17 -q e26:p3915       -z /cvmfs/larsoft.opensciencegrid.org/products
nurandom          v1_11_05        -f Linux64bit+3.10-2.17 -q e26:prof        -z /cvmfs/larsoft.opensciencegrid.org/products
nusimdata         v1_29_01        -f Linux64bit+3.10-2.17 -q e26:prof        -z /cvmfs/larsoft.opensciencegrid.org/products
nusystematics     v1_05_09        -f Linux64bit+3.10-2.17 -q e26:prof        -z /cvmfs/larsoft.opensciencegrid.org/products
nutools           v3_20_01        -f Linux64bit+3.10-2.17 -q e26:prof        -z /cvmfs/larsoft.opensciencegrid.org/products
openblas          v0_3_23         -f Linux64bit+3.10-2.17 -q e26             -z /cvmfs/larsoft.opensciencegrid.org/products
pandora           v04_16_02       -f Linux64bit+3.10-2.17 -q e26:prof        -z /cvmfs/larsoft.opensciencegrid.org/products
postgresql        v15_3           -f Linux64bit+3.10-2.17 -q p3915           -z /cvmfs/larsoft.opensciencegrid.org/products
ppfx              v02_20_06       -f Linux64bit+3.10-2.17 -q e26:prof        -z /cvmfs/larsoft.opensciencegrid.org/products
protobuf          v3_21_12a       -f Linux64bit+3.10-2.17 -q e26             -z /cvmfs/larsoft.opensciencegrid.org/products
protoduneana      v10_12_02d00    -f Linux64bit+3.10-2.17 -q e26:prof        -z /cvmfs/dune.opensciencegrid.org/products/dune
pygccxml          v2_2_1b         -f NULL                 -q p3915           -z /cvmfs/larsoft.opensciencegrid.org/products
pythia8           v8_3_15         -f Linux64bit+3.10-2.17 -q e26:p3915:prof  -z /cvmfs/larsoft.opensciencegrid.org/products
pythia            v6_4_28x        -f Linux64bit+3.10-2.17 -q e26:prof        -z /cvmfs/larsoft.opensciencegrid.org/products
python            v3_9_15         -f Linux64bit+3.10-2.17                    -z /cvmfs/larsoft.opensciencegrid.org/products
python_future_six_request  v1_3_1          -f Linux64bit+3.10-2.17 -q python3.9       -z /cvmfs/fermilab.opensciencegrid.org/products/common/db/
range             v3_0_12_0       -f NULL                                    -z /cvmfs/larsoft.opensciencegrid.org/products
root              v6_28_12        -f Linux64bit+3.10-2.17 -q e26:p3915:prof  -z /cvmfs/larsoft.opensciencegrid.org/products
rstartree         v2016_07        -f NULL                                    -z /cvmfs/larsoft.opensciencegrid.org/products
sam_web_client    v3_6            -f NULL                                    -z /cvmfs/fermilab.opensciencegrid.org/products/common/db/
scitokens_cpp     v1_0_1a         -f Linux64bit+3.10-2.17 -q e26:sql34001    -z /cvmfs/larsoft.opensciencegrid.org/products
spdlog            v1_9_2          -f Linux64bit+3.10-2.17 -q e26:prof        -z /cvmfs/larsoft.opensciencegrid.org/products
sqlite            v3_40_01_00     -f Linux64bit+3.10-2.17                    -z /cvmfs/larsoft.opensciencegrid.org/products
srproxy           v00.45          -f NULL                 -q py3915          -z /cvmfs/dune.opensciencegrid.org/products/dune
systematicstools  v01_04_04       -f Linux64bit+3.10-2.17 -q e26:prof        -z /cvmfs/larsoft.opensciencegrid.org/products
tbb               v2021_9_0       -f Linux64bit+3.10-2.17 -q e26             -z /cvmfs/larsoft.opensciencegrid.org/products
tensorflow        v2_15_0         -f Linux64bit+3.10-2.17 -q e26:p3915       -z /cvmfs/larsoft.opensciencegrid.org/products
torch_scatter     v2_1_2a         -f Linux64bit+3.10-2.17 -q e26             -z /cvmfs/larsoft.opensciencegrid.org/products
TRACE             v3_17_11        -f NULL                                    -z /cvmfs/larsoft.opensciencegrid.org/products
triton            v2_25_0d        -f Linux64bit+3.10-2.17 -q e26             -z /cvmfs/larsoft.opensciencegrid.org/products
ups               v6_0_7          -f Linux64bit+3.10-2.17                    -z /cvmfs/dune.opensciencegrid.org/products/dune
valgrind          v3_17_0         -f Linux64bit+3.10-2.17                    -z /cvmfs/larsoft.opensciencegrid.org/products
wirecell          v0_32_1         -f Linux64bit+3.10-2.17 -q e26:prof        -z /cvmfs/larsoft.opensciencegrid.org/products
xerces_c          v3_2_3e         -f Linux64bit+3.10-2.17 -q e26             -z /cvmfs/larsoft.opensciencegrid.org/products
xrootd            v5_5_5a         -f Linux64bit+3.10-2.17 -q e26:p3915:prof  -z /cvmfs/larsoft.opensciencegrid.org/products
LArPandoraContent: /cvmfs/larsoft.opensciencegrid.org/products/larpandoracontent/v04_17_03
/cvmfs/larsoft.opensciencegrid.org/products/xrootd/v5_5_5a/Linux64bit+3.10-2.17-e26-p3915-prof/lib/libXrdPosixPreload.so
lar exit code 0
mv: cannot stat 'trackCharacterisationTraining.root': No such file or directory
total 816
-rw-r--r-- 1 dunegli fnalgrid    212 Jan 27 10:10 all-input-dids.txt
-rw-r--r-- 1 dunegli fnalgrid      0 Jan 27 10:11 debugprod.log
-rw-r--r-- 1 dunegli fnalgrid  20948 Jan 27 10:13 jobscript.log
-rw-r--r-- 1 dunegli fnalgrid    185 Jan 27 10:13 justin-processed-pfns.txt
drwxr-xr-x 4 dunegli fnalgrid   4096 Jan 27 10:10 larpandoracontent
-rw-r--r-- 1 dunegli fnalgrid 359176 Jan 27 10:13 nue_dune10kt_1x2x6_1423_450_20230827T133953Z_gen_g4_detsim_hitreco__20240221T003651Z_reco2_reco2.root
-rw-r--r-- 1 dunegli fnalgrid 424832 Jan 27 10:13 nue_dune10kt_1x2x6_1423_450_20230827T133953Z_gen_g4_detsim_hitreco__20240221T003651Z_reco2_training_2026-01-27T_101101Z.log
-rw-r--r-- 1 dunegli fnalgrid    519 Jan 27 10:13 reco2_hist.root