Jobsub ID 21828.11@dunegpschedd02.fnal.gov

Jobsub ID	21828.11@dunegpschedd02.fnal.gov
Workflow ID	646
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-08-19 10:15:48
Site	NL_NIKHEF
Entry	VIRGO_NL_NIKHEF_juk
Last heartbeat	2025-08-19 10:18:56
From worker node	Hostname	wn-sate-036.farm.nikhef.nl
	cpuinfo	AMD EPYC 7551P 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	129600 (36 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-08-19 10:16:26
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6407968_279_20231203T040111Z_gen_g4_detsim_hitreco__20240508T044150Z_reco2.root
Jobscript	Exit code	0
	Real time	2m (130s)
	CPU time	0m (52s = 39%)
	Max RSS bytes	964812800 (920 MiB)
Outputting started	2025-08-19 10:18:37
Output files
Finished	2025-08-19 10:18:56
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org
Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/
Using CAF fcl: cafmaker_atmos_dune10kt_1x2x6_runreco-nuenergy-nuangular_geov5.fcl
Using custom sources from /cvmfs/fifeuser1.opensciencegrid.org/sw/dune/77bde15fa509d0f93131802e0d65821efa1ff0ff

MRB_PROJECT=larsoft
MRB_PROJECT_VERSION=
MRB_QUALS=e26:prof
MRB_TOP=/cvmfs/fifeuser1.opensciencegrid.org/sw/dune/77bde15fa509d0f93131802e0d65821efa1ff0ff/cafmaker_tests
MRB_SOURCE=/cvmfs/fifeuser1.opensciencegrid.org/sw/dune/77bde15fa509d0f93131802e0d65821efa1ff0ff/cafmaker_tests/srcs
MRB_BUILDDIR=/cvmfs/fifeuser1.opensciencegrid.org/sw/dune/77bde15fa509d0f93131802e0d65821efa1ff0ff/cafmaker_tests/build_slf7.x86_64
MRB_INSTALL=/cvmfs/fifeuser1.opensciencegrid.org/sw/dune/77bde15fa509d0f93131802e0d65821efa1ff0ff/cafmaker_tests/localProducts_larsoft__e26_prof

PRODUCTS=/cvmfs/fifeuser1.opensciencegrid.org/sw/dune/77bde15fa509d0f93131802e0d65821efa1ff0ff/cafmaker_tests/localProducts_larsoft__e26_prof:/cvmfs/dune.opensciencegrid.org/products/dune:/cvmfs/larsoft.opensciencegrid.org/products:/cvmfs/larsoft.opensciencegrid.org/packages:/cvmfs/fermilab.opensciencegrid.org/products/common/db/
CETPKG_INSTALL=/cvmfs/fifeuser1.opensciencegrid.org/sw/dune/77bde15fa509d0f93131802e0d65821efa1ff0ff/cafmaker_tests/localProducts_larsoft__e26_prof

local product directory is /cvmfs/fifeuser1.opensciencegrid.org/sw/dune/77bde15fa509d0f93131802e0d65821efa1ff0ff/cafmaker_tests/localProducts_larsoft__e26_prof
----------- this block should be empty ------------------
---------------------------------------------------------
Justin processors: 1
Will use justin-get-file
No files available from this stage
Could not get file
No files available from this stage
Could not get file
No files available from this stage
Could not get file
No files available from this stage
Could not get file
No files available from this stage
Could not get file
No files available from this stage
Could not get file
No files available from this stage
Could not get file
No files available from this stage
Could not get file
No files available from this stage
Could not get file
now run lar on these files
root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/b5/29/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6407968_279_20231203T040111Z_gen_g4_detsim_hitreco__20240508T044150Z_reco2.root
fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6407968_279_20231203T040111Z_gen_g4_detsim_hitreco__20240508T044150Z_reco2.root
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/fifeuser1.opensciencegrid.org/sw/dune/77bde15fa509d0f93131802e0d65821efa1ff0ff/cafmaker_tests/localProducts_larsoft__e26_prof/dunecore/v10_08_02d00/gdml/dune10kt_v6_refactored_1x2x6.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 1149
Info in <TGeoManager::CloseGeometry>: 67545 nodes/ 5439 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
lar exit code 0
=== Start last 100 lines of lar log file ===
tail: cannot open '_reco_20250819T101636Z.log' for reading: No such file or directory
=== End last 100 lines of lar log file ===
processed files
root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/b5/29/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6407968_279_20231203T040111Z_gen_g4_detsim_hitreco__20240508T044150Z_reco2.root
.:
total 5344
-rw-r--r--. 1 dune012 dune 1788917 Aug 19 12:18 flatcaf.root
-rw-r--r--. 1 dune012 dune 1350670 Aug 19 12:18 caf.root
-rw-r--r--. 1 dune012 dune 1350670 Aug 19 12:18 caf_fd_hd_atmo_646_20250819T101636Z.root
-rw-r--r--. 1 dune012 dune  931832 Aug 19 12:18 caf_20250819T101636Z.log
-rw-r--r--. 1 dune012 dune    4284 Aug 19 12:18 jobscript.log
-rw-r--r--. 1 dune012 dune     200 Aug 19 12:18 caf_20250819T101636Z.file
-rw-r--r--. 1 dune012 dune     200 Aug 19 12:18 caf_20250819T101636Z.pfns
-rw-r--r--. 1 dune012 dune     200 Aug 19 12:16 file.list
-rw-r--r--. 1 dune012 dune     200 Aug 19 12:18 justin-processed-pfns.txt
-rw-r--r--. 1 dune012 dune     137 Aug 19 12:16 all-input-dids.txt
-rw-r--r--. 1 dune012 dune     137 Aug 19 12:18 caf_20250819T101636Z.did
-rw-r--r--. 1 dune012 dune     137 Aug 19 12:16 did.list