Jobsub ID 15213.74@dunegpschedd02.fnal.gov

Jobsub ID	15213.74@dunegpschedd02.fnal.gov
Workflow ID	652
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-08-07 17:29:59
Site	US_UChicago
Entry	Engage_US_MWT2_uiuc_condce_mcore
Last heartbeat	2025-08-07 19:59:40
From worker node	Hostname	uct2-c625.mwt2.org
	cpuinfo	AMD EPYC 7302 16-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	86400 (24 hours)
	GPU
	Inner Apptainer?	True
Job state	finished
Started	2025-08-07 17:30:57
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50609765_682_20231207T075005Z_gen_g4_detsim_hitreco__20240510T023944Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6492237_346_20231208T054255Z_gen_g4_detsim_hitreco__20240510T045809Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50601160_712_20231205T031510Z_gen_g4_detsim_hitreco__20240510T020247Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66029763_89_20231207T030104Z_gen_g4_detsim_hitreco__20240510T025846Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74518431_373_20231204T212602Z_gen_g4_detsim_hitreco__20240510T030532Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472619_198_20231208T061723Z_gen_g4_detsim_hitreco__20240510T043410Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50601160_281_20231205T004844Z_gen_g4_detsim_hitreco__20240509T222436Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6374820_84_20231201T043454Z_gen_g4_detsim_hitreco__20240507T203726Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50557588_251_20231202T100322Z_gen_g4_detsim_hitreco__20240508T070543Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6491894_639_20231208T014623Z_gen_g4_detsim_hitreco__20240510T031452Z_reco2.root
Jobscript	Exit code	0
	Real time	2h (8905s)
	CPU time	6m (394s = 4%)
	Max RSS bytes	1161728000 (1107 MiB)
Outputting started	2025-08-07 19:59:22
Output files	https://fndcadoor.fnal.gov:2880/dune/scratch/users/pgranger/fnal/00652/1/001/caf_fd_hd_atmo_652_20250807T173108Z.root
Finished	2025-08-07 19:59:40
Saved logs	justin-logs:15213.74-dunegpschedd02.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org
Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/
Using CAF fcl: cafmaker_atmos_dune10kt_1x2x6_runreco-nuenergy-nuangular_geov5.fcl
Using custom sources from /cvmfs/fifeuser1.opensciencegrid.org/sw/dune/77bde15fa509d0f93131802e0d65821efa1ff0ff

MRB_PROJECT=larsoft
MRB_PROJECT_VERSION=
MRB_QUALS=e26:prof
MRB_TOP=/cvmfs/fifeuser1.opensciencegrid.org/sw/dune/77bde15fa509d0f93131802e0d65821efa1ff0ff/cafmaker_tests
MRB_SOURCE=/cvmfs/fifeuser1.opensciencegrid.org/sw/dune/77bde15fa509d0f93131802e0d65821efa1ff0ff/cafmaker_tests/srcs
MRB_BUILDDIR=/cvmfs/fifeuser1.opensciencegrid.org/sw/dune/77bde15fa509d0f93131802e0d65821efa1ff0ff/cafmaker_tests/build_slf7.x86_64
MRB_INSTALL=/cvmfs/fifeuser1.opensciencegrid.org/sw/dune/77bde15fa509d0f93131802e0d65821efa1ff0ff/cafmaker_tests/localProducts_larsoft__e26_prof

PRODUCTS=/cvmfs/fifeuser1.opensciencegrid.org/sw/dune/77bde15fa509d0f93131802e0d65821efa1ff0ff/cafmaker_tests/localProducts_larsoft__e26_prof:/cvmfs/dune.opensciencegrid.org/products/dune:/cvmfs/larsoft.opensciencegrid.org/products:/cvmfs/larsoft.opensciencegrid.org/packages:/cvmfs/fermilab.opensciencegrid.org/products/common/db/
CETPKG_INSTALL=/cvmfs/fifeuser1.opensciencegrid.org/sw/dune/77bde15fa509d0f93131802e0d65821efa1ff0ff/cafmaker_tests/localProducts_larsoft__e26_prof

local product directory is /cvmfs/fifeuser1.opensciencegrid.org/sw/dune/77bde15fa509d0f93131802e0d65821efa1ff0ff/cafmaker_tests/localProducts_larsoft__e26_prof
----------- this block should be empty ------------------
---------------------------------------------------------
Justin processors: 1
Will use justin-get-file
now run lar on these files
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/9d/78/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50609765_682_20231207T075005Z_gen_g4_detsim_hitreco__20240510T023944Z_reco2.root
root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/db/f3/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6492237_346_20231208T054255Z_gen_g4_detsim_hitreco__20240510T045809Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/f7/c6/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50601160_712_20231205T031510Z_gen_g4_detsim_hitreco__20240510T020247Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/d3/4b/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66029763_89_20231207T030104Z_gen_g4_detsim_hitreco__20240510T025846Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/54/f3/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74518431_373_20231204T212602Z_gen_g4_detsim_hitreco__20240510T030532Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/1b/d5/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472619_198_20231208T061723Z_gen_g4_detsim_hitreco__20240510T043410Z_reco2.root
root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/5b/3b/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50601160_281_20231205T004844Z_gen_g4_detsim_hitreco__20240509T222436Z_reco2.root
root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/12/e6/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6374820_84_20231201T043454Z_gen_g4_detsim_hitreco__20240507T203726Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/25/bc/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50557588_251_20231202T100322Z_gen_g4_detsim_hitreco__20240508T070543Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/04/bd/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6491894_639_20231208T014623Z_gen_g4_detsim_hitreco__20240510T031452Z_reco2.root
fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50609765_682_20231207T075005Z_gen_g4_detsim_hitreco__20240510T023944Z_reco2.root
fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6492237_346_20231208T054255Z_gen_g4_detsim_hitreco__20240510T045809Z_reco2.root
fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50601160_712_20231205T031510Z_gen_g4_detsim_hitreco__20240510T020247Z_reco2.root
fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66029763_89_20231207T030104Z_gen_g4_detsim_hitreco__20240510T025846Z_reco2.root
fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74518431_373_20231204T212602Z_gen_g4_detsim_hitreco__20240510T030532Z_reco2.root
fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472619_198_20231208T061723Z_gen_g4_detsim_hitreco__20240510T043410Z_reco2.root
fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50601160_281_20231205T004844Z_gen_g4_detsim_hitreco__20240509T222436Z_reco2.root
fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6374820_84_20231201T043454Z_gen_g4_detsim_hitreco__20240507T203726Z_reco2.root
fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50557588_251_20231202T100322Z_gen_g4_detsim_hitreco__20240508T070543Z_reco2.root
fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6491894_639_20231208T014623Z_gen_g4_detsim_hitreco__20240510T031452Z_reco2.root
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/fifeuser1.opensciencegrid.org/sw/dune/77bde15fa509d0f93131802e0d65821efa1ff0ff/cafmaker_tests/localProducts_larsoft__e26_prof/dunecore/v10_08_02d00/gdml/dune10kt_v6_refactored_1x2x6.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 1149
Info in <TGeoManager::CloseGeometry>: 67545 nodes/ 5439 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
lar exit code 0
=== Start last 100 lines of lar log file ===
tail: cannot open '_reco_20250807T173108Z.log' for reading: No such file or directory
=== End last 100 lines of lar log file ===
processed files
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/9d/78/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50609765_682_20231207T075005Z_gen_g4_detsim_hitreco__20240510T023944Z_reco2.root
root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/db/f3/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6492237_346_20231208T054255Z_gen_g4_detsim_hitreco__20240510T045809Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/f7/c6/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50601160_712_20231205T031510Z_gen_g4_detsim_hitreco__20240510T020247Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/d3/4b/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66029763_89_20231207T030104Z_gen_g4_detsim_hitreco__20240510T025846Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/54/f3/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74518431_373_20231204T212602Z_gen_g4_detsim_hitreco__20240510T030532Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/1b/d5/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472619_198_20231208T061723Z_gen_g4_detsim_hitreco__20240510T043410Z_reco2.root
root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/5b/3b/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50601160_281_20231205T004844Z_gen_g4_detsim_hitreco__20240509T222436Z_reco2.root
root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/12/e6/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6374820_84_20231201T043454Z_gen_g4_detsim_hitreco__20240507T203726Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/25/bc/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50557588_251_20231202T100322Z_gen_g4_detsim_hitreco__20240508T070543Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/04/bd/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6491894_639_20231208T014623Z_gen_g4_detsim_hitreco__20240510T031452Z_reco2.root
.:
total 52464
-rw-r--r--. 1 dune osgvo 16723183 Aug  7 14:59 flatcaf.root
-rw-r--r--. 1 dune osgvo 13949881 Aug  7 14:59 caf.root
-rw-r--r--. 1 dune osgvo 13949881 Aug  7 14:59 caf_fd_hd_atmo_652_20250807T173108Z.root
-rw-r--r--. 1 dune osgvo  9040374 Aug  7 14:59 caf_20250807T173108Z.log
-rw-r--r--. 1 dune osgvo     8483 Aug  7 14:59 jobscript.log
-rw-r--r--. 1 dune osgvo     1924 Aug  7 14:59 caf_20250807T173108Z.file
-rw-r--r--. 1 dune osgvo     1924 Aug  7 14:59 caf_20250807T173108Z.pfns
-rw-r--r--. 1 dune osgvo     1924 Aug  7 12:31 file.list
-rw-r--r--. 1 dune osgvo     1924 Aug  7 14:59 justin-processed-pfns.txt
-rw-r--r--. 1 dune osgvo     1374 Aug  7 12:31 all-input-dids.txt
-rw-r--r--. 1 dune osgvo     1374 Aug  7 14:59 caf_20250807T173108Z.did
-rw-r--r--. 1 dune osgvo     1374 Aug  7 12:31 did.list