justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 232423.1@dunegpschedd02.fnal.gov

Jobsub ID232423.1@dunegpschedd02.fnal.gov
Workflow ID9301
Stage ID1
User nameschellma@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-10-23 21:30:10
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce512
Last heartbeat2025-10-23 21:32:21
From worker nodeHostnameb9p29p2814.cern.ch
cpuinfoAMD EPYC 7302 16-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-10-23 21:31:33
Input filesmonte-carlo-009301-000001
JobscriptExit code0
Real time0m (29s)
CPU time0m (17s = 58%)
Max RSS bytes81494016 (77 MiB)
Outputting started2025-10-23 21:32:03
Output files
Finished2025-10-23 21:32:21
Saved logsjustin-logs:232423.1-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

fs/larsoft.opensciencegrid.org/products/larsoftobj/v09_31_00/slf7.x86_64.e20.prof/lib
BXDECAY0_DIR=/cvmfs/larsoft.opensciencegrid.org/products/bxdecay0/v1_1_0
PROTOBUF_VERSION=v3_15_8
NUEVDB_DIR=/cvmfs/larsoft.opensciencegrid.org/products/nuevdb/v1_08_01
LARANA_INC=/cvmfs/larsoft.opensciencegrid.org/products/larana/v09_14_00/include
SETUP_RSTARTREE=rstartree v2016_07 -f NULL -z /cvmfs/larsoft.opensciencegrid.org/products
IFDHC_LIB=/cvmfs/larsoft.opensciencegrid.org/products/ifdhc/v2_6_20/Linux64bit+3.10-2.17-e20-p3913-prof/lib
GENIE=/cvmfs/larsoft.opensciencegrid.org/products/genie/v3_04_00b/Linux64bit+3.10-2.17-e20-prof/GENIE-Generator
LHAPDF_VERSION=v6_5_3
LARDATA_INC=/cvmfs/larsoft.opensciencegrid.org/products/lardata/v09_15_00/include
LARDATAALG_INC=/cvmfs/larsoft.opensciencegrid.org/products/lardataalg/v09_15_00/include
SETUP_LARCOREALG=larcorealg v09_12_00 -f Linux64bit+3.10-2.17 -z /cvmfs/larsoft.opensciencegrid.org/products -q e20:prof
SETUP_G4TENDL=g4tendl v1_3_2 -f NULL -z /cvmfs/larsoft.opensciencegrid.org/products
CLHEP_LIB=CLHEP-2.4.5.3
SETUP_POSTGRESQL=postgresql v14_5 -f Linux64bit+3.10-2.17 -z /cvmfs/larsoft.opensciencegrid.org/products -q p3913
SQLITE_FQ=Linux64bit+3.10-2.17
GMP_DIR=/cvmfs/larsoft.opensciencegrid.org/products/gmp/v6_2_1
PRODUCTS=/cvmfs/dune.opensciencegrid.org/products/dune:/cvmfs/larsoft.opensciencegrid.org/products:/cvmfs/larsoft.opensciencegrid.org/packages:/cvmfs/fermilab.opensciencegrid.org/products/common/db/
GENIE_XSEC_TUNE=G18_10a_02_11a
SETUP_DUNECALIB=dunecalib v09_75_00d00 -f Linux64bit+3.10-2.17 -z /cvmfs/dune.opensciencegrid.org/products/dune -q e20:prof
PROTOBUF_LIB=/cvmfs/larsoft.opensciencegrid.org/products/protobuf/v3_15_8/Linux64bit+3.10-2.17-e20/lib
CRYDATAPATH=/cvmfs/larsoft.opensciencegrid.org/products/cry/v1_7p/Linux64bit+3.10-2.17-e20-prof/cry_v1.7/data
LHAPDF_LIB=/cvmfs/larsoft.opensciencegrid.org/products/lhapdf/v6_5_3/Linux64bit+3.10-2.17-e20-p3913-prof/lib
G4INCLDATA=/cvmfs/larsoft.opensciencegrid.org/products/g4incl/v1_0/G4INCL1.0
HEP_CONCURRENCY_FQ_DIR=/cvmfs/larsoft.opensciencegrid.org/products/hep_concurrency/v1_08_00/slf7.x86_64.e20.prof
CETLIB_DIR=/cvmfs/larsoft.opensciencegrid.org/products/cetlib/v3_16_00
ROOT_INC=/cvmfs/larsoft.opensciencegrid.org/products/root/v6_26_06b/Linux64bit+3.10-2.17-e20-p3913-prof/include
HUB_DIR=/cvmfs/larsoft.opensciencegrid.org/products/hub/v2_14_2
GEANT4REWEIGHT_DIR=/cvmfs/larsoft.opensciencegrid.org/products/geant4reweight/v01_18_01
DUNERECO_VERSION=v09_75_00d00
NLOHMANN_JSON_LIB=/cvmfs/dune.opensciencegrid.org/products/dune/nlohmann_json/v3_10_4_1/Linux64bit+3.10-2.17-e20-prof/lib64
LARPANDORA_LIB=/cvmfs/larsoft.opensciencegrid.org/products/larpandora/v09_21_00/slf7.x86_64.e20.prof/lib
IFDH_ART_INC=/cvmfs/larsoft.opensciencegrid.org/products/ifdh_art/v2_15_06/slf7.x86_64.e20.prof/inc
MARLEY_LIB=/cvmfs/larsoft.opensciencegrid.org/products/marley/v1_2_0i/Linux64bit+3.10-2.17-e20-prof/lib
SETUP_LAREVT=larevt v09_09_00 -f Linux64bit+3.10-2.17 -z /cvmfs/larsoft.opensciencegrid.org/products -q e20:prof
HDF5_INC=/cvmfs/larsoft.opensciencegrid.org/products/hdf5/v1_12_2a/Linux64bit+3.10-2.17-e20-prof/include
SETUP_CANVAS=canvas v3_14_00 -f Linux64bit+3.10-2.17 -z /cvmfs/larsoft.opensciencegrid.org/products -q e20:prof
DUNEDETDATAFORMATS_INC=/cvmfs/dune.opensciencegrid.org/products/dune/dunedetdataformats/v3_6_1/include
SETUP_TBB=tbb v2021_7_0 -f Linux64bit+3.10-2.17 -z /cvmfs/larsoft.opensciencegrid.org/products -q e20
PYTHIA_INC=/cvmfs/larsoft.opensciencegrid.org/products/pythia/v6_4_28w/Linux64bit+3.10-2.17-gcc930-prof/include
SETUP_DUNESW=dunesw v09_75_00d00 -f Linux64bit+3.10-2.17 -z /cvmfs/dune.opensciencegrid.org/products/dune -q e20:prof
PERL_MM_OPT=INSTALL_BASE=/cvmfs/larsoft.opensciencegrid.org/products/mrb/v6_09_11/slf7.x86_64/CPAN
GIT_EXEC_PATH=/cvmfs/larsoft.opensciencegrid.org/products/git/v2_45_1/Linux64bit+3.10-2.17/libexec/git-core
PROTODUNEANA_INC=/cvmfs/dune.opensciencegrid.org/products/dune/protoduneana/v09_75_00d00/include
DUNEDATAPREP_LIB=/cvmfs/dune.opensciencegrid.org/products/dune/dunedataprep/v09_75_00d00/slf7.x86_64.e20.prof/lib
DUNERECO_DIR=/cvmfs/dune.opensciencegrid.org/products/dune/dunereco/v09_75_00d00
WIRECELL_LIB=/cvmfs/larsoft.opensciencegrid.org/products/wirecell/v0_24_1b/Linux64bit+3.10-2.17-e20-prof/lib64
TRITON_INC=/cvmfs/larsoft.opensciencegrid.org/products/triton/v2_25_0a/Linux64bit+3.10-2.17-e20/include
LIBJPEG_TURBO_INC=/cvmfs/larsoft.opensciencegrid.org/products/libjpeg_turbo/v2_1_0/Linux64bit+3.10-2.17/include
LAREVENTDISPLAY_LIB=/cvmfs/larsoft.opensciencegrid.org/products/lareventdisplay/v09_10_00/slf7.x86_64.e20.prof/lib
NUG4_VERSION=v1_15_01
SETUP_LARDATAALG=lardataalg v09_15_00 -f Linux64bit+3.10-2.17 -z /cvmfs/larsoft.opensciencegrid.org/products -q e20:prof
LARCOREALG_INC=/cvmfs/larsoft.opensciencegrid.org/products/larcorealg/v09_12_00/include
G4PARTICLEHPDATA=/cvmfs/larsoft.opensciencegrid.org/products/g4tendl/v1_3_2/G4TENDL1.3.2
SETUP_LARANA=larana v09_14_00 -f Linux64bit+3.10-2.17 -z /cvmfs/larsoft.opensciencegrid.org/products -q e20:prof
SETUP_LARSOFT=larsoft v09_75_00 -f Linux64bit+3.10-2.17 -z /cvmfs/larsoft.opensciencegrid.org/products -q e20:prof
SETUP_CETLIB=cetlib v3_16_00 -f Linux64bit+3.10-2.17 -z /cvmfs/larsoft.opensciencegrid.org/products -q e20:prof
SETUP_FHICLCPP=fhiclcpp v4_17_00 -f Linux64bit+3.10-2.17 -z /cvmfs/larsoft.opensciencegrid.org/products -q e20:prof
OPENBLAS_INC=/cvmfs/larsoft.opensciencegrid.org/products/openblas/v0_3_21/Linux64bit+3.10-2.17-e20/include
PYTHON_INCLUDE=/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_13/Linux64bit+3.10-2.17/include/python3.9
_=/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_13/Linux64bit+3.10-2.17/bin/python3
fcldump: ERROR: Unable to find file cafmaker_dunevd10kt_1x8x6_3view_30deg_runreco-nuenergy_geov3.fcl
fcldump: Search path:
  .
  ./job
  /cvmfs/dune.opensciencegrid.org/products/dune/dunesw/v09_75_00d00/fcl
  /cvmfs/dune.opensciencegrid.org/products/dune/protoduneana/v09_75_00d00/job
  /cvmfs/larsoft.opensciencegrid.org/products/geant4reweight/v01_18_01/fcl
  /cvmfs/dune.opensciencegrid.org/products/dune/duneprototypes/v09_75_00d00/fcl
  /cvmfs/dune.opensciencegrid.org/products/dune/dunesim/v09_75_00d00/fcl
  /cvmfs/dune.opensciencegrid.org/products/dune/dunecalib/v09_75_00d00/fcl
  /cvmfs/dune.opensciencegrid.org/products/dune/dunedataprep/v09_75_00d00/fcl
  /cvmfs/dune.opensciencegrid.org/products/dune/duneana/v09_75_00d00/fcl
  /cvmfs/dune.opensciencegrid.org/products/dune/dunereco/v09_75_00d00/fcl
  /cvmfs/dune.opensciencegrid.org/products/dune/duneopdet/v09_75_00d00/fcl
  /cvmfs/dune.opensciencegrid.org/products/dune/dunecore/v09_75_00d00/fcl
  /cvmfs/larsoft.opensciencegrid.org/products/nusystematics/v01_03_07/fcl
  /cvmfs/larsoft.opensciencegrid.org/products/systematicstools/v01_03_01/fcl
  .
  ./job
  /cvmfs/larsoft.opensciencegrid.org/products/larwirecell/v09_14_00/fcl
  /cvmfs/larsoft.opensciencegrid.org/products/larsimdnn/v09_05_00/job
  /cvmfs/larsoft.opensciencegrid.org/products/larrecodnn/v09_21_00/job
  /cvmfs/larsoft.opensciencegrid.org/products/larpandora/v09_21_00/job
  /cvmfs/larsoft.opensciencegrid.org/products/larexamples/v09_08_00/job
  /cvmfs/larsoft.opensciencegrid.org/products/lareventdisplay/v09_10_00/job
  /cvmfs/larsoft.opensciencegrid.org/products/nuevdb/v1_08_01/fcl
  /cvmfs/larsoft.opensciencegrid.org/products/larana/v09_14_00/job
  /cvmfs/larsoft.opensciencegrid.org/products/larreco/v09_20_00/job
  /cvmfs/larsoft.opensciencegrid.org/products/larsim/v09_36_00/job
  /cvmfs/larsoft.opensciencegrid.org/products/nugen/v1_19_02/fcl
  /cvmfs/larsoft.opensciencegrid.org/products/larg4/v09_16_00/fcl
  /cvmfs/larsoft.opensciencegrid.org/products/nurandom/v1_10_01/fcl
  /cvmfs/larsoft.opensciencegrid.org/products/nug4/v1_15_01/fcl
  /cvmfs/larsoft.opensciencegrid.org/products/larevt/v09_09_00/job
  /cvmfs/larsoft.opensciencegrid.org/products/lardata/v09_15_00/job
  /cvmfs/larsoft.opensciencegrid.org/products/lardataalg/v09_15_00/job
  /cvmfs/larsoft.opensciencegrid.org/products/larcore/v09_10_00/job
  /cvmfs/larsoft.opensciencegrid.org/products/larcorealg/v09_12_00/job
  /cvmfs/larsoft.opensciencegrid.org/products/artg4tk/v11_00_01/fcl
  /cvmfs/dune.opensciencegrid.org/products/dune/dunepdlegacy/v1_00_03/fcl
Making local copy of input files in ./tmp:
  Copying nu_dunevd10kt_1x8x6_3view_30deg_1239_583_20230802T051827Z_gen_g4_detsim_hitreco__20240220T145932Z_reco2.root
  Copying nu_dunevd10kt_1x8x6_3view_30deg_1275_462_20230804T104546Z_gen_g4_detsim_hitreco__20240220T185027Z_reco2.root
  Copying nu_dunevd10kt_1x8x6_3view_30deg_952_602_20230803T053532Z_gen_g4_detsim_hitreco__20240219T225437Z_reco2.root
  Copying nu_dunevd10kt_1x8x6_3view_30deg_958_657_20230803T215910Z_gen_g4_detsim_hitreco__20240220T213859Z_reco2.root
Copied 4 files
Running check of environment:
env
Running fcldump command:
fcldump cafmaker_dunevd10kt_1x8x6_3view_30deg_runreco-nuenergy_geov3.fcl
Traceback (most recent call last):
  File "/cvmfs/fifeuser1.opensciencegrid.org/sw/dune/621bbf60e3e69a791b62714bd47aaac164c1b7ac/do_merge.py", line 168, in <module>
    main()
  File "/cvmfs/fifeuser1.opensciencegrid.org/sw/dune/621bbf60e3e69a791b62714bd47aaac164c1b7ac/do_merge.py", line 165, in main
    merge(config, outdir)
  File "/cvmfs/fifeuser1.opensciencegrid.org/sw/dune/621bbf60e3e69a791b62714bd47aaac164c1b7ac/do_merge.py", line 136, in merge
    merge_lar(output, inputs, lar_config)
  File "/cvmfs/fifeuser1.opensciencegrid.org/sw/dune/621bbf60e3e69a791b62714bd47aaac164c1b7ac/do_merge.py", line 49, in merge_lar
    result = subprocess.run(cmd, check=True)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_13/Linux64bit+3.10-2.17/lib/python3.9/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['fcldump', 'cafmaker_dunevd10kt_1x8x6_3view_30deg_runreco-nuenergy_geov3.fcl']' returned non-zero exit status 8.
merged 000001
justIN time: 2025-11-03 18:30:33 UTC       justIN version: 01.05.01