justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 237980.1@dunegpschedd01.fnal.gov

Jobsub ID237980.1@dunegpschedd01.fnal.gov
Workflow ID9299
Stage ID1
User nameschellma@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-10-23 20:02:05
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce512
Last heartbeat2025-10-23 20:04:05
From worker nodeHostnameb9p29p2814.cern.ch
cpuinfoAMD EPYC 7302 16-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-10-23 20:03:09
Input filesmonte-carlo-009299-000001
JobscriptExit code0
Real time0m (34s)
CPU time0m (22s = 64%)
Max RSS bytes81272832 (77 MiB)
Outputting started2025-10-23 20:03:44
Output files
Finished2025-10-23 20:04:05
Saved logsjustin-logs:237980.1-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

Config file: pass1_CERN_000001.json
Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org
Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/
{
  "name": "cafmaker_fd-vd_fd_mc_2023a_reco2_reco2_dunevd10kt_nu_1x8x6_3view_30deg_geov3.fcl_reco2_merged_20251023T195855.root",
  "namespace": "usertests",
  "metadata": {
    "core.first_event_number": 46201,
    "core.last_event_number": 65800,
    "core.event_count": 400,
    "core.runs": [
      952,
      1275,
      958,
      1239
    ],
    "core.runs_subruns": [
      123900001,
      95800001,
      127500001,
      95200001
    ],
    "dune.workflow": {
      "user": "benjamin@fnal.gov",
      "stage_id": 1,
      "os_release": "Scientific Linux release 7.9 (Nitrogen)"
    },
    "retention.status": "active",
    "core.group": "dune",
    "core.run_type": "fardet-vd",
    "dune.campaign": "fd_mc_2023a_reco2",
    "core.data_tier": "full-reconstructed",
    "core.file_type": "mc",
    "dune.requestid": "ritm1780305",
    "art.process_name": "Reco2",
    "core.application": "art.reco2",
    "core.data_stream": "out1",
    "core.file_format": "artroot",
    "dune.config_file": "reco2_dunevd10kt_nu_1x8x6_3view_30deg_geov3.fcl",
    "dune.output_status": "confirmed",
    "art.file_format_era": "ART_2011a",
    "core.application.name": "reco2",
    "dune_mc.detector_type": "fardet-vd",
    "art.file_format_version": 15,
    "core.application.family": "art",
    "core.application.version": "v09_81_00d02",
    "dune_mc.gen_fcl_filename": "prodgenie_nu_dunevd10kt_1x8x6_3view_30deg.fcl",
    "core.file_content_status": "good",
    "retention.class": "unknown",
    "merge.method": "lar",
    "merge.hash": "504b1ac2a373673cd6ea1149d7764460a3234733ace3a8155b982d56525e4e97",
    "merge.cfg": "cafmaker_dunevd10kt_1x8x6_3view_30deg_runreco-nuenergy_geov3.fcl"
  },
  "parents": [
    {
      "fid": "00Cad4lQRreV3yL5",
      "name": "nu_dunevd10kt_1x8x6_3view_30deg_1239_583_20230802T051827Z_gen_g4_detsim_hitreco__20240220T145932Z_reco2.root",
      "namespace": "fardet-vd"
    },
    {
      "fid": "000DPfuMQDKlS8wr",
      "name": "nu_dunevd10kt_1x8x6_3view_30deg_1275_462_20230804T104546Z_gen_g4_detsim_hitreco__20240220T185027Z_reco2.root",
      "namespace": "fardet-vd"
    },
    {
      "fid": "00KPulhuSruQj6fu",
      "name": "nu_dunevd10kt_1x8x6_3view_30deg_952_602_20230803T053532Z_gen_g4_detsim_hitreco__20240219T225437Z_reco2.root",
      "namespace": "fardet-vd"
    },
    {
      "fid": "00K23wChRfKbGcQX",
      "name": "nu_dunevd10kt_1x8x6_3view_30deg_958_657_20230803T215910Z_gen_g4_detsim_hitreco__20240220T213859Z_reco2.root",
      "namespace": "fardet-vd"
    }
  ],
  "inputs": [
    "root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-vd/43/f5/nu_dunevd10kt_1x8x6_3view_30deg_1239_583_20230802T051827Z_gen_g4_detsim_hitreco__20240220T145932Z_reco2.root",
    "root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-vd/c1/d2/nu_dunevd10kt_1x8x6_3view_30deg_1275_462_20230804T104546Z_gen_g4_detsim_hitreco__20240220T185027Z_reco2.root",
    "root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-vd/73/9f/nu_dunevd10kt_1x8x6_3view_30deg_952_602_20230803T053532Z_gen_g4_detsim_hitreco__20240219T225437Z_reco2.root",
    "root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-vd/06/1e/nu_dunevd10kt_1x8x6_3view_30deg_958_657_20230803T215910Z_gen_g4_detsim_hitreco__20240220T213859Z_reco2.root"
  ],
  "streaming": false
}fcldump: ERROR: Unable to find file cafmaker_dunevd10kt_1x8x6_3view_30deg_runreco-nuenergy_geov3.fcl
fcldump: Search path:
  .
  ./job
  /cvmfs/dune.opensciencegrid.org/products/dune/dunesw/v09_75_00d00/fcl
  /cvmfs/dune.opensciencegrid.org/products/dune/protoduneana/v09_75_00d00/job
  /cvmfs/larsoft.opensciencegrid.org/products/geant4reweight/v01_18_01/fcl
  /cvmfs/dune.opensciencegrid.org/products/dune/duneprototypes/v09_75_00d00/fcl
  /cvmfs/dune.opensciencegrid.org/products/dune/dunesim/v09_75_00d00/fcl
  /cvmfs/dune.opensciencegrid.org/products/dune/dunecalib/v09_75_00d00/fcl
  /cvmfs/dune.opensciencegrid.org/products/dune/dunedataprep/v09_75_00d00/fcl
  /cvmfs/dune.opensciencegrid.org/products/dune/duneana/v09_75_00d00/fcl
  /cvmfs/dune.opensciencegrid.org/products/dune/dunereco/v09_75_00d00/fcl
  /cvmfs/dune.opensciencegrid.org/products/dune/duneopdet/v09_75_00d00/fcl
  /cvmfs/dune.opensciencegrid.org/products/dune/dunecore/v09_75_00d00/fcl
  /cvmfs/larsoft.opensciencegrid.org/products/nusystematics/v01_03_07/fcl
  /cvmfs/larsoft.opensciencegrid.org/products/systematicstools/v01_03_01/fcl
  .
  ./job
  /cvmfs/larsoft.opensciencegrid.org/products/larwirecell/v09_14_00/fcl
  /cvmfs/larsoft.opensciencegrid.org/products/larsimdnn/v09_05_00/job
  /cvmfs/larsoft.opensciencegrid.org/products/larrecodnn/v09_21_00/job
  /cvmfs/larsoft.opensciencegrid.org/products/larpandora/v09_21_00/job
  /cvmfs/larsoft.opensciencegrid.org/products/larexamples/v09_08_00/job
  /cvmfs/larsoft.opensciencegrid.org/products/lareventdisplay/v09_10_00/job
  /cvmfs/larsoft.opensciencegrid.org/products/nuevdb/v1_08_01/fcl
  /cvmfs/larsoft.opensciencegrid.org/products/larana/v09_14_00/job
  /cvmfs/larsoft.opensciencegrid.org/products/larreco/v09_20_00/job
  /cvmfs/larsoft.opensciencegrid.org/products/larsim/v09_36_00/job
  /cvmfs/larsoft.opensciencegrid.org/products/nugen/v1_19_02/fcl
  /cvmfs/larsoft.opensciencegrid.org/products/larg4/v09_16_00/fcl
  /cvmfs/larsoft.opensciencegrid.org/products/nurandom/v1_10_01/fcl
  /cvmfs/larsoft.opensciencegrid.org/products/nug4/v1_15_01/fcl
  /cvmfs/larsoft.opensciencegrid.org/products/larevt/v09_09_00/job
  /cvmfs/larsoft.opensciencegrid.org/products/lardata/v09_15_00/job
  /cvmfs/larsoft.opensciencegrid.org/products/lardataalg/v09_15_00/job
  /cvmfs/larsoft.opensciencegrid.org/products/larcore/v09_10_00/job
  /cvmfs/larsoft.opensciencegrid.org/products/larcorealg/v09_12_00/job
  /cvmfs/larsoft.opensciencegrid.org/products/artg4tk/v11_00_01/fcl
  /cvmfs/dune.opensciencegrid.org/products/dune/dunepdlegacy/v1_00_03/fcl
Making local copy of input files in ./tmp:
  Copying nu_dunevd10kt_1x8x6_3view_30deg_1239_583_20230802T051827Z_gen_g4_detsim_hitreco__20240220T145932Z_reco2.root
  Copying nu_dunevd10kt_1x8x6_3view_30deg_1275_462_20230804T104546Z_gen_g4_detsim_hitreco__20240220T185027Z_reco2.root
  Copying nu_dunevd10kt_1x8x6_3view_30deg_952_602_20230803T053532Z_gen_g4_detsim_hitreco__20240219T225437Z_reco2.root
  Copying nu_dunevd10kt_1x8x6_3view_30deg_958_657_20230803T215910Z_gen_g4_detsim_hitreco__20240220T213859Z_reco2.root
Copied 4 files
Running fcldump command:
fcldump cafmaker_dunevd10kt_1x8x6_3view_30deg_runreco-nuenergy_geov3.fcl
Traceback (most recent call last):
  File "/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/6a37d53dfc7e355e452a07617aca11544677f353/do_merge.py", line 165, in <module>
    main()
  File "/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/6a37d53dfc7e355e452a07617aca11544677f353/do_merge.py", line 162, in main
    merge(config, outdir)
  File "/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/6a37d53dfc7e355e452a07617aca11544677f353/do_merge.py", line 133, in merge
    merge_lar(output, inputs, lar_config)
  File "/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/6a37d53dfc7e355e452a07617aca11544677f353/do_merge.py", line 46, in merge_lar
    result = subprocess.run(cmd, check=True)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_13/Linux64bit+3.10-2.17/lib/python3.9/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['fcldump', 'cafmaker_dunevd10kt_1x8x6_3view_30deg_runreco-nuenergy_geov3.fcl']' returned non-zero exit status 8.
merged 000001
justIN time: 2025-11-05 04:53:11 UTC       justIN version: 01.05.01