justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 237977.1@dunegpschedd01.fnal.gov

Jobsub ID237977.1@dunegpschedd01.fnal.gov
Workflow ID9297
Stage ID1
User nameschellma@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-10-23 19:02:02
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce512
Last heartbeat2025-10-23 19:04:08
From worker nodeHostnameb9p29p2814.cern.ch
cpuinfoAMD EPYC 7302 16-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-10-23 19:03:18
Input filesmonte-carlo-009297-000001
JobscriptExit code0
Real time0m (30s)
CPU time0m (19s = 63%)
Max RSS bytes81387520 (77 MiB)
Outputting started2025-10-23 19:03:49
Output files
Finished2025-10-23 19:04:08
Saved logsjustin-logs:237977.1-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

Config file: pass1_CERN_000001.json
Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org
Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/
{
  "name": "loose_consistency_fd-vd_fd_mc_2023a_reco2_reco2_dunevd10kt_nu_1x8x6_3view_30deg_geov3.fcl_reco2_merged_20251023T182705.root",
  "namespace": "usertests",
  "metadata": {
    "core.first_event_number": 46201,
    "core.last_event_number": 65800,
    "core.event_count": 400,
    "core.runs": [
      952,
      1275,
      958,
      1239
    ],
    "core.runs_subruns": [
      123900001,
      95800001,
      127500001,
      95200001
    ],
    "dune.workflow": {
      "user": "benjamin@fnal.gov",
      "stage_id": 1,
      "os_release": "Scientific Linux release 7.9 (Nitrogen)"
    },
    "retention.status": "active",
    "core.group": "dune",
    "core.run_type": "fardet-vd",
    "dune.campaign": "fd_mc_2023a_reco2",
    "core.data_tier": "full-reconstructed",
    "core.file_type": "mc",
    "dune.requestid": "ritm1780305",
    "art.process_name": "Reco2",
    "core.application": "art.reco2",
    "core.data_stream": "out1",
    "core.file_format": "artroot",
    "dune.config_file": "reco2_dunevd10kt_nu_1x8x6_3view_30deg_geov3.fcl",
    "dune.output_status": "confirmed",
    "art.file_format_era": "ART_2011a",
    "core.application.name": "reco2",
    "dune_mc.detector_type": "fardet-vd",
    "art.file_format_version": 15,
    "core.application.family": "art",
    "core.application.version": "v09_81_00d02",
    "dune_mc.gen_fcl_filename": "prodgenie_nu_dunevd10kt_1x8x6_3view_30deg.fcl",
    "core.file_content_status": "good",
    "retention.class": "unknown",
    "merge.method": "lar",
    "merge.hash": "504b1ac2a373673cd6ea1149d7764460a3234733ace3a8155b982d56525e4e97",
    "merge.cfg": "cafmaker_dunevd10kt_1x8x6_3view_30deg_runreco-nuenergy_geov3.fcl"
  },
  "parents": [
    {
      "fid": "00Cad4lQRreV3yL5",
      "name": "nu_dunevd10kt_1x8x6_3view_30deg_1239_583_20230802T051827Z_gen_g4_detsim_hitreco__20240220T145932Z_reco2.root",
      "namespace": "fardet-vd"
    },
    {
      "fid": "000DPfuMQDKlS8wr",
      "name": "nu_dunevd10kt_1x8x6_3view_30deg_1275_462_20230804T104546Z_gen_g4_detsim_hitreco__20240220T185027Z_reco2.root",
      "namespace": "fardet-vd"
    },
    {
      "fid": "00KPulhuSruQj6fu",
      "name": "nu_dunevd10kt_1x8x6_3view_30deg_952_602_20230803T053532Z_gen_g4_detsim_hitreco__20240219T225437Z_reco2.root",
      "namespace": "fardet-vd"
    },
    {
      "fid": "00K23wChRfKbGcQX",
      "name": "nu_dunevd10kt_1x8x6_3view_30deg_958_657_20230803T215910Z_gen_g4_detsim_hitreco__20240220T213859Z_reco2.root",
      "namespace": "fardet-vd"
    }
  ],
  "inputs": [
    "root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-vd/43/f5/nu_dunevd10kt_1x8x6_3view_30deg_1239_583_20230802T051827Z_gen_g4_detsim_hitreco__20240220T145932Z_reco2.root",
    "root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-vd/c1/d2/nu_dunevd10kt_1x8x6_3view_30deg_1275_462_20230804T104546Z_gen_g4_detsim_hitreco__20240220T185027Z_reco2.root",
    "root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-vd/73/9f/nu_dunevd10kt_1x8x6_3view_30deg_952_602_20230803T053532Z_gen_g4_detsim_hitreco__20240219T225437Z_reco2.root",
    "root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-vd/06/1e/nu_dunevd10kt_1x8x6_3view_30deg_958_657_20230803T215910Z_gen_g4_detsim_hitreco__20240220T213859Z_reco2.root"
  ],
  "streaming": false
}Failed to parse the configuration file 'cafmaker_dunevd10kt_1x8x6_3view_30deg_runreco-nuenergy_geov3.fcl' with exception
---- search_path BEGIN
  Can't find file "cafmaker_dunevd10kt_1x8x6_3view_30deg_runreco-nuenergy_geov3.fcl"
---- search_path END

Art has completed and will exit with status 90.
Making local copy of input files in ./tmp:
  Copying nu_dunevd10kt_1x8x6_3view_30deg_1239_583_20230802T051827Z_gen_g4_detsim_hitreco__20240220T145932Z_reco2.root
  Copying nu_dunevd10kt_1x8x6_3view_30deg_1275_462_20230804T104546Z_gen_g4_detsim_hitreco__20240220T185027Z_reco2.root
  Copying nu_dunevd10kt_1x8x6_3view_30deg_952_602_20230803T053532Z_gen_g4_detsim_hitreco__20240219T225437Z_reco2.root
  Copying nu_dunevd10kt_1x8x6_3view_30deg_958_657_20230803T215910Z_gen_g4_detsim_hitreco__20240220T213859Z_reco2.root
Copied 4 files
Running command:
lar -c cafmaker_dunevd10kt_1x8x6_3view_30deg_runreco-nuenergy_geov3.fcl -o ./loose_consistency_fd-vd_fd_mc_2023a_reco2_reco2_dunevd10kt_nu_1x8x6_3view_30deg_geov3.fcl_reco2_merged_20251023T182705.root ./tmp/nu_dunevd10kt_1x8x6_3view_30deg_1239_583_20230802T051827Z_gen_g4_detsim_hitreco__20240220T145932Z_reco2.root ./tmp/nu_dunevd10kt_1x8x6_3view_30deg_1275_462_20230804T104546Z_gen_g4_detsim_hitreco__20240220T185027Z_reco2.root ./tmp/nu_dunevd10kt_1x8x6_3view_30deg_952_602_20230803T053532Z_gen_g4_detsim_hitreco__20240219T225437Z_reco2.root ./tmp/nu_dunevd10kt_1x8x6_3view_30deg_958_657_20230803T215910Z_gen_g4_detsim_hitreco__20240220T213859Z_reco2.root
Traceback (most recent call last):
  File "/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/d9e47e6938c90437a81c015d8686bf3de3ee5352/do_merge.py", line 159, in <module>
    main()
  File "/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/d9e47e6938c90437a81c015d8686bf3de3ee5352/do_merge.py", line 156, in main
    merge(config, outdir)
  File "/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/d9e47e6938c90437a81c015d8686bf3de3ee5352/do_merge.py", line 127, in merge
    merge_lar(output, inputs, lar_config)
  File "/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/d9e47e6938c90437a81c015d8686bf3de3ee5352/do_merge.py", line 46, in merge_lar
    subprocess.run(cmd, check=True)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_13/Linux64bit+3.10-2.17/lib/python3.9/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['lar', '-c', 'cafmaker_dunevd10kt_1x8x6_3view_30deg_runreco-nuenergy_geov3.fcl', '-o', './loose_consistency_fd-vd_fd_mc_2023a_reco2_reco2_dunevd10kt_nu_1x8x6_3view_30deg_geov3.fcl_reco2_merged_20251023T182705.root', './tmp/nu_dunevd10kt_1x8x6_3view_30deg_1239_583_20230802T051827Z_gen_g4_detsim_hitreco__20240220T145932Z_reco2.root', './tmp/nu_dunevd10kt_1x8x6_3view_30deg_1275_462_20230804T104546Z_gen_g4_detsim_hitreco__20240220T185027Z_reco2.root', './tmp/nu_dunevd10kt_1x8x6_3view_30deg_952_602_20230803T053532Z_gen_g4_detsim_hitreco__20240219T225437Z_reco2.root', './tmp/nu_dunevd10kt_1x8x6_3view_30deg_958_657_20230803T215910Z_gen_g4_detsim_hitreco__20240220T213859Z_reco2.root']' returned non-zero exit status 90.
merged 000001
justIN time: 2025-11-03 18:30:06 UTC       justIN version: 01.05.01