Jobsub ID 20688.1@dunegpschedd01.fnal.gov
| Jobsub ID | 20688.1@dunegpschedd01.fnal.gov |
| Workflow ID | 272 |
| Stage ID | 1 |
| User name | gpaixao@fnal.gov |
| HTCondor Group | group_dune |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 80000 (22 hours) |
| Submitted time | 2025-08-03 19:22:54 |
| Site | UK_RAL-Tier1 |
| Entry | LIGO_UK_RAL_arc_ce01 |
| Last heartbeat | 2025-08-03 19:24:45 |
| From worker node | Hostname | dune001-8431582.0-lcg2445.gridpp.rl.ac.uk |
| cpuinfo | AMD EPYC 7452 32-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 216000 (60 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | jobscript_error |
| Started | 2025-08-03 19:23:44 |
| Input files | monte-carlo-000272-000004
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Max RSS bytes | 0 (0 MiB) |
| Outputting started | |
| Output files | |
| Finished | 2025-08-03 19:24:45 |
| Saved logs | justin-logs:20688.1-dunegpschedd01.fnal.gov.logs.tgz |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
sts/f5/50/H4_v34b_3GeV_-27.7_000257_20250801T200933Z_000257.root
hadd Source file 16: root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/usertests/d3/1b/H4_v34b_3GeV_-27.7_000260_20250801T193926Z_000260.root
hadd Source file 17: root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/usertests/cf/11/H4_v34b_3GeV_-27.7_000284_20250801T201543Z_000284.root
hadd Source file 18: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/usertests/6a/03/H4_v34b_3GeV_-27.7_000285_20250801T182801Z_000285.root
hadd Source file 19: root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/usertests/8b/a4/H4_v34b_3GeV_-27.7_000292_20250801T184826Z_000292.root
hadd Source file 20: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/usertests/34/75/H4_v34b_3GeV_-27.7_000308_20250801T195523Z_000308.root
hadd Source file 21: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/usertests/79/7e/H4_v34b_3GeV_-27.7_000347_20250802T040411Z_000347.root
hadd Source file 22: root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/usertests/d5/a1/H4_v34b_3GeV_-27.7_000361_20250801T173428Z_000361.root
hadd Source file 23: root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/usertests/d3/b9/H4_v34b_3GeV_-27.7_000373_20250801T191417Z_000373.root
hadd Source file 24: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/usertests/09/7b/H4_v34b_3GeV_-27.7_000374_20250801T172717Z_000374.root
hadd Source file 25: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/usertests/e7/b6/H4_v34b_3GeV_-27.7_000375_20250802T051042Z_000375.root
hadd Source file 26: root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/usertests/43/d3/H4_v34b_3GeV_-27.7_000376_20250802T010219Z_000376.root
hadd Source file 27: root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/usertests/2b/ea/H4_v34b_3GeV_-27.7_000381_20250801T195837Z_000381.root
hadd Source file 28: root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/usertests/c5/5b/H4_v34b_3GeV_-27.7_000385_20250801T190306Z_000385.root
hadd Source file 29: root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/usertests/cc/47/H4_v34b_3GeV_-27.7_000392_20250801T194659Z_000392.root
hadd Source file 30: root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/usertests/b9/d4/H4_v34b_3GeV_-27.7_000399_20250801T195912Z_000399.root
hadd Source file 31: root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/usertests/a9/66/H4_v34b_3GeV_-27.7_000404_20250801T191245Z_000404.root
hadd Source file 32: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/usertests/67/ac/H4_v34b_3GeV_-27.7_000413_20250801T175632Z_000413.root
hadd Source file 33: root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/usertests/ac/1e/H4_v34b_3GeV_-27.7_000441_20250802T041059Z_000441.root
hadd Source file 34: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/usertests/82/cb/H4_v34b_3GeV_-27.7_000445_20250801T171110Z_000445.root
hadd Source file 35: root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/usertests/6d/6a/H4_v34b_3GeV_-27.7_000447_20250801T201238Z_000447.root
hadd Source file 36: root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/usertests/e1/3e/H4_v34b_3GeV_-27.7_000448_20250801T200501Z_000448.root
hadd Source file 37: root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/usertests/c4/aa/H4_v34b_3GeV_-27.7_000452_20250801T201857Z_000452.root
hadd Source file 38: root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/usertests/c3/bb/H4_v34b_3GeV_-27.7_000461_20250801T195213Z_000461.root
hadd Source file 39: root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/usertests/5a/d8/H4_v34b_3GeV_-27.7_000475_20250801T173434Z_000475.root
hadd Source file 40: root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/usertests/53/dd/H4_v34b_3GeV_-27.7_000482_20250801T184420Z_000482.root
hadd Source file 41: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/usertests/a9/2d/H4_v34b_3GeV_-27.7_000483_20250802T060828Z_000483.root
hadd Source file 42: root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/usertests/46/b6/H4_v34b_3GeV_-27.7_000503_20250801T182257Z_000503.root
hadd Source file 43: root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/usertests/eb/96/H4_v34b_3GeV_-27.7_000505_20250801T194858Z_000505.root
hadd Source file 44: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/usertests/b7/5e/H4_v34b_3GeV_-27.7_000506_20250802T040414Z_000506.root
hadd Source file 45: root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/usertests/41/b6/H4_v34b_3GeV_-27.7_000507_20250801T163427Z_000507.root
hadd Source file 46: root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/usertests/1e/5e/H4_v34b_3GeV_-27.7_000508_20250801T193551Z_000508.root
hadd Source file 47: root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/usertests/1b/a7/H4_v34b_3GeV_-27.7_000521_20250801T194611Z_000521.root
hadd Source file 48: root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/usertests/10/0c/H4_v34b_3GeV_-27.7_000536_20250801T181706Z_000536.root
hadd Source file 49: root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/usertests/73/72/H4_v34b_3GeV_-27.7_000541_20250802T042429Z_000541.root
hadd Source file 50: root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/usertests/e1/7f/H4_v34b_3GeV_-27.7_000553_20250801T184037Z_000553.root
hadd Source file 51: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/usertests/ca/95/H4_v34b_3GeV_-27.7_000564_20250801T171037Z_000564.root
hadd Source file 52: root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/usertests/17/c8/H4_v34b_3GeV_-27.7_000569_20250802T011400Z_000569.root
hadd Source file 53: root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/usertests/3e/5f/H4_v34b_3GeV_-27.7_000583_20250801T182841Z_000583.root
hadd Source file 54: root://dcdndoor.sdcc.bnl.gov:1094/pnfs/sdcc.bnl.gov/data/dune/RSE/usertests/70/0b/H4_v34b_3GeV_-27.7_000585_20250801T183117Z_000585.root
hadd Source file 55: root://dcdndoor.sdcc.bnl.gov:1094/pnfs/sdcc.bnl.gov/data/dune/RSE/usertests/62/0c/H4_v34b_3GeV_-27.7_000598_20250801T173521Z_000598.root
hadd Source file 56: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/usertests/75/b2/H4_v34b_3GeV_-27.7_000599_20250802T024929Z_000599.root
hadd Source file 57: root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/usertests/1a/50/H4_v34b_3GeV_-27.7_000603_20250801T180613Z_000603.root
hadd Source file 58: root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/usertests/1d/1e/H4_v34b_3GeV_-27.7_000612_20250801T210415Z_000612.root
hadd Source file 59: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/usertests/fe/24/H4_v34b_3GeV_-27.7_000624_20250801T183304Z_000624.root
hadd Source file 60: root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/usertests/13/e4/H4_v34b_3GeV_-27.7_000640_20250801T175636Z_000640.root
hadd Source file 61: root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/usertests/b5/10/H4_v34b_3GeV_-27.7_000684_20250801T183257Z_000684.root
hadd Source file 62: root://dcdndoor.sdcc.bnl.gov:1094/pnfs/sdcc.bnl.gov/data/dune/RSE/usertests/85/e6/H4_v34b_3GeV_-27.7_000694_20250801T173437Z_000694.root
hadd Source file 63: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/usertests/d9/0c/H4_v34b_3GeV_-27.7_000695_20250802T040653Z_000695.root
hadd Source file 64: root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/usertests/60/48/H4_v34b_3GeV_-27.7_000701_20250801T174936Z_000701.root
hadd Source file 65: root://xgate.hec.lancs.ac.uk:1094//cephfs/grid/dune/usertests/1c/2a/H4_v34b_3GeV_-27.7_000708_20250801T170203Z_000708.root
hadd Source file 66: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/usertests/ac/9f/H4_v34b_3GeV_-27.7_000714_20250802T035020Z_000714.root
hadd Source file 67: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/usertests/41/f8/H4_v34b_3GeV_-27.7_000732_20250801T183235Z_000732.root
hadd Source file 68: root://dcdndoor.sdcc.bnl.gov:1094/pnfs/sdcc.bnl.gov/data/dune/RSE/usertests/ff/05/H4_v34b_3GeV_-27.7_000733_20250801T190535Z_000733.root
hadd Source file 69: root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/usertests/c7/49/H4_v34b_3GeV_-27.7_000745_20250802T043038Z_000745.root
hadd Source file 70: root://dcdndoor.sdcc.bnl.gov:1094/pnfs/sdcc.bnl.gov/data/dune/RSE/usertests/c9/2e/H4_v34b_3GeV_-27.7_000769_20250801T180018Z_000769.root
hadd Source file 71: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/usertests/cf/bf/H4_v34b_3GeV_-27.7_000784_20250801T182435Z_000784.root
hadd Source file 72: root://dcdndoor.sdcc.bnl.gov:1094/pnfs/sdcc.bnl.gov/data/dune/RSE/usertests/3c/d1/H4_v34b_3GeV_-27.7_000788_20250801T184502Z_000788.root
hadd Source file 73: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/usertests/c0/cc/H4_v34b_3GeV_-27.7_000793_20250801T183521Z_000793.root
hadd Source file 74: root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/usertests/45/93/H4_v34b_3GeV_-27.7_000801_20250801T172919Z_000801.root
hadd Source file 75: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/usertests/ce/a5/H4_v34b_3GeV_-27.7_000811_20250801T201107Z_000811.root
Error in <TNetXNGFile::Open>: [ERROR] Server responded with an error: [3011] Item not found.
Error in <TFileMerger::AddFile>: cannot open file root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/usertests/ce/a5/H4_v34b_3GeV_-27.7_000811_20250801T201107Z_000811.root
hadd exiting due to error in root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/usertests/ce/a5/H4_v34b_3GeV_-27.7_000811_20250801T201107Z_000811.root
Traceback (most recent call last):
File "/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/7f3f3008d21189b6ef9a78409a02e3f793fbbe00/merge_g4bl.py", line 511, in <module>
do_merge(args)
File "/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/7f3f3008d21189b6ef9a78409a02e3f793fbbe00/merge_g4bl.py", line 119, in do_merge
raise Exception('Error in hadd')
Exception: Error in hadd
Exiting with error 