justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 292870.0@dunegpschedd02.fnal.gov

Jobsub ID292870.0@dunegpschedd02.fnal.gov
Workflow TestingYes
Workflow ID1
Stage ID1
User nameamcnab@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes1073741824 (1024 MiB)
Wall seconds limit3600 (1 hours)
Submitted time2026-01-21 15:14:53
SiteCA_Victoria
EntryDUNE_CA_Victoria_dune-condor_whole
Last heartbeat2026-01-21 15:53:29
From worker nodeHostnamedune--arbutus-new--3456932580--30850055887008
cpuinfoIntel Xeon Processor (SapphireRapids)
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes1310720000 (1250 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2026-01-21 15:52:09
Input files
JobscriptExit code0
Real time1m (71s)
CPU time0m (3s = 4%)
Max RSS bytes32219136 (30 MiB)
Outputting started2026-01-21 15:53:21
Output files
Finished2026-01-21 15:53:29
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

-download-2023-03-07-01.txt_1749841165
'xrdcp --force --nopbar --verbose root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/tape_backed/dunepro//other/awt-staging/awt-download-2023-03-07-01.txt_1749841165 downloaded.txt' returns 0
../justin-jobscript: line 55: metacat: command not found
metacat file declare returns 127
GFAL_CONFIG_DIR:    GFAL_PLUGIN_DIR: 
justin-rucio-upload attempt 1
../justin-jobscript: line 64: justin-rucio-upload: command not found
'justin-rucio-upload --rse FNAL_DCACHE --protocol davs --scope testpro --dataset awt-uploads-202603 awt-1769010739-VqI0uYiDQO --timeout 1200' returns 127


---------------------------------------------------------------------
CA_Victoria NIKHEF davs root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/testpro/bb/7f/awt-download-2023-03-07-01.txt
'xrdcp --force --nopbar --verbose root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/testpro/bb/7f/awt-download-2023-03-07-01.txt downloaded.txt' returns 0
../justin-jobscript: line 55: metacat: command not found
metacat file declare returns 127
GFAL_CONFIG_DIR:    GFAL_PLUGIN_DIR: 
justin-rucio-upload attempt 1
../justin-jobscript: line 64: justin-rucio-upload: command not found
'justin-rucio-upload --rse NIKHEF --protocol davs --scope testpro --dataset awt-uploads-202603 awt-1769010739-Jh3sS7vxnP --timeout 1200' returns 127


---------------------------------------------------------------------
CA_Victoria PRAGUE davs root://se1.farm.particle.cz:1094//dune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt
'xrdcp --force --nopbar --verbose root://se1.farm.particle.cz:1094//dune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt downloaded.txt' returns 0
../justin-jobscript: line 55: metacat: command not found
metacat file declare returns 127
GFAL_CONFIG_DIR:    GFAL_PLUGIN_DIR: 
justin-rucio-upload attempt 1
../justin-jobscript: line 64: justin-rucio-upload: command not found
'justin-rucio-upload --rse PRAGUE --protocol davs --scope testpro --dataset awt-uploads-202603 awt-1769010739-0BaKCGT5HO --timeout 1200' returns 127


---------------------------------------------------------------------
CA_Victoria QMUL davs root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt
'xrdcp --force --nopbar --verbose root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt downloaded.txt' returns 0
../justin-jobscript: line 55: metacat: command not found
metacat file declare returns 127
GFAL_CONFIG_DIR:    GFAL_PLUGIN_DIR: 
justin-rucio-upload attempt 1
../justin-jobscript: line 64: justin-rucio-upload: command not found
'justin-rucio-upload --rse QMUL --protocol davs --scope testpro --dataset awt-uploads-202603 awt-1769010739-i3bbtvYYc6 --timeout 1200' returns 127


---------------------------------------------------------------------
CA_Victoria RAL-PP davs root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/testpro/bb/7f/awt-download-2023-03-07-01.txt
'xrdcp --force --nopbar --verbose root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/testpro/bb/7f/awt-download-2023-03-07-01.txt downloaded.txt' returns 0
../justin-jobscript: line 55: metacat: command not found
metacat file declare returns 127
GFAL_CONFIG_DIR:    GFAL_PLUGIN_DIR: 
justin-rucio-upload attempt 1
../justin-jobscript: line 64: justin-rucio-upload: command not found
'justin-rucio-upload --rse RAL-PP --protocol davs --scope testpro --dataset awt-uploads-202603 awt-1769010739-gQVHL3dsAL --timeout 1200' returns 127


---------------------------------------------------------------------
CA_Victoria RAL_ECHO davs root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt
'xrdcp --force --nopbar --verbose root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt downloaded.txt' returns 0
../justin-jobscript: line 55: metacat: command not found
metacat file declare returns 127
GFAL_CONFIG_DIR:    GFAL_PLUGIN_DIR: 
justin-rucio-upload attempt 1
../justin-jobscript: line 64: justin-rucio-upload: command not found
'justin-rucio-upload --rse RAL_ECHO --protocol davs --scope testpro --dataset awt-uploads-202603 awt-1769010739-BY6BBwW7Sr --timeout 1200' returns 127


---------------------------------------------------------------------
CA_Victoria SURFSARA davs root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt
'xrdcp --force --nopbar --verbose root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt downloaded.txt' returns 0
../justin-jobscript: line 55: metacat: command not found
metacat file declare returns 127
GFAL_CONFIG_DIR:    GFAL_PLUGIN_DIR: 
justin-rucio-upload attempt 1
../justin-jobscript: line 64: justin-rucio-upload: command not found
'justin-rucio-upload --rse SURFSARA --protocol davs --scope testpro --dataset awt-uploads-202603 awt-1769010739-URFLblQiga --timeout 1200' returns 127


---------------------------------------------------------------------
CA_Victoria T3_US_NERSC davs root://dtn14.nersc.gov:1094//global/cfs/cdirs/m3249/dune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt
'xrdcp --force --nopbar --verbose root://dtn14.nersc.gov:1094//global/cfs/cdirs/m3249/dune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt downloaded.txt' returns 0
../justin-jobscript: line 55: metacat: command not found
metacat file declare returns 127
GFAL_CONFIG_DIR:    GFAL_PLUGIN_DIR: 
justin-rucio-upload attempt 1
../justin-jobscript: line 64: justin-rucio-upload: command not found
'justin-rucio-upload --rse T3_US_NERSC --protocol davs --scope testpro --dataset awt-uploads-202603 awt-1769010739-HXpT3lFpOJ --timeout 1200' returns 127


subject   : /C=UK/O=eScience/OU=Manchester/L=HEP/CN=justin-jobs-production.dune.hep.ac.uk/CN=3462494137/CN=176901072906
issuer    : /C=UK/O=eScience/OU=Manchester/L=HEP/CN=justin-jobs-production.dune.hep.ac.uk/CN=3462494137
identity  : /C=UK/O=eScience/OU=Manchester/L=HEP/CN=justin-jobs-production.dune.hep.ac.uk/CN=3462494137
type      : RFC compliant proxy
strength  : 2048 bits
path      : /home/awt-proxy.pem
timeleft  : 167:58:49
key usage : Digital Signature, Key Encipherment, Key Agreement
=== VO dune extension information ===
VO        : dune
subject   : /C=UK/O=eScience/OU=Manchester/L=HEP/CN=justin-jobs-production.dune.hep.ac.uk
issuer    : /DC=org/DC=incommon/C=US/ST=Illinois/O=Fermi Research Alliance/CN=voms1.fnal.gov
attribute : /dune/Role=Production/Capability=NULL
attribute : /dune/Role=NULL/Capability=NULL
timeleft  : 155:29:41
uri       : voms1.fnal.gov:15042

===== Results =====

Download/upload commands:
xrdcp --force --nopbar --verbose $read_pfn downloaded.txt
echo '{"namespace":"testpro","name":"FILENAME","size":0}' >tmp.json
metacat file declare --json -f tmp.json "dune:all"
justin-rucio-upload --rse $rse_name --protocol $write_protocol --scope testpro --dataset awt-uploads-202603 --timeout 1200 FILENAME
Use the wrapper job link on the page for the job on the justIN Dashboard to find the full log file, with errors from these commands

Each line: $JUSTIN_SITE_NAME $rse_name $download_retval $upload_retval $read_pfn $write_protocol
==awt== CA_Victoria DUNE_CA_SFU 0 127 root://lcg-dunese1.sfu.computecanada.ca:1094//dune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== CA_Victoria DUNE_CERN_EOS 0 127 root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== CA_Victoria DUNE_ES_PIC 0 127 root://xrootd.pic.es:1094/pnfs/pic.es/data/dune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== CA_Victoria DUNE_FR_CCIN2P3_DISK 0 127 root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== CA_Victoria DUNE_IT_INFN_CNAF 0 127 root://xrootd-archive.cr.cnaf.infn.it:1096//dune/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== CA_Victoria DUNE_UK_GLASGOW 0 127 root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== CA_Victoria DUNE_UK_LANCASTER_CEPH 0 127 root://xgate.hec.lancs.ac.uk:1094//cephfs/grid/dune/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== CA_Victoria DUNE_UK_MANCHESTER_CEPH 0 127 root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== CA_Victoria DUNE_US_BNL_SDCC 0 127 root://dcdndoor.sdcc.bnl.gov:1094//pnfs/sdcc.bnl.gov/data/dune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== CA_Victoria DUNE_US_FNAL_DISK_STAGE 0 127 root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== CA_Victoria FNAL_DCACHE 0 127 root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/tape_backed/dunepro//other/awt-staging/awt-download-2023-03-07-01.txt_1749841165 davs
==awt== CA_Victoria NIKHEF 0 127 root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== CA_Victoria PRAGUE 0 127 root://se1.farm.particle.cz:1094//dune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== CA_Victoria QMUL 0 127 root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== CA_Victoria RAL-PP 0 127 root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== CA_Victoria RAL_ECHO 0 127 root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== CA_Victoria SURFSARA 0 127 root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== CA_Victoria T3_US_NERSC 0 127 root://dtn14.nersc.gov:1094//global/cfs/cdirs/m3249/dune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
justIN time: 2026-02-04 06:07:01 UTC       justIN version: 01.06.00