justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 20791.0@dunegpschedd01.fnal.gov

Jobsub ID20791.0@dunegpschedd01.fnal.gov
Workflow TestingYes
Workflow ID1
Stage ID1
User nameamcnab@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUYes
RSS bytes1073741824 (1024 MiB)
Wall seconds limit3600 (1 hours)
Submitted time2025-08-04 09:39:44
SiteUS_NERSC-GPU
Entrydune_t3_us_nersc_perlmutter_custom
Last heartbeat2025-08-04 10:13:02
From worker nodeHostnamenid008664
cpuinfoAMD EPYC 7763 64-Core Processor
OS releaseAlmaLinux release 9.5 (Teal Serval)
Processors1
RSS bytes1310720000 (1250 MiB)
Wall seconds limit84600 (23 hours)
GPUNVIDIA A100-SXM4-80GB 550.163.01 8.0 92.00.36.00.02 81037MiB
Inner Apptainer?False
Job stateoutputting_failed
Started2025-08-04 10:11:59
Input files
JobscriptExit code0
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started2025-08-04 10:12:58
Output files
Finished2025-08-04 10:13:02
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

e root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/testpro/bb/7f/awt-download-2023-03-07-01.txt downloaded.txt' returns 0
python3: error while loading shared libraries: libcrypt.so.1: cannot open shared object file: No such file or directory
metacat file declare returns 127
GFAL_CONFIG_DIR:    GFAL_PLUGIN_DIR: 
justin-rucio-upload attempt 1
python3: error while loading shared libraries: libcrypt.so.1: cannot open shared object file: No such file or directory
'justin-rucio-upload --rse NIKHEF --protocol davs --scope testpro --dataset awt-uploads-202531 awt-1754302321-vo46Zck1eI --timeout 1200' returns 127


---------------------------------------------------------------------
US_NERSC-GPU PRAGUE davs root://se1.farm.particle.cz:1094//dune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt
'xrdcp --force --nopbar --verbose root://se1.farm.particle.cz:1094//dune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt downloaded.txt' returns 0
python3: error while loading shared libraries: libcrypt.so.1: cannot open shared object file: No such file or directory
metacat file declare returns 127
GFAL_CONFIG_DIR:    GFAL_PLUGIN_DIR: 
justin-rucio-upload attempt 1
python3: error while loading shared libraries: libcrypt.so.1: cannot open shared object file: No such file or directory
'justin-rucio-upload --rse PRAGUE --protocol davs --scope testpro --dataset awt-uploads-202531 awt-1754302321-PPix7zXquM --timeout 1200' returns 127


---------------------------------------------------------------------
US_NERSC-GPU QMUL davs root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt
'xrdcp --force --nopbar --verbose root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt downloaded.txt' returns 0
python3: error while loading shared libraries: libcrypt.so.1: cannot open shared object file: No such file or directory
metacat file declare returns 127
GFAL_CONFIG_DIR:    GFAL_PLUGIN_DIR: 
justin-rucio-upload attempt 1
python3: error while loading shared libraries: libcrypt.so.1: cannot open shared object file: No such file or directory
'justin-rucio-upload --rse QMUL --protocol davs --scope testpro --dataset awt-uploads-202531 awt-1754302321-i6S6C3FKhT --timeout 1200' returns 127


---------------------------------------------------------------------
US_NERSC-GPU RAL-PP davs root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/testpro/bb/7f/awt-download-2023-03-07-01.txt
'xrdcp --force --nopbar --verbose root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/testpro/bb/7f/awt-download-2023-03-07-01.txt downloaded.txt' returns 0
python3: error while loading shared libraries: libcrypt.so.1: cannot open shared object file: No such file or directory
metacat file declare returns 127
GFAL_CONFIG_DIR:    GFAL_PLUGIN_DIR: 
justin-rucio-upload attempt 1
python3: error while loading shared libraries: libcrypt.so.1: cannot open shared object file: No such file or directory
'justin-rucio-upload --rse RAL-PP --protocol davs --scope testpro --dataset awt-uploads-202531 awt-1754302321-4Yudesk7f9 --timeout 1200' returns 127


---------------------------------------------------------------------
US_NERSC-GPU RAL_ECHO davs root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt
'xrdcp --force --nopbar --verbose root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt downloaded.txt' returns 0
python3: error while loading shared libraries: libcrypt.so.1: cannot open shared object file: No such file or directory
metacat file declare returns 127
GFAL_CONFIG_DIR:    GFAL_PLUGIN_DIR: 
justin-rucio-upload attempt 1
python3: error while loading shared libraries: libcrypt.so.1: cannot open shared object file: No such file or directory
'justin-rucio-upload --rse RAL_ECHO --protocol davs --scope testpro --dataset awt-uploads-202531 awt-1754302321-LYCxKDYrzF --timeout 1200' returns 127


---------------------------------------------------------------------
US_NERSC-GPU SURFSARA davs root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt
'xrdcp --force --nopbar --verbose root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt downloaded.txt' returns 0
python3: error while loading shared libraries: libcrypt.so.1: cannot open shared object file: No such file or directory
metacat file declare returns 127
GFAL_CONFIG_DIR:    GFAL_PLUGIN_DIR: 
justin-rucio-upload attempt 1
python3: error while loading shared libraries: libcrypt.so.1: cannot open shared object file: No such file or directory
'justin-rucio-upload --rse SURFSARA --protocol davs --scope testpro --dataset awt-uploads-202531 awt-1754302321-0PUBqhwkZA --timeout 1200' returns 127


---------------------------------------------------------------------
US_NERSC-GPU T3_US_NERSC davs root://dtn14.nersc.gov:1094//global/cfs/cdirs/m3249/dune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt
'xrdcp --force --nopbar --verbose root://dtn14.nersc.gov:1094//global/cfs/cdirs/m3249/dune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt downloaded.txt' returns 0
python3: error while loading shared libraries: libcrypt.so.1: cannot open shared object file: No such file or directory
metacat file declare returns 127
GFAL_CONFIG_DIR:    GFAL_PLUGIN_DIR: 
justin-rucio-upload attempt 1
python3: error while loading shared libraries: libcrypt.so.1: cannot open shared object file: No such file or directory
'justin-rucio-upload --rse T3_US_NERSC --protocol davs --scope testpro --dataset awt-uploads-202531 awt-1754302321-QjOQlqQO0F --timeout 1200' returns 127


subject   : /C=UK/O=eScience/OU=Manchester/L=HEP/CN=justin-jobs-production.dune.hep.ac.uk/CN=2236861484/CN=175430231983
issuer    : /C=UK/O=eScience/OU=Manchester/L=HEP/CN=justin-jobs-production.dune.hep.ac.uk/CN=2236861484
identity  : /C=UK/O=eScience/OU=Manchester/L=HEP/CN=justin-jobs-production.dune.hep.ac.uk/CN=2236861484
type      : RFC compliant proxy
strength  : 2048 bits
path      : /tmp/glide_wpBj96/execute/dir_168685/home/awt-proxy.pem
timeleft  : 167:59:01
key usage : Digital Signature, Key Encipherment, Key Agreement
=== VO dune extension information ===
VO        : dune
subject   : /C=UK/O=eScience/OU=Manchester/L=HEP/CN=justin-jobs-production.dune.hep.ac.uk
issuer    : /DC=org/DC=incommon/C=US/ST=Illinois/O=Fermi Research Alliance/CN=voms1.fnal.gov
attribute : /dune/Role=Production/Capability=NULL
attribute : /dune/Role=NULL/Capability=NULL
timeleft  : 160:58:03
uri       : voms1.fnal.gov:15042

===== Results =====

Download/upload commands:
xrdcp --force --nopbar --verbose $read_pfn downloaded.txt
echo '{"namespace":"testpro","name":"FILENAME","size":0}' >tmp.json
metacat file declare --json -f tmp.json "dune:all"
justin-rucio-upload --rse $rse_name --protocol $write_protocol --scope testpro --dataset awt-uploads-202531 --timeout 1200 FILENAME
Use the wrapper job link on the page for the job on the justIN Dashboard to find the full log file, with errors from these commands

Each line: $JUSTIN_SITE_NAME $rse_name $download_retval $upload_retval $read_pfn $write_protocol
==awt== US_NERSC-GPU DUNE_CA_SFU 0 127 root://lcg-dunese1.sfu.computecanada.ca:1094//dune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== US_NERSC-GPU DUNE_CERN_EOS 0 127 root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== US_NERSC-GPU DUNE_ES_PIC 0 127 root://xrootd.pic.es:1094/pnfs/pic.es/data/dune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== US_NERSC-GPU DUNE_FR_CCIN2P3_DISK 0 127 root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== US_NERSC-GPU DUNE_IT_INFN_CNAF 0 127 root://xrootd-archive.cr.cnaf.infn.it:1096//dune/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== US_NERSC-GPU DUNE_UK_GLASGOW 0 127 root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== US_NERSC-GPU DUNE_UK_LANCASTER_CEPH 0 127 root://xgate.hec.lancs.ac.uk:1094//cephfs/grid/dune/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== US_NERSC-GPU DUNE_UK_MANCHESTER_CEPH 0 127 root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== US_NERSC-GPU DUNE_US_BNL_SDCC 0 127 root://dcdndoor.sdcc.bnl.gov:1094//pnfs/sdcc.bnl.gov/data/dune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== US_NERSC-GPU DUNE_US_FNAL_DISK_STAGE 0 127 root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== US_NERSC-GPU FNAL_DCACHE 0 127 root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/tape_backed/dunepro//other/awt-staging/awt-download-2023-03-07-01.txt_1749841165 davs
==awt== US_NERSC-GPU NIKHEF 0 127 root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== US_NERSC-GPU PRAGUE 0 127 root://se1.farm.particle.cz:1094//dune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== US_NERSC-GPU QMUL 0 127 root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== US_NERSC-GPU RAL-PP 0 127 root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== US_NERSC-GPU RAL_ECHO 0 127 root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== US_NERSC-GPU SURFSARA 0 127 root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== US_NERSC-GPU T3_US_NERSC 0 127 root://dtn14.nersc.gov:1094//global/cfs/cdirs/m3249/dune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
justIN time: 2025-08-04 14:13:36 UTC       justIN version: 01.04.00