Jobsub ID 254464.6@dunegpschedd02.fnal.gov
| Jobsub ID | 254464.6@dunegpschedd02.fnal.gov |
| Workflow ID | 10867 |
| Stage ID | 1 |
| User name | mkramer@fnal.gov |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 2097152000 (2000 MiB) |
| Wall seconds limit | 12000 (3 hours) |
| Submitted time | 2025-11-28 00:17:14 |
| Site | UK_RAL-Tier1 |
| Entry | LIGO_UK_RAL_arc_ce01 |
| Last heartbeat | 2025-11-28 01:10:18 |
| From worker node | Hostname | dune001-505328.0-lcg2695.gridpp.rl.ac.uk |
| cpuinfo | AMD EPYC 9654 96-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 2097152000 (2000 MiB) |
| Wall seconds limit | 216000 (60 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | jobscript_error |
| Started | 2025-11-28 00:17:49 |
| Input files | neardet-2x2-lar-charge:binary-0020114-2024_11_09_01_00_02_CET.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Max RSS bytes | 0 (0 MiB) |
| Outputting started | |
| Output files | |
| Finished | 2025-11-28 01:10:18 |
| Saved logs | justin-logs:254464.6-dunegpschedd02.fnal.gov.logs.tgz |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
The node working directory /home/workspace
host is dune001-505328.0-lcg2695.gridpp.rl.ac.uk
justin site is UK_RAL-Tier1
the current directory is /home/workspace
The workspace directory is /home/workspace
Retrieving the file from the path [/home].
The file data identifier (DID) is [neardet-2x2-lar-charge:binary-0020114-2024_11_09_01_00_02_CET.hdf5]
The file physical file name (PFN) is [root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/neardet-2x2-lar-charge/14/55/binary-0020114-2024_11_09_01_00_02_CET.hdf5]
The file Rucio storage element (RSE) is [DUNE_US_FNAL_DISK_STAGE]
The input file is binary-0020114-2024_11_09_01_00_02_CET.hdf5
Using rucio to download file [neardet-2x2-lar-charge:binary-0020114-2024_11_09_01_00_02_CET.hdf5]
Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org
Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/
[32;1m2025-11-28 00:17:53,983 INFO Processing 1 item(s) for input[0m
[32;1m2025-11-28 00:17:55,023 INFO No preferred protocol impl in rucio.cfg: No section: 'download'[0m
[32;1m2025-11-28 00:17:55,024 INFO Using main thread to download 1 file(s)[0m
[32;1m2025-11-28 00:17:55,024 INFO Preparing download of neardet-2x2-lar-charge:binary-0020114-2024_11_09_01_00_02_CET.hdf5[0m
[32;1m2025-11-28 00:17:55,580 INFO Trying to download with root and timeout of 646s from DUNE_US_FNAL_DISK_STAGE: neardet-2x2-lar-charge:binary-0020114-2024_11_09_01_00_02_CET.hdf5 [0m
[32;1m2025-11-28 00:17:55,630 INFO Using PFN: root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/neardet-2x2-lar-charge/14/55/binary-0020114-2024_11_09_01_00_02_CET.hdf5[0m
error in request of checking file QoS: is not a valid HTTP or Webdav URL
(process:994): GLib-WARNING **: 00:17:55.631: GError set over the top of a previous GError or uninitialized memory.
This indicates a bug in someone's code. You must ensure an error is NULL before it's set.
The overwriting error message was: \x01 is not a valid HTTP or Webdav URL
[33;1m2025-11-28 00:32:11,750 WARNING Download attempt failed. Try 1/2[0m
/cvmfs/dune.opensciencegrid.org/products/dune/rucio/v38_1_0/NULL/lib/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'dune-rucio.fnal.gov'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
warnings.warn(
error in request of checking file QoS: is not a valid HTTP or Webdav URL
(process:994): GLib-WARNING **: 00:32:12.285: GError set over the top of a previous GError or uninitialized memory.
This indicates a bug in someone's code. You must ensure an error is NULL before it's set.
The overwriting error message was: \x01 is not a valid HTTP or Webdav URL
[33;1m2025-11-28 00:46:28,084 WARNING Download attempt failed. Try 2/2[0m
/cvmfs/dune.opensciencegrid.org/products/dune/rucio/v38_1_0/NULL/lib/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'dune-rucio.fnal.gov'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
warnings.warn(
[32;1m2025-11-28 00:46:29,096 INFO Trying to download with davs and timeout of 646s from DUNE_US_FNAL_DISK_STAGE: neardet-2x2-lar-charge:binary-0020114-2024_11_09_01_00_02_CET.hdf5 [0m
[32;1m2025-11-28 00:46:29,104 INFO Using PFN: davs://fndcadoor.fnal.gov:2880/dune/persistent/staging/neardet-2x2-lar-charge/14/55/binary-0020114-2024_11_09_01_00_02_CET.hdf5[0m
error in request of checking file QoS: is not a valid HTTP or Webdav URL
(process:994): GLib-WARNING **: 00:46:29.105: GError set over the top of a previous GError or uninitialized memory.
This indicates a bug in someone's code. You must ensure an error is NULL before it's set.
The overwriting error message was: \x01 is not a valid HTTP or Webdav URL
[33;1m2025-11-28 00:57:15,553 WARNING Download attempt failed. Try 1/2[0m
/cvmfs/dune.opensciencegrid.org/products/dune/rucio/v38_1_0/NULL/lib/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'dune-rucio.fnal.gov'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
warnings.warn(
error in request of checking file QoS: is not a valid HTTP or Webdav URL
(process:994): GLib-WARNING **: 00:57:16.163: GError set over the top of a previous GError or uninitialized memory.
This indicates a bug in someone's code. You must ensure an error is NULL before it's set.
The overwriting error message was: \x01 is not a valid HTTP or Webdav URL
[33;1m2025-11-28 01:08:02,664 WARNING Download attempt failed. Try 2/2[0m
/cvmfs/dune.opensciencegrid.org/products/dune/rucio/v38_1_0/NULL/lib/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'dune-rucio.fnal.gov'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
warnings.warn(
[31;1m2025-11-28 01:08:03,192 ERROR Failed to download file neardet-2x2-lar-charge:binary-0020114-2024_11_09_01_00_02_CET.hdf5[0m
[31;1m2025-11-28 01:08:03,193 ERROR None of the requested files have been downloaded.[0m
Downloading the matching light files for the charge+light combination workflow.
Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org
Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/
Running the command [ python /cvmfs/dune.opensciencegrid.org/dunend/2x2//releases/v1.5.4p1/ndlar_prod_scripts/ND_Production/toolbox/scripts/GetInputList.py --file=neardet-2x2-lar-charge:binary-0020114-2024_11_09_01_00_02_CET.hdf5 --light ].
Enter getting the input list of matching files for the ndlar production jobs
The input file is [neardet-2x2-lar-charge:binary-0020114-2024_11_09_01_00_02_CET.hdf5]
The matching runs/subruns list : [(20114, 0), (20114, 1)]
Successfully download the file [neardet-2x2-lar-light:mpd_run_data_rctl_224_p0.data].
Successfully download the file [neardet-2x2-lar-light:mpd_run_data_rctl_224_p1.data].
Exit getting the input list of matching files.
The number of files is [2].
The parent files are [neardet-2x2-lar-charge:binary-0020114-2024_11_09_01_00_02_CET.hdf5 neardet-2x2-lar-light:mpd_run_data_rctl_224_p0.data neardet-2x2-lar-light:mpd_run_data_rctl_224_p1.data].
The output files are placed in the directory [/home/workspace]
Setup and enter the miniforge conda environment for the software release [v1.5.4p1]
Running [ source /cvmfs/dune.opensciencegrid.org/dunend/2x2//miniforge/v25.3.0-3/miniforge3/etc/profile.d/conda.sh ]
The current conda virtual environment is activated: [/cvmfs/dune.opensciencegrid.org/dunend/2x2/miniforge/v25.3.0-3/conda_envs/ndlar_flow_v1.5.4p1]
Copying the configuration directory [/cvmfs/dune.opensciencegrid.org/dunend/2x2//releases/v1.5.4p1/ndlar_flow/yamls/fsd_flow] to the workspace [/home/workspace/ndlar_flow/yamls/fsd_flow]
Copying the constants directory [/cvmfs/dune.opensciencegrid.org/dunend/2x2//releases/v1.5.4p1/ndlar_flow/data/fsd_flow] to the workspace [/home/workspace/ndlar_flow/data/fsd_flow]
Enter executing the charge raw files to hdf5 packet workflow for data stream [combined]
Running the charge raw data to packet data conversion workflow.
[ python /cvmfs/dune.opensciencegrid.org/dunend/2x2//releases/v1.5.4p1/larpix-control/scripts/convert_rawhdf5_to_hdf5.py -i /home/workspace/binary-0020114-2024_11_09_01_00_02_CET.hdf5 -o packet-0020114-2025_11_28_01_09_59_CDT.hdf5 --direct ]
Traceback (most recent call last):
File "/cvmfs/dune.opensciencegrid.org/dunend/2x2//releases/v1.5.4p1/larpix-control/scripts/convert_rawhdf5_to_hdf5.py", line 6, in <module>
import h5py
File "/cvmfs/dune.opensciencegrid.org/dunend/2x2/miniforge/v25.3.0-3/conda_envs/ndlar_flow_v1.5.4p1/lib/python3.11/site-packages/h5py/__init__.py", line 25, in <module>
from . import _errors
ImportError: /lib64/libm.so.6: version `GLIBC_2.27' not found (required by /cvmfs/dune.opensciencegrid.org/dunend/2x2/miniforge/v25.3.0-3/conda_envs/ndlar_flow_v1.5.4p1/lib/python3.11/site-packages/h5py/../h5py.libs/libhdf5-7d060e2f.so.310.5.1)
FATAL::The file [packet-0020114-2025_11_28_01_09_59_CDT.hdf5] does not exist! Will not continue. 