Jobsub ID 303713.13@dunegpschedd01.fnal.gov

Jobsub ID	303713.13@dunegpschedd01.fnal.gov
Workflow ID	12721
Stage ID	1
User name	calcuttj@fnal.gov
Requested	Processors	1
	GPU	No
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2026-02-04 00:46:29
Site	CERN
Entry	CMSHTPC_T2_CH_CERN_ce513
Last heartbeat	2026-02-04 00:49:14
From worker node	Hostname	b9p17p7474.cern.ch
	cpuinfo	Intel(R) Xeon(R) Silver 4216 CPU @ 2.10GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	343800 (95 hours)
	GPU
	Inner Apptainer?	True
Job state	none_processed
Started	2026-02-04 00:47:29
Input files	vd-protodune:np02vd_raw_run040380_0007_df-s04-d0_dw_0_20251105T175934.hdf5
Outputting started	2026-02-04 00:48:53
Output files
Finished	2026-02-04 00:49:14
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org
Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/
Pip installing snakemake
  WARNING: The script pulptest is installed in '/home/.local/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
  WARNING: The script tabulate is installed in '/home/.local/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
  WARNING: The script humanfriendly is installed in '/home/.local/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
  WARNING: The scripts docutils, rst2html, rst2html4, rst2html5, rst2latex, rst2man, rst2odt, rst2pseudoxml, rst2s5, rst2xetex and rst2xml are installed in '/home/.local/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
  WARNING: The script normalizer is installed in '/home/.local/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
  WARNING: The script yte is installed in '/home/.local/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
  WARNING: The scripts jupyter, jupyter-migrate and jupyter-troubleshoot are installed in '/home/.local/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
  WARNING: The script jsonschema is installed in '/home/.local/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
  WARNING: The script jupyter-trust is installed in '/home/.local/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
  WARNING: The scripts snakemake and snakemake-bash-completion are installed in '/home/.local/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.

[notice] A new release of pip is available: 23.0 -> 26.0
[notice] To update, run: pip install --upgrade pip
7.32.4
workflow/Snakefile
Building DAG of jobs...
Using shell: /usr/bin/bash
Provided cores: 1 (use --cores to define parallelism)
Rules claiming more threads will be scaled down.
Singularity containers: ignored
Job stats:
job      count
-----  -------
all          1
jgf          1
total        2

Select jobs to execute...
Calling justin get file -- 0
0
justin-get-file successful: vd-protodune:np02vd_raw_run040380_0007_df-s04-d0_dw_0_20251105T175934.hdf5 root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/vd-protodune/12/2d/np02vd_raw_run040380_0007_df-s04-d0_dw_0_20251105T175934.hdf5 DUNE_CERN_EOS
Select jobs to execute...
input:
/usr/bin/bash: line 3: xrootd_posix_preload: unbound variable
[Wed Feb  4 01:48:53 2026]
Error in rule test:
    jobid: 5
    input: justin_input_pfns.txt
    output: test.txt
    shell:
        
        echo "input:"
        input_file=$(head -n 1 justin_input_pfns.txt)
        echo "xrd posix preload: ${xrootd_posix_preload}"

        if [ -z "${xrootd_posix_preload}" ]; then
          export xrootd_posix_preload=$(find /home/ubuntu/lardon/.pixi -name "libXrdPosixPreload.so")
          echo "Found xrootd_posix_preload: ${xrootd_posix_preload}"
        fi

        LD_PRELOAD=$xrootd_posix_preload lardon -file $input_file -out lardon -trk
        output_file=$(ls *lardon.h5 | head -n 1)
        echo $output_file > test.txt
        
        (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: .snakemake/log/2026-02-04T014851.924479.snakemake.log
Done running snakemake