justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Workflow 11969, Stage 1

Workflow11969
Campaign367
Priority50
Processors1
Wall seconds80000
Image/cvmfs/singularity.opensciencegrid.org/fermilab/fnal-wn-sl7:latest
RSS bytes8387559424 (7999 MiB)
Max distance for inputs100.0
Enabled input RSEs CERN_PDUNE_EOS, DUNE_CA_SFU, DUNE_CERN_EOS, DUNE_ES_PIC, DUNE_FR_CCIN2P3_DISK, DUNE_IN_TIFR, DUNE_IT_INFN_CNAF, DUNE_UK_GLASGOW, DUNE_UK_LANCASTER_CEPH, DUNE_UK_MANCHESTER_CEPH, DUNE_US_BNL_SDCC, DUNE_US_FNAL_DISK_STAGE, FNAL_DCACHE, FNAL_DCACHE_STAGING, FNAL_DCACHE_TEST, MONTECARLO, NIKHEF, PRAGUE, QMUL, RAL-PP, RAL_ECHO, SURFSARA, T3_US_NERSC
Enabled output RSEs CERN_PDUNE_EOS, DUNE_CA_SFU, DUNE_CERN_EOS, DUNE_ES_PIC, DUNE_FR_CCIN2P3_DISK, DUNE_IN_TIFR, DUNE_IT_INFN_CNAF, DUNE_UK_GLASGOW, DUNE_UK_LANCASTER_CEPH, DUNE_UK_MANCHESTER_CEPH, DUNE_US_BNL_SDCC, DUNE_US_FNAL_DISK_STAGE, FNAL_DCACHE, FNAL_DCACHE_STAGING, FNAL_DCACHE_TEST, NIKHEF, PRAGUE, QMUL, RAL-PP, RAL_ECHO, SURFSARA, T3_US_NERSC
Enabled sites BR_CBPF, CA_SFU, CA_Victoria, CERN, CH_UNIBE-LHEP, CZ_FZU, ES_CIEMAT, ES_PIC, FR_CCIN2P3, IT_CNAF, NL_NIKHEF, NL_SURFsara, UK_Bristol, UK_Brunel, UK_Durham, UK_Edinburgh, UK_Glasgow, UK_Imperial, UK_Lancaster, UK_Liverpool, UK_Manchester, UK_Oxford, UK_QMUL, UK_RAL-PPD, UK_RAL-Tier1, UK_Sheffield, US_BNL, US_Colorado, US_FNAL-FermiGrid, US_FNAL-T1, US_Michigan, US_NotreDame, US_PuertoRico, US_SU-ITS, US_Swan, US_UChicago, US_UConn-HPC, US_UCSD, US_Wisconsin
Scopeusertests
Events for this stage

Output patterns

 DestinationPatternLifetimeFor next stageRSE expression
1https://fndcadoor.fnal.gov:2880/dune/scratch/users/jjo/fnal/11969/1*_sp.root
2https://fndcadoor.fnal.gov:2880/dune/scratch/users/jjo/fnal/11969/1*_g4-tru-anode*.h5

Environment variables

NameValue
INPUT_TAR_DIR_LOCAL/cvmfs/fifeuser4.opensciencegrid.org/sw/dune/a4216cf97a94593d2b86b6ceaaa9de83d513111b
NUM_EVENTS10

File states

Total filesFindingUnallocatedAllocatedOutputtingProcessedNot foundFailed
100008700013000

Job states

TotalSubmittedStartedProcessingOutputtingFinishedNotusedAbortedStalledJobscript errorOutputting failedNone processed
1500000013081404966540
Files processed001010202030304040505060607070Jan-15 18:00Jan-15 19:00Jan-15 20:00Jan-15 21:00Files processedBin start timesNumber per binUK_OxfordUK_RAL-PPDUS_UChicagoCZ_FZUUS_FNAL-T1US_FNAL-FermiG…US_FNAL-FermiGridUK_QMULUK_BrunelES_PICIT_CNAF

RSEs used

NameInputsOutputs
MONTECARLO6820
None0130

Stats of processed input files as CSV or JSON, and of uploaded output files as CSV or JSON (up to 10000 files included)

File reset events, by site

SiteAllocatedOutputting
CERN12226
US_UChicago1060
CZ_FZU640
UK_Brunel320
US_FNAL-FermiGrid310
IT_CNAF300
UK_QMUL240
NL_SURFsara2312
CA_SFU190
US_FNAL-T1100
ES_PIC90
UK_RAL-PPD80
UK_Oxford60
FR_CCIN2P3412
UK_Sheffield32
UK_Bristol12

Jobscript

#!/bin/bash
#
# fdhd_test.jobscript
# Simple testing jobscript to run a short FD-VD chain (20 events per file)
#

# ---- Setup environment ----
source /cvmfs/dune.opensciencegrid.org/products/dune/setup_dune.sh

# Use requested version and qualifier
setup dunesw v10_16_00d00 -q e26:prof
if [ $? -ne 0 ]; then
  echo "Error setting up dunesw v10_16_00d00"
  exit 1
fi

# Prefer uploaded FHiCLs and WireCell configs if provided
if [ -n "${INPUT_TAR_DIR_LOCAL:-}" ]; then
    # Tell art/lar where to find .fcl files
    export FW_SEARCH_PATH="${INPUT_TAR_DIR_LOCAL}/cffm-if/dune10kt-1x2x6:${FW_SEARCH_PATH}"
    export FCL_SEARCH_PATH="${INPUT_TAR_DIR_LOCAL}/cffm-if/dune10kt-1x2x6:${FCL_SEARCH_PATH}"
    export FHICL_FILE_PATH="${INPUT_TAR_DIR_LOCAL}/cffm-if/dune10kt-1x2x6:${FHICL_FILE_PATH}"

    # WireCell lookup
    export WIRECELL_PATH="${INPUT_TAR_DIR_LOCAL}/cffm-if/dune10kt-1x2x6:${WIRECELL_PATH}"
fi

echo "FW_SEARCH_PATH=${FW_SEARCH_PATH}"
echo "FCL_SEARCH_PATH=${FCL_SEARCH_PATH}"
echo "FHICL_FILE_PATH=${FHICL_FILE_PATH}"
echo "WIRECELL_PATH=${WIRECELL_PATH}"

# Parallelism
if [ -z "${JUSTIN_PROCESSORS}" ]; then
  JUSTIN_PROCESSORS=1
fi
export TF_NUM_THREADS=${JUSTIN_PROCESSORS}
export OPENBLAS_NUM_THREADS=${JUSTIN_PROCESSORS}
export JULIA_NUM_THREADS=${JUSTIN_PROCESSORS}
export MKL_NUM_THREADS=${JUSTIN_PROCESSORS}
export NUMEXPR_NUM_THREADS=${JUSTIN_PROCESSORS}
export OMP_NUM_THREADS=${JUSTIN_PROCESSORS}

# ---- Get one input file (MC counter or real file, depending on submission) ----
did_pfn_rse="$($JUSTIN_PATH/justin-get-file)"
if [ -z "${did_pfn_rse}" ] ; then
  echo "No unprocessed files available, exiting."
  exit 0
fi

did=$(echo "$did_pfn_rse" | cut -f1 -d' ')
pfn=$(echo "$did_pfn_rse" | cut -f2 -d' ')
rse=$(echo "$did_pfn_rse" | cut -f3 -d' ')
name_only=$(echo "$did" | cut -f2 -d':' )

echo "Processing file: ${pfn}"
echo "RSE: ${rse}"
echo "DID: ${did}"

# ---- Set event count ----
nevents="${NUM_EVENTS:-10}"

# ---- Timestamp for unique output names ----
now=$(date -u +"%Y%m%dT%H%M%SZ")
jobid=$(echo "${JUSTIN_JOBSUB_ID:-1}" | cut -f1 -d'@' | sed -e "s/\./_/")
stageid=${JUSTIN_STAGE_ID:-1}

# ---- LArSoft chain ----

# 1. GEN (pure MC)
lar -n "${nevents}" -c gen_genie.fcl -o gen.root
if [ $? -ne 0 ]; then echo "GEN failed"; exit 1; fi

# 2. G4
lar -c g4.fcl -s gen.root -o g4.root
if [ $? -ne 0 ]; then echo "G4 failed"; exit 1; fi

# 3. WireCell Sim SP
outFile=${name_only}_${jobid}_${stageid}_${now}_sp.root
lar -c wcls_sim_sp.fcl -s g4.root -o ${outFile}
if [ $? -ne 0 ]; then echo "WCLS failed"; exit 1; fi

# 4. Truth labelling
label_tag="${name_only}_${jobid}_${stageid}_${now}"
mkdir -p truth_${label_tag}
pushd truth_${label_tag}

# Run on the SP file from step 3 (path relative to this dir)
lar -c wcls-labelling2d_sep.fcl -s "../${outFile}"
rc=$?

if [ $rc -ne 0 ]; then
  echo "Truth labeling failed"
  exit 1
fi

# Stamp the produced files with your tag
for f in g4-tru-anode*.h5; do
  mv "$f" "${label_tag}_${f}"
done

popd

# ---- Mark as processed ----
echo "${pfn}" > justin-processed-pfns.txt

echo "Job completed successfully."
ls -lh *.root

exit 0
justIN time: 2026-02-04 05:42:07 UTC       justIN version: 01.06.00