Workflow 12522, Stage 1
| Workflow | 12522 |
| Campaign | 868 |
| Priority | 50 |
| Processors | 1 |
| Wall seconds | 80000 |
| Image | /cvmfs/singularity.opensciencegrid.org/fermilab/fnal-wn-sl7:latest |
| RSS bytes | 2097152000 (2000 MiB) |
| Max distance for inputs | 100.0 |
| Enabled input RSEs |
CERN_PDUNE_EOS, DUNE_CA_SFU, DUNE_CERN_EOS, DUNE_ES_PIC, DUNE_FR_CCIN2P3_DISK, DUNE_IN_TIFR, DUNE_IT_INFN_CNAF, DUNE_UK_GLASGOW, DUNE_UK_LANCASTER_CEPH, DUNE_UK_MANCHESTER_CEPH, DUNE_US_BNL_SDCC, DUNE_US_FNAL_DISK_STAGE, FNAL_DCACHE, FNAL_DCACHE_STAGING, FNAL_DCACHE_TEST, MONTECARLO, NIKHEF, PRAGUE, QMUL, RAL-PP, RAL_ECHO, SURFSARA, T3_US_NERSC |
| Enabled output RSEs |
CERN_PDUNE_EOS, DUNE_CA_SFU, DUNE_CERN_EOS, DUNE_ES_PIC, DUNE_FR_CCIN2P3_DISK, DUNE_IN_TIFR, DUNE_IT_INFN_CNAF, DUNE_UK_GLASGOW, DUNE_UK_LANCASTER_CEPH, DUNE_UK_MANCHESTER_CEPH, DUNE_US_BNL_SDCC, DUNE_US_FNAL_DISK_STAGE, FNAL_DCACHE, FNAL_DCACHE_STAGING, FNAL_DCACHE_TEST, NIKHEF, PRAGUE, QMUL, RAL-PP, RAL_ECHO, SURFSARA, T3_US_NERSC |
| Enabled sites |
CERN |
| Scope | usertests |
| Events for this stage |
Output patterns
| | Destination | Pattern | Lifetime | For next stage | RSE expression |
|---|
| 1 | Rucio usertests:fnal-w12522s1p1 | prodmarley_nue_flat_es_dune10kt_1x2x2__trg_mc_2025a_tpg__tpg_dune10kt_1x2x2__reco__v10_12_01d01__triggerana_dune10kt_1x2x2__v10_12_01d01__ntuple__TEST-CC2_l000020_20260129T134317*.root | 2592000 | False | |
Environment variables
| Name | Value |
|---|
| CONFIG_DIR | /cvmfs/fifeuser4.opensciencegrid.org/sw/dune/aa5da5f6a707c00866a9d6109cbb797bbdc1f28c |
| DUNE_QUALIFIER | e26:prof |
| DUNE_VERSION | v10_12_01d01 |
| MERGE_CONFIG | pass1_CERN |
File states
| Total files | Finding | Unallocated | Allocated | Outputting | Processed | Not found | Failed |
|---|
|
| 1 | 0 | 0 | 0 | 0 | 0 | 0 | 1 |
Job states
| Total | Submitted | Started | Processing | Outputting | Finished | Notused | Aborted | Stalled | Jobscript error | Outputting failed | None processed |
|---|
| 8 | 0 | 0 | 0 | 0 | 2 | 0 | 0 | 0 | 6 | 0 | 0 |
RSEs used
| Name | Inputs | Outputs |
|---|
| MONTECARLO | 6 | 0 |
Stats of processed input files as CSV or JSON, and of uploaded output files as CSV or JSON (up to 10000 files included)
Jobscript
#!/bin/sh
: <<'EOF'
Hello World jobscript for justIN
Submit a workflow like this to run 10 jobs:
justin simple-workflow --monte-carlo 10 --jobscript hello-world.jobscript
Or like this to run jobs and put the output file into Rucio-managed storage:
justin simple-workflow \
--monte-carlo 10 \
--jobscript hello-world.jobscript \
--description 'Hello World!!!' \
--scope usertests \
--output-pattern 'hello-world-*.txt:output-test-01'
EOF
# Try to get an unprocessed file from this stage
did_pfn_rse=`$JUSTIN_PATH/justin-get-file`
if [ "$did_pfn_rse" == "" ]; then
echo "Nothing to process - exit jobscript"
exit 0
fi
# extract job key from the did_pfn_rse string, which is of the form:
# monte-carlo-000001-000001 000001 MONTECARLO
jobkey=`echo $did_pfn_rse | cut -f2 -d' '`
# Get job configuration
while [ ! -d $CONFIG_DIR ]; do
echo "Waiting for config directory $CONFIG_DIR to be created"
sleep 5
done
config="${MERGE_CONFIG}_${jobkey}.json"
echo "Config file: $config"
cat $CONFIG_DIR/$config
# Setup DUNE environment
#DUNE_VERSION=`grep 'dune_version' $CONFIG_DIR/$config | cut -d'"' -f4`
#DUNE_QUALIFIER=`grep 'dune_qualifier' $CONFIG_DIR/$config | cut -d'"' -f4`
DUNE_VERSION=${DUNE_VERSION:-v10_12_01d00}
DUNE_QUALIFIER=${DUNE_QUALIFIER:-e26:prof}
source /cvmfs/dune.opensciencegrid.org/products/dune/setup_dune.sh
setup dunesw "$DUNE_VERSION" -q "$DUNE_QUALIFIER"
python3 $CONFIG_DIR/do_merge.py $CONFIG_DIR/$config $OUT_DIR
exitcode=$?
if [ $exitcode -ne 0 ];
then
echo "failed to merge $config $exitcode"
exit $exitcode
fi
echo "$jobkey" > justin-processed-pfns.txt
echo "merged $config"
echo "merged $config" >merge-`date +%s.%N.txt`
exit 0
