Workflow 9216, Stage 1

Priority	50
Processors	1
Wall seconds	80000
Image	/cvmfs/singularity.opensciencegrid.org/fermilab/fnal-wn-sl7:latest
RSS bytes	2097152000 (2000 MiB)
Max distance for inputs	100.0
Enabled input RSEs	CERN_PDUNE_EOS, DUNE_CA_SFU, DUNE_CERN_EOS, DUNE_ES_PIC, DUNE_FR_CCIN2P3_DISK, DUNE_IN_TIFR, DUNE_IT_INFN_CNAF, DUNE_UK_GLASGOW, DUNE_UK_LANCASTER_CEPH, DUNE_UK_MANCHESTER_CEPH, DUNE_US_BNL_SDCC, DUNE_US_FNAL_DISK_STAGE, FNAL_DCACHE, FNAL_DCACHE_STAGING, FNAL_DCACHE_TEST, MONTECARLO, NIKHEF, PRAGUE, QMUL, RAL-PP, RAL_ECHO, SURFSARA, T3_US_NERSC
Enabled output RSEs	CERN_PDUNE_EOS, DUNE_CA_SFU, DUNE_CERN_EOS, DUNE_ES_PIC, DUNE_FR_CCIN2P3_DISK, DUNE_IN_TIFR, DUNE_IT_INFN_CNAF, DUNE_UK_GLASGOW, DUNE_UK_LANCASTER_CEPH, DUNE_UK_MANCHESTER_CEPH, DUNE_US_BNL_SDCC, DUNE_US_FNAL_DISK_STAGE, FNAL_DCACHE, FNAL_DCACHE_STAGING, FNAL_DCACHE_TEST, NIKHEF, PRAGUE, QMUL, RAL-PP, RAL_ECHO, SURFSARA, T3_US_NERSC
Enabled sites	US_FNAL-FermiGrid
Scope	usertests
Events for this stage

Output patterns

	Destination	Pattern	Lifetime	For next stage	RSE expression
1	Rucio usertests:fnal-w9216s1p1	_merged_	604800	False

Environment variables

Name	Value
CONFIG_DIR	/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/8bf94f14876074236592bba035ded6e5a1cdac5a
MERGE_CONFIG	pass1_US_FNAL-FermiGrid

File states

Total files	Finding	Unallocated	Allocated	Outputting	Processed	Not found	Failed
2	0	0	0	0	2	0	0

Job states

Total	Submitted	Started	Processing	Outputting	Finished	Notused	Aborted	Stalled	Jobscript error	Outputting failed	None processed
3	0	0	0	0	3	0	0	0	0	0	0

RSEs used

Name	Inputs	Outputs
MONTECARLO	2	0
DUNE_US_FNAL_DISK_STAGE	0	4

Stats of processed input files as CSV or JSON, and of uploaded output files as CSV or JSON (up to 10000 files included)

Jobscript

#!/bin/sh
: <<'EOF'

Hello World jobscript for justIN

Submit a workflow like this to run 10 jobs:

justin simple-workflow --monte-carlo 10 --jobscript hello-world.jobscript


Or like this to run jobs and put the output file into Rucio-managed storage:

justin simple-workflow \
     --monte-carlo 10 \
     --jobscript hello-world.jobscript \
     --description 'Hello World!!!' \
     --scope usertests \
     --output-pattern 'hello-world-*.txt:output-test-01'

EOF

DUNE_VERSION=${DUNE_VERSION:-v09_75_00d00}
DUNE_QUALIFIER=${DUNE_QUALIFIER:-e20:prof}
#OUT_DIR=${OUT_DIR:.}

# Try to get an unprocessed file from this stage
did_pfn_rse=`$JUSTIN_PATH/justin-get-file`
if [ "$did_pfn_rse" == "" ]; then
  echo "Nothing to process - exit jobscript"
  exit 0
fi
# extract job key from the did_pfn_rse string, which is of the form:
# monte-carlo-000001-000001 000001 MONTECARLO
jobkey=`echo $did_pfn_rse | cut -f2 -d' '`
config="${MERGE_CONFIG}_${jobkey}.json"
echo "Config file: $config"

# Setup DUNE environment
source /cvmfs/dune.opensciencegrid.org/products/dune/setup_dune.sh
setup dunesw "$DUNE_VERSION" -q "$DUNE_QUALIFIER"

while [ ! -d $CONFIG_DIR ]; do
  echo "Waiting for config directory $CONFIG_DIR to be created"
  sleep 5
done
cat $CONFIG_DIR/$config

python3 $CONFIG_DIR/do_merge.py $CONFIG_DIR/$config $OUT_DIR

echo "$jobkey" > justin-processed-pfns.txt

echo "merged $jobkey"
echo "merged $jobkey" >merge-`date +%s.%N.txt`

exit 0