justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 279845.0@dunegpschedd01.fnal.gov

Jobsub ID279845.0@dunegpschedd01.fnal.gov
Workflow ID11986
Stage ID3
User nameusher@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit28800 (8 hours)
Submitted time2026-01-17 15:46:34
SiteUK_QMUL
EntryDUNE_UK_London_QMUL_arcce02
Last heartbeat2026-01-17 16:07:25
From worker nodeHostnamecn017.htc.esc.qmul
cpuinfoIntel(R) Xeon(R) Gold 6248R CPU @ 3.00GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Started2026-01-17 15:47:17
Input filesusertests:mpvmpr_monte-carlo-011986-000033_gen_g4.root
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2026-01-17 16:07:25
Saved logsjustin-logs:279845.0-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

of  -z /cvmfs/larsoft.opensciencegrid.org/products
pythia            v6_4_28x        -f Linux64bit+3.10-2.17 -q e26:prof        -z /cvmfs/larsoft.opensciencegrid.org/products
python            v3_9_15         -f Linux64bit+3.10-2.17                    -z /cvmfs/larsoft.opensciencegrid.org/products
python_future_six_request  v1_3_1          -f Linux64bit+3.10-2.17 -q python3.9       -z /cvmfs/fermilab.opensciencegrid.org/products/common/db/
range             v3_0_12_0       -f NULL                                    -z /cvmfs/larsoft.opensciencegrid.org/products
root              v6_28_12        -f Linux64bit+3.10-2.17 -q e26:p3915:prof  -z /cvmfs/larsoft.opensciencegrid.org/products
rstartree         v2016_07        -f NULL                                    -z /cvmfs/larsoft.opensciencegrid.org/products
sam_web_client    v3_6            -f NULL                                    -z /cvmfs/fermilab.opensciencegrid.org/products/common/db/
scitokens_cpp     v1_0_1a         -f Linux64bit+3.10-2.17 -q e26:sql34001    -z /cvmfs/larsoft.opensciencegrid.org/products
spdlog            v1_9_2          -f Linux64bit+3.10-2.17 -q e26:prof        -z /cvmfs/larsoft.opensciencegrid.org/products
sqlite            v3_40_01_00     -f Linux64bit+3.10-2.17                    -z /cvmfs/larsoft.opensciencegrid.org/products
srproxy           v00.45          -f NULL                 -q py3915          -z /cvmfs/dune.opensciencegrid.org/products/dune
systematicstools  v01_04_04       -f Linux64bit+3.10-2.17 -q e26:prof        -z /cvmfs/larsoft.opensciencegrid.org/products
tbb               v2021_9_0       -f Linux64bit+3.10-2.17 -q e26             -z /cvmfs/larsoft.opensciencegrid.org/products
tensorflow        v2_15_0         -f Linux64bit+3.10-2.17 -q e26:p3915       -z /cvmfs/larsoft.opensciencegrid.org/products
torch_scatter     v2_1_2a         -f Linux64bit+3.10-2.17 -q e26             -z /cvmfs/larsoft.opensciencegrid.org/products
TRACE             v3_17_11        -f NULL                                    -z /cvmfs/larsoft.opensciencegrid.org/products
triton            v2_25_0d        -f Linux64bit+3.10-2.17 -q e26             -z /cvmfs/larsoft.opensciencegrid.org/products
ups               v6_0_7          -f Linux64bit+3.10-2.17                    -z /cvmfs/dune.opensciencegrid.org/products/dune
valgrind          v3_17_0         -f Linux64bit+3.10-2.17                    -z /cvmfs/larsoft.opensciencegrid.org/products
wirecell          v0_32_1         -f Linux64bit+3.10-2.17 -q e26:prof        -z /cvmfs/larsoft.opensciencegrid.org/products
xerces_c          v3_2_3e         -f Linux64bit+3.10-2.17 -q e26             -z /cvmfs/larsoft.opensciencegrid.org/products
xrootd            v5_5_5a         -f Linux64bit+3.10-2.17 -q e26:p3915:prof  -z /cvmfs/larsoft.opensciencegrid.org/products
art 3.14.04
Running lar DETSIM:
  FCL:     standard_detsim_dune10kt_1x2x6.fcl
  Input:   root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/usertests/d9/a6/mpvmpr_monte-carlo-011986-000033_gen_g4.root
  Output:  mpvmpr_monte-carlo-011986-000033_gen_g4_detsim.root
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/dune.opensciencegrid.org/products/dune/dunecore/v10_16_00d00/gdml/dune10kt_v6_refactored_1x2x6.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Error: Unsupported GDML Tag Used :gdml_simple_extension. Please Check Geometry/Schema.
Error: Unsupported GDML Tag Used :extension. Please Check Geometry/Schema.
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 1149
Info in <TGeoManager::CloseGeometry>: 67545 nodes/ 5439 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
registering to  primaryGeneratorActionsMap_
registering to   eventActionsMap_
registering to   trackingActionsMap_
registering to  steppingActionsMap_
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
Using TESTbench spe!!! 
With: SPE_DAPHNE2_FBK_2022.dat
[15:47:44.322] I [  wct   ] loading compressed json file: /cvmfs/dune.opensciencegrid.org/products/dune/dunereco/v10_16_00d00/wire-cell-cfg/dune10kt-1x2x6-wires-larsoft-v1.json.bz2
[15:47:55.363] I [  wct   ] loading compressed json file: /cvmfs/dune.opensciencegrid.org/products/dune/dunereco/v10_16_00d00/wire-cell-cfg/dune-garfield-1d60563.json.bz2
[15:47:58.315] I [  wct   ] loading compressed json file: /cvmfs/dune.opensciencegrid.org/products/dune/dune_pardata/v01_84_00/WireCellData/protodune-noise-spectra-v1.json.bz2
[15:47:58.327] W [sigproc ] <OmnibusSigProc:apa0sigproc0> The 'wiener_threshold_tag' is obsolete, thresholds in summary on 'wiener' tagged traces 
[15:47:58.334] W [sigproc ] <OmnibusSigProc:apa1sigproc1> The 'wiener_threshold_tag' is obsolete, thresholds in summary on 'wiener' tagged traces 
[15:47:58.341] W [sigproc ] <OmnibusSigProc:apa2sigproc2> The 'wiener_threshold_tag' is obsolete, thresholds in summary on 'wiener' tagged traces 
[15:47:58.348] W [sigproc ] <OmnibusSigProc:apa3sigproc3> The 'wiener_threshold_tag' is obsolete, thresholds in summary on 'wiener' tagged traces 
[15:47:58.354] W [sigproc ] <OmnibusSigProc:apa4sigproc4> The 'wiener_threshold_tag' is obsolete, thresholds in summary on 'wiener' tagged traces 
[15:47:58.361] W [sigproc ] <OmnibusSigProc:apa5sigproc5> The 'wiener_threshold_tag' is obsolete, thresholds in summary on 'wiener' tagged traces 
[15:47:58.368] W [sigproc ] <OmnibusSigProc:apa6sigproc6> The 'wiener_threshold_tag' is obsolete, thresholds in summary on 'wiener' tagged traces 
[15:47:58.375] W [sigproc ] <OmnibusSigProc:apa7sigproc7> The 'wiener_threshold_tag' is obsolete, thresholds in summary on 'wiener' tagged traces 
[15:47:58.382] W [sigproc ] <OmnibusSigProc:apa8sigproc8> The 'wiener_threshold_tag' is obsolete, thresholds in summary on 'wiener' tagged traces 
[15:47:58.389] W [sigproc ] <OmnibusSigProc:apa9sigproc9> The 'wiener_threshold_tag' is obsolete, thresholds in summary on 'wiener' tagged traces 
[15:47:58.396] W [sigproc ] <OmnibusSigProc:apa10sigproc10> The 'wiener_threshold_tag' is obsolete, thresholds in summary on 'wiener' tagged traces 
[15:47:58.403] W [sigproc ] <OmnibusSigProc:apa11sigproc11> The 'wiener_threshold_tag' is obsolete, thresholds in summary on 'wiener' tagged traces 
Inputer: "wclsSimDepoSetSource:"
Outputer: "wclsDepoFluxWriter:postdrift"
Outputer: "wclsFrameSaver:spsignals"
wclsFrameSaver: promising to produce recob::Wires named "gauss"
wclsFrameSaver: promising to produce recob::Wires named "wiener"
wclsFrameSaver: promising to produce recob::Wires named "dnnsp"
17-Jan-2026 15:47:58 GMT  Initiating request to open input file "root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/usertests/d9/a6/mpvmpr_monte-carlo-011986-000033_gen_g4.root"
17-Jan-2026 15:54:39 GMT  Opened input file "root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/usertests/d9/a6/mpvmpr_monte-carlo-011986-000033_gen_g4.root"
Begin processing the 1st record. run: 200033 subRun: 0 event: 1 at 17-Jan-2026 15:55:01 GMT
17-Jan-2026 16:03:42 GMT  Opened output file with pattern "mpvmpr_monte-carlo-011986-000033_gen_g4_detsim.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
[ No processed events ]
====================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 2647.03 MB
  Peak resident set size usage (VmHWM): 950.448 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 17-Jan-2026 16:03:43 GMT ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- FileReadError BEGIN
      ---- FatalRootError BEGIN
        Fatal Root Error: TNetXNGFile::TNetXNGFile
        The remote file is not open
        ROOT severity: 3000
      ---- FatalRootError END
      
      The above exception was thrown while processing module WireCellToolkit/tpcrawdecoder run: 200033 subRun: 0 event: 1
    ---- FileReadError END
    Exception going through path simulate
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
  Fatal Root Error: TNetXNGFile::TNetXNGFile
  The remote file is not open
  ROOT severity: 3000
---- FatalRootError END
---- FatalRootError BEGIN
  Fatal Root Error: TNetXNGFile::Close
  [FATAL] Socket timeout
  ROOT severity: 3000
---- FatalRootError END
---- FatalRootError BEGIN
  Fatal Root Error: TNetXNGFile::Close
  [FATAL] Socket timeout
  ROOT severity: 3000
---- FatalRootError END
%MSG
../justin-jobscript: line 83:  1146 Segmentation fault      (core dumped) lar -c "${FCL}" -s "${INPUT_G4}" -o "${OUTFILE}" -n -1
ERROR: Output file mpvmpr_monte-carlo-011986-000033_gen_g4_detsim.root not produced!
justIN time: 2026-02-04 07:19:55 UTC       justIN version: 01.06.00