justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 273857.1@dunegpschedd02.fnal.gov

Jobsub ID273857.1@dunegpschedd02.fnal.gov
Workflow ID11995
Stage ID3
User nameusher@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes6291456000 (6000 MiB)
Wall seconds limit28800 (8 hours)
Submitted time2026-01-16 22:27:39
SiteUK_QMUL
EntryDUNE_UK_London_QMUL_arcce02
Last heartbeat2026-01-16 23:01:44
From worker nodeHostnamecn017.htc.esc.qmul
cpuinfoIntel(R) Xeon(R) Gold 6248R CPU @ 3.00GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes7864320000 (7500 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Started2026-01-16 22:37:18
Input filesusertests:mpvmpr_monte-carlo-011995-000041_gen_g4.root
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2026-01-16 23:01:44
Saved logsjustin-logs:273857.1-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

of  -z /cvmfs/larsoft.opensciencegrid.org/products
pythia            v6_4_28x        -f Linux64bit+3.10-2.17 -q e26:prof        -z /cvmfs/larsoft.opensciencegrid.org/products
python            v3_9_15         -f Linux64bit+3.10-2.17                    -z /cvmfs/larsoft.opensciencegrid.org/products
python_future_six_request  v1_3_1          -f Linux64bit+3.10-2.17 -q python3.9       -z /cvmfs/fermilab.opensciencegrid.org/products/common/db/
range             v3_0_12_0       -f NULL                                    -z /cvmfs/larsoft.opensciencegrid.org/products
root              v6_28_12        -f Linux64bit+3.10-2.17 -q e26:p3915:prof  -z /cvmfs/larsoft.opensciencegrid.org/products
rstartree         v2016_07        -f NULL                                    -z /cvmfs/larsoft.opensciencegrid.org/products
sam_web_client    v3_6            -f NULL                                    -z /cvmfs/fermilab.opensciencegrid.org/products/common/db/
scitokens_cpp     v1_0_1a         -f Linux64bit+3.10-2.17 -q e26:sql34001    -z /cvmfs/larsoft.opensciencegrid.org/products
spdlog            v1_9_2          -f Linux64bit+3.10-2.17 -q e26:prof        -z /cvmfs/larsoft.opensciencegrid.org/products
sqlite            v3_40_01_00     -f Linux64bit+3.10-2.17                    -z /cvmfs/larsoft.opensciencegrid.org/products
srproxy           v00.45          -f NULL                 -q py3915          -z /cvmfs/dune.opensciencegrid.org/products/dune
systematicstools  v01_04_04       -f Linux64bit+3.10-2.17 -q e26:prof        -z /cvmfs/larsoft.opensciencegrid.org/products
tbb               v2021_9_0       -f Linux64bit+3.10-2.17 -q e26             -z /cvmfs/larsoft.opensciencegrid.org/products
tensorflow        v2_15_0         -f Linux64bit+3.10-2.17 -q e26:p3915       -z /cvmfs/larsoft.opensciencegrid.org/products
torch_scatter     v2_1_2a         -f Linux64bit+3.10-2.17 -q e26             -z /cvmfs/larsoft.opensciencegrid.org/products
TRACE             v3_17_11        -f NULL                                    -z /cvmfs/larsoft.opensciencegrid.org/products
triton            v2_25_0d        -f Linux64bit+3.10-2.17 -q e26             -z /cvmfs/larsoft.opensciencegrid.org/products
ups               v6_0_7          -f Linux64bit+3.10-2.17                    -z /cvmfs/dune.opensciencegrid.org/products/dune
valgrind          v3_17_0         -f Linux64bit+3.10-2.17                    -z /cvmfs/larsoft.opensciencegrid.org/products
wirecell          v0_32_1         -f Linux64bit+3.10-2.17 -q e26:prof        -z /cvmfs/larsoft.opensciencegrid.org/products
xerces_c          v3_2_3e         -f Linux64bit+3.10-2.17 -q e26             -z /cvmfs/larsoft.opensciencegrid.org/products
xrootd            v5_5_5a         -f Linux64bit+3.10-2.17 -q e26:p3915:prof  -z /cvmfs/larsoft.opensciencegrid.org/products
art 3.14.04
Running lar DETSIM:
  FCL:     standard_detsim_dune10kt_1x2x6.fcl
  Input:   root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/usertests/e7/35/mpvmpr_monte-carlo-011995-000041_gen_g4.root
  Output:  mpvmpr_monte-carlo-011995-000041_gen_g4_detsim.root
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/dune.opensciencegrid.org/products/dune/dunecore/v10_16_00d00/gdml/dune10kt_v6_refactored_1x2x6.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Error: Unsupported GDML Tag Used :gdml_simple_extension. Please Check Geometry/Schema.
Error: Unsupported GDML Tag Used :extension. Please Check Geometry/Schema.
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 1149
Info in <TGeoManager::CloseGeometry>: 67545 nodes/ 5439 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
registering to  primaryGeneratorActionsMap_
registering to   eventActionsMap_
registering to   trackingActionsMap_
registering to  steppingActionsMap_
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
Using TESTbench spe!!! 
With: SPE_DAPHNE2_FBK_2022.dat
[22:37:50.511] I [  wct   ] loading compressed json file: /cvmfs/dune.opensciencegrid.org/products/dune/dunereco/v10_16_00d00/wire-cell-cfg/dune10kt-1x2x6-wires-larsoft-v1.json.bz2
[22:38:01.618] I [  wct   ] loading compressed json file: /cvmfs/dune.opensciencegrid.org/products/dune/dunereco/v10_16_00d00/wire-cell-cfg/dune-garfield-1d60563.json.bz2
[22:38:04.502] I [  wct   ] loading compressed json file: /cvmfs/dune.opensciencegrid.org/products/dune/dune_pardata/v01_84_00/WireCellData/protodune-noise-spectra-v1.json.bz2
[22:38:04.515] W [sigproc ] <OmnibusSigProc:apa0sigproc0> The 'wiener_threshold_tag' is obsolete, thresholds in summary on 'wiener' tagged traces 
[22:38:04.522] W [sigproc ] <OmnibusSigProc:apa1sigproc1> The 'wiener_threshold_tag' is obsolete, thresholds in summary on 'wiener' tagged traces 
[22:38:04.529] W [sigproc ] <OmnibusSigProc:apa2sigproc2> The 'wiener_threshold_tag' is obsolete, thresholds in summary on 'wiener' tagged traces 
[22:38:04.536] W [sigproc ] <OmnibusSigProc:apa3sigproc3> The 'wiener_threshold_tag' is obsolete, thresholds in summary on 'wiener' tagged traces 
[22:38:04.542] W [sigproc ] <OmnibusSigProc:apa4sigproc4> The 'wiener_threshold_tag' is obsolete, thresholds in summary on 'wiener' tagged traces 
[22:38:04.549] W [sigproc ] <OmnibusSigProc:apa5sigproc5> The 'wiener_threshold_tag' is obsolete, thresholds in summary on 'wiener' tagged traces 
[22:38:04.556] W [sigproc ] <OmnibusSigProc:apa6sigproc6> The 'wiener_threshold_tag' is obsolete, thresholds in summary on 'wiener' tagged traces 
[22:38:04.563] W [sigproc ] <OmnibusSigProc:apa7sigproc7> The 'wiener_threshold_tag' is obsolete, thresholds in summary on 'wiener' tagged traces 
[22:38:04.570] W [sigproc ] <OmnibusSigProc:apa8sigproc8> The 'wiener_threshold_tag' is obsolete, thresholds in summary on 'wiener' tagged traces 
[22:38:04.576] W [sigproc ] <OmnibusSigProc:apa9sigproc9> The 'wiener_threshold_tag' is obsolete, thresholds in summary on 'wiener' tagged traces 
[22:38:04.584] W [sigproc ] <OmnibusSigProc:apa10sigproc10> The 'wiener_threshold_tag' is obsolete, thresholds in summary on 'wiener' tagged traces 
[22:38:04.591] W [sigproc ] <OmnibusSigProc:apa11sigproc11> The 'wiener_threshold_tag' is obsolete, thresholds in summary on 'wiener' tagged traces 
Inputer: "wclsSimDepoSetSource:"
Outputer: "wclsDepoFluxWriter:postdrift"
Outputer: "wclsFrameSaver:spsignals"
wclsFrameSaver: promising to produce recob::Wires named "gauss"
wclsFrameSaver: promising to produce recob::Wires named "wiener"
wclsFrameSaver: promising to produce recob::Wires named "dnnsp"
16-Jan-2026 22:38:04 GMT  Initiating request to open input file "root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/usertests/e7/35/mpvmpr_monte-carlo-011995-000041_gen_g4.root"
16-Jan-2026 22:40:47 GMT  Opened input file "root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/usertests/e7/35/mpvmpr_monte-carlo-011995-000041_gen_g4.root"
Begin processing the 1st record. run: 200041 subRun: 0 event: 1 at 16-Jan-2026 22:41:04 GMT
16-Jan-2026 22:50:22 GMT  Opened output file with pattern "mpvmpr_monte-carlo-011995-000041_gen_g4_detsim.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
[ No processed events ]
====================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 2647.29 MB
  Peak resident set size usage (VmHWM): 955.515 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 16-Jan-2026 22:50:23 GMT ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- FileReadError BEGIN
      ---- FatalRootError BEGIN
        Fatal Root Error: TNetXNGFile::TNetXNGFile
        The remote file is not open
        ROOT severity: 3000
      ---- FatalRootError END
      
      The above exception was thrown while processing module WireCellToolkit/tpcrawdecoder run: 200041 subRun: 0 event: 1
    ---- FileReadError END
    Exception going through path simulate
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
  Fatal Root Error: TNetXNGFile::TNetXNGFile
  The remote file is not open
  ROOT severity: 3000
---- FatalRootError END
---- FatalRootError BEGIN
  Fatal Root Error: TNetXNGFile::Close
  [FATAL] Socket timeout
  ROOT severity: 3000
---- FatalRootError END
---- FatalRootError BEGIN
  Fatal Root Error: TNetXNGFile::Close
  [FATAL] Socket timeout
  ROOT severity: 3000
---- FatalRootError END
%MSG
../justin-jobscript: line 83:  1146 Segmentation fault      (core dumped) lar -c "${FCL}" -s "${INPUT_G4}" -o "${OUTFILE}" -n -1
ERROR: Output file mpvmpr_monte-carlo-011995-000041_gen_g4_detsim.root not produced!
justIN time: 2026-02-04 08:57:04 UTC       justIN version: 01.06.00