Jobsub ID 273990.0@dunegpschedd02.fnal.gov
| Jobsub ID | 273990.0@dunegpschedd02.fnal.gov |
| Workflow ID | 11986 |
| Stage ID | 3 |
| User name | usher@fnal.gov |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 28800 (8 hours) |
| Submitted time | 2026-01-17 11:58:22 |
| Site | UK_QMUL |
| Entry | DUNE_UK_London_QMUL_arcce02 |
| Last heartbeat | 2026-01-17 13:23:48 |
| From worker node | Hostname | cn017.htc.esc.qmul |
| cpuinfo | Intel(R) Xeon(R) Gold 6248R CPU @ 3.00GHz |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 171000 (47 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | jobscript_error |
| Started | 2026-01-17 12:23:02 |
| Input files | usertests:mpvmpr_monte-carlo-011986-000033_gen_g4.root
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Max RSS bytes | 0 (0 MiB) |
| Outputting started | |
| Output files | |
| Finished | 2026-01-17 13:23:48 |
| Saved logs | justin-logs:273990.0-dunegpschedd02.fnal.gov.logs.tgz |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
fs/larsoft.opensciencegrid.org/products
srproxy v00.45 -f NULL -q py3915 -z /cvmfs/dune.opensciencegrid.org/products/dune
systematicstools v01_04_04 -f Linux64bit+3.10-2.17 -q e26:prof -z /cvmfs/larsoft.opensciencegrid.org/products
tbb v2021_9_0 -f Linux64bit+3.10-2.17 -q e26 -z /cvmfs/larsoft.opensciencegrid.org/products
tensorflow v2_15_0 -f Linux64bit+3.10-2.17 -q e26:p3915 -z /cvmfs/larsoft.opensciencegrid.org/products
torch_scatter v2_1_2a -f Linux64bit+3.10-2.17 -q e26 -z /cvmfs/larsoft.opensciencegrid.org/products
TRACE v3_17_11 -f NULL -z /cvmfs/larsoft.opensciencegrid.org/products
triton v2_25_0d -f Linux64bit+3.10-2.17 -q e26 -z /cvmfs/larsoft.opensciencegrid.org/products
ups v6_0_7 -f Linux64bit+3.10-2.17 -z /cvmfs/dune.opensciencegrid.org/products/dune
valgrind v3_17_0 -f Linux64bit+3.10-2.17 -z /cvmfs/larsoft.opensciencegrid.org/products
wirecell v0_32_1 -f Linux64bit+3.10-2.17 -q e26:prof -z /cvmfs/larsoft.opensciencegrid.org/products
xerces_c v3_2_3e -f Linux64bit+3.10-2.17 -q e26 -z /cvmfs/larsoft.opensciencegrid.org/products
xrootd v5_5_5a -f Linux64bit+3.10-2.17 -q e26:p3915:prof -z /cvmfs/larsoft.opensciencegrid.org/products
art 3.14.04
Running lar DETSIM:
FCL: standard_detsim_dune10kt_1x2x6.fcl
Input: root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/usertests/d9/a6/mpvmpr_monte-carlo-011986-000033_gen_g4.root
Output: mpvmpr_monte-carlo-011986-000033_gen_g4_detsim.root
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/dune.opensciencegrid.org/products/dune/dunecore/v10_16_00d00/gdml/dune10kt_v6_refactored_1x2x6.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Error: Unsupported GDML Tag Used :gdml_simple_extension. Please Check Geometry/Schema.
Error: Unsupported GDML Tag Used :extension. Please Check Geometry/Schema.
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 1149
Info in <TGeoManager::CloseGeometry>: 67545 nodes/ 5439 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
registering to primaryGeneratorActionsMap_
registering to eventActionsMap_
registering to trackingActionsMap_
registering to steppingActionsMap_
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
Using TESTbench spe!!!
With: SPE_DAPHNE2_FBK_2022.dat
[12:23:44.817] I [ wct ] loading compressed json file: /cvmfs/dune.opensciencegrid.org/products/dune/dunereco/v10_16_00d00/wire-cell-cfg/dune10kt-1x2x6-wires-larsoft-v1.json.bz2
[12:24:03.254] I [ wct ] loading compressed json file: /cvmfs/dune.opensciencegrid.org/products/dune/dunereco/v10_16_00d00/wire-cell-cfg/dune-garfield-1d60563.json.bz2
[12:24:08.373] I [ wct ] loading compressed json file: /cvmfs/dune.opensciencegrid.org/products/dune/dune_pardata/v01_84_00/WireCellData/protodune-noise-spectra-v1.json.bz2
[12:24:08.397] W [sigproc ] <OmnibusSigProc:apa0sigproc0> The 'wiener_threshold_tag' is obsolete, thresholds in summary on 'wiener' tagged traces
[12:24:08.410] W [sigproc ] <OmnibusSigProc:apa1sigproc1> The 'wiener_threshold_tag' is obsolete, thresholds in summary on 'wiener' tagged traces
[12:24:08.423] W [sigproc ] <OmnibusSigProc:apa2sigproc2> The 'wiener_threshold_tag' is obsolete, thresholds in summary on 'wiener' tagged traces
[12:24:08.436] W [sigproc ] <OmnibusSigProc:apa3sigproc3> The 'wiener_threshold_tag' is obsolete, thresholds in summary on 'wiener' tagged traces
[12:24:08.459] W [sigproc ] <OmnibusSigProc:apa4sigproc4> The 'wiener_threshold_tag' is obsolete, thresholds in summary on 'wiener' tagged traces
[12:24:08.472] W [sigproc ] <OmnibusSigProc:apa5sigproc5> The 'wiener_threshold_tag' is obsolete, thresholds in summary on 'wiener' tagged traces
[12:24:08.485] W [sigproc ] <OmnibusSigProc:apa6sigproc6> The 'wiener_threshold_tag' is obsolete, thresholds in summary on 'wiener' tagged traces
[12:24:08.498] W [sigproc ] <OmnibusSigProc:apa7sigproc7> The 'wiener_threshold_tag' is obsolete, thresholds in summary on 'wiener' tagged traces
[12:24:08.512] W [sigproc ] <OmnibusSigProc:apa8sigproc8> The 'wiener_threshold_tag' is obsolete, thresholds in summary on 'wiener' tagged traces
[12:24:08.525] W [sigproc ] <OmnibusSigProc:apa9sigproc9> The 'wiener_threshold_tag' is obsolete, thresholds in summary on 'wiener' tagged traces
[12:24:08.538] W [sigproc ] <OmnibusSigProc:apa10sigproc10> The 'wiener_threshold_tag' is obsolete, thresholds in summary on 'wiener' tagged traces
[12:24:08.550] W [sigproc ] <OmnibusSigProc:apa11sigproc11> The 'wiener_threshold_tag' is obsolete, thresholds in summary on 'wiener' tagged traces
Inputer: "wclsSimDepoSetSource:"
Outputer: "wclsDepoFluxWriter:postdrift"
Outputer: "wclsFrameSaver:spsignals"
wclsFrameSaver: promising to produce recob::Wires named "gauss"
wclsFrameSaver: promising to produce recob::Wires named "wiener"
wclsFrameSaver: promising to produce recob::Wires named "dnnsp"
17-Jan-2026 12:24:08 GMT Initiating request to open input file "root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/usertests/d9/a6/mpvmpr_monte-carlo-011986-000033_gen_g4.root"
17-Jan-2026 12:27:11 GMT Opened input file "root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/usertests/d9/a6/mpvmpr_monte-carlo-011986-000033_gen_g4.root"
Begin processing the 1st record. run: 200033 subRun: 0 event: 1 at 17-Jan-2026 12:30:57 GMT
SimDepoSetSource got 15240 depos from art tag "InputTag: label = 'IonAndScint', instance = ''" returns: okay
SimDepoSetSource: ready with 15240 depos spanning: [7.33254, 7.34512]us
[12:41:42.802] W [ gen ] <IncoherentAddNoise:addnoiseapa4> model EmpiricalNoiseModel:empericalnoisemeganodes produced 6174 samples instead of expected 6000, future warnings muted
Retagger: tagging trace set: wiener with 5120 traces, 5120 summary
Retagger: tagging trace set: gauss with 5120 traces, 0 summary
wclsFrameSaver: saving 5120 traces tagged "gauss"
FrameSaver: q=434492 n=56452 tag=gauss
wclsFrameSaver: saving 5120 traces tagged "wiener"
FrameSaver: q=450605 n=53712 tag=wiener
wclsFrameSaver: no traces tagged "dnnsp"
FrameSaver: q=0 n=0 tag=dnnsp
17-Jan-2026 12:42:29 GMT Opened output file with pattern "mpvmpr_monte-carlo-011986-000033_gen_g4_detsim.root"
================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
================================================================================================================================
Full event 386.177 386.177 386.177 386.177 0 1
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.527761 0.527761 0.527761 0.527761 0 1
simulate:tpcrawdecoder:WireCellToolkit 385.156 385.156 385.156 385.156 0 1
simulate:opdigi:OpDetDigitizerDUNE 0.491447 0.491447 0.491447 0.491447 0 1
simulate:rns:RandomNumberSaver 0.000515978 0.000515978 0.000515978 0.000515978 0 1
[art]:TriggerResults:TriggerResultInserter 6.701e-05 6.701e-05 6.701e-05 6.701e-05 0 1
end_path:out1:RootOutput 3.2427e-05 3.2427e-05 3.2427e-05 3.2427e-05 0 1
================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 2963.89 MB
Peak resident set size usage (VmHWM): 1270.33 MB
====================================================================================================
%MSG-s ArtException: PostEndJob 17-Jan-2026 13:20:46 GMT ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- FatalRootError BEGIN
Fatal Root Error: TNetXNGFile::ReadBuffer
[FATAL] Socket timeout
ROOT severity: 3000
---- FatalRootError END
---- EventProcessorFailure END
---- FatalRootError BEGIN
Fatal Root Error: TNetXNGFile::Close
[FATAL] Socket timeout
ROOT severity: 3000
---- FatalRootError END
---- FatalRootError BEGIN
Fatal Root Error: TNetXNGFile::Close
[FATAL] Socket timeout
ROOT severity: 3000
---- FatalRootError END
%MSG
../justin-jobscript: line 83: 1146 Segmentation fault (core dumped) lar -c "${FCL}" -s "${INPUT_G4}" -o "${OUTFILE}" -n -1
ERROR: Output file mpvmpr_monte-carlo-011986-000033_gen_g4_detsim.root not produced! 