justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 236116.0@dunegpschedd02.fnal.gov

Jobsub ID236116.0@dunegpschedd02.fnal.gov
Workflow ID9405
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-11-01 21:08:41
SiteNL_NIKHEF
EntryVIRGO_NL_NIKHEF_juk
Last heartbeat2025-11-01 21:15:27
From worker nodeHostnamewn-pep-013.farm.nikhef.nl
cpuinfoIntel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Started2025-11-01 21:10:14
Input filesvd-protodune:np02vd_raw_run040285_0003_df-s04-d2_dw_0_20251028T162653.hdf5
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-11-01 21:15:27
Saved logsjustin-logs:236116.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

lter inputs:[ WireCell::IFrame ] outputs:[ WireCell::IFrame ](WireCell::IFrame:0)
[22:10:40.646] D [ pgraph ] connect <Node  type:WireCell::SigProc::ChannelSelector cat:3 sig:WireCell::IFrameFilter inputs:[ WireCell::IFrame ] outputs:[ WireCell::IFrame ]:(WireCell::IFrame:0) -> <Node  type:WireCell::SigProc::OmnibusSigProc cat:3 sig:WireCell::IFrameFilter inputs:[ WireCell::IFrame ] outputs:[ WireCell::IFrame ](WireCell::IFrame:0)
[22:10:40.646] D [ pgraph ] connect <Node  type:WireCell::SigProc::ChannelSelector cat:3 sig:WireCell::IFrameFilter inputs:[ WireCell::IFrame ] outputs:[ WireCell::IFrame ]:(WireCell::IFrame:0) -> <Node  type:WireCell::SigProc::OmnibusSigProc cat:3 sig:WireCell::IFrameFilter inputs:[ WireCell::IFrame ] outputs:[ WireCell::IFrame ](WireCell::IFrame:0)
Inputer: "wclsRawFrameSource"
Outputer: "wclsFrameSaver:spsaver"
wclsFrameSaver: promising to produce recob::Wires named "gauss"
wclsFrameSaver: promising to produce recob::Wires named "wiener"
db: runtime1761668813
Scale HitLimit based on readout window size 10000
HitLimit = 33333
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=8000 keeping as is
[22:12:20.697] D [  main  ] executing 1 apps, thread limit 0:
[22:12:20.697] D [  main  ] executing 1 apps, thread limit 0:
[22:12:20.697] D [  main  ] executing app: "Pgrapher"
[22:12:20.697] D [ pgraph ] <Pgrapher:> executing graph 
[22:12:20.697] D [ pgraph ] executing with 26 nodes
[22:12:20.700] D [  glue  ] <FrameFanout:nfsp> call=0: input: frame: ident=582 time=0 tick=512 with 12288 traces.  frame tags:[ "orig" ] 0 tagged trace sets:[ ] cmm:[ ] output 0: frame: ident=582 time=0 tick=512 with 12288 traces.  frame tags:[ "orig0" ] 0 tagged trace sets:[ ] cmm:[ ] output 1: frame: ident=582 time=0 tick=512 with 12288 traces.  frame tags:[ "orig1" ] 0 tagged trace sets:[ ] cmm:[ ] output 2: frame: ident=582 time=0 tick=512 with 12288 traces.  frame tags:[ "orig2" ] 0 tagged trace sets:[ ] cmm:[ ] output 3: frame: ident=582 time=0 tick=512 with 12288 traces.  frame tags:[ "orig3" ] 0 tagged trace sets:[ ] cmm:[ ] output 4: frame: ident=582 time=0 tick=512 with 12288 traces.  frame tags:[ "orig4" ] 0 tagged trace sets:[ ] cmm:[ ] output 5: frame: ident=582 time=0 tick=512 with 12288 traces.  frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] output 6: frame: ident=582 time=0 tick=512 with 12288 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] output 7: frame: ident=582 time=0 tick=512 with 12288 traces.  frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ]  
[22:12:20.701] W [  glue  ] <ChannelSelector:chsel7> Untagged summary not supported, summary will be dropped. 
[22:12:20.701] D [  glue  ] <ChannelSelector:chsel7> input frame: ident=582 time=0 tick=512 with 12288 traces.  frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=582 time=0 tick=512 with 1536 traces.  frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ] 
[22:12:20.702] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=0 input frame: frame: ident=582 time=0 tick=512 with 1536 traces.  frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ] 
[22:12:20.702] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=0 init nticks=8000 tbinmin=0 tbinmax=8000 
[22:12:20.759] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=0 load plane index: 0, ntraces=1536, input bad regions: 0 
[22:12:22.247] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=0 load plane index: 1, ntraces=1536, input bad regions: 0 
[22:12:23.770] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=0 load plane index: 2, ntraces=1536, input bad regions: 0 
[22:12:59.715] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=0 save plane index: 0, Qtot=41170355244 Qloss=-76661614054, 5989 indices spanning [15300,21288] "wiener" 
[22:12:59.916] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=0 save plane index: 0, Qtot=39686979609 Qloss=-76748473904, 5099 indices spanning [21289,26387] "gauss" 
[22:13:00.538] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=0 save plane index: 1, Qtot=15584993880 Qloss=-5011130936, 6964 indices spanning [26388,33351] "wiener" 
[22:13:00.733] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=0 save plane index: 1, Qtot=13470156196 Qloss=-5162459364, 6437 indices spanning [33352,39788] "gauss" 
[22:13:01.093] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=0 save plane index: 2, Qtot=5457835393 Qloss=-4878789537, 21519 indices spanning [39789,61307] "wiener" 
[22:13:01.457] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=0 save plane index: 2, Qtot=4422847382 Qloss=-3875293880, 21596 indices spanning [61308,82903] "gauss" 
[22:13:01.458] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=0 produce 82904 traces: 34472 wiener7, 0 decon_charge7, 33132 gauss7, frame tag: sigproc 
[22:13:01.458] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=0 output frame: frame: ident=582 time=0 tick=512 with 82904 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss7":33132 [0] "mp2_roi7":11231 [0] "mp3_roi7":4069 [0] "wiener7":34472 [34472] ] cmm:[ ] 
[22:13:07.325] W [  glue  ] <ChannelSelector:chsel6> Untagged summary not supported, summary will be dropped. 
[22:13:07.327] D [  glue  ] <ChannelSelector:chsel6> input frame: ident=582 time=0 tick=512 with 12288 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=582 time=0 tick=512 with 1536 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] 
[22:13:07.327] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=0 input frame: frame: ident=582 time=0 tick=512 with 1536 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] 
[22:13:07.327] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=0 init nticks=8000 tbinmin=0 tbinmax=8000 
[22:13:07.364] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=0 load plane index: 0, ntraces=1536, input bad regions: 0 
[22:13:09.169] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=0 load plane index: 1, ntraces=1536, input bad regions: 0 
[22:13:11.137] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=0 load plane index: 2, ntraces=1536, input bad regions: 0 
[22:13:14.671] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=0 save plane index: 0, Qtot=130139104 Qloss=-21997880, 2226 indices spanning [11192,13417] "wiener" 
[22:13:14.869] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=0 save plane index: 0, Qtot=126920944 Qloss=-19192292, 1817 indices spanning [13418,15234] "gauss" 
[22:13:15.217] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=0 save plane index: 1, Qtot=115468701 Qloss=-19498245, 2864 indices spanning [15235,18098] "wiener" 
[22:13:15.419] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=0 save plane index: 1, Qtot=109835102 Qloss=-16381127, 2402 indices spanning [18099,20500] "gauss" 
[22:13:15.792] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=0 save plane index: 2, Qtot=105164418 Qloss=-8234988, 4296 indices spanning [20501,24796] "wiener" 
[22:13:16.175] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=0 save plane index: 2, Qtot=101502947 Qloss=-4497306, 3512 indices spanning [24797,28308] "gauss" 
[22:13:16.175] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=0 produce 28309 traces: 9386 wiener6, 0 decon_charge6, 7731 gauss6, frame tag: sigproc 
[22:13:16.175] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=0 output frame: frame: ident=582 time=0 tick=512 with 28309 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss6":7731 [0] "mp2_roi6":9206 [0] "mp3_roi6":1986 [0] "wiener6":9386 [9386] ] cmm:[ ] 
[22:13:16.239] W [  glue  ] <ChannelSelector:chsel5> Untagged summary not supported, summary will be dropped. 
[22:13:16.240] D [  glue  ] <ChannelSelector:chsel5> input frame: ident=582 time=0 tick=512 with 12288 traces.  frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=582 time=0 tick=512 with 1536 traces.  frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] 
[22:13:16.240] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=0 input frame: frame: ident=582 time=0 tick=512 with 1536 traces.  frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] 
[22:13:16.240] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=0 init nticks=8000 tbinmin=0 tbinmax=8000 
[22:13:16.315] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=0 load plane index: 0, ntraces=1536, input bad regions: 0 
[22:13:18.195] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=0 load plane index: 1, ntraces=1536, input bad regions: 0 
[22:13:20.160] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=0 load plane index: 2, ntraces=1536, input bad regions: 0 
01-Nov-2025 22:15:07 CET  Opened output file with pattern "%ifb_reco_stage1_%tc_keepup.root"
Malformed TimeTracker database.  The TimeEvent table is empty, but
the TimeModule table is not.  This can happen if an exception has
been thrown from a module while processing the first event.  Any
saved database file is suspect and should not be used.

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8589.8 MB
  Peak resident set size usage (VmHWM): 6600.93 MB
  Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException:  PostEndJob 01-Nov-2025 22:15:09 CET ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- BadAlloc BEGIN
      A bad_alloc exception was thrown while processing module WireCellToolkit/wclsdatavd run: 40285 subRun: 1 event: 582
      The job has probably exhausted the virtual memory available to the process.
    ---- BadAlloc END
    Exception going through path produce
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
%MSG
Art has completed and will exit with status 1.
Error in reco1
justIN time: 2025-11-04 01:33:48 UTC       justIN version: 01.05.01