justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 241774.1@dunegpschedd01.fnal.gov

Jobsub ID241774.1@dunegpschedd01.fnal.gov
Workflow ID9373
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-11-01 21:04:41
SiteNL_NIKHEF
EntryVIRGO_NL_NIKHEF_juk
Last heartbeat2025-11-01 21:12:23
From worker nodeHostnamewn-pep-011.farm.nikhef.nl
cpuinfoIntel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Started2025-11-01 21:08:14
Input filesvd-protodune:np02vd_raw_run040266_0161_df-s04-d2_dw_0_20251025T014050.hdf5
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-11-01 21:12:23
Saved logsjustin-logs:241774.1-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

chsel6> input frame: ident=29742 time=0 tick=512 with 12288 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=29742 time=0 tick=512 with 1536 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] 
[22:10:17.108] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=0 input frame: frame: ident=29742 time=0 tick=512 with 1536 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] 
[22:10:17.108] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=0 init nticks=8864 tbinmin=0 tbinmax=8864 
[22:10:17.142] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=0 load plane index: 0, ntraces=1536, input bad regions: 0 
[22:10:19.871] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=0 load plane index: 1, ntraces=1536, input bad regions: 0 
[22:10:22.686] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=0 load plane index: 2, ntraces=1536, input bad regions: 0 
[22:10:27.173] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=0 save plane index: 0, Qtot=139904337 Qloss=-21973046, 3042 indices spanning [19892,22933] "wiener" 
[22:10:27.463] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=0 save plane index: 0, Qtot=134741811 Qloss=-18573142, 2438 indices spanning [22934,25371] "gauss" 
[22:10:27.934] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=0 save plane index: 1, Qtot=116622360 Qloss=-18780325, 3437 indices spanning [25372,28808] "wiener" 
[22:10:28.225] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=0 save plane index: 1, Qtot=108276595 Qloss=-15479645, 2776 indices spanning [28809,31584] "gauss" 
[22:10:28.691] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=0 save plane index: 2, Qtot=115909905 Qloss=-8503677, 4164 indices spanning [31585,35748] "wiener" 
[22:10:29.140] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=0 save plane index: 2, Qtot=111641517 Qloss=-4175959, 3399 indices spanning [35749,39147] "gauss" 
[22:10:29.141] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=0 produce 39148 traces: 10643 wiener6, 0 decon_charge6, 8613 gauss6, frame tag: sigproc 
[22:10:29.141] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=0 output frame: frame: ident=29742 time=0 tick=512 with 39148 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss6":8613 [0] "mp2_roi6":17102 [0] "mp3_roi6":2790 [0] "wiener6":10643 [10643] ] cmm:[ ] 
[22:10:29.182] W [  glue  ] <ChannelSelector:chsel5> Untagged summary not supported, summary will be dropped. 
[22:10:29.182] D [  glue  ] <ChannelSelector:chsel5> input frame: ident=29742 time=0 tick=512 with 12288 traces.  frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=29742 time=0 tick=512 with 1536 traces.  frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] 
[22:10:29.183] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=0 input frame: frame: ident=29742 time=0 tick=512 with 1536 traces.  frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] 
[22:10:29.183] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=0 init nticks=8864 tbinmin=0 tbinmax=8864 
[22:10:29.223] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=0 load plane index: 0, ntraces=1536, input bad regions: 0 
[22:10:31.923] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=0 load plane index: 1, ntraces=1536, input bad regions: 0 
[22:10:34.663] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=0 load plane index: 2, ntraces=1536, input bad regions: 0 
[22:10:39.929] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=0 save plane index: 0, Qtot=181974172 Qloss=-29315163, 3860 indices spanning [21267,25126] "wiener" 
[22:10:40.216] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=0 save plane index: 0, Qtot=174776806 Qloss=-24192117, 3150 indices spanning [25127,28276] "gauss" 
[22:10:40.710] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=0 save plane index: 1, Qtot=170351235 Qloss=-26670054, 4556 indices spanning [28277,32832] "wiener" 
[22:10:40.998] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=0 save plane index: 1, Qtot=160286060 Qloss=-21306470, 3749 indices spanning [32833,36581] "gauss" 
[22:10:41.463] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=0 save plane index: 2, Qtot=168011874 Qloss=-14586177, 6324 indices spanning [36582,42905] "wiener" 
[22:10:41.911] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=0 save plane index: 2, Qtot=161651199 Qloss=-8018879, 5166 indices spanning [42906,48071] "gauss" 
[22:10:41.911] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=0 produce 48072 traces: 14740 wiener5, 0 decon_charge5, 12065 gauss5, frame tag: sigproc 
[22:10:41.911] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=0 output frame: frame: ident=29742 time=0 tick=512 with 48072 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss5":12065 [0] "mp2_roi5":17798 [0] "mp3_roi5":3469 [0] "wiener5":14740 [14740] ] cmm:[ ] 
[22:10:41.975] W [  glue  ] <ChannelSelector:chsel4> Untagged summary not supported, summary will be dropped. 
[22:10:41.975] D [  glue  ] <ChannelSelector:chsel4> input frame: ident=29742 time=0 tick=512 with 12288 traces.  frame tags:[ "orig4" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=29742 time=0 tick=512 with 1536 traces.  frame tags:[ "orig4" ] 0 tagged trace sets:[ ] cmm:[ ] 
[22:10:41.976] D [sigproc ] <OmnibusSigProc:anode4sigproc4> call=0 input frame: frame: ident=29742 time=0 tick=512 with 1536 traces.  frame tags:[ "orig4" ] 0 tagged trace sets:[ ] cmm:[ ] 
[22:10:41.976] D [sigproc ] <OmnibusSigProc:anode4sigproc4> call=0 init nticks=8864 tbinmin=0 tbinmax=8864 
[22:10:42.012] D [sigproc ] <OmnibusSigProc:anode4sigproc4> call=0 load plane index: 0, ntraces=1536, input bad regions: 0 
[22:10:44.698] D [sigproc ] <OmnibusSigProc:anode4sigproc4> call=0 load plane index: 1, ntraces=1536, input bad regions: 0 
[22:10:47.612] D [sigproc ] <OmnibusSigProc:anode4sigproc4> call=0 load plane index: 2, ntraces=1536, input bad regions: 0 
[22:10:52.110] D [sigproc ] <OmnibusSigProc:anode4sigproc4> call=0 save plane index: 0, Qtot=136356294 Qloss=-20785240, 3046 indices spanning [17660,20705] "wiener" 
[22:10:52.396] D [sigproc ] <OmnibusSigProc:anode4sigproc4> call=0 save plane index: 0, Qtot=130749214 Qloss=-16651096, 2437 indices spanning [20706,23142] "gauss" 
[22:10:52.849] D [sigproc ] <OmnibusSigProc:anode4sigproc4> call=0 save plane index: 1, Qtot=131211465 Qloss=-20647682, 3461 indices spanning [23143,26603] "wiener" 
[22:10:53.129] D [sigproc ] <OmnibusSigProc:anode4sigproc4> call=0 save plane index: 1, Qtot=122761300 Qloss=-16483235, 2896 indices spanning [26604,29499] "gauss" 
[22:10:53.598] D [sigproc ] <OmnibusSigProc:anode4sigproc4> call=0 save plane index: 2, Qtot=123256155 Qloss=-11345720, 4710 indices spanning [29500,34209] "wiener" 
[22:10:54.049] D [sigproc ] <OmnibusSigProc:anode4sigproc4> call=0 save plane index: 2, Qtot=118148651 Qloss=-6033912, 3844 indices spanning [34210,38053] "gauss" 
[22:10:54.050] D [sigproc ] <OmnibusSigProc:anode4sigproc4> call=0 produce 38054 traces: 11217 wiener4, 0 decon_charge4, 9177 gauss4, frame tag: sigproc 
[22:10:54.050] D [sigproc ] <OmnibusSigProc:anode4sigproc4> call=0 output frame: frame: ident=29742 time=0 tick=512 with 38054 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss4":9177 [0] "mp2_roi4":15041 [0] "mp3_roi4":2619 [0] "wiener4":11217 [11217] ] cmm:[ ] 
[22:10:54.096] W [  glue  ] <ChannelSelector:chsel3> Untagged summary not supported, summary will be dropped. 
[22:10:54.097] D [  glue  ] <ChannelSelector:chsel3> input frame: ident=29742 time=0 tick=512 with 12288 traces.  frame tags:[ "orig3" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=29742 time=0 tick=512 with 1536 traces.  frame tags:[ "orig3" ] 0 tagged trace sets:[ ] cmm:[ ] 
[22:10:54.098] D [  aux   ] <Resampler:resmp3> first ch=4539 Ts=512 Ns=8657 Ns_pad=8750 Nrat=125 Tr=500 Nr=8960 Nout=8864 padding:linear 
[22:10:54.358] D [  aux   ] <Resampler:resmp3> resample 1536 traces at call=0 
[22:10:54.358] D [sigproc ] <OmnibusSigProc:anode3sigproc3> call=0 input frame: frame: ident=29742 time=0 tick=500 with 1536 traces.  frame tags:[ "orig3" ] 0 tagged trace sets:[ ] cmm:[ ] 
[22:10:54.359] D [sigproc ] <OmnibusSigProc:anode3sigproc3> call=0 init nticks=8864 tbinmin=0 tbinmax=8864 
[22:10:54.391] D [sigproc ] <OmnibusSigProc:anode3sigproc3> call=0 load plane index: 0, ntraces=1536, input bad regions: 0 
[22:10:57.042] D [sigproc ] <OmnibusSigProc:anode3sigproc3> call=0 load plane index: 1, ntraces=1536, input bad regions: 0 
[22:10:59.735] D [sigproc ] <OmnibusSigProc:anode3sigproc3> call=0 load plane index: 2, ntraces=1536, input bad regions: 0 
01-Nov-2025 22:12:04 CET  Opened output file with pattern "%ifb_reco_stage1_%tc_keepup.root"
Malformed TimeTracker database.  The TimeEvent table is empty, but
the TimeModule table is not.  This can happen if an exception has
been thrown from a module while processing the first event.  Any
saved database file is suspect and should not be used.

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8589.84 MB
  Peak resident set size usage (VmHWM): 6597.08 MB
  Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException:  PostEndJob 01-Nov-2025 22:12:05 CET ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- BadAlloc BEGIN
      A bad_alloc exception was thrown while processing module WireCellToolkit/wclsdatavd run: 40266 subRun: 1 event: 29742
      The job has probably exhausted the virtual memory available to the process.
    ---- BadAlloc END
    Exception going through path produce
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
%MSG
Art has completed and will exit with status 1.
Error in reco1
justIN time: 2025-11-04 01:28:09 UTC       justIN version: 01.05.01