justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 47246.1@dunegpschedd01.fnal.gov

Jobsub ID47246.1@dunegpschedd01.fnal.gov
Workflow ID2576
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-09-16 23:52:29
SiteNL_NIKHEF
EntryVIRGO_NL_NIKHEF_brug
Last heartbeat2025-09-17 01:09:48
From worker nodeHostnamewn-sate-030.farm.nikhef.nl
cpuinfoAMD EPYC 7551P 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Started2025-09-16 23:53:24
Input filesvd-protodune:np02vd_raw_run039275_0146_df-s02-d1_dw_0_20250901T191605.hdf5
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-09-17 01:09:48
Saved logsjustin-logs:47246.1-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

17-Sep-2025 03:08:24 CEST  run: 39275 subRun: 1 event: 52843
Error! unrecognized channel number -1. Ignoring pulse
%MSG
%MSG-e OpHitFinder:  OpHitFinder:ophit@BeginModule 17-Sep-2025 03:08:24 CEST  run: 39275 subRun: 1 event: 52843
Error! unrecognized channel number -1. Ignoring pulse
%MSG
%MSG-e OpHitFinder:  OpHitFinder:ophit@BeginModule 17-Sep-2025 03:08:24 CEST  run: 39275 subRun: 1 event: 52843
Error! unrecognized channel number -1. Ignoring pulse
%MSG
%MSG-e OpHitFinder:  OpHitFinder:ophit@BeginModule 17-Sep-2025 03:08:24 CEST  run: 39275 subRun: 1 event: 52843
Error! unrecognized channel number -1. Ignoring pulse
%MSG
%MSG-e OpHitFinder:  OpHitFinder:ophit@BeginModule 17-Sep-2025 03:08:24 CEST  run: 39275 subRun: 1 event: 52843
Error! unrecognized channel number -1. Ignoring pulse
%MSG
%MSG-e OpHitFinder:  OpHitFinder:ophit@BeginModule 17-Sep-2025 03:08:24 CEST  run: 39275 subRun: 1 event: 52843
Error! unrecognized channel number -1. Ignoring pulse
%MSG
%MSG-e OpHitFinder:  OpHitFinder:ophit@BeginModule 17-Sep-2025 03:08:24 CEST  run: 39275 subRun: 1 event: 52843
Error! unrecognized channel number -1. Ignoring pulse
%MSG
%MSG-e OpHitFinder:  OpHitFinder:ophit@BeginModule 17-Sep-2025 03:08:24 CEST  run: 39275 subRun: 1 event: 52843
Error! unrecognized channel number -1. Ignoring pulse
%MSG
%MSG-e OpHitFinder:  OpHitFinder:ophit@BeginModule 17-Sep-2025 03:08:24 CEST  run: 39275 subRun: 1 event: 52843
Error! unrecognized channel number -1. Ignoring pulse
%MSG
%MSG-e OpHitFinder:  OpHitFinder:ophit@BeginModule 17-Sep-2025 03:08:24 CEST  run: 39275 subRun: 1 event: 52843
Error! unrecognized channel number -1. Ignoring pulse
%MSG
%MSG-e OpHitFinder:  OpHitFinder:ophit@BeginModule 17-Sep-2025 03:08:24 CEST  run: 39275 subRun: 1 event: 52843
Error! unrecognized channel number -1. Ignoring pulse
%MSG
%MSG-e OpHitFinder:  OpHitFinder:ophit@BeginModule 17-Sep-2025 03:08:24 CEST  run: 39275 subRun: 1 event: 52843
Error! unrecognized channel number -1. Ignoring pulse
%MSG
%MSG-e OpHitFinder:  OpHitFinder:ophit@BeginModule 17-Sep-2025 03:08:24 CEST  run: 39275 subRun: 1 event: 52843
Error! unrecognized channel number -1. Ignoring pulse
%MSG
%MSG-e OpHitFinder:  OpHitFinder:ophit@BeginModule 17-Sep-2025 03:08:24 CEST  run: 39275 subRun: 1 event: 52843
Error! unrecognized channel number -1. Ignoring pulse
%MSG
%MSG-e OpHitFinder:  OpHitFinder:ophit@BeginModule 17-Sep-2025 03:08:24 CEST  run: 39275 subRun: 1 event: 52843
Error! unrecognized channel number -1. Ignoring pulse
%MSG
%MSG-e OpHitFinder:  OpHitFinder:ophit@BeginModule 17-Sep-2025 03:08:24 CEST  run: 39275 subRun: 1 event: 52843
Error! unrecognized channel number -1. Ignoring pulse
%MSG
%MSG-e OpHitFinder:  OpHitFinder:ophit@BeginModule 17-Sep-2025 03:08:24 CEST  run: 39275 subRun: 1 event: 52843
Error! unrecognized channel number -1. Ignoring pulse
%MSG
%MSG-e OpHitFinder:  OpHitFinder:ophit@BeginModule 17-Sep-2025 03:08:24 CEST  run: 39275 subRun: 1 event: 52843
Error! unrecognized channel number -1. Ignoring pulse
%MSG
%MSG-e OpHitFinder:  OpHitFinder:ophit@BeginModule 17-Sep-2025 03:08:24 CEST  run: 39275 subRun: 1 event: 52843
Error! unrecognized channel number -1. Ignoring pulse
%MSG
%MSG-e OpHitFinder:  OpHitFinder:ophit@BeginModule 17-Sep-2025 03:08:24 CEST  run: 39275 subRun: 1 event: 52843
Error! unrecognized channel number -1. Ignoring pulse
%MSG
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=10048 keeping as is
[03:08:24.451] D [  main  ] executing 1 apps, thread limit 0:
[03:08:24.451] D [  main  ] executing 1 apps, thread limit 0:
[03:08:24.451] D [  main  ] executing app: "Pgrapher"
[03:08:24.451] D [ pgraph ] <Pgrapher:> executing graph 
[03:08:24.451] D [ pgraph ] executing with 26 nodes
[03:08:24.454] D [  glue  ] <FrameFanout:nfsp> call=28: input: frame: ident=52843 time=340 tick=512 with 12288 traces.  frame tags:[ "orig" ] 0 tagged trace sets:[ ] cmm:[ ] output 0: frame: ident=52843 time=340 tick=512 with 12288 traces.  frame tags:[ "orig0" ] 0 tagged trace sets:[ ] cmm:[ ] output 1: frame: ident=52843 time=340 tick=512 with 12288 traces.  frame tags:[ "orig1" ] 0 tagged trace sets:[ ] cmm:[ ] output 2: frame: ident=52843 time=340 tick=512 with 12288 traces.  frame tags:[ "orig2" ] 0 tagged trace sets:[ ] cmm:[ ] output 3: frame: ident=52843 time=340 tick=512 with 12288 traces.  frame tags:[ "orig3" ] 0 tagged trace sets:[ ] cmm:[ ] output 4: frame: ident=52843 time=340 tick=512 with 12288 traces.  frame tags:[ "orig4" ] 0 tagged trace sets:[ ] cmm:[ ] output 5: frame: ident=52843 time=340 tick=512 with 12288 traces.  frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] output 6: frame: ident=52843 time=340 tick=512 with 12288 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] output 7: frame: ident=52843 time=340 tick=512 with 12288 traces.  frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ]  
[03:08:24.454] W [  glue  ] <ChannelSelector:chsel7> Untagged summary not supported, summary will be dropped. 
[03:08:24.455] D [  glue  ] <ChannelSelector:chsel7> input frame: ident=52843 time=340 tick=512 with 12288 traces.  frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=52843 time=340 tick=512 with 1536 traces.  frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ] 
[03:08:24.455] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=28 input frame: frame: ident=52843 time=340 tick=512 with 1536 traces.  frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ] 
[03:08:24.455] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=28 init nticks=10048 tbinmin=0 tbinmax=10048 
[03:08:24.493] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=28 load plane index: 0, ntraces=1536, input bad regions: 0 
[03:08:27.162] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=28 load plane index: 1, ntraces=1536, input bad regions: 0 
[03:08:29.970] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=28 load plane index: 2, ntraces=1536, input bad regions: 0 

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                      9.3786e-05      287.498       428.308       292.36        91.4021        15     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                      6.1666e-05    8.47561e-05   0.000234501   7.4901e-05    4.09657e-05      15     
produce:tpcrawdecoder:PDVDTPCReader               84.7985       103.712       124.997       99.7116       12.1077        15     
produce:triggerrawdecoder:PDVDTriggerReader4     0.0344433     0.0348665     0.0366494     0.0346862    0.000531404      15     
produce:pdvddaphne:DAPHNEReaderPDVD               9.39895       14.1595       17.0671       14.0078       2.0028         15     
produce:ophit:OpHitFinder                        0.062698      0.0719497     0.0840949     0.0716306    0.00584864       15     
produce:opflash:OpFlashFinderVerticalDrift       0.0133636     0.0164539     0.0262474     0.0155156    0.00317946       15     
produce:wclsdatavd:WireCellToolkit                76.2801       87.0832       97.2072       87.3315       4.4575         14     
produce:gaushit:GausHitFinder                    0.976586       1.53031       2.23099       1.50468      0.329977        14     
produce:nhitsfilter:NumberOfHitsFilter          0.000368894   0.000484632   0.000943956   0.000465541   0.000138184      14     
produce:reco3d:SpacePointSolver                   9.20951       18.2247       31.1326       18.5421       5.56647        14     
produce:hitpdune:DisambigFromSpacePoints         0.145755      0.257339      0.479721      0.253924      0.092298        14     
produce:pandora:StandardPandora                   28.8451       76.5571       169.714       69.1628       37.9393        14     
produce:pandoraTrack:LArPandoraTrackCreation     0.653718       1.69804       2.91239       1.49967      0.740342        14     
produce:pandoraGnocalo:GnocchiCalorimetry        0.0243664     0.0411317     0.0561338     0.0427207     0.0102861       14     
[art]:TriggerResults:TriggerResultInserter      2.0819e-05    3.49092e-05   9.5399e-05    2.6044e-05    2.24444e-05      14     
end_path:out1:RootOutput                         8.096e-06    1.25434e-05   3.3352e-05    1.01695e-05   6.57434e-06      14     
end_path:out1:RootOutput(write)                   5.93453       6.28053       6.6477        6.30255      0.220578        14     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8589.89 MB
  Peak resident set size usage (VmHWM): 6703.97 MB
  Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException:  PostEndJob 17-Sep-2025 03:09:27 CEST ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- BadAlloc BEGIN
      A bad_alloc exception was thrown while processing module WireCellToolkit/wclsdatavd run: 39275 subRun: 1 event: 52843
      The job has probably exhausted the virtual memory available to the process.
    ---- BadAlloc END
    Exception going through path produce
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
%MSG
Art has completed and will exit with status 1.
Error in reco1
justIN time: 2025-09-18 19:25:47 UTC       justIN version: 01.05.00