justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 40377.0@dunegpschedd02.fnal.gov

Jobsub ID40377.0@dunegpschedd02.fnal.gov
Workflow ID2332
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod.mcsim
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-09-16 10:34:49
SiteNL_NIKHEF
EntryVIRGO_NL_NIKHEF_juk
Last heartbeat2025-09-16 10:58:27
From worker nodeHostnamewn-choc-034.farm.nikhef.nl
cpuinfoIntel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Started2025-09-16 10:36:06
Input filesvd-protodune:np02vd_raw_run039343_0032_df-s05-d2_dw_0_20250908T125053.hdf5
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-09-16 10:58:27
Saved logsjustin-logs:40377.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

r:  OpHitFinder:ophit@BeginModule 16-Sep-2025 12:57:04 CEST  run: 39343 subRun: 1 event: 17116
Error! unrecognized channel number -1. Ignoring pulse
%MSG
%MSG-e OpHitFinder:  OpHitFinder:ophit@BeginModule 16-Sep-2025 12:57:04 CEST  run: 39343 subRun: 1 event: 17116
Error! unrecognized channel number -1. Ignoring pulse
%MSG
%MSG-e OpHitFinder:  OpHitFinder:ophit@BeginModule 16-Sep-2025 12:57:04 CEST  run: 39343 subRun: 1 event: 17116
Error! unrecognized channel number -1. Ignoring pulse
%MSG
%MSG-e OpHitFinder:  OpHitFinder:ophit@BeginModule 16-Sep-2025 12:57:04 CEST  run: 39343 subRun: 1 event: 17116
Error! unrecognized channel number -1. Ignoring pulse
%MSG
%MSG-e OpHitFinder:  OpHitFinder:ophit@BeginModule 16-Sep-2025 12:57:04 CEST  run: 39343 subRun: 1 event: 17116
Error! unrecognized channel number -1. Ignoring pulse
%MSG
%MSG-e OpHitFinder:  OpHitFinder:ophit@BeginModule 16-Sep-2025 12:57:04 CEST  run: 39343 subRun: 1 event: 17116
Error! unrecognized channel number -1. Ignoring pulse
%MSG
%MSG-e OpHitFinder:  OpHitFinder:ophit@BeginModule 16-Sep-2025 12:57:04 CEST  run: 39343 subRun: 1 event: 17116
Error! unrecognized channel number -1. Ignoring pulse
%MSG
%MSG-e OpHitFinder:  OpHitFinder:ophit@BeginModule 16-Sep-2025 12:57:04 CEST  run: 39343 subRun: 1 event: 17116
Error! unrecognized channel number -1. Ignoring pulse
%MSG
%MSG-e OpHitFinder:  OpHitFinder:ophit@BeginModule 16-Sep-2025 12:57:04 CEST  run: 39343 subRun: 1 event: 17116
Error! unrecognized channel number -1. Ignoring pulse
%MSG
%MSG-e OpHitFinder:  OpHitFinder:ophit@BeginModule 16-Sep-2025 12:57:04 CEST  run: 39343 subRun: 1 event: 17116
Error! unrecognized channel number -1. Ignoring pulse
%MSG
%MSG-e OpHitFinder:  OpHitFinder:ophit@BeginModule 16-Sep-2025 12:57:04 CEST  run: 39343 subRun: 1 event: 17116
Error! unrecognized channel number -1. Ignoring pulse
%MSG
%MSG-e OpHitFinder:  OpHitFinder:ophit@BeginModule 16-Sep-2025 12:57:04 CEST  run: 39343 subRun: 1 event: 17116
Error! unrecognized channel number -1. Ignoring pulse
%MSG
%MSG-e OpHitFinder:  OpHitFinder:ophit@BeginModule 16-Sep-2025 12:57:04 CEST  run: 39343 subRun: 1 event: 17116
Error! unrecognized channel number -1. Ignoring pulse
%MSG
%MSG-e OpHitFinder:  OpHitFinder:ophit@BeginModule 16-Sep-2025 12:57:04 CEST  run: 39343 subRun: 1 event: 17116
Error! unrecognized channel number -1. Ignoring pulse
%MSG
%MSG-e OpHitFinder:  OpHitFinder:ophit@BeginModule 16-Sep-2025 12:57:04 CEST  run: 39343 subRun: 1 event: 17116
Error! unrecognized channel number -1. Ignoring pulse
%MSG
%MSG-e OpHitFinder:  OpHitFinder:ophit@BeginModule 16-Sep-2025 12:57:04 CEST  run: 39343 subRun: 1 event: 17116
Error! unrecognized channel number -1. Ignoring pulse
%MSG
%MSG-e OpHitFinder:  OpHitFinder:ophit@BeginModule 16-Sep-2025 12:57:04 CEST  run: 39343 subRun: 1 event: 17116
Error! unrecognized channel number -1. Ignoring pulse
%MSG
%MSG-e OpHitFinder:  OpHitFinder:ophit@BeginModule 16-Sep-2025 12:57:04 CEST  run: 39343 subRun: 1 event: 17116
Error! unrecognized channel number -1. Ignoring pulse
%MSG
%MSG-e OpHitFinder:  OpHitFinder:ophit@BeginModule 16-Sep-2025 12:57:04 CEST  run: 39343 subRun: 1 event: 17116
Error! unrecognized channel number -1. Ignoring pulse
%MSG
%MSG-e OpHitFinder:  OpHitFinder:ophit@BeginModule 16-Sep-2025 12:57:04 CEST  run: 39343 subRun: 1 event: 17116
Error! unrecognized channel number -1. Ignoring pulse
%MSG
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6400 keeping as is
[12:57:04.910] D [  main  ] executing 1 apps, thread limit 0:
[12:57:04.910] D [  main  ] executing 1 apps, thread limit 0:
[12:57:04.910] D [  main  ] executing app: "Pgrapher"
[12:57:04.911] D [ pgraph ] <Pgrapher:> executing graph 
[12:57:04.911] D [ pgraph ] executing with 26 nodes
[12:57:04.912] D [  glue  ] <FrameFanout:nfsp> call=6: input: frame: ident=17116 time=1 tick=512 with 12288 traces.  frame tags:[ "orig" ] 0 tagged trace sets:[ ] cmm:[ ] output 0: frame: ident=17116 time=1 tick=512 with 12288 traces.  frame tags:[ "orig0" ] 0 tagged trace sets:[ ] cmm:[ ] output 1: frame: ident=17116 time=1 tick=512 with 12288 traces.  frame tags:[ "orig1" ] 0 tagged trace sets:[ ] cmm:[ ] output 2: frame: ident=17116 time=1 tick=512 with 12288 traces.  frame tags:[ "orig2" ] 0 tagged trace sets:[ ] cmm:[ ] output 3: frame: ident=17116 time=1 tick=512 with 12288 traces.  frame tags:[ "orig3" ] 0 tagged trace sets:[ ] cmm:[ ] output 4: frame: ident=17116 time=1 tick=512 with 12288 traces.  frame tags:[ "orig4" ] 0 tagged trace sets:[ ] cmm:[ ] output 5: frame: ident=17116 time=1 tick=512 with 12288 traces.  frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] output 6: frame: ident=17116 time=1 tick=512 with 12288 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] output 7: frame: ident=17116 time=1 tick=512 with 12288 traces.  frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ]  
[12:57:04.913] W [  glue  ] <ChannelSelector:chsel7> Untagged summary not supported, summary will be dropped. 
[12:57:04.913] D [  glue  ] <ChannelSelector:chsel7> input frame: ident=17116 time=1 tick=512 with 12288 traces.  frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=17116 time=1 tick=512 with 1536 traces.  frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ] 
[12:57:04.914] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=6 input frame: frame: ident=17116 time=1 tick=512 with 1536 traces.  frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ] 
[12:57:04.914] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=6 init nticks=6400 tbinmin=0 tbinmax=6400 
[12:57:04.956] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=6 load plane index: 0, ntraces=1536, input bad regions: 0 
[12:57:06.580] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=6 load plane index: 1, ntraces=1536, input bad regions: 0 
[12:57:08.120] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=6 load plane index: 2, ntraces=1536, input bad regions: 0 

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                      6.0995e-05      287.905       721.545       215.037       266.724         4     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       5.536e-05    0.000107329   0.000257306   5.83245e-05   8.66184e-05       4     
produce:tpcrawdecoder:PDVDTPCReader               53.4416       55.5921       59.1543       54.8862       2.27561         4     
produce:triggerrawdecoder:PDVDTriggerReader4      0.5304       0.573151      0.593207      0.584498      0.0253996        4     
produce:pdvddaphne:DAPHNEReaderPDVD               8.02411       9.1722        10.1334       9.26567      0.800249         4     
produce:ophit:OpHitFinder                        0.0400454     0.0447589     0.0472404     0.045875     0.00278156        4     
produce:opflash:OpFlashFinderVerticalDrift      0.00944217     0.0142358     0.0163173     0.015592      0.0027835        4     
produce:wclsdatavd:WireCellToolkit                57.1284       76.3213       112.355       59.4807       25.4977         3     
produce:gaushit:GausHitFinder                     1.18643       3.42371       7.4349        1.64979       2.84265         3     
produce:nhitsfilter:NumberOfHitsFilter          0.000296891   0.000489893   0.000594841   0.000577947   0.000136647       3     
produce:reco3d:SpacePointSolver                   9.20396       17.7407       27.6253       16.3928       7.58062         3     
produce:hitpdune:DisambigFromSpacePoints         0.143994      0.314955      0.492873      0.307999      0.142514         3     
produce:pandora:StandardPandora                   39.6561       214.723       502.774       101.739       205.253         3     
produce:pandoraTrack:LArPandoraTrackCreation     0.975578       1.34114       1.78793       1.25991      0.336576         3     
produce:pandoraGnocalo:GnocchiCalorimetry        0.0239344     0.0353051     0.0427257     0.0392552    0.00816416        3     
[art]:TriggerResults:TriggerResultInserter      2.1507e-05    3.9144e-05    6.1872e-05    3.4053e-05    1.68676e-05       3     
end_path:out1:RootOutput                         7.57e-06     1.31903e-05   2.4096e-05     7.905e-06    7.71268e-06       3     
end_path:out1:RootOutput(write)                   4.13221       4.5744        4.96647       4.62451      0.342422         3     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8588.91 MB
  Peak resident set size usage (VmHWM): 6662.14 MB
  Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException:  PostEndJob 16-Sep-2025 12:58:11 CEST ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- BadAlloc BEGIN
      A bad_alloc exception was thrown while processing module WireCellToolkit/wclsdatavd run: 39343 subRun: 1 event: 17116
      The job has probably exhausted the virtual memory available to the process.
    ---- BadAlloc END
    Exception going through path produce
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
%MSG
Art has completed and will exit with status 1.
Error in reco1
justIN time: 2025-09-18 19:58:05 UTC       justIN version: 01.05.00