justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 241376.0@dunegpschedd01.fnal.gov

Jobsub ID241376.0@dunegpschedd01.fnal.gov
Workflow ID9405
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-10-31 12:28:57
SiteNL_NIKHEF
EntryVIRGO_NL_NIKHEF_juk
Last heartbeat2025-10-31 13:07:52
From worker nodeHostnamewn-pep-011.farm.nikhef.nl
cpuinfoIntel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Started2025-10-31 12:30:38
Input filesvd-protodune:np02vd_raw_run040270_0672_df-s04-d0_dw_0_20251028T073207.hdf5
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-10-31 13:07:52
Saved logsjustin-logs:241376.0-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

nd 1626 XVs -> 94 XUVs
C:0 T:3 306 XUs and 221 XVs -> 8 XUVs
C:0 T:4 1123 XUs and 1290 XVs -> 43 XUVs
C:0 T:5 2843 XUs and 3621 XVs -> 94 XUVs
C:0 T:6 458 XUs and 526 XVs -> 27 XUVs
C:0 T:7 300 XUs and 260 XVs -> 7 XUVs
C:0 T:8 640 XUs and 875 XVs -> 48 XUVs
C:0 T:9 1529 XUs and 1810 XVs -> 92 XUVs
C:0 T:10 444 XUs and 861 XVs -> 17 XUVs
C:0 T:11 855 XUs and 1167 XVs -> 46 XUVs
C:0 T:12 1013 XUs and 1156 XVs -> 56 XUVs
C:0 T:13 476 XUs and 594 XVs -> 23 XUVs
C:0 T:14 854 XUs and 1134 XVs -> 85 XUVs
C:0 T:15 998 XUs and 1208 XVs -> 52 XUVs
1067 XUVs total
855 collection wire objects
1067 potential space points
Neighbour search...
11381 tests to find 3750 neighbours
Iterating with no regularization...
Begin: 5.60815e+10
0 5.52667e+10
1 5.51355e+10
2 5.51352e+10
Now with regularization...
Begin: 5.42847e+10
0 5.42837e+10
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=8000 keeping as is
[14:05:05.930] D [  main  ] executing 1 apps, thread limit 0:
[14:05:05.930] D [  main  ] executing 1 apps, thread limit 0:
[14:05:05.930] D [  main  ] executing app: "Pgrapher"
[14:05:05.930] D [ pgraph ] <Pgrapher:> executing graph 
[14:05:05.930] D [ pgraph ] executing with 26 nodes
[14:05:05.932] D [  glue  ] <FrameFanout:nfsp> call=24: input: frame: ident=124596 time=52 tick=512 with 12288 traces.  frame tags:[ "orig" ] 0 tagged trace sets:[ ] cmm:[ ] output 0: frame: ident=124596 time=52 tick=512 with 12288 traces.  frame tags:[ "orig0" ] 0 tagged trace sets:[ ] cmm:[ ] output 1: frame: ident=124596 time=52 tick=512 with 12288 traces.  frame tags:[ "orig1" ] 0 tagged trace sets:[ ] cmm:[ ] output 2: frame: ident=124596 time=52 tick=512 with 12288 traces.  frame tags:[ "orig2" ] 0 tagged trace sets:[ ] cmm:[ ] output 3: frame: ident=124596 time=52 tick=512 with 12288 traces.  frame tags:[ "orig3" ] 0 tagged trace sets:[ ] cmm:[ ] output 4: frame: ident=124596 time=52 tick=512 with 12288 traces.  frame tags:[ "orig4" ] 0 tagged trace sets:[ ] cmm:[ ] output 5: frame: ident=124596 time=52 tick=512 with 12288 traces.  frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] output 6: frame: ident=124596 time=52 tick=512 with 12288 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] output 7: frame: ident=124596 time=52 tick=512 with 12288 traces.  frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ]  
[14:05:05.932] W [  glue  ] <ChannelSelector:chsel7> Untagged summary not supported, summary will be dropped. 
[14:05:05.933] D [  glue  ] <ChannelSelector:chsel7> input frame: ident=124596 time=52 tick=512 with 12288 traces.  frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=124596 time=52 tick=512 with 1536 traces.  frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ] 
[14:05:05.933] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=24 input frame: frame: ident=124596 time=52 tick=512 with 1536 traces.  frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ] 
[14:05:05.933] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=24 init nticks=8000 tbinmin=0 tbinmax=8000 
[14:05:05.969] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=24 load plane index: 0, ntraces=1536, input bad regions: 0 
[14:05:07.746] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=24 load plane index: 1, ntraces=1536, input bad regions: 0 
[14:05:09.581] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=24 load plane index: 2, ntraces=1536, input bad regions: 0 
[14:05:57.779] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=24 save plane index: 0, Qtot=45558865080 Qloss=-53990727154, 4678 indices spanning [16921,21598] "wiener" 
[14:05:58.000] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=24 save plane index: 0, Qtot=44462063814 Qloss=-54367582704, 4187 indices spanning [21599,25785] "gauss" 
[14:05:59.155] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=24 save plane index: 1, Qtot=71949570152 Qloss=-7693252662, 11774 indices spanning [25786,37559] "wiener" 
[14:05:59.363] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=24 save plane index: 1, Qtot=66629671632 Qloss=-8056997496, 11209 indices spanning [37560,48768] "gauss" 
[14:05:59.728] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=24 save plane index: 2, Qtot=2697060153 Qloss=-2743497776, 11497 indices spanning [48769,60265] "wiener" 
[14:06:00.097] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=24 save plane index: 2, Qtot=2201475693 Qloss=-2253214251, 12628 indices spanning [60266,72893] "gauss" 
[14:06:00.097] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=24 produce 72894 traces: 27949 wiener7, 0 decon_charge7, 28024 gauss7, frame tag: sigproc 
[14:06:00.097] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=24 output frame: frame: ident=124596 time=52 tick=512 with 72894 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss7":28024 [0] "mp2_roi7":12546 [0] "mp3_roi7":4375 [0] "wiener7":27949 [27949] ] cmm:[ ] 
[14:06:06.342] W [  glue  ] <ChannelSelector:chsel6> Untagged summary not supported, summary will be dropped. 
[14:06:06.343] D [  glue  ] <ChannelSelector:chsel6> input frame: ident=124596 time=52 tick=512 with 12288 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=124596 time=52 tick=512 with 1536 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] 
[14:06:06.343] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=24 input frame: frame: ident=124596 time=52 tick=512 with 1536 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] 
[14:06:06.343] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=24 init nticks=8000 tbinmin=0 tbinmax=8000 
[14:06:06.382] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=24 load plane index: 0, ntraces=1536, input bad regions: 0 
[14:06:08.450] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=24 load plane index: 1, ntraces=1536, input bad regions: 0 
[14:06:10.493] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=24 load plane index: 2, ntraces=1536, input bad regions: 0 

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                      7.6816e-05      154.263       268.125       160.171       65.573         13     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                      5.9196e-05    7.98011e-05   0.000165168   7.3259e-05    2.56289e-05      13     
produce:tpcrawdecoder:PDVDTPCReader               12.7913       41.0236       147.53        19.7505       43.3141        13     
produce:triggerrawdecoder:PDVDTriggerReader4     0.292726      0.322304      0.470063      0.304968      0.0451205       13     
produce:pdvddaphne:DAPHNEReaderPDVD             0.000390444   0.000443808   0.000709902   0.000410117   8.98555e-05      13     
produce:ophit:OpHitFinder                       5.6737e-05    9.52889e-05   0.000476847   6.0679e-05    0.000110306      13     
produce:opflash:OpFlashFinderVerticalDrift      4.8286e-05    7.38878e-05   0.000293564    5.542e-05    6.36607e-05      13     
produce:wclsdatavd:WireCellToolkit                62.5407       76.6485       124.752       67.3029       19.0355        12     
produce:gaushit:GausHitFinder                     1.03887       1.37323       1.89557       1.2796       0.239901        12     
produce:nhitsfilter:NumberOfHitsFilter          0.000363509   0.000449467   0.000685522   0.000413524   8.97004e-05      12     
produce:reco3d:SpacePointSolver                   8.01857       11.3522       16.9054       10.6563       2.60497        12     
produce:hitpdune:DisambigFromSpacePoints         0.146975      0.210862      0.303452      0.195224      0.0546094       12     
produce:pandora:StandardPandora                   15.8881       29.1549       49.7618       24.6774       10.7479        12     
produce:pandoraTrack:LArPandoraTrackCreation     0.687929       1.38781       2.37031       1.28608      0.520041        12     
produce:pandoraGnocalo:GnocchiCalorimetry        0.0238881     0.0326762     0.041929      0.0333981    0.00610922       12     
[art]:TriggerResults:TriggerResultInserter      2.9489e-05    3.51633e-05   7.2048e-05    3.10695e-05   1.13208e-05      12     
end_path:out1:RootOutput                         6.834e-06    9.79825e-06   2.5261e-05     8.522e-06    4.69427e-06      12     
end_path:out1:RootOutput(write)                   4.08571       4.36431       4.77804       4.23225      0.240904        12     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8589.92 MB
  Peak resident set size usage (VmHWM): 6703.31 MB
  Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException:  PostEndJob 31-Oct-2025 14:07:31 CET ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- BadAlloc BEGIN
      A bad_alloc exception was thrown while processing module WireCellToolkit/wclsdatavd run: 40270 subRun: 1 event: 124596
      The job has probably exhausted the virtual memory available to the process.
    ---- BadAlloc END
    Exception going through path produce
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
%MSG
Art has completed and will exit with status 1.
Error in reco1
justIN time: 2025-11-05 08:42:19 UTC       justIN version: 01.05.01