justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 48330.2@dunegpschedd01.fnal.gov

Jobsub ID48330.2@dunegpschedd01.fnal.gov
Workflow ID2862
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-09-18 10:34:21
SiteNL_NIKHEF
EntryVIRGO_NL_NIKHEF_klomp
Last heartbeat2025-09-18 11:29:45
From worker nodeHostnamewn-sate-050.farm.nikhef.nl
cpuinfoAMD EPYC 7551P 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Started2025-09-18 10:35:33
Input filesvd-protodune:np02vd_raw_run039433_0289_df-s05-d0_dw_0_20250918T031730.hdf5
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-09-18 11:29:45
Saved logsjustin-logs:48330.2-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

.  frame tags:[ "orig2" ] 0 tagged trace sets:[ ] cmm:[ ] 
[13:28:11.263] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=46 init nticks=6500 tbinmin=0 tbinmax=6500 
[13:28:11.286] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=46 load plane index: 0, ntraces=1536, input bad regions: 0 
[13:28:12.614] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=46 load plane index: 1, ntraces=1536, input bad regions: 0 
[13:28:13.815] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=46 load plane index: 2, ntraces=1536, input bad regions: 0 
[13:28:16.968] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=46 save plane index: 0, Qtot=131482860 Qloss=-7400306, 3355 indices spanning [17020,20374] "wiener" 
[13:28:17.100] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=46 save plane index: 0, Qtot=123728282 Qloss=-5848163, 2607 indices spanning [20375,22981] "gauss" 
[13:28:17.471] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=46 save plane index: 1, Qtot=123077591 Qloss=-9356354, 3756 indices spanning [22982,26737] "wiener" 
[13:28:17.603] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=46 save plane index: 1, Qtot=106844228 Qloss=-7876751, 2979 indices spanning [26738,29716] "gauss" 
[13:28:17.885] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=46 save plane index: 2, Qtot=158044128 Qloss=-2546575, 3818 indices spanning [29717,33534] "wiener" 
[13:28:18.099] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=46 save plane index: 2, Qtot=156611727 Qloss=-1126937, 2821 indices spanning [33535,36355] "gauss" 
[13:28:18.099] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=46 produce 36356 traces: 10929 wiener2, 0 decon_charge2, 8407 gauss2, frame tag: sigproc 
[13:28:18.099] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=46 output frame: frame: ident=83130 time=66 tick=500 with 36356 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss2":8407 [0] "mp2_roi2":13978 [0] "mp3_roi2":3042 [0] "wiener2":10929 [10929] ] cmm:[ ] 
[13:28:18.192] W [  glue  ] <ChannelSelector:chsel1> Untagged summary not supported, summary will be dropped. 
[13:28:18.193] D [  glue  ] <ChannelSelector:chsel1> input frame: ident=83130 time=66 tick=512 with 6144 traces.  frame tags:[ "orig1" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=83130 time=66 tick=512 with 1536 traces.  frame tags:[ "orig1" ] 0 tagged trace sets:[ ] cmm:[ ] 
[13:28:18.193] D [  aux   ] <Resampler:resmp1> first ch=617 Ts=512 Ns=6348 Ns_pad=6375 Nrat=125 Tr=500 Nr=6528 Nout=6500 padding:linear 
[13:28:18.534] D [  aux   ] <Resampler:resmp1> resample 1536 traces at call=46 
[13:28:18.534] D [sigproc ] <OmnibusSigProc:anode1sigproc1> call=46 input frame: frame: ident=83130 time=66 tick=500 with 1536 traces.  frame tags:[ "orig1" ] 0 tagged trace sets:[ ] cmm:[ ] 
[13:28:18.534] D [sigproc ] <OmnibusSigProc:anode1sigproc1> call=46 init nticks=6500 tbinmin=0 tbinmax=6500 
[13:28:18.557] D [sigproc ] <OmnibusSigProc:anode1sigproc1> call=46 load plane index: 0, ntraces=1536, input bad regions: 0 
[13:28:19.949] D [sigproc ] <OmnibusSigProc:anode1sigproc1> call=46 load plane index: 1, ntraces=1536, input bad regions: 0 
[13:28:21.158] D [sigproc ] <OmnibusSigProc:anode1sigproc1> call=46 load plane index: 2, ntraces=1536, input bad regions: 0 
[13:28:23.269] D [sigproc ] <OmnibusSigProc:anode1sigproc1> call=46 save plane index: 0, Qtot=51061924 Qloss=-5546619, 3158 indices spanning [19197,22354] "wiener" 
[13:28:23.405] D [sigproc ] <OmnibusSigProc:anode1sigproc1> call=46 save plane index: 0, Qtot=46763409 Qloss=-3902021, 2218 indices spanning [22355,24572] "gauss" 
[13:28:23.763] D [sigproc ] <OmnibusSigProc:anode1sigproc1> call=46 save plane index: 1, Qtot=62061780 Qloss=-5775614, 2944 indices spanning [24573,27516] "wiener" 
[13:28:23.895] D [sigproc ] <OmnibusSigProc:anode1sigproc1> call=46 save plane index: 1, Qtot=53340024 Qloss=-4569386, 2094 indices spanning [27517,29610] "gauss" 
[13:28:24.111] D [sigproc ] <OmnibusSigProc:anode1sigproc1> call=46 save plane index: 2, Qtot=48448372 Qloss=-2536893, 3681 indices spanning [29611,33291] "wiener" 
[13:28:24.324] D [sigproc ] <OmnibusSigProc:anode1sigproc1> call=46 save plane index: 2, Qtot=46715873 Qloss=-744833, 2703 indices spanning [33292,35994] "gauss" 
[13:28:24.324] D [sigproc ] <OmnibusSigProc:anode1sigproc1> call=46 produce 35995 traces: 9783 wiener1, 0 decon_charge1, 7015 gauss1, frame tag: sigproc 
[13:28:24.324] D [sigproc ] <OmnibusSigProc:anode1sigproc1> call=46 output frame: frame: ident=83130 time=66 tick=500 with 35995 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss1":7015 [0] "mp2_roi1":17163 [0] "mp3_roi1":2034 [0] "wiener1":9783 [9783] ] cmm:[ ] 
[13:28:24.349] W [  glue  ] <ChannelSelector:chsel0> Untagged summary not supported, summary will be dropped. 
[13:28:24.350] D [  glue  ] <ChannelSelector:chsel0> input frame: ident=83130 time=66 tick=512 with 6144 traces.  frame tags:[ "orig0" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=83130 time=66 tick=512 with 1536 traces.  frame tags:[ "orig0" ] 0 tagged trace sets:[ ] cmm:[ ] 
[13:28:24.351] D [  aux   ] <Resampler:resmp0> first ch=1388 Ts=512 Ns=6348 Ns_pad=6375 Nrat=125 Tr=500 Nr=6528 Nout=6500 padding:linear 
[13:28:24.692] D [  aux   ] <Resampler:resmp0> resample 1536 traces at call=46 
[13:28:24.692] D [sigproc ] <OmnibusSigProc:anode0sigproc0> call=46 input frame: frame: ident=83130 time=66 tick=500 with 1536 traces.  frame tags:[ "orig0" ] 0 tagged trace sets:[ ] cmm:[ ] 
[13:28:24.692] D [sigproc ] <OmnibusSigProc:anode0sigproc0> call=46 init nticks=6500 tbinmin=0 tbinmax=6500 
[13:28:24.716] D [sigproc ] <OmnibusSigProc:anode0sigproc0> call=46 load plane index: 0, ntraces=1536, input bad regions: 0 
[13:28:26.029] D [sigproc ] <OmnibusSigProc:anode0sigproc0> call=46 load plane index: 1, ntraces=1536, input bad regions: 0 
[13:28:27.195] D [sigproc ] <OmnibusSigProc:anode0sigproc0> call=46 load plane index: 2, ntraces=1536, input bad regions: 0 

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                      4.6137e-05      127.452       576.023       100.955       109.601        24     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                      4.6137e-05    5.52492e-05   0.000134233   5.3485e-05    1.71981e-05      24     
produce:tpcrawdecoder:PDVDTPCReader               25.9688       41.4313       62.7232       41.3824       8.78408        24     
produce:triggerrawdecoder:PDVDTriggerReader4     0.290381      0.398613       1.2657        0.35898      0.200542        24     
produce:pdvddaphne:DAPHNEReaderPDVD               5.30535       8.99617       12.284        8.95959       1.75102        24     
produce:ophit:OpHitFinder                        0.0249781     0.0397454     0.0606692     0.0390469    0.00642455       24     
produce:opflash:OpFlashFinderVerticalDrift      0.00257387    0.00851166     0.0140208    0.00810623     0.002439        24     
produce:wclsdatavd:WireCellToolkit                23.3223       33.5881       52.5306       33.4264       8.86652        23     
produce:gaushit:GausHitFinder                    0.324976      0.604374       1.46803      0.527868      0.273918        23     
produce:nhitsfilter:NumberOfHitsFilter          7.4701e-05    0.000127596   0.000329851   0.000101912   7.24497e-05      23     
produce:reco3d:SpacePointSolver                   2.36288       5.67834       23.9706       4.33287       4.64738        23     
produce:hitpdune:DisambigFromSpacePoints         0.0249571     0.0782419     0.413635      0.0456314     0.0972783       23     
produce:pandora:StandardPandora                   4.43631       39.2309       441.726       10.226        96.0413        23     
produce:pandoraTrack:LArPandoraTrackCreation     0.0798645     0.416843       3.44829      0.208394      0.693441        23     
produce:pandoraGnocalo:GnocchiCalorimetry       0.00724906     0.0146554     0.0345164     0.0136125    0.00585272       23     
[art]:TriggerResults:TriggerResultInserter      1.4387e-05    1.86899e-05   4.8471e-05    1.7803e-05     6.761e-06       23     
end_path:out1:RootOutput                         3.887e-06    7.55335e-06   2.0739e-05     7.293e-06    2.98467e-06      23     
end_path:out1:RootOutput(write)                   2.00353       2.44294       3.0773        2.2608       0.354131        23     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8589.86 MB
  Peak resident set size usage (VmHWM): 6713.23 MB
  Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException:  PostEndJob 18-Sep-2025 13:29:27 CEST ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- BadAlloc BEGIN
      A bad_alloc exception was thrown while processing module WireCellToolkit/wclsdatavd run: 39433 subRun: 1 event: 83130
      The job has probably exhausted the virtual memory available to the process.
    ---- BadAlloc END
    Exception going through path produce
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
%MSG
Art has completed and will exit with status 1.
Error in reco1
justIN time: 2025-09-18 18:00:15 UTC       justIN version: 01.05.00