justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 241334.12@dunegpschedd01.fnal.gov

Jobsub ID241334.12@dunegpschedd01.fnal.gov
Workflow ID9374
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-10-31 07:50:43
SiteNL_NIKHEF
EntryVIRGO_NL_NIKHEF_klomp
Last heartbeat2025-10-31 08:50:46
From worker nodeHostnamewn-pep-014.farm.nikhef.nl
cpuinfoIntel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Started2025-10-31 07:52:33
Input filesvd-protodune:np02vd_raw_run040266_0685_df-s04-d3_dw_0_20251025T162348.hdf5
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-10-31 08:50:46
Saved logsjustin-logs:241334.12-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

glue  ] <ChannelSelector:chsel6> input frame: ident=126511 time=89 tick=512 with 12288 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=126511 time=89 tick=512 with 1536 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] 
[09:48:05.139] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=38 input frame: frame: ident=126511 time=89 tick=512 with 1536 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] 
[09:48:05.139] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=38 init nticks=8642 tbinmin=0 tbinmax=8642 
[09:48:05.177] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=38 load plane index: 0, ntraces=1536, input bad regions: 0 
[09:48:08.681] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=38 load plane index: 1, ntraces=1536, input bad regions: 0 
[09:48:12.173] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=38 load plane index: 2, ntraces=1536, input bad regions: 0 
[09:48:17.061] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=38 save plane index: 0, Qtot=83817591 Qloss=-11342459, 1848 indices spanning [12994,14841] "wiener" 
[09:48:17.441] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=38 save plane index: 0, Qtot=81277884 Qloss=-9574804, 1430 indices spanning [14842,16271] "gauss" 
[09:48:18.035] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=38 save plane index: 1, Qtot=90304805 Qloss=-12736807, 3249 indices spanning [16272,19520] "wiener" 
[09:48:18.414] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=38 save plane index: 1, Qtot=83771465 Qloss=-8845725, 2627 indices spanning [19521,22147] "gauss" 
[09:48:19.079] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=38 save plane index: 2, Qtot=79915022 Qloss=-6382232, 3079 indices spanning [22148,25226] "wiener" 
[09:48:19.676] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=38 save plane index: 2, Qtot=76881791 Qloss=-3278784, 2585 indices spanning [25227,27811] "gauss" 
[09:48:19.676] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=38 produce 27812 traces: 8176 wiener6, 0 decon_charge6, 6642 gauss6, frame tag: sigproc 
[09:48:19.676] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=38 output frame: frame: ident=126511 time=89 tick=512 with 27812 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss6":6642 [0] "mp2_roi6":10956 [0] "mp3_roi6":2038 [0] "wiener6":8176 [8176] ] cmm:[ ] 
[09:48:19.705] W [  glue  ] <ChannelSelector:chsel5> Untagged summary not supported, summary will be dropped. 
[09:48:19.706] D [  glue  ] <ChannelSelector:chsel5> input frame: ident=126511 time=89 tick=512 with 12288 traces.  frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=126511 time=89 tick=512 with 1536 traces.  frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] 
[09:48:19.707] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=38 input frame: frame: ident=126511 time=89 tick=512 with 1536 traces.  frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] 
[09:48:19.707] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=38 init nticks=8642 tbinmin=0 tbinmax=8642 
[09:48:19.747] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=38 load plane index: 0, ntraces=1536, input bad regions: 0 
[09:48:23.288] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=38 load plane index: 1, ntraces=1536, input bad regions: 0 
[09:48:26.853] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=38 load plane index: 2, ntraces=1536, input bad regions: 0 
[09:48:50.184] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=38 save plane index: 0, Qtot=2314987021 Qloss=-70403028, 9453 indices spanning [39848,49300] "wiener" 
[09:48:50.572] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=38 save plane index: 0, Qtot=2223156689 Qloss=-62193459, 7396 indices spanning [49301,56696] "gauss" 
[09:48:51.547] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=38 save plane index: 1, Qtot=988458432 Qloss=-56199278, 12161 indices spanning [56697,68857] "wiener" 
[09:48:51.931] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=38 save plane index: 1, Qtot=849323640 Qloss=-43446891, 9653 indices spanning [68858,78510] "gauss" 
[09:48:52.608] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=38 save plane index: 2, Qtot=1097113780 Qloss=-38806947, 10397 indices spanning [78511,88907] "wiener" 
[09:48:53.219] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=38 save plane index: 2, Qtot=1074987027 Qloss=-17668206, 8013 indices spanning [88908,96920] "gauss" 
[09:48:53.220] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=38 produce 96921 traces: 32011 wiener5, 0 decon_charge5, 25062 gauss5, frame tag: sigproc 
[09:48:53.220] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=38 output frame: frame: ident=126511 time=89 tick=512 with 96921 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss5":25062 [0] "mp2_roi5":28338 [0] "mp3_roi5":11510 [0] "wiener5":32011 [32011] ] cmm:[ ] 
[09:48:56.253] W [  glue  ] <ChannelSelector:chsel4> Untagged summary not supported, summary will be dropped. 
[09:48:56.254] D [  glue  ] <ChannelSelector:chsel4> input frame: ident=126511 time=89 tick=512 with 12288 traces.  frame tags:[ "orig4" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=126511 time=89 tick=512 with 1536 traces.  frame tags:[ "orig4" ] 0 tagged trace sets:[ ] cmm:[ ] 
[09:48:56.255] D [sigproc ] <OmnibusSigProc:anode4sigproc4> call=38 input frame: frame: ident=126511 time=89 tick=512 with 1536 traces.  frame tags:[ "orig4" ] 0 tagged trace sets:[ ] cmm:[ ] 
[09:48:56.255] D [sigproc ] <OmnibusSigProc:anode4sigproc4> call=38 init nticks=8642 tbinmin=0 tbinmax=8642 
[09:48:56.297] D [sigproc ] <OmnibusSigProc:anode4sigproc4> call=38 load plane index: 0, ntraces=1536, input bad regions: 0 
[09:48:59.797] D [sigproc ] <OmnibusSigProc:anode4sigproc4> call=38 load plane index: 1, ntraces=1536, input bad regions: 0 
[09:49:03.377] D [sigproc ] <OmnibusSigProc:anode4sigproc4> call=38 load plane index: 2, ntraces=1536, input bad regions: 0 

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                      7.8634e-05      162.321       239.982       154.171       53.9418        20     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                      6.1299e-05    8.45256e-05   0.000173473   7.7631e-05    2.40694e-05      20     
produce:tpcrawdecoder:PDVDTPCReader               12.0174       20.8608       34.6103       18.3191       6.02913        20     
produce:triggerrawdecoder:PDVDTriggerReader4     0.508735      0.539255      0.606304      0.533081      0.0274672       20     
produce:pdvddaphne:DAPHNEReaderPDVD             0.000380658   0.000416491   0.000697746   0.000401835   6.52864e-05      20     
produce:ophit:OpHitFinder                       6.1102e-05    9.0678e-05    0.000472874   6.69105e-05   8.83771e-05      20     
produce:opflash:OpFlashFinderVerticalDrift      4.2917e-05    6.0933e-05    0.00028471    4.9883e-05    5.14554e-05      20     
produce:wclsdatavd:WireCellToolkit                62.4903       86.8663       132.081       76.124        20.0008        19     
produce:gaushit:GausHitFinder                     1.07264       1.47219       1.83901       1.50586      0.241325        19     
produce:nhitsfilter:NumberOfHitsFilter          0.000370159   0.000487384   0.00070745    0.000470909   8.69627e-05      19     
produce:reco3d:SpacePointSolver                   8.14072       12.9257       19.6431       13.2651       3.34976        19     
produce:hitpdune:DisambigFromSpacePoints         0.139233      0.251065      0.460941      0.235857      0.090314        19     
produce:pandora:StandardPandora                   17.1701       41.6034       108.09        34.6073       22.4915        19     
produce:pandoraTrack:LArPandoraTrackCreation     0.791567       2.16565       7.13983       1.81548       1.45189        19     
produce:pandoraGnocalo:GnocchiCalorimetry        0.0259277     0.0356603     0.0469451     0.0335689    0.00712931       19     
[art]:TriggerResults:TriggerResultInserter      2.7507e-05    3.89355e-05   8.1815e-05    3.7858e-05    1.08638e-05      19     
end_path:out1:RootOutput                         3.88e-06     6.68368e-06   1.9903e-05     5.331e-06    3.43111e-06      19     
end_path:out1:RootOutput(write)                   3.98993       4.46384       5.45546       4.29684      0.362808        19     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8589.92 MB
  Peak resident set size usage (VmHWM): 6700.24 MB
  Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException:  PostEndJob 31-Oct-2025 09:50:22 CET ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- BadAlloc BEGIN
      A bad_alloc exception was thrown while processing module WireCellToolkit/wclsdatavd run: 40266 subRun: 1 event: 126511
      The job has probably exhausted the virtual memory available to the process.
    ---- BadAlloc END
    Exception going through path produce
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
%MSG
Art has completed and will exit with status 1.
Error in reco1
justIN time: 2025-11-05 07:26:01 UTC       justIN version: 01.05.01