justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 241234.0@dunegpschedd01.fnal.gov

Jobsub ID241234.0@dunegpschedd01.fnal.gov
Workflow ID9374
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-10-31 00:08:18
SiteNL_NIKHEF
EntryVIRGO_NL_NIKHEF_juk
Last heartbeat2025-10-31 01:32:48
From worker nodeHostnamewn-pep-011.farm.nikhef.nl
cpuinfoIntel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Started2025-10-31 00:09:02
Input filesvd-protodune:np02vd_raw_run040266_0287_df-s03-d3_dw_0_20251025T051201.hdf5
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-10-31 01:32:48
Saved logsjustin-logs:241234.0-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

glue  ] <ChannelSelector:chsel7> input frame: ident=53027 time=91 tick=512 with 12288 traces.  frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=53027 time=91 tick=512 with 1536 traces.  frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ] 
[02:30:20.183] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=40 input frame: frame: ident=53027 time=91 tick=512 with 1536 traces.  frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ] 
[02:30:20.183] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=40 init nticks=8000 tbinmin=0 tbinmax=8000 
[02:30:20.221] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=40 load plane index: 0, ntraces=1536, input bad regions: 0 
[02:30:22.379] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=40 load plane index: 1, ntraces=1536, input bad regions: 0 
[02:30:24.550] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=40 load plane index: 2, ntraces=1536, input bad regions: 0 
[02:30:42.976] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=40 save plane index: 0, Qtot=18146415585 Qloss=-10411980611, 4530 indices spanning [11334,15863] "wiener" 
[02:30:43.201] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=40 save plane index: 0, Qtot=17356880185 Qloss=-10450103872, 3926 indices spanning [15864,19789] "gauss" 
[02:30:43.806] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=40 save plane index: 1, Qtot=7428063763 Qloss=-4513799102, 3878 indices spanning [19790,23667] "wiener" 
[02:30:44.037] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=40 save plane index: 1, Qtot=6677570590 Qloss=-4604616181, 3375 indices spanning [23668,27042] "gauss" 
[02:30:44.476] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=40 save plane index: 2, Qtot=2886373915 Qloss=-1733410938, 10400 indices spanning [27043,37442] "wiener" 
[02:30:44.898] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=40 save plane index: 2, Qtot=2489947281 Qloss=-1336828040, 11561 indices spanning [37443,49003] "gauss" 
[02:30:44.898] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=40 produce 49004 traces: 18808 wiener7, 0 decon_charge7, 18862 gauss7, frame tag: sigproc 
[02:30:44.899] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=40 output frame: frame: ident=53027 time=91 tick=512 with 49004 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss7":18862 [0] "mp2_roi7":8735 [0] "mp3_roi7":2599 [0] "wiener7":18808 [18808] ] cmm:[ ] 
[02:30:47.947] W [  glue  ] <ChannelSelector:chsel6> Untagged summary not supported, summary will be dropped. 
[02:30:47.948] D [  glue  ] <ChannelSelector:chsel6> input frame: ident=53027 time=91 tick=512 with 12288 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=53027 time=91 tick=512 with 1536 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] 
[02:30:47.949] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=40 input frame: frame: ident=53027 time=91 tick=512 with 1536 traces.  frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] 
[02:30:47.949] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=40 init nticks=8000 tbinmin=0 tbinmax=8000 
[02:30:47.985] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=40 load plane index: 0, ntraces=1536, input bad regions: 0 
[02:30:50.078] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=40 load plane index: 1, ntraces=1536, input bad regions: 0 
[02:30:52.219] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=40 load plane index: 2, ntraces=1536, input bad regions: 0 
[02:30:55.359] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=40 save plane index: 0, Qtot=73427209 Qloss=-14135621, 1423 indices spanning [7475,8897] "wiener" 
[02:30:55.590] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=40 save plane index: 0, Qtot=71315959 Qloss=-12468977, 1151 indices spanning [8898,10048] "gauss" 
[02:30:55.958] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=40 save plane index: 1, Qtot=75357021 Qloss=-11969639, 2376 indices spanning [10049,12424] "wiener" 
[02:30:56.192] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=40 save plane index: 1, Qtot=69928268 Qloss=-9900914, 2017 indices spanning [12425,14441] "gauss" 
[02:30:56.601] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=40 save plane index: 2, Qtot=66179436 Qloss=-4546122, 2675 indices spanning [14442,17116] "wiener" 
[02:30:57.004] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=40 save plane index: 2, Qtot=63847560 Qloss=-2181090, 2222 indices spanning [17117,19338] "gauss" 
[02:30:57.004] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=40 produce 19339 traces: 6474 wiener6, 0 decon_charge6, 5390 gauss6, frame tag: sigproc 
[02:30:57.004] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=40 output frame: frame: ident=53027 time=91 tick=512 with 19339 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss6":5390 [0] "mp2_roi6":6177 [0] "mp3_roi6":1298 [0] "wiener6":6474 [6474] ] cmm:[ ] 
[02:30:57.029] W [  glue  ] <ChannelSelector:chsel5> Untagged summary not supported, summary will be dropped. 
[02:30:57.030] D [  glue  ] <ChannelSelector:chsel5> input frame: ident=53027 time=91 tick=512 with 12288 traces.  frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=53027 time=91 tick=512 with 1536 traces.  frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] 
[02:30:57.031] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=40 input frame: frame: ident=53027 time=91 tick=512 with 1536 traces.  frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] 
[02:30:57.031] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=40 init nticks=8000 tbinmin=0 tbinmax=8000 
[02:30:57.071] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=40 load plane index: 0, ntraces=1536, input bad regions: 0 
[02:30:59.211] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=40 load plane index: 1, ntraces=1536, input bad regions: 0 
[02:31:01.400] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=40 load plane index: 2, ntraces=1536, input bad regions: 0 

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                       8.67e-05       226.722       320.087       235.425       63.0036        21     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                      5.7943e-05    8.20513e-05   0.000157726   7.0984e-05    2.55663e-05      21     
produce:tpcrawdecoder:PDVDTPCReader               49.9708       87.424        126.374       84.1243       18.5157        21     
produce:triggerrawdecoder:PDVDTriggerReader4     0.525707      0.666244      0.766531      0.697599      0.0759183       21     
produce:pdvddaphne:DAPHNEReaderPDVD             0.000383157   0.000414603   0.000739461   0.000398336   7.31878e-05      21     
produce:ophit:OpHitFinder                       6.3993e-05    9.65589e-05   0.00054006    7.0374e-05    9.9677e-05       21     
produce:opflash:OpFlashFinderVerticalDrift      5.0444e-05    7.17826e-05   0.000315357   5.8149e-05    5.47584e-05      21     
produce:wclsdatavd:WireCellToolkit                59.6321       94.5082       141.454       94.3598       22.8202        20     
produce:gaushit:GausHitFinder                    0.706964       1.44829       2.00696       1.4873       0.305075        20     
produce:nhitsfilter:NumberOfHitsFilter          0.000302651   0.000491814   0.000772248   0.000468701   9.97308e-05      20     
produce:reco3d:SpacePointSolver                   4.83655       12.7005       18.2639       12.0568       3.55171        20     
produce:hitpdune:DisambigFromSpacePoints         0.0900119     0.240224      0.423286       0.23783      0.0860007       20     
produce:pandora:StandardPandora                   13.1215       33.8591       64.0061       32.383        12.488         20     
produce:pandoraTrack:LArPandoraTrackCreation     0.513188       2.08531       4.67976       1.8103        1.04154        20     
produce:pandoraGnocalo:GnocchiCalorimetry        0.0187218     0.0393505     0.086269      0.0349673     0.0145696       20     
[art]:TriggerResults:TriggerResultInserter      3.1529e-05    4.26536e-05   7.9567e-05    4.1126e-05    1.02782e-05      20     
end_path:out1:RootOutput                         4.681e-06    9.5656e-06    2.8018e-05     8.645e-06    4.78954e-06      20     
end_path:out1:RootOutput(write)                   3.90105       4.43513       5.38132       4.4298        0.38673        20     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8589.91 MB
  Peak resident set size usage (VmHWM): 6710.71 MB
  Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException:  PostEndJob 31-Oct-2025 02:32:20 CET ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- BadAlloc BEGIN
      A bad_alloc exception was thrown while processing module WireCellToolkit/wclsdatavd run: 40266 subRun: 1 event: 53027
      The job has probably exhausted the virtual memory available to the process.
    ---- BadAlloc END
    Exception going through path produce
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
%MSG
Art has completed and will exit with status 1.
Error in reco1
justIN time: 2025-11-05 07:26:21 UTC       justIN version: 01.05.01