Jobsub ID 234515.2@dunegpschedd02.fnal.gov
| Jobsub ID | 234515.2@dunegpschedd02.fnal.gov |
| Workflow ID | 9374 |
| Stage ID | 1 |
| User name | ykermaid@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 18000 (5 hours) |
| Submitted time | 2025-10-29 10:40:18 |
| Site | NL_NIKHEF |
| Entry | VIRGO_NL_NIKHEF_klomp |
| Last heartbeat | 2025-10-29 12:03:16 |
| From worker node | Hostname | wn-lot-036.farm.nikhef.nl |
| cpuinfo | AMD EPYC 7702P 64-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 129600 (36 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | aborted |
| Started | 2025-10-29 11:44:17 |
| Input files | vd-protodune:np02vd_raw_run040266_0666_df-s04-d1_dw_0_20251025T155100.hdf5
|
| Outputting started | |
| Output files | |
| Finished | 2025-10-29 12:03:16 |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
XUs and 3997 XVs -> 175 XUVs
C:0 T:3 369 XUs and 440 XVs -> 22 XUVs
C:0 T:4 200 XUs and 449 XVs -> 11 XUVs
C:0 T:5 809 XUs and 647 XVs -> 46 XUVs
C:0 T:6 22236 XUs and 34973 XVs -> 1994 XUVs
C:0 T:7 365 XUs and 381 XVs -> 21 XUVs
C:0 T:8 49 XUs and 82 XVs -> 2 XUVs
C:0 T:9 1611 XUs and 1514 XVs -> 79 XUVs
C:0 T:10 3145 XUs and 2952 XVs -> 209 XUVs
C:0 T:11 1291 XUs and 1588 XVs -> 46 XUVs
C:0 T:12 1771 XUs and 2390 XVs -> 177 XUVs
C:0 T:13 3648 XUs and 3789 XVs -> 177 XUVs
C:0 T:14 1295 XUs and 1693 XVs -> 68 XUVs
C:0 T:15 243 XUs and 268 XVs -> 18 XUVs
3196 XUVs total
1827 collection wire objects
3196 potential space points
Neighbour search...
87926 tests to find 22598 neighbours
Iterating with no regularization...
Begin: 1.27666e+10
0 1.14705e+10
1 1.13949e+10
2 1.13915e+10
Now with regularization...
Begin: 1.12513e+10
0 1.12506e+10
RawFrameSource: got 12288 raw::RawDigit objects
input nticks=8656 keeping as is
[12:52:46.663] D [ main ] executing 1 apps, thread limit 0:
[12:52:46.663] D [ main ] executing 1 apps, thread limit 0:
[12:52:46.663] D [ main ] executing app: "Pgrapher"
[12:52:46.663] D [ pgraph ] <Pgrapher:> executing graph
[12:52:46.663] D [ pgraph ] executing with 26 nodes
[12:52:46.664] D [ glue ] <FrameFanout:nfsp> call=6: input: frame: ident=122773 time=19 tick=512 with 12288 traces. frame tags:[ "orig" ] 0 tagged trace sets:[ ] cmm:[ ] output 0: frame: ident=122773 time=19 tick=512 with 12288 traces. frame tags:[ "orig0" ] 0 tagged trace sets:[ ] cmm:[ ] output 1: frame: ident=122773 time=19 tick=512 with 12288 traces. frame tags:[ "orig1" ] 0 tagged trace sets:[ ] cmm:[ ] output 2: frame: ident=122773 time=19 tick=512 with 12288 traces. frame tags:[ "orig2" ] 0 tagged trace sets:[ ] cmm:[ ] output 3: frame: ident=122773 time=19 tick=512 with 12288 traces. frame tags:[ "orig3" ] 0 tagged trace sets:[ ] cmm:[ ] output 4: frame: ident=122773 time=19 tick=512 with 12288 traces. frame tags:[ "orig4" ] 0 tagged trace sets:[ ] cmm:[ ] output 5: frame: ident=122773 time=19 tick=512 with 12288 traces. frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] output 6: frame: ident=122773 time=19 tick=512 with 12288 traces. frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] output 7: frame: ident=122773 time=19 tick=512 with 12288 traces. frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ]
[12:52:46.664] W [ glue ] <ChannelSelector:chsel7> Untagged summary not supported, summary will be dropped.
[12:52:46.664] D [ glue ] <ChannelSelector:chsel7> input frame: ident=122773 time=19 tick=512 with 12288 traces. frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=122773 time=19 tick=512 with 1536 traces. frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ]
[12:52:46.665] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=6 input frame: frame: ident=122773 time=19 tick=512 with 1536 traces. frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ]
[12:52:46.665] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=6 init nticks=8656 tbinmin=0 tbinmax=8656
[12:52:46.692] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=6 load plane index: 0, ntraces=1536, input bad regions: 0
[12:52:49.106] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=6 load plane index: 1, ntraces=1536, input bad regions: 0
[12:52:51.662] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=6 load plane index: 2, ntraces=1536, input bad regions: 0
[12:53:29.433] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=6 save plane index: 0, Qtot=31521022762 Qloss=-50424274277, 6475 indices spanning [22743,29217] "wiener"
[12:53:29.699] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=6 save plane index: 0, Qtot=29896173606 Qloss=-50659621880, 5775 indices spanning [29218,34992] "gauss"
[12:53:30.969] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=6 save plane index: 1, Qtot=39776687544 Qloss=-15703387161, 10640 indices spanning [34993,45632] "wiener"
[12:53:31.230] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=6 save plane index: 1, Qtot=36151776980 Qloss=-16113055499, 10043 indices spanning [45633,55675] "gauss"
[12:53:31.724] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=6 save plane index: 2, Qtot=2938107977 Qloss=-3607649470, 11306 indices spanning [55676,66981] "wiener"
[12:53:32.226] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=6 save plane index: 2, Qtot=2546256394 Qloss=-3220623379, 13421 indices spanning [66982,80402] "gauss"
[12:53:32.226] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=6 produce 80403 traces: 28421 wiener7, 0 decon_charge7, 29239 gauss7, frame tag: sigproc
[12:53:32.227] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=6 output frame: frame: ident=122773 time=19 tick=512 with 80403 traces. frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss7":29239 [0] "mp2_roi7":16957 [0] "mp3_roi7":5786 [0] "wiener7":28421 [28421] ] cmm:[ ]
[12:53:36.144] W [ glue ] <ChannelSelector:chsel6> Untagged summary not supported, summary will be dropped.
[12:53:36.144] D [ glue ] <ChannelSelector:chsel6> input frame: ident=122773 time=19 tick=512 with 12288 traces. frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=122773 time=19 tick=512 with 1536 traces. frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ]
[12:53:36.145] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=6 input frame: frame: ident=122773 time=19 tick=512 with 1536 traces. frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ]
[12:53:36.145] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=6 init nticks=8656 tbinmin=0 tbinmax=8656
[12:53:36.173] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=6 load plane index: 0, ntraces=1536, input bad regions: 0
[12:53:38.613] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=6 load plane index: 1, ntraces=1536, input bad regions: 0
[12:53:41.162] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=6 load plane index: 2, ntraces=1536, input bad regions: 0
==================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
==================================================================================================================================
Full event 5.4923e-05 115.739 173.719 144.618 69.3481 4
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read) 5.4923e-05 8.89143e-05 0.000167604 6.6565e-05 4.59117e-05 4
produce:tpcrawdecoder:PDVDTPCReader 12.6997 12.9635 13.5755 12.7893 0.359981 4
produce:triggerrawdecoder:PDVDTriggerReader4 0.509877 0.528939 0.552923 0.526477 0.0191922 4
produce:pdvddaphne:DAPHNEReaderPDVD 0.000302829 0.000387031 0.000606108 0.000319594 0.000126889 4
produce:ophit:OpHitFinder 4.8801e-05 0.000148964 0.000443522 5.1767e-05 0.000170068 4
produce:opflash:OpFlashFinderVerticalDrift 3.7761e-05 9.63585e-05 0.000269807 3.8933e-05 0.000100144 4
produce:wclsdatavd:WireCellToolkit 56.2421 77.7552 100.328 76.6952 18.0137 3
produce:gaushit:GausHitFinder 0.98589 1.23174 1.49336 1.21598 0.207474 3
produce:nhitsfilter:NumberOfHitsFilter 0.000296727 0.00033526 0.000410581 0.000298471 5.3265e-05 3
produce:reco3d:SpacePointSolver 10.0475 12.4428 14.2392 13.0417 1.76286 3
produce:hitpdune:DisambigFromSpacePoints 0.18052 0.24366 0.341288 0.209173 0.0700173 3
produce:pandora:StandardPandora 31.0642 43.5711 61.1353 38.5137 12.7867 3
produce:pandoraTrack:LArPandoraTrackCreation 0.748471 1.02956 1.27741 1.0628 0.217216 3
produce:pandoraGnocalo:GnocchiCalorimetry 0.029386 0.0327989 0.0348739 0.0341368 0.00243195 3
[art]:TriggerResults:TriggerResultInserter 1.4768e-05 2.96927e-05 5.3941e-05 2.0369e-05 1.7298e-05 3
end_path:out1:RootOutput 3.507e-06 9.31767e-06 2.0679e-05 3.767e-06 8.03438e-06 3
end_path:out1:RootOutput(write) 4.47199 4.66823 5.04986 4.48284 0.269889 3
==================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 8589.9 MB
Peak resident set size usage (VmHWM): 6705.38 MB
Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException: PostEndJob 29-Oct-2025 12:54:48 CET ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- ScheduleExecutionFailure BEGIN
Path: ProcessingStopped.
---- BadAlloc BEGIN
A bad_alloc exception was thrown while processing module WireCellToolkit/wclsdatavd run: 40266 subRun: 1 event: 122773
The job has probably exhausted the virtual memory available to the process.
---- BadAlloc END
Exception going through path produce
---- ScheduleExecutionFailure END
---- EventProcessorFailure END
%MSG
Art has completed and will exit with status 1.
Error in reco1 