Jobsub ID 241373.5@dunegpschedd01.fnal.gov
| Jobsub ID | 241373.5@dunegpschedd01.fnal.gov |
| Workflow ID | 9405 |
| Stage ID | 1 |
| User name | ykermaid@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 18000 (5 hours) |
| Submitted time | 2025-10-31 11:50:55 |
| Site | NL_NIKHEF |
| Entry | VIRGO_NL_NIKHEF_juk |
| Last heartbeat | 2025-10-31 12:28:01 |
| From worker node | Hostname | wn-pep-011.farm.nikhef.nl |
| cpuinfo | Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 129600 (36 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | jobscript_error |
| Started | 2025-10-31 11:52:33 |
| Input files | vd-protodune:np02vd_raw_run040270_0672_df-s04-d0_dw_0_20251028T073207.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Max RSS bytes | 0 (0 MiB) |
| Outputting started | |
| Output files | |
| Finished | 2025-10-31 12:28:01 |
| Saved logs | justin-logs:241373.5-dunegpschedd01.fnal.gov.logs.tgz |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
nd 1626 XVs -> 94 XUVs
C:0 T:3 306 XUs and 221 XVs -> 8 XUVs
C:0 T:4 1123 XUs and 1290 XVs -> 43 XUVs
C:0 T:5 2843 XUs and 3621 XVs -> 94 XUVs
C:0 T:6 458 XUs and 526 XVs -> 27 XUVs
C:0 T:7 300 XUs and 260 XVs -> 7 XUVs
C:0 T:8 640 XUs and 875 XVs -> 48 XUVs
C:0 T:9 1529 XUs and 1810 XVs -> 92 XUVs
C:0 T:10 444 XUs and 861 XVs -> 17 XUVs
C:0 T:11 855 XUs and 1167 XVs -> 46 XUVs
C:0 T:12 1013 XUs and 1156 XVs -> 56 XUVs
C:0 T:13 476 XUs and 594 XVs -> 23 XUVs
C:0 T:14 854 XUs and 1134 XVs -> 85 XUVs
C:0 T:15 998 XUs and 1208 XVs -> 52 XUVs
1067 XUVs total
855 collection wire objects
1067 potential space points
Neighbour search...
11381 tests to find 3750 neighbours
Iterating with no regularization...
Begin: 5.60815e+10
0 5.52667e+10
1 5.51355e+10
2 5.51352e+10
Now with regularization...
Begin: 5.42847e+10
0 5.42837e+10
RawFrameSource: got 12288 raw::RawDigit objects
input nticks=8000 keeping as is
[13:24:51.005] D [ main ] executing 1 apps, thread limit 0:
[13:24:51.005] D [ main ] executing 1 apps, thread limit 0:
[13:24:51.005] D [ main ] executing app: "Pgrapher"
[13:24:51.005] D [ pgraph ] <Pgrapher:> executing graph
[13:24:51.005] D [ pgraph ] executing with 26 nodes
[13:24:51.007] D [ glue ] <FrameFanout:nfsp> call=24: input: frame: ident=124596 time=52 tick=512 with 12288 traces. frame tags:[ "orig" ] 0 tagged trace sets:[ ] cmm:[ ] output 0: frame: ident=124596 time=52 tick=512 with 12288 traces. frame tags:[ "orig0" ] 0 tagged trace sets:[ ] cmm:[ ] output 1: frame: ident=124596 time=52 tick=512 with 12288 traces. frame tags:[ "orig1" ] 0 tagged trace sets:[ ] cmm:[ ] output 2: frame: ident=124596 time=52 tick=512 with 12288 traces. frame tags:[ "orig2" ] 0 tagged trace sets:[ ] cmm:[ ] output 3: frame: ident=124596 time=52 tick=512 with 12288 traces. frame tags:[ "orig3" ] 0 tagged trace sets:[ ] cmm:[ ] output 4: frame: ident=124596 time=52 tick=512 with 12288 traces. frame tags:[ "orig4" ] 0 tagged trace sets:[ ] cmm:[ ] output 5: frame: ident=124596 time=52 tick=512 with 12288 traces. frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] output 6: frame: ident=124596 time=52 tick=512 with 12288 traces. frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] output 7: frame: ident=124596 time=52 tick=512 with 12288 traces. frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ]
[13:24:51.007] W [ glue ] <ChannelSelector:chsel7> Untagged summary not supported, summary will be dropped.
[13:24:51.008] D [ glue ] <ChannelSelector:chsel7> input frame: ident=124596 time=52 tick=512 with 12288 traces. frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=124596 time=52 tick=512 with 1536 traces. frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ]
[13:24:51.008] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=24 input frame: frame: ident=124596 time=52 tick=512 with 1536 traces. frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ]
[13:24:51.008] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=24 init nticks=8000 tbinmin=0 tbinmax=8000
[13:24:51.043] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=24 load plane index: 0, ntraces=1536, input bad regions: 0
[13:24:52.656] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=24 load plane index: 1, ntraces=1536, input bad regions: 0
[13:24:54.262] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=24 load plane index: 2, ntraces=1536, input bad regions: 0
[13:25:47.652] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=24 save plane index: 0, Qtot=52701181951 Qloss=-36211882115, 5348 indices spanning [19054,24401] "wiener"
[13:25:47.861] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=24 save plane index: 0, Qtot=50668934368 Qloss=-37137061211, 5002 indices spanning [24402,29403] "gauss"
[13:25:48.951] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=24 save plane index: 1, Qtot=98784878404 Qloss=-2681666232, 12406 indices spanning [29404,41809] "wiener"
[13:25:49.156] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=24 save plane index: 1, Qtot=91344935143 Qloss=-2775099508, 11935 indices spanning [41810,53744] "gauss"
[13:25:49.507] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=24 save plane index: 2, Qtot=2470314042 Qloss=-3010223762, 11233 indices spanning [53745,64977] "wiener"
[13:25:49.872] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=24 save plane index: 2, Qtot=1971990436 Qloss=-2520674706, 13310 indices spanning [64978,78287] "gauss"
[13:25:49.873] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=24 produce 78288 traces: 28987 wiener7, 0 decon_charge7, 30247 gauss7, frame tag: sigproc
[13:25:49.873] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=24 output frame: frame: ident=124596 time=52 tick=512 with 78288 traces. frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss7":30247 [0] "mp2_roi7":13945 [0] "mp3_roi7":5109 [0] "wiener7":28987 [28987] ] cmm:[ ]
[13:25:56.509] W [ glue ] <ChannelSelector:chsel6> Untagged summary not supported, summary will be dropped.
[13:25:56.510] D [ glue ] <ChannelSelector:chsel6> input frame: ident=124596 time=52 tick=512 with 12288 traces. frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=124596 time=52 tick=512 with 1536 traces. frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ]
[13:25:56.510] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=24 input frame: frame: ident=124596 time=52 tick=512 with 1536 traces. frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ]
[13:25:56.510] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=24 init nticks=8000 tbinmin=0 tbinmax=8000
[13:25:56.546] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=24 load plane index: 0, ntraces=1536, input bad regions: 0
[13:25:58.447] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=24 load plane index: 1, ntraces=1536, input bad regions: 0
[13:26:00.225] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=24 load plane index: 2, ntraces=1536, input bad regions: 0
==================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
==================================================================================================================================
Full event 6.8366e-05 142.82 232.174 140.069 54.2376 13
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read) 6.0344e-05 8.85245e-05 0.000173363 6.8501e-05 3.26902e-05 13
produce:tpcrawdecoder:PDVDTPCReader 16.7768 34.1609 48.7905 37.6265 12.8602 13
produce:triggerrawdecoder:PDVDTriggerReader4 0.288719 0.30615 0.364403 0.298458 0.0226222 13
produce:pdvddaphne:DAPHNEReaderPDVD 0.000373468 0.000423048 0.000722313 0.000396891 8.85105e-05 13
produce:ophit:OpHitFinder 5.2663e-05 0.000100229 0.000491152 6.3038e-05 0.000113628 13
produce:opflash:OpFlashFinderVerticalDrift 4.1017e-05 6.91845e-05 0.000328163 4.8247e-05 7.48686e-05 13
produce:wclsdatavd:WireCellToolkit 60.3727 74.5639 120.556 64.4917 18.7427 12
produce:gaushit:GausHitFinder 0.997804 1.37022 2.25344 1.24689 0.326596 12
produce:nhitsfilter:NumberOfHitsFilter 0.000345424 0.000440183 0.000668931 0.000414513 8.92448e-05 12
produce:reco3d:SpacePointSolver 7.8801 11.1394 16.8034 10.2225 2.64841 12
produce:hitpdune:DisambigFromSpacePoints 0.142157 0.205409 0.303296 0.184097 0.0567064 12
produce:pandora:StandardPandora 15.8712 28.3476 50.0933 23.7476 10.6026 12
produce:pandoraTrack:LArPandoraTrackCreation 0.67235 1.19283 1.92776 1.10929 0.418322 12
produce:pandoraGnocalo:GnocchiCalorimetry 0.0233346 0.0311705 0.0396691 0.0314042 0.00555109 12
[art]:TriggerResults:TriggerResultInserter 2.368e-05 3.16617e-05 7.2384e-05 2.87095e-05 1.25981e-05 12
end_path:out1:RootOutput 7.078e-06 1.0363e-05 3.4071e-05 8.0165e-06 7.26404e-06 12
end_path:out1:RootOutput(write) 3.99057 4.24545 4.73497 4.10463 0.252053 12
==================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 8589.88 MB
Peak resident set size usage (VmHWM): 6702.91 MB
Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException: PostEndJob 31-Oct-2025 13:27:40 CET ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- ScheduleExecutionFailure BEGIN
Path: ProcessingStopped.
---- BadAlloc BEGIN
A bad_alloc exception was thrown while processing module WireCellToolkit/wclsdatavd run: 40270 subRun: 1 event: 124596
The job has probably exhausted the virtual memory available to the process.
---- BadAlloc END
Exception going through path produce
---- ScheduleExecutionFailure END
---- EventProcessorFailure END
%MSG
Art has completed and will exit with status 1.
Error in reco1 