Jobsub ID 241373.3@dunegpschedd01.fnal.gov
| Jobsub ID | 241373.3@dunegpschedd01.fnal.gov |
| Workflow ID | 9405 |
| Stage ID | 1 |
| User name | ykermaid@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 18000 (5 hours) |
| Submitted time | 2025-10-31 11:50:55 |
| Site | NL_NIKHEF |
| Entry | VIRGO_NL_NIKHEF_klomp |
| Last heartbeat | 2025-10-31 12:11:26 |
| From worker node | Hostname | wn-pep-014.farm.nikhef.nl |
| cpuinfo | Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 129600 (36 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | jobscript_error |
| Started | 2025-10-31 11:52:30 |
| Input files | vd-protodune:np02vd_raw_run040270_0587_df-s04-d2_dw_0_20251028T050817.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Max RSS bytes | 0 (0 MiB) |
| Outputting started | |
| Output files | |
| Finished | 2025-10-31 12:11:26 |
| Saved logs | justin-logs:241373.3-dunegpschedd01.fnal.gov.logs.tgz |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
UVs
C:0 T:3 2578 XUs and 1717 XVs -> 169 XUVs
C:0 T:4 943 XUs and 2450 XVs -> 101 XUVs
C:0 T:5 1312 XUs and 1874 XVs -> 55 XUVs
C:0 T:6 1040 XUs and 1642 XVs -> 116 XUVs
C:0 T:7 757 XUs and 1081 XVs -> 71 XUVs
C:0 T:8 1348 XUs and 2188 XVs -> 226 XUVs
C:0 T:9 5342 XUs and 8652 XVs -> 2415 XUVs
C:0 T:10 257 XUs and 317 XVs -> 27 XUVs
C:0 T:11 820 XUs and 723 XVs -> 78 XUVs
C:0 T:12 1486 XUs and 1823 XVs -> 484 XUVs
C:0 T:13 1524 XUs and 1386 XVs -> 86 XUVs
C:0 T:14 1588 XUs and 980 XVs -> 95 XUVs
C:0 T:15 8896 XUs and 6905 XVs -> 771 XUVs
7765 XUVs total
2679 collection wire objects
7765 potential space points
Neighbour search...
657453 tests to find 262098 neighbours
Iterating with no regularization...
Begin: 7.93987e+10
0 7.70626e+10
1 7.69794e+10
2 7.69774e+10
Now with regularization...
Begin: 7.61679e+10
0 7.61662e+10
RawFrameSource: got 12288 raw::RawDigit objects
input nticks=8000 keeping as is
[13:09:15.497] D [ main ] executing 1 apps, thread limit 0:
[13:09:15.497] D [ main ] executing 1 apps, thread limit 0:
[13:09:15.497] D [ main ] executing app: "Pgrapher"
[13:09:15.497] D [ pgraph ] <Pgrapher:> executing graph
[13:09:15.497] D [ pgraph ] executing with 26 nodes
[13:09:15.498] D [ glue ] <FrameFanout:nfsp> call=12: input: frame: ident=108662 time=27 tick=512 with 12288 traces. frame tags:[ "orig" ] 0 tagged trace sets:[ ] cmm:[ ] output 0: frame: ident=108662 time=27 tick=512 with 12288 traces. frame tags:[ "orig0" ] 0 tagged trace sets:[ ] cmm:[ ] output 1: frame: ident=108662 time=27 tick=512 with 12288 traces. frame tags:[ "orig1" ] 0 tagged trace sets:[ ] cmm:[ ] output 2: frame: ident=108662 time=27 tick=512 with 12288 traces. frame tags:[ "orig2" ] 0 tagged trace sets:[ ] cmm:[ ] output 3: frame: ident=108662 time=27 tick=512 with 12288 traces. frame tags:[ "orig3" ] 0 tagged trace sets:[ ] cmm:[ ] output 4: frame: ident=108662 time=27 tick=512 with 12288 traces. frame tags:[ "orig4" ] 0 tagged trace sets:[ ] cmm:[ ] output 5: frame: ident=108662 time=27 tick=512 with 12288 traces. frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] output 6: frame: ident=108662 time=27 tick=512 with 12288 traces. frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] output 7: frame: ident=108662 time=27 tick=512 with 12288 traces. frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ]
[13:09:15.499] W [ glue ] <ChannelSelector:chsel7> Untagged summary not supported, summary will be dropped.
[13:09:15.500] D [ glue ] <ChannelSelector:chsel7> input frame: ident=108662 time=27 tick=512 with 12288 traces. frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=108662 time=27 tick=512 with 1536 traces. frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ]
[13:09:15.500] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=12 input frame: frame: ident=108662 time=27 tick=512 with 1536 traces. frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ]
[13:09:15.500] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=12 init nticks=8000 tbinmin=0 tbinmax=8000
[13:09:15.535] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=12 load plane index: 0, ntraces=1536, input bad regions: 0
[13:09:17.486] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=12 load plane index: 1, ntraces=1536, input bad regions: 0
[13:09:19.458] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=12 load plane index: 2, ntraces=1536, input bad regions: 0
[13:09:51.494] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=12 save plane index: 0, Qtot=12802629100 Qloss=-3127276296, 9404 indices spanning [15077,24480] "wiener"
[13:09:51.707] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=12 save plane index: 0, Qtot=9374588237 Qloss=-3271041700, 9085 indices spanning [24481,33565] "gauss"
[13:09:53.074] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=12 save plane index: 1, Qtot=33800313185 Qloss=-1071315778, 13790 indices spanning [33566,47355] "wiener"
[13:09:53.287] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=12 save plane index: 1, Qtot=27373045917 Qloss=-1158635013, 13594 indices spanning [47356,60949] "gauss"
[13:09:53.720] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=12 save plane index: 2, Qtot=3225794708 Qloss=-2370351739, 14373 indices spanning [60950,75322] "wiener"
[13:09:54.139] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=12 save plane index: 2, Qtot=2446326828 Qloss=-1585775619, 16039 indices spanning [75323,91361] "gauss"
[13:09:54.140] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=12 produce 91362 traces: 37567 wiener7, 0 decon_charge7, 38718 gauss7, frame tag: sigproc
[13:09:54.140] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=12 output frame: frame: ident=108662 time=27 tick=512 with 91362 traces. frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss7":38718 [0] "mp2_roi7":10067 [0] "mp3_roi7":5010 [0] "wiener7":37567 [37567] ] cmm:[ ]
[13:09:58.772] W [ glue ] <ChannelSelector:chsel6> Untagged summary not supported, summary will be dropped.
[13:09:58.773] D [ glue ] <ChannelSelector:chsel6> input frame: ident=108662 time=27 tick=512 with 12288 traces. frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=108662 time=27 tick=512 with 1536 traces. frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ]
[13:09:58.773] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=12 input frame: frame: ident=108662 time=27 tick=512 with 1536 traces. frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ]
[13:09:58.773] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=12 init nticks=8000 tbinmin=0 tbinmax=8000
[13:09:58.820] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=12 load plane index: 0, ntraces=1536, input bad regions: 0
[13:10:00.511] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=12 load plane index: 1, ntraces=1536, input bad regions: 0
[13:10:02.293] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=12 load plane index: 2, ntraces=1536, input bad regions: 0
==================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
==================================================================================================================================
Full event 7.1029e-05 129.855 247.002 124.846 69.4167 7
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read) 5.9151e-05 8.46527e-05 0.000184421 7.1029e-05 4.09432e-05 7
produce:tpcrawdecoder:PDVDTPCReader 11.581 26.4224 61.5948 16.875 18.7399 7
produce:triggerrawdecoder:PDVDTriggerReader4 0.29065 0.302216 0.329426 0.292681 0.0139751 7
produce:pdvddaphne:DAPHNEReaderPDVD 0.00037677 0.000429814 0.000685928 0.000389084 0.000104647 7
produce:ophit:OpHitFinder 5.9762e-05 0.000123324 0.000447038 6.5955e-05 0.000132581 7
produce:opflash:OpFlashFinderVerticalDrift 4.9435e-05 8.96474e-05 0.000298691 5.5088e-05 8.54415e-05 7
produce:wclsdatavd:WireCellToolkit 59.3368 81.4012 131.058 72.5163 24.8526 6
produce:gaushit:GausHitFinder 1.05447 1.28936 1.75625 1.16279 0.255941 6
produce:nhitsfilter:NumberOfHitsFilter 0.000360088 0.000448228 0.000600451 0.000406504 8.3369e-05 6
produce:reco3d:SpacePointSolver 8.34278 11.5937 17.6847 10.9494 2.97753 6
produce:hitpdune:DisambigFromSpacePoints 0.138296 0.207023 0.401948 0.174989 0.0892272 6
produce:pandora:StandardPandora 18.3085 30.2192 66.4684 22.1234 16.8347 6
produce:pandoraTrack:LArPandoraTrackCreation 0.788309 1.37189 3.14514 1.07265 0.804608 6
produce:pandoraGnocalo:GnocchiCalorimetry 0.0255621 0.031954 0.0423802 0.0311405 0.00613812 6
[art]:TriggerResults:TriggerResultInserter 2.8479e-05 3.99142e-05 8.4201e-05 3.17905e-05 1.98733e-05 6
end_path:out1:RootOutput 6.99e-06 1.03233e-05 2.5156e-05 7.2285e-06 6.64916e-06 6
end_path:out1:RootOutput(write) 4.03511 4.4829 5.0116 4.3972 0.41397 6
==================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 8589.93 MB
Peak resident set size usage (VmHWM): 6706.75 MB
Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException: PostEndJob 31-Oct-2025 13:11:06 CET ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- ScheduleExecutionFailure BEGIN
Path: ProcessingStopped.
---- BadAlloc BEGIN
A bad_alloc exception was thrown while processing module WireCellToolkit/wclsdatavd run: 40270 subRun: 1 event: 108662
The job has probably exhausted the virtual memory available to the process.
---- BadAlloc END
Exception going through path produce
---- ScheduleExecutionFailure END
---- EventProcessorFailure END
%MSG
Art has completed and will exit with status 1.
Error in reco1 