Jobsub ID 241040.0@dunegpschedd01.fnal.gov
| Jobsub ID | 241040.0@dunegpschedd01.fnal.gov |
| Workflow ID | 9405 |
| Stage ID | 1 |
| User name | ykermaid@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 18000 (5 hours) |
| Submitted time | 2025-10-30 16:59:54 |
| Site | NL_NIKHEF |
| Entry | VIRGO_NL_NIKHEF_juk |
| Last heartbeat | 2025-10-30 17:09:34 |
| From worker node | Hostname | wn-sate-037.farm.nikhef.nl |
| cpuinfo | AMD EPYC 7551P 32-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 129600 (36 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | jobscript_error |
| Started | 2025-10-30 17:01:15 |
| Input files | vd-protodune:np02vd_raw_run040270_0597_df-s04-d2_dw_0_20251028T052512.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Max RSS bytes | 0 (0 MiB) |
| Outputting started | |
| Output files | |
| Finished | 2025-10-30 17:09:34 |
| Saved logs | justin-logs:241040.0-dunegpschedd01.fnal.gov.logs.tgz |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
s and 837 XVs -> 53 XUVs
C:0 T:4 56 XUs and 79 XVs -> 2 XUVs
C:0 T:5 1667 XUs and 1864 XVs -> 104 XUVs
C:0 T:6 274 XUs and 244 XVs -> 8 XUVs
C:0 T:7 847 XUs and 1005 XVs -> 34 XUVs
C:0 T:8 1016 XUs and 1821 XVs -> 86 XUVs
C:0 T:9 506 XUs and 1024 XVs -> 67 XUVs
C:0 T:10 607 XUs and 827 XVs -> 68 XUVs
C:0 T:11 279 XUs and 450 XVs -> 26 XUVs
C:0 T:12 327 XUs and 941 XVs -> 25 XUVs
C:0 T:13 744 XUs and 814 XVs -> 60 XUVs
C:0 T:14 1495 XUs and 1708 XVs -> 133 XUVs
C:0 T:15 828 XUs and 944 XVs -> 73 XUVs
815 XUVs total
663 collection wire objects
815 potential space points
Neighbour search...
5955 tests to find 3094 neighbours
Iterating with no regularization...
Begin: 7.61597e+08
0 7.52672e+08
1 7.5259e+08
Now with regularization...
Begin: 7.38024e+08
0 7.3801e+08
30-Oct-2025 18:05:04 CET Opened output file with pattern "%ifb_reco_stage1_%tc_keepup.root"
RawFrameSource: got 12288 raw::RawDigit objects
input nticks=8000 keeping as is
[18:06:16.559] D [ main ] executing 1 apps, thread limit 0:
[18:06:16.559] D [ main ] executing 1 apps, thread limit 0:
[18:06:16.559] D [ main ] executing app: "Pgrapher"
[18:06:16.559] D [ pgraph ] <Pgrapher:> executing graph
[18:06:16.559] D [ pgraph ] executing with 26 nodes
[18:06:16.561] D [ glue ] <FrameFanout:nfsp> call=2: input: frame: ident=110470 time=4 tick=512 with 12288 traces. frame tags:[ "orig" ] 0 tagged trace sets:[ ] cmm:[ ] output 0: frame: ident=110470 time=4 tick=512 with 12288 traces. frame tags:[ "orig0" ] 0 tagged trace sets:[ ] cmm:[ ] output 1: frame: ident=110470 time=4 tick=512 with 12288 traces. frame tags:[ "orig1" ] 0 tagged trace sets:[ ] cmm:[ ] output 2: frame: ident=110470 time=4 tick=512 with 12288 traces. frame tags:[ "orig2" ] 0 tagged trace sets:[ ] cmm:[ ] output 3: frame: ident=110470 time=4 tick=512 with 12288 traces. frame tags:[ "orig3" ] 0 tagged trace sets:[ ] cmm:[ ] output 4: frame: ident=110470 time=4 tick=512 with 12288 traces. frame tags:[ "orig4" ] 0 tagged trace sets:[ ] cmm:[ ] output 5: frame: ident=110470 time=4 tick=512 with 12288 traces. frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] output 6: frame: ident=110470 time=4 tick=512 with 12288 traces. frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] output 7: frame: ident=110470 time=4 tick=512 with 12288 traces. frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ]
[18:06:16.562] W [ glue ] <ChannelSelector:chsel7> Untagged summary not supported, summary will be dropped.
[18:06:16.563] D [ glue ] <ChannelSelector:chsel7> input frame: ident=110470 time=4 tick=512 with 12288 traces. frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=110470 time=4 tick=512 with 1536 traces. frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ]
[18:06:16.564] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=2 input frame: frame: ident=110470 time=4 tick=512 with 1536 traces. frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ]
[18:06:16.564] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=2 init nticks=8000 tbinmin=0 tbinmax=8000
[18:06:16.603] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=2 load plane index: 0, ntraces=1536, input bad regions: 0
[18:06:18.076] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=2 load plane index: 1, ntraces=1536, input bad regions: 0
[18:06:19.782] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=2 load plane index: 2, ntraces=1536, input bad regions: 0
[18:07:13.300] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=2 save plane index: 0, Qtot=53864387404 Qloss=-69815874406, 6664 indices spanning [14163,20826] "wiener"
[18:07:13.511] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=2 save plane index: 0, Qtot=52067772516 Qloss=-70076921097, 5977 indices spanning [20827,26803] "gauss"
[18:07:14.960] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=2 save plane index: 1, Qtot=32980864622 Qloss=-19869420936, 11728 indices spanning [26804,38531] "wiener"
[18:07:15.168] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=2 save plane index: 1, Qtot=29012281446 Qloss=-20023978496, 10722 indices spanning [38532,49253] "gauss"
[18:07:15.546] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=2 save plane index: 2, Qtot=4122204369 Qloss=-1947731851, 12317 indices spanning [49254,61570] "wiener"
[18:07:15.934] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=2 save plane index: 2, Qtot=3624344061 Qloss=-1450296174, 14051 indices spanning [61571,75621] "gauss"
[18:07:15.934] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=2 produce 75622 traces: 30709 wiener7, 0 decon_charge7, 30750 gauss7, frame tag: sigproc
[18:07:15.934] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=2 output frame: frame: ident=110470 time=4 tick=512 with 75622 traces. frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss7":30750 [0] "mp2_roi7":9514 [0] "mp3_roi7":4649 [0] "wiener7":30709 [30709] ] cmm:[ ]
[18:07:24.977] W [ glue ] <ChannelSelector:chsel6> Untagged summary not supported, summary will be dropped.
[18:07:24.978] D [ glue ] <ChannelSelector:chsel6> input frame: ident=110470 time=4 tick=512 with 12288 traces. frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=110470 time=4 tick=512 with 1536 traces. frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ]
[18:07:24.978] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=2 input frame: frame: ident=110470 time=4 tick=512 with 1536 traces. frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ]
[18:07:24.978] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=2 init nticks=8000 tbinmin=0 tbinmax=8000
[18:07:25.012] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=2 load plane index: 0, ntraces=1536, input bad regions: 0
[18:07:27.020] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=2 load plane index: 1, ntraces=1536, input bad regions: 0
[18:07:29.242] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=2 load plane index: 2, ntraces=1536, input bad regions: 0
==================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
==================================================================================================================================
Full event 6.2086e-05 99.9142 199.828 99.9142 99.9142 2
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read) 6.2086e-05 0.000116744 0.000171402 0.000116744 5.4658e-05 2
produce:tpcrawdecoder:PDVDTPCReader 64.0289 73.1608 82.2927 73.1608 9.1319 2
produce:triggerrawdecoder:PDVDTriggerReader4 0.296206 0.310238 0.32427 0.310238 0.0140319 2
produce:pdvddaphne:DAPHNEReaderPDVD 0.000391887 0.000530583 0.000669279 0.000530583 0.000138696 2
produce:ophit:OpHitFinder 6.0795e-05 0.000277317 0.000493839 0.000277317 0.000216522 2
produce:opflash:OpFlashFinderVerticalDrift 5.0555e-05 0.000172169 0.000293782 0.000172169 0.000121613 2
produce:wclsdatavd:WireCellToolkit 71.1236 71.1236 71.1236 71.1236 0 1
produce:gaushit:GausHitFinder 1.16017 1.16017 1.16017 1.16017 0 1
produce:nhitsfilter:NumberOfHitsFilter 0.000660773 0.000660773 0.000660773 0.000660773 0 1
produce:reco3d:SpacePointSolver 10.0342 10.0342 10.0342 10.0342 0 1
produce:hitpdune:DisambigFromSpacePoints 0.146171 0.146171 0.146171 0.146171 0 1
produce:pandora:StandardPandora 28.2363 28.2363 28.2363 28.2363 0 1
produce:pandoraTrack:LArPandoraTrackCreation 0.75721 0.75721 0.75721 0.75721 0 1
produce:pandoraGnocalo:GnocchiCalorimetry 0.0350557 0.0350557 0.0350557 0.0350557 0 1
[art]:TriggerResults:TriggerResultInserter 6.2517e-05 6.2517e-05 6.2517e-05 6.2517e-05 0 1
end_path:out1:RootOutput 2.5078e-05 2.5078e-05 2.5078e-05 2.5078e-05 0 1
end_path:out1:RootOutput(write) 5.71882 5.71882 5.71882 5.71882 0 1
==================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 8589.83 MB
Peak resident set size usage (VmHWM): 6705.52 MB
Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException: PostEndJob 30-Oct-2025 18:09:05 CET ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- ScheduleExecutionFailure BEGIN
Path: ProcessingStopped.
---- BadAlloc BEGIN
A bad_alloc exception was thrown while processing module WireCellToolkit/wclsdatavd run: 40270 subRun: 1 event: 110470
The job has probably exhausted the virtual memory available to the process.
---- BadAlloc END
Exception going through path produce
---- ScheduleExecutionFailure END
---- EventProcessorFailure END
%MSG
Art has completed and will exit with status 1.
Error in reco1 