Jobsub ID 236254.3@dunegpschedd02.fnal.gov
| Jobsub ID | 236254.3@dunegpschedd02.fnal.gov |
| Workflow ID | 9405 |
| Stage ID | 1 |
| User name | ykermaid@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 18000 (5 hours) |
| Submitted time | 2025-11-02 08:01:16 |
| Site | NL_NIKHEF |
| Entry | VIRGO_NL_NIKHEF_juk |
| Last heartbeat | 2025-11-02 08:10:09 |
| From worker node | Hostname | wn-sate-050.farm.nikhef.nl |
| cpuinfo | AMD EPYC 7551P 32-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 129600 (36 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | jobscript_error |
| Started | 2025-11-02 08:01:29 |
| Input files | vd-protodune:np02vd_raw_run040270_0624_df-s03-d3_dw_0_20251028T061135.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Max RSS bytes | 0 (0 MiB) |
| Outputting started | |
| Output files | |
| Finished | 2025-11-02 08:10:09 |
| Saved logs | justin-logs:236254.3-dunegpschedd02.fnal.gov.logs.tgz |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
s -> 36 XUVs
C:0 T:4 585 XUs and 520 XVs -> 14 XUVs
C:0 T:5 1342 XUs and 2031 XVs -> 46 XUVs
C:0 T:6 16631 XUs and 20684 XVs -> 1474 XUVs
C:0 T:7 373 XUs and 493 XVs -> 14 XUVs
C:0 T:8 1058 XUs and 1309 XVs -> 75 XUVs
C:0 T:9 332 XUs and 291 XVs -> 6 XUVs
C:0 T:10 1745 XUs and 1901 XVs -> 67 XUVs
C:0 T:11 939 XUs and 1194 XVs -> 89 XUVs
C:0 T:12 1580 XUs and 1876 XVs -> 158 XUVs
C:0 T:13 763 XUs and 1024 XVs -> 60 XUVs
C:0 T:14 388 XUs and 518 XVs -> 23 XUVs
C:0 T:15 1046 XUs and 1250 XVs -> 87 XUVs
2227 XUVs total
1352 collection wire objects
2227 potential space points
Neighbour search...
72509 tests to find 18124 neighbours
Iterating with no regularization...
Begin: 8.19477e+10
0 8.08484e+10
1 8.0791e+10
Now with regularization...
Begin: 8.01684e+10
0 8.01677e+10
02-Nov-2025 09:05:51 CET Opened output file with pattern "%ifb_reco_stage1_%tc_keepup.root"
RawFrameSource: got 12288 raw::RawDigit objects
input nticks=8642 keeping as is
[09:07:28.978] D [ main ] executing 1 apps, thread limit 0:
[09:07:28.978] D [ main ] executing 1 apps, thread limit 0:
[09:07:28.978] D [ main ] executing app: "Pgrapher"
[09:07:28.978] D [ pgraph ] <Pgrapher:> executing graph
[09:07:28.978] D [ pgraph ] executing with 26 nodes
[09:07:28.980] D [ glue ] <FrameFanout:nfsp> call=2: input: frame: ident=115555 time=4 tick=512 with 12288 traces. frame tags:[ "orig" ] 0 tagged trace sets:[ ] cmm:[ ] output 0: frame: ident=115555 time=4 tick=512 with 12288 traces. frame tags:[ "orig0" ] 0 tagged trace sets:[ ] cmm:[ ] output 1: frame: ident=115555 time=4 tick=512 with 12288 traces. frame tags:[ "orig1" ] 0 tagged trace sets:[ ] cmm:[ ] output 2: frame: ident=115555 time=4 tick=512 with 12288 traces. frame tags:[ "orig2" ] 0 tagged trace sets:[ ] cmm:[ ] output 3: frame: ident=115555 time=4 tick=512 with 12288 traces. frame tags:[ "orig3" ] 0 tagged trace sets:[ ] cmm:[ ] output 4: frame: ident=115555 time=4 tick=512 with 12288 traces. frame tags:[ "orig4" ] 0 tagged trace sets:[ ] cmm:[ ] output 5: frame: ident=115555 time=4 tick=512 with 12288 traces. frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] output 6: frame: ident=115555 time=4 tick=512 with 12288 traces. frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] output 7: frame: ident=115555 time=4 tick=512 with 12288 traces. frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ]
[09:07:28.980] W [ glue ] <ChannelSelector:chsel7> Untagged summary not supported, summary will be dropped.
[09:07:28.981] D [ glue ] <ChannelSelector:chsel7> input frame: ident=115555 time=4 tick=512 with 12288 traces. frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=115555 time=4 tick=512 with 1536 traces. frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ]
[09:07:28.981] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=2 input frame: frame: ident=115555 time=4 tick=512 with 1536 traces. frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ]
[09:07:28.981] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=2 init nticks=8642 tbinmin=0 tbinmax=8642
[09:07:29.012] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=2 load plane index: 0, ntraces=1536, input bad regions: 0
[09:07:31.765] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=2 load plane index: 1, ntraces=1536, input bad regions: 0
[09:07:34.542] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=2 load plane index: 2, ntraces=1536, input bad regions: 0
[09:08:14.841] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=2 save plane index: 0, Qtot=26462367879 Qloss=-6021980142, 4626 indices spanning [13392,18017] "wiener"
[09:08:15.180] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=2 save plane index: 0, Qtot=25038808219 Qloss=-6352028614, 4302 indices spanning [18018,22319] "gauss"
[09:08:16.389] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=2 save plane index: 1, Qtot=46106745212 Qloss=-15363404212, 6350 indices spanning [22320,28669] "wiener"
[09:08:16.725] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=2 save plane index: 1, Qtot=43479456978 Qloss=-15585413851, 5812 indices spanning [28670,34481] "gauss"
[09:08:17.269] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=2 save plane index: 2, Qtot=1836442293 Qloss=-1537083112, 7219 indices spanning [34482,41700] "wiener"
[09:08:17.814] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=2 save plane index: 2, Qtot=1579233415 Qloss=-1281346610, 8010 indices spanning [41701,49710] "gauss"
[09:08:17.814] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=2 produce 49711 traces: 18195 wiener7, 0 decon_charge7, 18124 gauss7, frame tag: sigproc
[09:08:17.814] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=2 output frame: frame: ident=115555 time=4 tick=512 with 49711 traces. frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss7":18124 [0] "mp2_roi7":9102 [0] "mp3_roi7":4290 [0] "wiener7":18195 [18195] ] cmm:[ ]
[09:08:21.618] W [ glue ] <ChannelSelector:chsel6> Untagged summary not supported, summary will be dropped.
[09:08:21.619] D [ glue ] <ChannelSelector:chsel6> input frame: ident=115555 time=4 tick=512 with 12288 traces. frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=115555 time=4 tick=512 with 1536 traces. frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ]
[09:08:21.619] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=2 input frame: frame: ident=115555 time=4 tick=512 with 1536 traces. frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ]
[09:08:21.619] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=2 init nticks=8642 tbinmin=0 tbinmax=8642
[09:08:21.651] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=2 load plane index: 0, ntraces=1536, input bad regions: 0
[09:08:24.947] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=2 load plane index: 1, ntraces=1536, input bad regions: 0
[09:08:28.150] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=2 load plane index: 2, ntraces=1536, input bad regions: 0
==================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
==================================================================================================================================
Full event 5.7728e-05 112.448 224.897 112.448 112.448 2
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read) 5.7728e-05 0.00013849 0.000219252 0.00013849 8.0762e-05 2
produce:tpcrawdecoder:PDVDTPCReader 70.7885 80.2609 89.7334 80.2609 9.47249 2
produce:triggerrawdecoder:PDVDTriggerReader4 0.329094 0.338797 0.3485 0.338797 0.00970301 2
produce:pdvddaphne:DAPHNEReaderPDVD 0.000409721 0.000589333 0.000768946 0.000589333 0.000179613 2
produce:ophit:OpHitFinder 7.3678e-05 0.000314626 0.000555574 0.000314626 0.000240948 2
produce:opflash:OpFlashFinderVerticalDrift 6.3188e-05 0.000204574 0.000345961 0.000204574 0.000141386 2
produce:wclsdatavd:WireCellToolkit 87.6681 87.6681 87.6681 87.6681 0 1
produce:gaushit:GausHitFinder 1.48372 1.48372 1.48372 1.48372 0 1
produce:nhitsfilter:NumberOfHitsFilter 0.000693073 0.000693073 0.000693073 0.000693073 0 1
produce:reco3d:SpacePointSolver 13.705 13.705 13.705 13.705 0 1
produce:hitpdune:DisambigFromSpacePoints 0.244518 0.244518 0.244518 0.244518 0 1
produce:pandora:StandardPandora 43.6531 43.6531 43.6531 43.6531 0 1
produce:pandoraTrack:LArPandoraTrackCreation 1.53511 1.53511 1.53511 1.53511 0 1
produce:pandoraGnocalo:GnocchiCalorimetry 0.0460646 0.0460646 0.0460646 0.0460646 0 1
[art]:TriggerResults:TriggerResultInserter 0.000122831 0.000122831 0.000122831 0.000122831 0 1
end_path:out1:RootOutput 3.1148e-05 3.1148e-05 3.1148e-05 3.1148e-05 0 1
end_path:out1:RootOutput(write) 5.38847 5.38847 5.38847 5.38847 0 1
==================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 8589.91 MB
Peak resident set size usage (VmHWM): 6707.61 MB
Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException: PostEndJob 02-Nov-2025 09:09:46 CET ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- ScheduleExecutionFailure BEGIN
Path: ProcessingStopped.
---- BadAlloc BEGIN
A bad_alloc exception was thrown while processing module WireCellToolkit/wclsdatavd run: 40270 subRun: 1 event: 115555
The job has probably exhausted the virtual memory available to the process.
---- BadAlloc END
Exception going through path produce
---- ScheduleExecutionFailure END
---- EventProcessorFailure END
%MSG
Art has completed and will exit with status 1.
Error in reco1 