Jobsub ID 41115.26@dunegpschedd02.fnal.gov
| Jobsub ID | 41115.26@dunegpschedd02.fnal.gov |
| Workflow ID | 2742 |
| Stage ID | 1 |
| User name | ykermaid@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 18000 (5 hours) |
| Submitted time | 2025-09-17 17:17:36 |
| Site | NL_NIKHEF |
| Entry | VIRGO_NL_NIKHEF_juk |
| Last heartbeat | 2025-09-17 17:40:51 |
| From worker node | Hostname | wn-sate-052.farm.nikhef.nl |
| cpuinfo | AMD EPYC 7551P 32-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 129600 (36 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | jobscript_error |
| Started | 2025-09-17 17:20:18 |
| Input files | vd-protodune:np02vd_raw_run039433_0006_df-s05-d4_dw_0_20250917T155308.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Max RSS bytes | 0 (0 MiB) |
| Outputting started | |
| Output files | |
| Finished | 2025-09-17 17:40:51 |
| Saved logs | justin-logs:41115.26-dunegpschedd02.fnal.gov.logs.tgz |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
e: ident=1756 time=27 tick=500 with 1536 traces. frame tags:[ "orig3" ] 0 tagged trace sets:[ ] cmm:[ ]
[19:38:57.615] D [sigproc ] <OmnibusSigProc:anode3sigproc3> call=16 init nticks=6944 tbinmin=0 tbinmax=6944
[19:38:57.639] D [sigproc ] <OmnibusSigProc:anode3sigproc3> call=16 load plane index: 0, ntraces=1536, input bad regions: 0
[19:38:59.215] D [sigproc ] <OmnibusSigProc:anode3sigproc3> call=16 load plane index: 1, ntraces=1536, input bad regions: 0
[19:39:00.820] D [sigproc ] <OmnibusSigProc:anode3sigproc3> call=16 load plane index: 2, ntraces=1536, input bad regions: 0
[19:39:03.043] D [sigproc ] <OmnibusSigProc:anode3sigproc3> call=16 save plane index: 0, Qtot=27125214 Qloss=-3508891, 836 indices spanning [3171,4006] "wiener"
[19:39:03.187] D [sigproc ] <OmnibusSigProc:anode3sigproc3> call=16 save plane index: 0, Qtot=26527859 Qloss=-3071926, 687 indices spanning [4007,4693] "gauss"
[19:39:03.447] D [sigproc ] <OmnibusSigProc:anode3sigproc3> call=16 save plane index: 1, Qtot=31081120 Qloss=-4182263, 1546 indices spanning [4694,6239] "wiener"
[19:39:03.593] D [sigproc ] <OmnibusSigProc:anode3sigproc3> call=16 save plane index: 1, Qtot=29348911 Qloss=-2954966, 1278 indices spanning [6240,7517] "gauss"
[19:39:03.876] D [sigproc ] <OmnibusSigProc:anode3sigproc3> call=16 save plane index: 2, Qtot=26904707 Qloss=-1109790, 1810 indices spanning [7518,9327] "wiener"
[19:39:04.160] D [sigproc ] <OmnibusSigProc:anode3sigproc3> call=16 save plane index: 2, Qtot=26335845 Qloss=-541726, 1448 indices spanning [9328,10775] "gauss"
[19:39:04.161] D [sigproc ] <OmnibusSigProc:anode3sigproc3> call=16 produce 10776 traces: 4192 wiener3, 0 decon_charge3, 3413 gauss3, frame tag: sigproc
[19:39:04.161] D [sigproc ] <OmnibusSigProc:anode3sigproc3> call=16 output frame: frame: ident=1756 time=27 tick=500 with 10776 traces. frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss3":3413 [0] "mp2_roi3":2213 [0] "mp3_roi3":958 [0] "wiener3":4192 [4192] ] cmm:[ ]
[19:39:04.166] W [ glue ] <ChannelSelector:chsel2> Untagged summary not supported, summary will be dropped.
[19:39:04.167] D [ glue ] <ChannelSelector:chsel2> input frame: ident=1756 time=27 tick=512 with 6144 traces. frame tags:[ "orig2" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=1756 time=27 tick=512 with 1536 traces. frame tags:[ "orig2" ] 0 tagged trace sets:[ ] cmm:[ ]
[19:39:04.167] D [ aux ] <Resampler:resmp2> first ch=3406 Ts=512 Ns=6782 Ns_pad=6875 Nrat=125 Tr=500 Nr=7040 Nout=6944 padding:linear
[19:39:04.391] D [ aux ] <Resampler:resmp2> resample 1536 traces at call=16
[19:39:04.391] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=16 input frame: frame: ident=1756 time=27 tick=500 with 1536 traces. frame tags:[ "orig2" ] 0 tagged trace sets:[ ] cmm:[ ]
[19:39:04.391] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=16 init nticks=6944 tbinmin=0 tbinmax=6944
[19:39:04.415] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=16 load plane index: 0, ntraces=1536, input bad regions: 0
[19:39:06.022] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=16 load plane index: 1, ntraces=1536, input bad regions: 0
[19:39:07.660] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=16 load plane index: 2, ntraces=1536, input bad regions: 0
[19:39:10.010] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=16 save plane index: 0, Qtot=27466718 Qloss=-3726868, 819 indices spanning [3957,4775] "wiener"
[19:39:10.153] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=16 save plane index: 0, Qtot=26559391 Qloss=-3344782, 681 indices spanning [4776,5456] "gauss"
[19:39:10.419] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=16 save plane index: 1, Qtot=29493395 Qloss=-4803002, 1352 indices spanning [5457,6808] "wiener"
[19:39:10.565] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=16 save plane index: 1, Qtot=26855465 Qloss=-3775347, 1103 indices spanning [6809,7911] "gauss"
[19:39:10.848] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=16 save plane index: 2, Qtot=24761817 Qloss=-1737709, 2126 indices spanning [7912,10037] "wiener"
[19:39:11.131] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=16 save plane index: 2, Qtot=23675367 Qloss=-651792, 1673 indices spanning [10038,11710] "gauss"
[19:39:11.133] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=16 produce 11711 traces: 4297 wiener2, 0 decon_charge2, 3457 gauss2, frame tag: sigproc
[19:39:11.133] D [sigproc ] <OmnibusSigProc:anode2sigproc2> call=16 output frame: frame: ident=1756 time=27 tick=500 with 11711 traces. frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss2":3457 [0] "mp2_roi2":3110 [0] "mp3_roi2":847 [0] "wiener2":4297 [4297] ] cmm:[ ]
[19:39:11.140] W [ glue ] <ChannelSelector:chsel1> Untagged summary not supported, summary will be dropped.
[19:39:11.140] D [ glue ] <ChannelSelector:chsel1> input frame: ident=1756 time=27 tick=512 with 6144 traces. frame tags:[ "orig1" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=1756 time=27 tick=512 with 1536 traces. frame tags:[ "orig1" ] 0 tagged trace sets:[ ] cmm:[ ]
[19:39:11.141] D [ aux ] <Resampler:resmp1> first ch=617 Ts=512 Ns=6782 Ns_pad=6875 Nrat=125 Tr=500 Nr=7040 Nout=6944 padding:linear
[19:39:11.364] D [ aux ] <Resampler:resmp1> resample 1536 traces at call=16
[19:39:11.364] D [sigproc ] <OmnibusSigProc:anode1sigproc1> call=16 input frame: frame: ident=1756 time=27 tick=500 with 1536 traces. frame tags:[ "orig1" ] 0 tagged trace sets:[ ] cmm:[ ]
[19:39:11.364] D [sigproc ] <OmnibusSigProc:anode1sigproc1> call=16 init nticks=6944 tbinmin=0 tbinmax=6944
[19:39:11.389] D [sigproc ] <OmnibusSigProc:anode1sigproc1> call=16 load plane index: 0, ntraces=1536, input bad regions: 0
[19:39:12.902] D [sigproc ] <OmnibusSigProc:anode1sigproc1> call=16 load plane index: 1, ntraces=1536, input bad regions: 0
[19:39:14.432] D [sigproc ] <OmnibusSigProc:anode1sigproc1> call=16 load plane index: 2, ntraces=1536, input bad regions: 0
==================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
==================================================================================================================================
Full event 6.0675e-05 81.5925 129.844 82.1674 34.8144 9
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read) 5.2328e-05 6.8686e-05 0.000136878 5.8921e-05 2.52423e-05 9
produce:tpcrawdecoder:PDVDTPCReader 25.8878 36.0746 62.6412 33.4028 11.4042 9
produce:triggerrawdecoder:PDVDTriggerReader4 0.289511 0.313899 0.366583 0.307838 0.0244162 9
produce:pdvddaphne:DAPHNEReaderPDVD 4.2583 6.9213 9.50149 6.13946 1.93614 9
produce:ophit:OpHitFinder 0.0373688 0.0426613 0.0499496 0.042865 0.00365972 9
produce:opflash:OpFlashFinderVerticalDrift 0.00836968 0.0111359 0.0145308 0.0111995 0.00190323 9
produce:wclsdatavd:WireCellToolkit 21.4917 28.2917 42.8152 23.5366 8.01208 8
produce:gaushit:GausHitFinder 0.386099 0.535311 0.739879 0.482516 0.117658 8
produce:nhitsfilter:NumberOfHitsFilter 9.1712e-05 0.000165147 0.00031388 0.000128737 7.53735e-05 8
produce:reco3d:SpacePointSolver 2.95353 4.61792 8.42355 4.02421 1.8078 8
produce:hitpdune:DisambigFromSpacePoints 0.0325963 0.0527606 0.0839771 0.0521968 0.0175815 8
produce:pandora:StandardPandora 4.90107 11.0757 22.6695 8.65585 5.81477 8
produce:pandoraTrack:LArPandoraTrackCreation 0.161045 0.246576 0.460575 0.217326 0.0964666 8
produce:pandoraGnocalo:GnocchiCalorimetry 0.00976031 0.0138997 0.0207075 0.0124912 0.00369915 8
[art]:TriggerResults:TriggerResultInserter 1.549e-05 2.30899e-05 5.5946e-05 1.77785e-05 1.26928e-05 8
end_path:out1:RootOutput 6.032e-06 8.9905e-06 2.0338e-05 6.707e-06 4.52868e-06 8
end_path:out1:RootOutput(write) 2.06652 2.31665 2.95377 2.29267 0.26391 8
==================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 8589.88 MB
Peak resident set size usage (VmHWM): 6708.83 MB
Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException: PostEndJob 17-Sep-2025 19:40:33 CEST ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- ScheduleExecutionFailure BEGIN
Path: ProcessingStopped.
---- BadAlloc BEGIN
A bad_alloc exception was thrown while processing module WireCellToolkit/wclsdatavd run: 39433 subRun: 1 event: 1756
The job has probably exhausted the virtual memory available to the process.
---- BadAlloc END
Exception going through path produce
---- ScheduleExecutionFailure END
---- EventProcessorFailure END
%MSG
Art has completed and will exit with status 1.
Error in reco1 