Jobsub ID 47052.3@dunegpschedd01.fnal.gov
| Jobsub ID | 47052.3@dunegpschedd01.fnal.gov |
| Workflow ID | 2332 |
| Stage ID | 1 |
| User name | ykermaid@fnal.gov |
| HTCondor Group | group_dune.prod.mcsim |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 18000 (5 hours) |
| Submitted time | 2025-09-16 16:42:07 |
| Site | NL_NIKHEF |
| Entry | VIRGO_NL_NIKHEF_klomp |
| Last heartbeat | 2025-09-16 17:08:30 |
| From worker node | Hostname | wn-snel-030.farm.nikhef.nl |
| cpuinfo | AMD EPYC 7H12 64-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 129600 (36 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | jobscript_error |
| Started | 2025-09-16 16:42:25 |
| Input files | vd-protodune:np02vd_raw_run039343_0008_df-s03-d0_dw_0_20250908T122407.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Max RSS bytes | 0 (0 MiB) |
| Outputting started | |
| Output files | |
| Finished | 2025-09-16 17:08:30 |
| Saved logs | justin-logs:47052.3-dunegpschedd01.fnal.gov.logs.tgz |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
SG-e OpHitFinder: OpHitFinder:ophit@BeginModule 16-Sep-2025 19:07:19 CEST run: 39343 subRun: 1 event: 4366
Error! unrecognized channel number -1. Ignoring pulse
%MSG
%MSG-e OpHitFinder: OpHitFinder:ophit@BeginModule 16-Sep-2025 19:07:19 CEST run: 39343 subRun: 1 event: 4366
Error! unrecognized channel number -1. Ignoring pulse
%MSG
%MSG-e OpHitFinder: OpHitFinder:ophit@BeginModule 16-Sep-2025 19:07:19 CEST run: 39343 subRun: 1 event: 4366
Error! unrecognized channel number -1. Ignoring pulse
%MSG
%MSG-e OpHitFinder: OpHitFinder:ophit@BeginModule 16-Sep-2025 19:07:19 CEST run: 39343 subRun: 1 event: 4366
Error! unrecognized channel number -1. Ignoring pulse
%MSG
%MSG-e OpHitFinder: OpHitFinder:ophit@BeginModule 16-Sep-2025 19:07:19 CEST run: 39343 subRun: 1 event: 4366
Error! unrecognized channel number -1. Ignoring pulse
%MSG
%MSG-e OpHitFinder: OpHitFinder:ophit@BeginModule 16-Sep-2025 19:07:19 CEST run: 39343 subRun: 1 event: 4366
Error! unrecognized channel number -1. Ignoring pulse
%MSG
%MSG-e OpHitFinder: OpHitFinder:ophit@BeginModule 16-Sep-2025 19:07:19 CEST run: 39343 subRun: 1 event: 4366
Error! unrecognized channel number -1. Ignoring pulse
%MSG
%MSG-e OpHitFinder: OpHitFinder:ophit@BeginModule 16-Sep-2025 19:07:19 CEST run: 39343 subRun: 1 event: 4366
Error! unrecognized channel number -1. Ignoring pulse
%MSG
%MSG-e OpHitFinder: OpHitFinder:ophit@BeginModule 16-Sep-2025 19:07:19 CEST run: 39343 subRun: 1 event: 4366
Error! unrecognized channel number -1. Ignoring pulse
%MSG
%MSG-e OpHitFinder: OpHitFinder:ophit@BeginModule 16-Sep-2025 19:07:19 CEST run: 39343 subRun: 1 event: 4366
Error! unrecognized channel number -1. Ignoring pulse
%MSG
%MSG-e OpHitFinder: OpHitFinder:ophit@BeginModule 16-Sep-2025 19:07:19 CEST run: 39343 subRun: 1 event: 4366
Error! unrecognized channel number -1. Ignoring pulse
%MSG
%MSG-e OpHitFinder: OpHitFinder:ophit@BeginModule 16-Sep-2025 19:07:19 CEST run: 39343 subRun: 1 event: 4366
Error! unrecognized channel number -1. Ignoring pulse
%MSG
%MSG-e OpHitFinder: OpHitFinder:ophit@BeginModule 16-Sep-2025 19:07:19 CEST run: 39343 subRun: 1 event: 4366
Error! unrecognized channel number -1. Ignoring pulse
%MSG
%MSG-e OpHitFinder: OpHitFinder:ophit@BeginModule 16-Sep-2025 19:07:19 CEST run: 39343 subRun: 1 event: 4366
Error! unrecognized channel number -1. Ignoring pulse
%MSG
%MSG-e OpHitFinder: OpHitFinder:ophit@BeginModule 16-Sep-2025 19:07:19 CEST run: 39343 subRun: 1 event: 4366
Error! unrecognized channel number -1. Ignoring pulse
%MSG
%MSG-e OpHitFinder: OpHitFinder:ophit@BeginModule 16-Sep-2025 19:07:19 CEST run: 39343 subRun: 1 event: 4366
Error! unrecognized channel number -1. Ignoring pulse
%MSG
%MSG-e OpHitFinder: OpHitFinder:ophit@BeginModule 16-Sep-2025 19:07:19 CEST run: 39343 subRun: 1 event: 4366
Error! unrecognized channel number -1. Ignoring pulse
%MSG
%MSG-e OpHitFinder: OpHitFinder:ophit@BeginModule 16-Sep-2025 19:07:19 CEST run: 39343 subRun: 1 event: 4366
Error! unrecognized channel number -1. Ignoring pulse
%MSG
%MSG-e OpHitFinder: OpHitFinder:ophit@BeginModule 16-Sep-2025 19:07:19 CEST run: 39343 subRun: 1 event: 4366
Error! unrecognized channel number -1. Ignoring pulse
%MSG
%MSG-e OpHitFinder: OpHitFinder:ophit@BeginModule 16-Sep-2025 19:07:19 CEST run: 39343 subRun: 1 event: 4366
Error! unrecognized channel number -1. Ignoring pulse
%MSG
RawFrameSource: got 12288 raw::RawDigit objects
input nticks=6400 keeping as is
[19:07:19.802] D [ main ] executing 1 apps, thread limit 0:
[19:07:19.802] D [ main ] executing 1 apps, thread limit 0:
[19:07:19.802] D [ main ] executing app: "Pgrapher"
[19:07:19.802] D [ pgraph ] <Pgrapher:> executing graph
[19:07:19.802] D [ pgraph ] executing with 26 nodes
[19:07:19.803] D [ glue ] <FrameFanout:nfsp> call=10: input: frame: ident=4366 time=23 tick=512 with 12288 traces. frame tags:[ "orig" ] 0 tagged trace sets:[ ] cmm:[ ] output 0: frame: ident=4366 time=23 tick=512 with 12288 traces. frame tags:[ "orig0" ] 0 tagged trace sets:[ ] cmm:[ ] output 1: frame: ident=4366 time=23 tick=512 with 12288 traces. frame tags:[ "orig1" ] 0 tagged trace sets:[ ] cmm:[ ] output 2: frame: ident=4366 time=23 tick=512 with 12288 traces. frame tags:[ "orig2" ] 0 tagged trace sets:[ ] cmm:[ ] output 3: frame: ident=4366 time=23 tick=512 with 12288 traces. frame tags:[ "orig3" ] 0 tagged trace sets:[ ] cmm:[ ] output 4: frame: ident=4366 time=23 tick=512 with 12288 traces. frame tags:[ "orig4" ] 0 tagged trace sets:[ ] cmm:[ ] output 5: frame: ident=4366 time=23 tick=512 with 12288 traces. frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] output 6: frame: ident=4366 time=23 tick=512 with 12288 traces. frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] output 7: frame: ident=4366 time=23 tick=512 with 12288 traces. frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ]
[19:07:19.803] W [ glue ] <ChannelSelector:chsel7> Untagged summary not supported, summary will be dropped.
[19:07:19.803] D [ glue ] <ChannelSelector:chsel7> input frame: ident=4366 time=23 tick=512 with 12288 traces. frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=4366 time=23 tick=512 with 1536 traces. frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ]
[19:07:19.804] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=10 input frame: frame: ident=4366 time=23 tick=512 with 1536 traces. frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ]
[19:07:19.804] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=10 init nticks=6400 tbinmin=0 tbinmax=6400
[19:07:19.824] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=10 load plane index: 0, ntraces=1536, input bad regions: 0
[19:07:20.966] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=10 load plane index: 1, ntraces=1536, input bad regions: 0
[19:07:22.235] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=10 load plane index: 2, ntraces=1536, input bad regions: 0
==================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
==================================================================================================================================
Full event 7.0582e-05 240.429 878.531 129.908 290.674 6
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read) 6.6264e-05 9.5798e-05 0.000209473 7.4495e-05 5.1032e-05 6
produce:tpcrawdecoder:PDVDTPCReader 13.072 15.1089 20.091 13.5696 2.65984 6
produce:triggerrawdecoder:PDVDTriggerReader4 0.514054 0.530372 0.548819 0.530828 0.0129484 6
produce:pdvddaphne:DAPHNEReaderPDVD 3.11351 3.29294 3.46813 3.28122 0.104599 6
produce:ophit:OpHitFinder 0.0300124 0.0352321 0.0427342 0.0349109 0.00409276 6
produce:opflash:OpFlashFinderVerticalDrift 0.00764255 0.0102453 0.0158955 0.00898618 0.00284503 6
produce:wclsdatavd:WireCellToolkit 36.6789 49.3724 71.0802 46.4866 11.5172 5
produce:gaushit:GausHitFinder 0.74639 1.38074 2.61541 1.16625 0.640591 5
produce:nhitsfilter:NumberOfHitsFilter 0.000302788 0.000399527 0.000488666 0.000400671 7.98136e-05 5
produce:reco3d:SpacePointSolver 6.89707 14.2832 24.225 13.7221 5.62418 5
produce:hitpdune:DisambigFromSpacePoints 0.0894309 0.271145 0.64815 0.20278 0.194668 5
produce:pandora:StandardPandora 16.1581 199.526 781.729 49.9788 292.422 5
produce:pandoraTrack:LArPandoraTrackCreation 0.506215 0.659149 0.961772 0.620429 0.156993 5
produce:pandoraGnocalo:GnocchiCalorimetry 0.0191639 0.0268061 0.0325044 0.027561 0.00440914 5
[art]:TriggerResults:TriggerResultInserter 1.4247e-05 2.16208e-05 4.6988e-05 1.5569e-05 1.26951e-05 5
end_path:out1:RootOutput 3.597e-06 6.875e-06 1.9316e-05 3.787e-06 6.22168e-06 5
end_path:out1:RootOutput(write) 3.29874 3.58868 4.35479 3.39933 0.395466 5
==================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 8589.85 MB
Peak resident set size usage (VmHWM): 6712.34 MB
Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException: PostEndJob 16-Sep-2025 19:08:10 CEST ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- ScheduleExecutionFailure BEGIN
Path: ProcessingStopped.
---- BadAlloc BEGIN
A bad_alloc exception was thrown while processing module WireCellToolkit/wclsdatavd run: 39343 subRun: 1 event: 4366
The job has probably exhausted the virtual memory available to the process.
---- BadAlloc END
Exception going through path produce
---- ScheduleExecutionFailure END
---- EventProcessorFailure END
%MSG
Art has completed and will exit with status 1.
Error in reco1 