Jobsub ID 233605.0@dunegpschedd02.fnal.gov
| Jobsub ID | 233605.0@dunegpschedd02.fnal.gov |
| Workflow ID | 9152 |
| Stage ID | 1 |
| User name | ykermaid@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 18000 (5 hours) |
| Submitted time | 2025-10-28 10:30:03 |
| Site | NL_NIKHEF |
| Entry | VIRGO_NL_NIKHEF_juk |
| Last heartbeat | 2025-10-28 12:06:12 |
| From worker node | Hostname | wn-lot-030.farm.nikhef.nl |
| cpuinfo | AMD EPYC 7702P 64-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 129600 (36 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | jobscript_error |
| Started | 2025-10-28 10:30:18 |
| Input files | vd-protodune:np02vd_raw_run040140_1655_df-s04-d1_dw_0_20251019T134635.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Max RSS bytes | 0 (0 MiB) |
| Outputting started | |
| Output files | |
| Finished | 2025-10-28 12:06:12 |
| Saved logs | justin-logs:233605.0-dunegpschedd02.fnal.gov.logs.tgz |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
OS
[12:54:28.940] D [ pgraph ] <Pgrapher:> graph execution complete
[12:54:28.940] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 42.51 sec
[12:54:28.940] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 18.09 sec
[12:54:28.940] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 12.98 sec
[12:54:28.940] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 12.8 sec
[12:54:28.940] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 12.55 sec
[12:54:28.940] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 12.52 sec
[12:54:28.940] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 11.91 sec
[12:54:28.940] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 11.75 sec
[12:54:28.940] I [ timer ] Timer: WireCell::Aux::Resampler : 0.69 sec
[12:54:28.940] I [ timer ] Timer: WireCell::Aux::Resampler : 0.68 sec
[12:54:28.940] I [ timer ] Timer: WireCell::Aux::Resampler : 0.68 sec
[12:54:28.940] I [ timer ] Timer: WireCell::Aux::Resampler : 0.67 sec
[12:54:28.940] I [ timer ] Timer: WireCell::Gen::FrameFanin : 0.04 sec
[12:54:28.940] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[12:54:28.940] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[12:54:28.940] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[12:54:28.941] I [ timer ] Timer: WireCell::Gen::Retagger : 0.01 sec
[12:54:28.941] I [ timer ] Timer: WireCell::Gen::FrameFanout : 0.01 sec
[12:54:28.941] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[12:54:28.941] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[12:54:28.941] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[12:54:28.941] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[12:54:28.941] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[12:54:28.941] I [ timer ] Timer: WireCell::Gen::DumpFrames : 0 sec
[12:54:28.941] I [ timer ] Timer: wcls::RawFrameSource : 0 sec
[12:54:28.941] I [ timer ] Timer: wcls::FrameSaver : 0 sec
[12:54:28.941] I [ timer ] Timer: Total node execution : 137.9199987296015 sec
wclsFrameSaver saving cooked to 10000 ticks
wclsFrameSaver: saving 106392 traces tagged "gauss"
FrameSaver: q=7.40912e+07 n=2883769 tag=gauss
wclsFrameSaver: saving 134922 traces tagged "wiener"
FrameSaver: q=7.69596e+07 n=2761739 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 2859 XUs and 2828 XVs -> 152 XUVs
C:0 T:1 2261 XUs and 3084 XVs -> 132 XUVs
C:0 T:2 1241 XUs and 1489 XVs -> 32 XUVs
C:0 T:3 1509 XUs and 1859 XVs -> 59 XUVs
C:0 T:4 52340 XUs and 64868 XVs -> 3940 XUVs
C:0 T:5 1685 XUs and 2724 XVs -> 75 XUVs
C:0 T:6 12532 XUs and 12458 XVs -> 614 XUVs
C:0 T:7 1496 XUs and 2063 XVs -> 57 XUVs
C:0 T:8 1771 XUs and 2088 XVs -> 91 XUVs
C:0 T:9 3566 XUs and 3755 XVs -> 240 XUVs
C:0 T:10 1276 XUs and 1316 XVs -> 95 XUVs
C:0 T:11 976 XUs and 1075 XVs -> 69 XUVs
C:0 T:12 76883 XUs and 135912 XVs -> 5637 XUVs
C:0 T:13 7253 XUs and 6881 XVs -> 297 XUVs
C:0 T:14 4576 XUs and 4651 XVs -> 319 XUVs
C:0 T:15 2064 XUs and 1565 XVs -> 113 XUVs
11922 XUVs total
6485 collection wire objects
11922 potential space points
Neighbour search...
252618 tests to find 65386 neighbours
Iterating with no regularization...
Begin: 4.43441e+12
0 4.36916e+12
1 4.36851e+12
Now with regularization...
Begin: 4.36205e+12
0 4.36204e+12
RawFrameSource: got 12288 raw::RawDigit objects
input nticks=8000 keeping as is
[13:05:03.133] D [ main ] executing 1 apps, thread limit 0:
[13:05:03.134] D [ main ] executing 1 apps, thread limit 0:
[13:05:03.134] D [ main ] executing app: "Pgrapher"
[13:05:03.134] D [ pgraph ] <Pgrapher:> executing graph
[13:05:03.134] D [ pgraph ] executing with 26 nodes
[13:05:03.135] D [ glue ] <FrameFanout:nfsp> call=38: input: frame: ident=290597 time=83 tick=512 with 12288 traces. frame tags:[ "orig" ] 0 tagged trace sets:[ ] cmm:[ ] output 0: frame: ident=290597 time=83 tick=512 with 12288 traces. frame tags:[ "orig0" ] 0 tagged trace sets:[ ] cmm:[ ] output 1: frame: ident=290597 time=83 tick=512 with 12288 traces. frame tags:[ "orig1" ] 0 tagged trace sets:[ ] cmm:[ ] output 2: frame: ident=290597 time=83 tick=512 with 12288 traces. frame tags:[ "orig2" ] 0 tagged trace sets:[ ] cmm:[ ] output 3: frame: ident=290597 time=83 tick=512 with 12288 traces. frame tags:[ "orig3" ] 0 tagged trace sets:[ ] cmm:[ ] output 4: frame: ident=290597 time=83 tick=512 with 12288 traces. frame tags:[ "orig4" ] 0 tagged trace sets:[ ] cmm:[ ] output 5: frame: ident=290597 time=83 tick=512 with 12288 traces. frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] output 6: frame: ident=290597 time=83 tick=512 with 12288 traces. frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] output 7: frame: ident=290597 time=83 tick=512 with 12288 traces. frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ]
[13:05:03.135] W [ glue ] <ChannelSelector:chsel7> Untagged summary not supported, summary will be dropped.
[13:05:03.136] D [ glue ] <ChannelSelector:chsel7> input frame: ident=290597 time=83 tick=512 with 12288 traces. frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=290597 time=83 tick=512 with 1536 traces. frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ]
[13:05:03.136] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=38 input frame: frame: ident=290597 time=83 tick=512 with 1536 traces. frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ]
[13:05:03.136] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=38 init nticks=8000 tbinmin=0 tbinmax=8000
[13:05:03.165] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=38 load plane index: 0, ntraces=1536, input bad regions: 0
[13:05:04.826] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=38 load plane index: 1, ntraces=1536, input bad regions: 0
[13:05:06.457] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=38 load plane index: 2, ntraces=1536, input bad regions: 0
==================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
==================================================================================================================================
Full event 8.8636e-05 280.261 1825.47 134.188 391.82 20
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read) 6.3088e-05 8.76087e-05 0.000217626 8.11315e-05 3.1108e-05 20
produce:tpcrawdecoder:PDVDTPCReader 17.0307 30.7437 76.6901 23.9898 15.5303 20
produce:triggerrawdecoder:PDVDTriggerReader4 0.292811 0.314179 0.354022 0.311808 0.0133153 20
produce:pdvddaphne:DAPHNEReaderPDVD 3.31182 3.97872 5.7556 3.90702 0.590291 20
produce:ophit:OpHitFinder 0.0406384 0.0494577 0.0603477 0.049576 0.00511892 20
produce:opflash:OpFlashFinderVerticalDrift 0.000572539 0.00669357 0.0158551 0.00651825 0.00348707 20
produce:wclsdatavd:WireCellToolkit 49.5747 77.1254 139.77 59.5274 27.4881 19
produce:gaushit:GausHitFinder 0.873017 1.29565 2.76965 1.13715 0.448088 19
produce:nhitsfilter:NumberOfHitsFilter 0.000242603 0.000438334 0.0017972 0.000336779 0.000332606 19
produce:reco3d:SpacePointSolver 8.63056 15.7091 52.0633 11.8709 10.5467 19
produce:hitpdune:DisambigFromSpacePoints 0.0979448 0.270297 1.1616 0.173858 0.251395 19
produce:pandora:StandardPandora 14.8757 159.49 1623.72 25.2587 368.816 19
produce:pandoraTrack:LArPandoraTrackCreation 0.370433 1.73263 12.45 0.795228 2.64409 19
produce:pandoraGnocalo:GnocchiCalorimetry 0.0209817 0.0338613 0.0728864 0.0327556 0.0109207 19
[art]:TriggerResults:TriggerResultInserter 1.5088e-05 2.05958e-05 5.1897e-05 1.8094e-05 7.7909e-06 19
end_path:out1:RootOutput 4.028e-06 5.22989e-06 2.1069e-05 4.358e-06 3.73871e-06 19
end_path:out1:RootOutput(write) 4.00497 4.45079 5.29986 4.29872 0.423099 19
==================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 8589.88 MB
Peak resident set size usage (VmHWM): 6702.58 MB
Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException: PostEndJob 28-Oct-2025 13:05:48 CET ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- ScheduleExecutionFailure BEGIN
Path: ProcessingStopped.
---- BadAlloc BEGIN
A bad_alloc exception was thrown while processing module WireCellToolkit/wclsdatavd run: 40140 subRun: 1 event: 290597
The job has probably exhausted the virtual memory available to the process.
---- BadAlloc END
Exception going through path produce
---- ScheduleExecutionFailure END
---- EventProcessorFailure END
%MSG
Art has completed and will exit with status 1.
Error in reco1 