Jobsub ID 241423.1@dunegpschedd01.fnal.gov
| Jobsub ID | 241423.1@dunegpschedd01.fnal.gov |
| Workflow ID | 9374 |
| Stage ID | 1 |
| User name | ykermaid@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 18000 (5 hours) |
| Submitted time | 2025-10-31 14:51:05 |
| Site | NL_NIKHEF |
| Entry | VIRGO_NL_NIKHEF_juk |
| Last heartbeat | 2025-10-31 15:15:28 |
| From worker node | Hostname | wn-sate-051.farm.nikhef.nl |
| cpuinfo | AMD EPYC 7551P 32-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 129600 (36 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | jobscript_error |
| Started | 2025-10-31 14:52:26 |
| Input files | vd-protodune:np02vd_raw_run040266_0650_df-s03-d1_dw_0_20251025T152515.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Max RSS bytes | 0 (0 MiB) |
| Outputting started | |
| Output files | |
| Finished | 2025-10-31 15:15:28 |
| Saved logs | justin-logs:241423.1-dunegpschedd01.fnal.gov.logs.tgz |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
S at call=13 with 8
[16:12:27.575] D [ glue ] frame sink sees EOS
[16:12:27.575] D [ pgraph ] <Pgrapher:> graph execution complete
[16:12:27.575] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 8.94 sec
[16:12:27.575] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 8.69 sec
[16:12:27.575] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 7.93 sec
[16:12:27.575] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 7.85 sec
[16:12:27.575] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 7.84 sec
[16:12:27.575] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 7.78 sec
[16:12:27.575] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 7.7 sec
[16:12:27.575] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 7.4 sec
[16:12:27.575] I [ timer ] Timer: WireCell::Aux::Resampler : 0.25 sec
[16:12:27.575] I [ timer ] Timer: WireCell::Aux::Resampler : 0.25 sec
[16:12:27.575] I [ timer ] Timer: WireCell::Aux::Resampler : 0.25 sec
[16:12:27.575] I [ timer ] Timer: WireCell::Aux::Resampler : 0.23 sec
[16:12:27.575] I [ timer ] Timer: WireCell::Gen::FrameFanin : 0.02 sec
[16:12:27.575] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[16:12:27.575] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[16:12:27.575] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[16:12:27.576] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[16:12:27.576] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[16:12:27.576] I [ timer ] Timer: WireCell::Gen::DumpFrames : 0 sec
[16:12:27.576] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[16:12:27.576] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[16:12:27.576] I [ timer ] Timer: WireCell::Gen::Retagger : 0 sec
[16:12:27.576] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[16:12:27.576] I [ timer ] Timer: WireCell::Gen::FrameFanout : 0 sec
[16:12:27.576] I [ timer ] Timer: wcls::RawFrameSource : 0 sec
[16:12:27.576] I [ timer ] Timer: wcls::FrameSaver : 0 sec
[16:12:27.576] I [ timer ] Timer: Total node execution : 65.13999916426837 sec
wclsFrameSaver saving cooked to 10000 ticks
wclsFrameSaver: saving 43515 traces tagged "gauss"
FrameSaver: q=9.90619e+06 n=891559 tag=gauss
wclsFrameSaver: saving 52916 traces tagged "wiener"
FrameSaver: q=1.05964e+07 n=853416 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 383 XUs and 535 XVs -> 5 XUVs
C:0 T:1 288 XUs and 267 XVs -> 4 XUVs
C:0 T:2 539 XUs and 1006 XVs -> 32 XUVs
C:0 T:3 573 XUs and 597 XVs -> 33 XUVs
C:0 T:4 304 XUs and 316 XVs -> 9 XUVs
C:0 T:5 459 XUs and 470 XVs -> 19 XUVs
C:0 T:6 9573 XUs and 10622 XVs -> 965 XUVs
C:0 T:7 2989 XUs and 1834 XVs -> 82 XUVs
C:0 T:8 1112 XUs and 1502 XVs -> 65 XUVs
C:0 T:9 620 XUs and 586 XVs -> 37 XUVs
C:0 T:10 547 XUs and 484 XVs -> 25 XUVs
C:0 T:11 537 XUs and 705 XVs -> 46 XUVs
C:0 T:12 828 XUs and 1125 XVs -> 47 XUVs
C:0 T:13 322 XUs and 472 XVs -> 44 XUVs
C:0 T:14 1955 XUs and 2113 XVs -> 96 XUVs
C:0 T:15 1882 XUs and 2805 XVs -> 94 XUVs
1603 XUVs total
1115 collection wire objects
1603 potential space points
Neighbour search...
41243 tests to find 10822 neighbours
Iterating with no regularization...
Begin: 4.87683e+09
0 4.76892e+09
1 4.76747e+09
Now with regularization...
Begin: 4.58783e+09
0 4.58729e+09
RawFrameSource: got 12288 raw::RawDigit objects
input nticks=9545 keeping as is
[16:13:43.245] D [ main ] executing 1 apps, thread limit 0:
[16:13:43.245] D [ main ] executing 1 apps, thread limit 0:
[16:13:43.245] D [ main ] executing app: "Pgrapher"
[16:13:43.245] D [ pgraph ] <Pgrapher:> executing graph
[16:13:43.245] D [ pgraph ] executing with 26 nodes
[16:13:43.246] D [ glue ] <FrameFanout:nfsp> call=14: input: frame: ident=119937 time=28 tick=512 with 12288 traces. frame tags:[ "orig" ] 0 tagged trace sets:[ ] cmm:[ ] output 0: frame: ident=119937 time=28 tick=512 with 12288 traces. frame tags:[ "orig0" ] 0 tagged trace sets:[ ] cmm:[ ] output 1: frame: ident=119937 time=28 tick=512 with 12288 traces. frame tags:[ "orig1" ] 0 tagged trace sets:[ ] cmm:[ ] output 2: frame: ident=119937 time=28 tick=512 with 12288 traces. frame tags:[ "orig2" ] 0 tagged trace sets:[ ] cmm:[ ] output 3: frame: ident=119937 time=28 tick=512 with 12288 traces. frame tags:[ "orig3" ] 0 tagged trace sets:[ ] cmm:[ ] output 4: frame: ident=119937 time=28 tick=512 with 12288 traces. frame tags:[ "orig4" ] 0 tagged trace sets:[ ] cmm:[ ] output 5: frame: ident=119937 time=28 tick=512 with 12288 traces. frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] output 6: frame: ident=119937 time=28 tick=512 with 12288 traces. frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] output 7: frame: ident=119937 time=28 tick=512 with 12288 traces. frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ]
[16:13:43.247] W [ glue ] <ChannelSelector:chsel7> Untagged summary not supported, summary will be dropped.
[16:13:43.247] D [ glue ] <ChannelSelector:chsel7> input frame: ident=119937 time=28 tick=512 with 12288 traces. frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=119937 time=28 tick=512 with 1536 traces. frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ]
[16:13:43.248] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=14 input frame: frame: ident=119937 time=28 tick=512 with 1536 traces. frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ]
[16:13:43.248] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=14 init nticks=9545 tbinmin=0 tbinmax=9545
[16:13:43.277] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=14 load plane index: 0, ntraces=1536, input bad regions: 0
[16:13:47.693] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=14 load plane index: 1, ntraces=1536, input bad regions: 0
[16:13:52.045] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=14 load plane index: 2, ntraces=1536, input bad regions: 0
==================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
==================================================================================================================================
Full event 7.6153e-05 150.193 228.807 158.859 62.2269 8
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read) 6.1585e-05 9.25561e-05 0.00018635 8.22795e-05 3.6327e-05 8
produce:tpcrawdecoder:PDVDTPCReader 25.7658 38.4513 66.1943 35.1812 11.9178 8
produce:triggerrawdecoder:PDVDTriggerReader4 0.523548 0.586884 0.856626 0.548027 0.103831 8
produce:pdvddaphne:DAPHNEReaderPDVD 0.000376256 0.000449372 0.00090625 0.000381505 0.000172873 8
produce:ophit:OpHitFinder 5.869e-05 0.000119443 0.000497734 6.4771e-05 0.000143107 8
produce:opflash:OpFlashFinderVerticalDrift 5.2158e-05 8.93675e-05 0.000333295 5.46975e-05 9.2223e-05 8
produce:wclsdatavd:WireCellToolkit 62.9482 76.196 96.4358 69.3345 11.8577 7
produce:gaushit:GausHitFinder 0.955273 1.32323 1.73573 1.27082 0.277736 7
produce:nhitsfilter:NumberOfHitsFilter 0.000251692 0.000304938 0.000414428 0.000289092 4.8885e-05 7
produce:reco3d:SpacePointSolver 7.53141 12.8764 18.0887 12.3701 3.48326 7
produce:hitpdune:DisambigFromSpacePoints 0.126654 0.204006 0.29197 0.16212 0.069736 7
produce:pandora:StandardPandora 14.8328 34.3451 72.4343 26.4259 17.6255 7
produce:pandoraTrack:LArPandoraTrackCreation 0.854371 1.37793 2.13832 1.11399 0.480363 7
produce:pandoraGnocalo:GnocchiCalorimetry 0.0309509 0.0379639 0.0515143 0.0360212 0.00636377 7
[art]:TriggerResults:TriggerResultInserter 2.2061e-05 3.92349e-05 0.000105208 2.7632e-05 2.7171e-05 7
end_path:out1:RootOutput 6.682e-06 1.24933e-05 4.4072e-05 7.113e-06 1.29042e-05 7
end_path:out1:RootOutput(write) 4.65586 5.00387 5.39278 5.05833 0.246391 7
==================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 8589.83 MB
Peak resident set size usage (VmHWM): 6701.12 MB
Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException: PostEndJob 31-Oct-2025 16:15:02 CET ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- ScheduleExecutionFailure BEGIN
Path: ProcessingStopped.
---- BadAlloc BEGIN
A bad_alloc exception was thrown while processing module WireCellToolkit/wclsdatavd run: 40266 subRun: 1 event: 119937
The job has probably exhausted the virtual memory available to the process.
---- BadAlloc END
Exception going through path produce
---- ScheduleExecutionFailure END
---- EventProcessorFailure END
%MSG
Art has completed and will exit with status 1.
Error in reco1 