Jobsub ID 241139.2@dunegpschedd01.fnal.gov
| Jobsub ID | 241139.2@dunegpschedd01.fnal.gov |
| Workflow ID | 9375 |
| Stage ID | 1 |
| User name | ykermaid@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 18000 (5 hours) |
| Submitted time | 2025-10-30 19:22:00 |
| Site | NL_NIKHEF |
| Entry | VIRGO_NL_NIKHEF_juk |
| Last heartbeat | 2025-10-30 20:09:26 |
| From worker node | Hostname | wn-pep-014.farm.nikhef.nl |
| cpuinfo | Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 129600 (36 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | jobscript_error |
| Started | 2025-10-30 19:22:40 |
| Input files | vd-protodune:np02vd_raw_run040267_0002_df-s03-d0_dw_0_20251025T183635.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Max RSS bytes | 0 (0 MiB) |
| Outputting started | |
| Output files | |
| Finished | 2025-10-30 20:09:26 |
| Saved logs | justin-logs:241139.2-dunegpschedd01.fnal.gov.logs.tgz |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
at call=15 with 8
[20:48:06.356] D [ glue ] frame sink sees EOS
[20:48:06.356] D [ pgraph ] <Pgrapher:> graph execution complete
[20:48:06.356] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 9.17 sec
[20:48:06.356] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 9.08 sec
[20:48:06.356] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 7.85 sec
[20:48:06.356] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 7.82 sec
[20:48:06.356] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 7.74 sec
[20:48:06.356] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 7.72 sec
[20:48:06.356] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 7.54 sec
[20:48:06.356] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 7.31 sec
[20:48:06.356] I [ timer ] Timer: WireCell::Aux::Resampler : 0.27 sec
[20:48:06.356] I [ timer ] Timer: WireCell::Aux::Resampler : 0.26 sec
[20:48:06.356] I [ timer ] Timer: WireCell::Aux::Resampler : 0.26 sec
[20:48:06.356] I [ timer ] Timer: WireCell::Aux::Resampler : 0.25 sec
[20:48:06.356] I [ timer ] Timer: WireCell::Gen::FrameFanin : 0.05 sec
[20:48:06.356] I [ timer ] Timer: WireCell::Gen::Retagger : 0.02 sec
[20:48:06.356] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[20:48:06.356] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[20:48:06.356] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[20:48:06.356] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[20:48:06.356] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[20:48:06.356] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[20:48:06.356] I [ timer ] Timer: WireCell::Gen::DumpFrames : 0 sec
[20:48:06.356] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[20:48:06.356] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[20:48:06.356] I [ timer ] Timer: WireCell::Gen::FrameFanout : 0 sec
[20:48:06.356] I [ timer ] Timer: wcls::RawFrameSource : 0 sec
[20:48:06.356] I [ timer ] Timer: wcls::FrameSaver : 0 sec
[20:48:06.356] I [ timer ] Timer: Total node execution : 65.35999953374267 sec
wclsFrameSaver saving cooked to 10000 ticks
wclsFrameSaver: saving 62904 traces tagged "gauss"
FrameSaver: q=6.54391e+07 n=1332601 tag=gauss
wclsFrameSaver: saving 77517 traces tagged "wiener"
FrameSaver: q=6.64407e+07 n=1274302 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 1262 XUs and 1307 XVs -> 66 XUVs
C:0 T:1 1397 XUs and 1568 XVs -> 40 XUVs
C:0 T:2 12849 XUs and 15823 XVs -> 1119 XUVs
C:0 T:3 823 XUs and 1375 XVs -> 28 XUVs
C:0 T:4 586 XUs and 737 XVs -> 17 XUVs
C:0 T:5 962 XUs and 1554 XVs -> 46 XUVs
C:0 T:6 972 XUs and 1044 XVs -> 29 XUVs
C:0 T:7 375 XUs and 494 XVs -> 7 XUVs
C:0 T:8 1206 XUs and 1477 XVs -> 92 XUVs
C:0 T:9 1509 XUs and 1736 XVs -> 79 XUVs
C:0 T:10 1077 XUs and 1433 XVs -> 82 XUVs
C:0 T:11 2031 XUs and 2365 XVs -> 82 XUVs
C:0 T:12 1625 XUs and 2148 XVs -> 147 XUVs
C:0 T:13 2103 XUs and 2336 XVs -> 87 XUVs
C:0 T:14 6029 XUs and 6919 XVs -> 384 XUVs
C:0 T:15 1086 XUs and 926 XVs -> 47 XUVs
2352 XUVs total
1653 collection wire objects
2352 potential space points
Neighbour search...
62676 tests to find 16652 neighbours
Iterating with no regularization...
Begin: 3.67742e+09
0 3.5418e+09
1 3.53346e+09
2 3.53336e+09
Now with regularization...
Begin: 3.43818e+09
0 3.43786e+09
RawFrameSource: got 12288 raw::RawDigit objects
input nticks=8000 keeping as is
[20:50:15.083] D [ main ] executing 1 apps, thread limit 0:
[20:50:15.083] D [ main ] executing 1 apps, thread limit 0:
[20:50:15.083] D [ main ] executing app: "Pgrapher"
[20:50:15.083] D [ pgraph ] <Pgrapher:> executing graph
[20:50:15.083] D [ pgraph ] executing with 26 nodes
[20:50:15.085] D [ glue ] <FrameFanout:nfsp> call=16: input: frame: ident=448 time=37 tick=512 with 12288 traces. frame tags:[ "orig" ] 0 tagged trace sets:[ ] cmm:[ ] output 0: frame: ident=448 time=37 tick=512 with 12288 traces. frame tags:[ "orig0" ] 0 tagged trace sets:[ ] cmm:[ ] output 1: frame: ident=448 time=37 tick=512 with 12288 traces. frame tags:[ "orig1" ] 0 tagged trace sets:[ ] cmm:[ ] output 2: frame: ident=448 time=37 tick=512 with 12288 traces. frame tags:[ "orig2" ] 0 tagged trace sets:[ ] cmm:[ ] output 3: frame: ident=448 time=37 tick=512 with 12288 traces. frame tags:[ "orig3" ] 0 tagged trace sets:[ ] cmm:[ ] output 4: frame: ident=448 time=37 tick=512 with 12288 traces. frame tags:[ "orig4" ] 0 tagged trace sets:[ ] cmm:[ ] output 5: frame: ident=448 time=37 tick=512 with 12288 traces. frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] output 6: frame: ident=448 time=37 tick=512 with 12288 traces. frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] output 7: frame: ident=448 time=37 tick=512 with 12288 traces. frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ]
[20:50:15.085] W [ glue ] <ChannelSelector:chsel7> Untagged summary not supported, summary will be dropped.
[20:50:15.086] D [ glue ] <ChannelSelector:chsel7> input frame: ident=448 time=37 tick=512 with 12288 traces. frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=448 time=37 tick=512 with 1536 traces. frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ]
[20:50:15.086] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=16 input frame: frame: ident=448 time=37 tick=512 with 1536 traces. frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ]
[20:50:15.086] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=16 init nticks=8000 tbinmin=0 tbinmax=8000
[20:50:15.121] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=16 load plane index: 0, ntraces=1536, input bad regions: 0
[20:50:16.801] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=16 load plane index: 1, ntraces=1536, input bad regions: 0
[20:50:18.512] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=16 load plane index: 2, ntraces=1536, input bad regions: 0
==================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
==================================================================================================================================
Full event 0.000114356 173.495 229.843 191.131 63.9308 9
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read) 5.03e-05 8.26158e-05 0.000186206 6.5146e-05 4.08365e-05 9
produce:tpcrawdecoder:PDVDTPCReader 38.6365 63.9708 98.5723 63.3416 16.3123 9
produce:triggerrawdecoder:PDVDTriggerReader4 0.291184 0.331525 0.392524 0.327625 0.0355998 9
produce:pdvddaphne:DAPHNEReaderPDVD 0.000384228 0.000435421 0.000772199 0.000393018 0.000119164 9
produce:ophit:OpHitFinder 5.7716e-05 0.000116762 0.000549407 6.2655e-05 0.000153021 9
produce:opflash:OpFlashFinderVerticalDrift 4.7838e-05 8.14249e-05 0.000322951 5.0851e-05 8.54347e-05 9
produce:wclsdatavd:WireCellToolkit 62.3664 78.2605 98.9089 77.966 11.8737 8
produce:gaushit:GausHitFinder 0.997863 1.43154 1.85887 1.42424 0.28685 8
produce:nhitsfilter:NumberOfHitsFilter 0.000366677 0.000506443 0.000749347 0.00045151 0.000125249 8
produce:reco3d:SpacePointSolver 8.37781 11.7398 16.9286 10.5834 3.06809 8
produce:hitpdune:DisambigFromSpacePoints 0.149854 0.217491 0.330643 0.199267 0.0623397 8
produce:pandora:StandardPandora 18.0774 31.7531 45.1725 30.0866 8.65233 8
produce:pandoraTrack:LArPandoraTrackCreation 0.936808 2.03641 5.17758 1.34111 1.37451 8
produce:pandoraGnocalo:GnocchiCalorimetry 0.0244622 0.0343189 0.0463549 0.0323198 0.00754611 8
[art]:TriggerResults:TriggerResultInserter 3.0097e-05 4.31338e-05 9.4422e-05 3.757e-05 1.97041e-05 8
end_path:out1:RootOutput 3.971e-06 9.3355e-06 2.4879e-05 8.1375e-06 6.12732e-06 8
end_path:out1:RootOutput(write) 3.81792 4.34065 5.02081 4.30065 0.315 8
==================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 8589.85 MB
Peak resident set size usage (VmHWM): 6704.14 MB
Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException: PostEndJob 30-Oct-2025 20:51:25 CET ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- ScheduleExecutionFailure BEGIN
Path: ProcessingStopped.
---- BadAlloc BEGIN
A bad_alloc exception was thrown while processing module WireCellToolkit/wclsdatavd run: 40267 subRun: 1 event: 448
The job has probably exhausted the virtual memory available to the process.
---- BadAlloc END
Exception going through path produce
---- ScheduleExecutionFailure END
---- EventProcessorFailure END
%MSG
Art has completed and will exit with status 1.
Error in reco1 