Jobsub ID 231192.14@dunegpschedd02.fnal.gov
| Jobsub ID | 231192.14@dunegpschedd02.fnal.gov |
| Workflow ID | 9173 |
| Stage ID | 1 |
| User name | ykermaid@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 18000 (5 hours) |
| Submitted time | 2025-10-20 08:38:40 |
| Site | FR_CCIN2P3 |
| Entry | DUNE_FR_CCIN2P3_cccondorce02 |
| Last heartbeat | 2025-10-20 08:51:21 |
| From worker node | Hostname | ccwcondor0012 |
| cpuinfo | AMD EPYC 9334 32-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 106200 (29 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | jobscript_error |
| Started | 2025-10-20 08:39:50 |
| Input files | vd-protodune:np02vd_raw_run040140_2307_df-s04-d0_dw_0_20251020T071245.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Max RSS bytes | 0 (0 MiB) |
| Outputting started | |
| Output files | |
| Finished | 2025-10-20 08:51:21 |
| Saved logs | justin-logs:231192.14-dunegpschedd02.fnal.gov.logs.tgz |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
window for channel 12280
wclsFrameSaver: no samples within desired window for channel 12280
wclsFrameSaver: no samples within desired window for channel 12280
wclsFrameSaver: no samples within desired window for channel 12280
wclsFrameSaver: no samples within desired window for channel 12281
wclsFrameSaver: no samples within desired window for channel 12281
wclsFrameSaver: no samples within desired window for channel 12281
wclsFrameSaver: no samples within desired window for channel 12281
wclsFrameSaver: no samples within desired window for channel 12282
wclsFrameSaver: no samples within desired window for channel 12282
wclsFrameSaver: no samples within desired window for channel 12282
wclsFrameSaver: no samples within desired window for channel 12282
wclsFrameSaver: no samples within desired window for channel 12282
wclsFrameSaver: no samples within desired window for channel 12283
wclsFrameSaver: no samples within desired window for channel 12283
wclsFrameSaver: no samples within desired window for channel 12283
wclsFrameSaver: no samples within desired window for channel 12283
wclsFrameSaver: no samples within desired window for channel 12283
wclsFrameSaver: no samples within desired window for channel 12284
wclsFrameSaver: no samples within desired window for channel 12284
wclsFrameSaver: no samples within desired window for channel 12284
wclsFrameSaver: no samples within desired window for channel 12285
wclsFrameSaver: no samples within desired window for channel 12285
wclsFrameSaver: no samples within desired window for channel 12285
wclsFrameSaver: no samples within desired window for channel 12285
wclsFrameSaver: no samples within desired window for channel 12285
wclsFrameSaver: no samples within desired window for channel 12286
wclsFrameSaver: no samples within desired window for channel 12286
wclsFrameSaver: no samples within desired window for channel 12286
wclsFrameSaver: no samples within desired window for channel 12286
wclsFrameSaver: no samples within desired window for channel 12286
wclsFrameSaver: no samples within desired window for channel 12286
wclsFrameSaver: no samples within desired window for channel 12286
wclsFrameSaver: no samples within desired window for channel 12286
wclsFrameSaver: no samples within desired window for channel 12287
wclsFrameSaver: no samples within desired window for channel 12287
wclsFrameSaver: no samples within desired window for channel 12287
wclsFrameSaver: no samples within desired window for channel 12287
wclsFrameSaver: no samples within desired window for channel 12287
FrameSaver: q=1.8718e+07 n=1333245 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 1451 XUs and 1623 XVs -> 98 XUVs
C:0 T:1 11331 XUs and 13267 XVs -> 869 XUVs
C:0 T:2 343 XUs and 291 XVs -> 9 XUVs
C:0 T:3 964 XUs and 846 XVs -> 39 XUVs
C:0 T:4 3233 XUs and 4154 XVs -> 157 XUVs
C:0 T:5 862 XUs and 1290 XVs -> 33 XUVs
C:0 T:6 1351 XUs and 1529 XVs -> 87 XUVs
C:0 T:7 11819 XUs and 15691 XVs -> 1181 XUVs
C:0 T:8 1406 XUs and 1421 XVs -> 76 XUVs
C:0 T:9 2305 XUs and 2352 XVs -> 109 XUVs
C:0 T:10 1357 XUs and 1028 XVs -> 50 XUVs
C:0 T:11 1132 XUs and 1544 XVs -> 41 XUVs
C:0 T:12 2006 XUs and 2116 XVs -> 124 XUVs
C:0 T:13 522 XUs and 413 XVs -> 23 XUVs
C:0 T:14 1190 XUs and 1315 XVs -> 100 XUVs
C:0 T:15 2592 XUs and 2015 XVs -> 124 XUVs
3120 XUVs total
1765 collection wire objects
3120 potential space points
Neighbour search...
88696 tests to find 22310 neighbours
Iterating with no regularization...
Begin: 7.47639e+09
0 7.06857e+09
1 7.04355e+09
2 7.04248e+09
Now with regularization...
Begin: 6.91489e+09
0 6.91451e+09
RawFrameSource: got 12288 raw::RawDigit objects
input nticks=128 keeping as is
[10:50:55.111] D [ main ] executing 1 apps, thread limit 0:
[10:50:55.111] D [ main ] executing 1 apps, thread limit 0:
[10:50:55.111] D [ main ] executing app: "Pgrapher"
[10:50:55.111] D [ pgraph ] <Pgrapher:> executing graph
[10:50:55.111] D [ pgraph ] executing with 26 nodes
[10:50:55.112] D [ glue ] <FrameFanout:nfsp> call=14: input: frame: ident=404500 time=22 tick=512 with 12288 traces. frame tags:[ "orig" ] 0 tagged trace sets:[ ] cmm:[ ] output 0: frame: ident=404500 time=22 tick=512 with 12288 traces. frame tags:[ "orig0" ] 0 tagged trace sets:[ ] cmm:[ ] output 1: frame: ident=404500 time=22 tick=512 with 12288 traces. frame tags:[ "orig1" ] 0 tagged trace sets:[ ] cmm:[ ] output 2: frame: ident=404500 time=22 tick=512 with 12288 traces. frame tags:[ "orig2" ] 0 tagged trace sets:[ ] cmm:[ ] output 3: frame: ident=404500 time=22 tick=512 with 12288 traces. frame tags:[ "orig3" ] 0 tagged trace sets:[ ] cmm:[ ] output 4: frame: ident=404500 time=22 tick=512 with 12288 traces. frame tags:[ "orig4" ] 0 tagged trace sets:[ ] cmm:[ ] output 5: frame: ident=404500 time=22 tick=512 with 12288 traces. frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] output 6: frame: ident=404500 time=22 tick=512 with 12288 traces. frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] output 7: frame: ident=404500 time=22 tick=512 with 12288 traces. frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ]
[10:50:55.112] W [ glue ] <ChannelSelector:chsel7> Untagged summary not supported, summary will be dropped.
[10:50:55.113] D [ glue ] <ChannelSelector:chsel7> input frame: ident=404500 time=22 tick=512 with 12288 traces. frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=404500 time=22 tick=512 with 1536 traces. frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ]
[10:50:55.113] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=14 input frame: frame: ident=404500 time=22 tick=512 with 1536 traces. frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ]
[10:50:55.113] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=14 init nticks=128 tbinmin=0 tbinmax=128
[10:50:55.126] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=14 load plane index: 0, ntraces=1536, input bad regions: 0
==================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
==================================================================================================================================
Full event 61.5011 90.8571 174.232 75.9594 36.6207 7
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read) 6.4488e-05 9.10433e-05 0.000211431 7.46585e-05 4.62568e-05 8
produce:tpcrawdecoder:PDVDTPCReader 1.24938 5.67847 10.8115 5.30539 2.47668 8
produce:triggerrawdecoder:PDVDTriggerReader4 0.046196 0.0495098 0.0565602 0.0475221 0.0038262 8
produce:pdvddaphne:DAPHNEReaderPDVD 0.00026386 1.53778 1.99327 1.79833 0.610201 8
produce:ophit:OpHitFinder 6.2074e-05 0.0254714 0.0361656 0.0291911 0.0103095 8
produce:opflash:OpFlashFinderVerticalDrift 3.963e-05 0.00234883 0.00398556 0.00265118 0.00138162 8
produce:wclsdatavd:WireCellToolkit 34.2118 47.3757 95.975 36.5719 20.8682 7
produce:gaushit:GausHitFinder 0.448778 0.822741 1.14735 0.805973 0.24675 7
produce:nhitsfilter:NumberOfHitsFilter 0.000116767 0.000260444 0.00045483 0.000236048 0.000125752 7
produce:reco3d:SpacePointSolver 4.62038 8.70379 14.1233 9.22987 2.8611 7
produce:hitpdune:DisambigFromSpacePoints 0.0996679 0.146605 0.209432 0.13402 0.0334224 7
produce:pandora:StandardPandora 8.94731 20.1641 38.7197 21.099 9.59125 7
produce:pandoraTrack:LArPandoraTrackCreation 0.384 0.87729 2.14317 0.787593 0.552344 7
produce:pandoraGnocalo:GnocchiCalorimetry 0.0126796 0.0260276 0.0563121 0.0216442 0.0133685 7
[art]:TriggerResults:TriggerResultInserter 1.9279e-05 4.27163e-05 9.6497e-05 2.633e-05 2.90942e-05 7
end_path:out1:RootOutput 7.652e-06 1.19079e-05 2.5108e-05 8.382e-06 6.2104e-06 7
end_path:out1:RootOutput(write) 2.99111 4.51719 8.95574 4.10553 1.90527 7
==================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 5898.04 MB
Peak resident set size usage (VmHWM): 3943.28 MB
Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException: PostEndJob 20-Oct-2025 10:50:55 CEST ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- ScheduleExecutionFailure BEGIN
Path: ProcessingStopped.
---- BadAlloc BEGIN
A bad_alloc exception was thrown while processing module WireCellToolkit/wclsdatavd run: 40140 subRun: 1 event: 404500
The job has probably exhausted the virtual memory available to the process.
---- BadAlloc END
Exception going through path produce
---- ScheduleExecutionFailure END
---- EventProcessorFailure END
%MSG
Art has completed and will exit with status 1.
Error in reco1 