Jobsub ID 228001.0@dunegpschedd02.fnal.gov
| Jobsub ID | 228001.0@dunegpschedd02.fnal.gov |
| Workflow ID | 8924 |
| Stage ID | 1 |
| User name | ykermaid@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 18000 (5 hours) |
| Submitted time | 2025-10-13 20:47:17 |
| Site | FR_CCIN2P3 |
| Entry | DUNE_FR_CCIN2P3_cccondorce01 |
| Last heartbeat | 2025-10-13 21:54:05 |
| From worker node | Hostname | ccwcondor0692 |
| cpuinfo | AMD EPYC 7302 16-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 106200 (29 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | jobscript_error |
| Started | 2025-10-13 20:48:45 |
| Input files | vd-protodune:np02vd_raw_run040033_0034_df-s04-d0_dw_0_20251013T170354.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Max RSS bytes | 0 (0 MiB) |
| Outputting started | |
| Output files | |
| Finished | 2025-10-13 21:54:05 |
| Saved logs | justin-logs:228001.0-dunegpschedd02.fnal.gov.logs.tgz |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
w for channel 6143
FrameSaver: q=1.1049e+07 n=876759 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 3745 XUs and 2890 XVs -> 303 XUVs
C:0 T:1 3130 XUs and 4054 XVs -> 225 XUVs
C:0 T:2 3590 XUs and 2322 XVs -> 111 XUVs
C:0 T:3 4644 XUs and 4918 XVs -> 185 XUVs
C:0 T:4 1142 XUs and 2016 XVs -> 49 XUVs
C:0 T:5 2743 XUs and 3145 XVs -> 111 XUVs
C:0 T:6 18424 XUs and 20456 XVs -> 1272 XUVs
C:0 T:7 1956 XUs and 2308 XVs -> 85 XUVs
2341 XUVs total
1578 collection wire objects
2341 potential space points
Neighbour search...
59701 tests to find 17092 neighbours
Iterating with no regularization...
Begin: 6.12279e+09
0 6.01021e+09
1 6.00616e+09
Now with regularization...
Begin: 5.90204e+09
0 5.90183e+09
RawFrameSource: got 6144 raw::RawDigit objects
input nticks=168 keeping as is
[23:53:36.276] D [ main ] executing 1 apps, thread limit 0:
[23:53:36.276] D [ main ] executing 1 apps, thread limit 0:
[23:53:36.276] D [ main ] executing app: "Pgrapher"
[23:53:36.276] D [ pgraph ] <Pgrapher:> executing graph
[23:53:36.277] D [ pgraph ] executing with 26 nodes
[23:53:36.277] D [ glue ] <FrameFanout:nfsp> call=66: input: frame: ident=5668 time=78 tick=512 with 6144 traces. frame tags:[ "orig" ] 0 tagged trace sets:[ ] cmm:[ ] output 0: frame: ident=5668 time=78 tick=512 with 6144 traces. frame tags:[ "orig0" ] 0 tagged trace sets:[ ] cmm:[ ] output 1: frame: ident=5668 time=78 tick=512 with 6144 traces. frame tags:[ "orig1" ] 0 tagged trace sets:[ ] cmm:[ ] output 2: frame: ident=5668 time=78 tick=512 with 6144 traces. frame tags:[ "orig2" ] 0 tagged trace sets:[ ] cmm:[ ] output 3: frame: ident=5668 time=78 tick=512 with 6144 traces. frame tags:[ "orig3" ] 0 tagged trace sets:[ ] cmm:[ ] output 4: frame: ident=5668 time=78 tick=512 with 6144 traces. frame tags:[ "orig4" ] 0 tagged trace sets:[ ] cmm:[ ] output 5: frame: ident=5668 time=78 tick=512 with 6144 traces. frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] output 6: frame: ident=5668 time=78 tick=512 with 6144 traces. frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] output 7: frame: ident=5668 time=78 tick=512 with 6144 traces. frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ]
[23:53:36.279] W [ glue ] <ChannelSelector:chsel7> Untagged summary not supported, summary will be dropped.
[23:53:36.280] D [ glue ] <ChannelSelector:chsel7> input frame: ident=5668 time=78 tick=512 with 6144 traces. frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=5668 time=78 tick=512 with 0 traces. frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ]
[23:53:36.280] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=66 input frame: frame: ident=5668 time=78 tick=512 with 0 traces. frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ]
[23:53:36.280] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=66 forwarding empty frame=5668
[23:53:36.281] W [ glue ] <ChannelSelector:chsel6> Untagged summary not supported, summary will be dropped.
[23:53:36.282] D [ glue ] <ChannelSelector:chsel6> input frame: ident=5668 time=78 tick=512 with 6144 traces. frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=5668 time=78 tick=512 with 0 traces. frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ]
[23:53:36.282] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=66 input frame: frame: ident=5668 time=78 tick=512 with 0 traces. frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ]
[23:53:36.282] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=66 forwarding empty frame=5668
[23:53:36.282] W [ glue ] <ChannelSelector:chsel5> Untagged summary not supported, summary will be dropped.
[23:53:36.284] D [ glue ] <ChannelSelector:chsel5> input frame: ident=5668 time=78 tick=512 with 6144 traces. frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=5668 time=78 tick=512 with 0 traces. frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ]
[23:53:36.284] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=66 input frame: frame: ident=5668 time=78 tick=512 with 0 traces. frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ]
[23:53:36.284] D [sigproc ] <OmnibusSigProc:anode5sigproc5> call=66 forwarding empty frame=5668
[23:53:36.284] W [ glue ] <ChannelSelector:chsel4> Untagged summary not supported, summary will be dropped.
[23:53:36.284] D [ glue ] <ChannelSelector:chsel4> input frame: ident=5668 time=78 tick=512 with 6144 traces. frame tags:[ "orig4" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=5668 time=78 tick=512 with 0 traces. frame tags:[ "orig4" ] 0 tagged trace sets:[ ] cmm:[ ]
[23:53:36.285] D [sigproc ] <OmnibusSigProc:anode4sigproc4> call=66 input frame: frame: ident=5668 time=78 tick=512 with 0 traces. frame tags:[ "orig4" ] 0 tagged trace sets:[ ] cmm:[ ]
[23:53:36.285] D [sigproc ] <OmnibusSigProc:anode4sigproc4> call=66 forwarding empty frame=5668
[23:53:36.285] W [ glue ] <ChannelSelector:chsel3> Untagged summary not supported, summary will be dropped.
[23:53:36.286] D [ glue ] <ChannelSelector:chsel3> input frame: ident=5668 time=78 tick=512 with 6144 traces. frame tags:[ "orig3" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=5668 time=78 tick=512 with 1536 traces. frame tags:[ "orig3" ] 0 tagged trace sets:[ ] cmm:[ ]
[23:53:36.287] D [ aux ] <Resampler:resmp3> first ch=4539 Ts=512 Ns=168 Ns_pad=250 Nrat=125 Tr=500 Nr=256 Nout=172 padding:linear
[23:53:36.297] D [ aux ] <Resampler:resmp3> resample 1536 traces at call=66
[23:53:36.297] D [sigproc ] <OmnibusSigProc:anode3sigproc3> call=66 input frame: frame: ident=5668 time=78 tick=500 with 1536 traces. frame tags:[ "orig3" ] 0 tagged trace sets:[ ] cmm:[ ]
[23:53:36.297] D [sigproc ] <OmnibusSigProc:anode3sigproc3> call=66 init nticks=172 tbinmin=0 tbinmax=172
[23:53:36.320] D [sigproc ] <OmnibusSigProc:anode3sigproc3> call=66 load plane index: 0, ntraces=1536, input bad regions: 0
==================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
==================================================================================================================================
Full event 59.931 115.725 375.166 77.317 73.5501 33
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read) 8.7675e-05 0.000127151 0.000269518 0.000108935 4.08223e-05 34
produce:tpcrawdecoder:PDVDTPCReader 0.861072 3.42304 7.2228 3.08228 1.10364 34
produce:triggerrawdecoder:PDVDTriggerReader4 0.0613681 0.066843 0.0755037 0.06654 0.0041679 34
produce:pdvddaphne:DAPHNEReaderPDVD 0.00034496 2.48297 2.80134 2.57073 0.458602 34
produce:ophit:OpHitFinder 0.000101822 0.0365844 0.0433618 0.038208 0.00724173 34
produce:opflash:OpFlashFinderVerticalDrift 7.2427e-05 0.0138168 0.0281862 0.0139157 0.00483453 34
produce:wclsdatavd:WireCellToolkit 30.9007 48.5251 105.216 37.8029 19.0639 33
produce:gaushit:GausHitFinder 0.445812 0.800685 1.9745 0.69717 0.308577 33
produce:nhitsfilter:NumberOfHitsFilter 0.000137669 0.00027644 0.000568823 0.000259299 9.54475e-05 33
produce:reco3d:SpacePointSolver 3.58371 7.88252 21.3251 6.2918 4.51693 33
produce:hitpdune:DisambigFromSpacePoints 0.0296385 0.108329 0.328814 0.0858403 0.0672451 33
produce:pandora:StandardPandora 9.29087 48.8101 237.694 25.0101 53.0603 33
produce:pandoraTrack:LArPandoraTrackCreation 0.119209 0.454037 1.52905 0.38674 0.315272 33
produce:pandoraGnocalo:GnocchiCalorimetry 0.00922831 0.0187126 0.03726 0.0185813 0.00568706 33
[art]:TriggerResults:TriggerResultInserter 1.9757e-05 2.78132e-05 6.0093e-05 2.6941e-05 7.95497e-06 33
end_path:out1:RootOutput 4.268e-06 8.91464e-06 2.4316e-05 8.597e-06 3.32805e-06 33
end_path:out1:RootOutput(write) 2.2686 2.91557 5.42449 2.86643 0.512139 33
==================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 7150.6 MB
Peak resident set size usage (VmHWM): 5185.47 MB
Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException: PostEndJob 13-Oct-2025 23:53:36 CEST ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- ScheduleExecutionFailure BEGIN
Path: ProcessingStopped.
---- BadAlloc BEGIN
A bad_alloc exception was thrown while processing module WireCellToolkit/wclsdatavd run: 40033 subRun: 1 event: 5668
The job has probably exhausted the virtual memory available to the process.
---- BadAlloc END
Exception going through path produce
---- ScheduleExecutionFailure END
---- EventProcessorFailure END
%MSG
Art has completed and will exit with status 1.
Error in reco1 