Jobsub ID 236259.1@dunegpschedd02.fnal.gov
| Jobsub ID | 236259.1@dunegpschedd02.fnal.gov |
| Workflow ID | 9405 |
| Stage ID | 1 |
| User name | ykermaid@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 18000 (5 hours) |
| Submitted time | 2025-11-02 08:11:16 |
| Site | NL_NIKHEF |
| Entry | VIRGO_NL_NIKHEF_juk |
| Last heartbeat | 2025-11-02 08:21:22 |
| From worker node | Hostname | wn-sate-050.farm.nikhef.nl |
| cpuinfo | AMD EPYC 7551P 32-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 129600 (36 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | jobscript_error |
| Started | 2025-11-02 08:12:12 |
| Input files | vd-protodune:np02vd_raw_run040270_0624_df-s03-d3_dw_0_20251028T061135.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Max RSS bytes | 0 (0 MiB) |
| Outputting started | |
| Output files | |
| Finished | 2025-11-02 08:21:22 |
| Saved logs | justin-logs:236259.1-dunegpschedd02.fnal.gov.logs.tgz |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
-> 36 XUVs
C:0 T:4 585 XUs and 520 XVs -> 14 XUVs
C:0 T:5 1342 XUs and 2031 XVs -> 46 XUVs
C:0 T:6 16631 XUs and 20684 XVs -> 1474 XUVs
C:0 T:7 373 XUs and 493 XVs -> 14 XUVs
C:0 T:8 1058 XUs and 1309 XVs -> 75 XUVs
C:0 T:9 332 XUs and 291 XVs -> 6 XUVs
C:0 T:10 1745 XUs and 1901 XVs -> 67 XUVs
C:0 T:11 939 XUs and 1194 XVs -> 89 XUVs
C:0 T:12 1580 XUs and 1876 XVs -> 158 XUVs
C:0 T:13 763 XUs and 1024 XVs -> 60 XUVs
C:0 T:14 388 XUs and 518 XVs -> 23 XUVs
C:0 T:15 1046 XUs and 1250 XVs -> 87 XUVs
2227 XUVs total
1352 collection wire objects
2227 potential space points
Neighbour search...
72509 tests to find 18124 neighbours
Iterating with no regularization...
Begin: 8.19477e+10
0 8.08484e+10
1 8.0791e+10
Now with regularization...
Begin: 8.01684e+10
0 8.01677e+10
02-Nov-2025 09:16:55 CET Opened output file with pattern "%ifb_reco_stage1_%tc_keepup.root"
RawFrameSource: got 12288 raw::RawDigit objects
input nticks=8642 keeping as is
[09:18:56.567] D [ main ] executing 1 apps, thread limit 0:
[09:18:56.567] D [ main ] executing 1 apps, thread limit 0:
[09:18:56.567] D [ main ] executing app: "Pgrapher"
[09:18:56.567] D [ pgraph ] <Pgrapher:> executing graph
[09:18:56.567] D [ pgraph ] executing with 26 nodes
[09:18:56.568] D [ glue ] <FrameFanout:nfsp> call=2: input: frame: ident=115555 time=4 tick=512 with 12288 traces. frame tags:[ "orig" ] 0 tagged trace sets:[ ] cmm:[ ] output 0: frame: ident=115555 time=4 tick=512 with 12288 traces. frame tags:[ "orig0" ] 0 tagged trace sets:[ ] cmm:[ ] output 1: frame: ident=115555 time=4 tick=512 with 12288 traces. frame tags:[ "orig1" ] 0 tagged trace sets:[ ] cmm:[ ] output 2: frame: ident=115555 time=4 tick=512 with 12288 traces. frame tags:[ "orig2" ] 0 tagged trace sets:[ ] cmm:[ ] output 3: frame: ident=115555 time=4 tick=512 with 12288 traces. frame tags:[ "orig3" ] 0 tagged trace sets:[ ] cmm:[ ] output 4: frame: ident=115555 time=4 tick=512 with 12288 traces. frame tags:[ "orig4" ] 0 tagged trace sets:[ ] cmm:[ ] output 5: frame: ident=115555 time=4 tick=512 with 12288 traces. frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] output 6: frame: ident=115555 time=4 tick=512 with 12288 traces. frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] output 7: frame: ident=115555 time=4 tick=512 with 12288 traces. frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ]
[09:18:56.568] W [ glue ] <ChannelSelector:chsel7> Untagged summary not supported, summary will be dropped.
[09:18:56.569] D [ glue ] <ChannelSelector:chsel7> input frame: ident=115555 time=4 tick=512 with 12288 traces. frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=115555 time=4 tick=512 with 1536 traces. frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ]
[09:18:56.569] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=2 input frame: frame: ident=115555 time=4 tick=512 with 1536 traces. frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ]
[09:18:56.569] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=2 init nticks=8642 tbinmin=0 tbinmax=8642
[09:18:56.600] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=2 load plane index: 0, ntraces=1536, input bad regions: 0
[09:18:59.388] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=2 load plane index: 1, ntraces=1536, input bad regions: 0
[09:19:02.157] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=2 load plane index: 2, ntraces=1536, input bad regions: 0
[09:19:38.527] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=2 save plane index: 0, Qtot=19445010073 Qloss=-3169881499, 4975 indices spanning [11875,16849] "wiener"
[09:19:38.863] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=2 save plane index: 0, Qtot=18036468951 Qloss=-3223164247, 4441 indices spanning [16850,21290] "gauss"
[09:19:40.147] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=2 save plane index: 1, Qtot=28311665519 Qloss=-23529604342, 6650 indices spanning [21291,27940] "wiener"
[09:19:40.476] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=2 save plane index: 1, Qtot=25673020852 Qloss=-23960808888, 6046 indices spanning [27941,33986] "gauss"
[09:19:41.011] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=2 save plane index: 2, Qtot=2317890597 Qloss=-1513696014, 8826 indices spanning [33987,42812] "wiener"
[09:19:41.555] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=2 save plane index: 2, Qtot=1989768615 Qloss=-1185609725, 10065 indices spanning [42813,52877] "gauss"
[09:19:41.555] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=2 produce 52878 traces: 20451 wiener7, 0 decon_charge7, 20552 gauss7, frame tag: sigproc
[09:19:41.555] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=2 output frame: frame: ident=115555 time=4 tick=512 with 52878 traces. frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss7":20552 [0] "mp2_roi7":8139 [0] "mp3_roi7":3736 [0] "wiener7":20451 [20451] ] cmm:[ ]
[09:19:45.356] W [ glue ] <ChannelSelector:chsel6> Untagged summary not supported, summary will be dropped.
[09:19:45.356] D [ glue ] <ChannelSelector:chsel6> input frame: ident=115555 time=4 tick=512 with 12288 traces. frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=115555 time=4 tick=512 with 1536 traces. frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ]
[09:19:45.357] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=2 input frame: frame: ident=115555 time=4 tick=512 with 1536 traces. frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ]
[09:19:45.357] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=2 init nticks=8642 tbinmin=0 tbinmax=8642
[09:19:45.391] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=2 load plane index: 0, ntraces=1536, input bad regions: 0
[09:19:48.664] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=2 load plane index: 1, ntraces=1536, input bad regions: 0
[09:19:52.062] D [sigproc ] <OmnibusSigProc:anode6sigproc6> call=2 load plane index: 2, ntraces=1536, input bad regions: 0
==================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
==================================================================================================================================
Full event 5.8891e-05 124.426 248.851 124.426 124.425 2
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read) 5.8891e-05 0.000155708 0.000252525 0.000155708 9.6817e-05 2
produce:tpcrawdecoder:PDVDTPCReader 95.5557 104.453 113.351 104.453 8.89774 2
produce:triggerrawdecoder:PDVDTriggerReader4 0.365873 0.366996 0.368118 0.366996 0.00112295 2
produce:pdvddaphne:DAPHNEReaderPDVD 0.000394933 0.000607993 0.000821054 0.000607993 0.000213061 2
produce:ophit:OpHitFinder 6.7276e-05 0.000330306 0.000593336 0.000330306 0.00026303 2
produce:opflash:OpFlashFinderVerticalDrift 5.293e-05 0.000211799 0.000370667 0.000211799 0.000158869 2
produce:wclsdatavd:WireCellToolkit 87.081 87.081 87.081 87.081 0 1
produce:gaushit:GausHitFinder 1.4867 1.4867 1.4867 1.4867 0 1
produce:nhitsfilter:NumberOfHitsFilter 0.000629844 0.000629844 0.000629844 0.000629844 0 1
produce:reco3d:SpacePointSolver 13.71 13.71 13.71 13.71 0 1
produce:hitpdune:DisambigFromSpacePoints 0.238829 0.238829 0.238829 0.238829 0 1
produce:pandora:StandardPandora 43.4682 43.4682 43.4682 43.4682 0 1
produce:pandoraTrack:LArPandoraTrackCreation 1.46666 1.46666 1.46666 1.46666 0 1
produce:pandoraGnocalo:GnocchiCalorimetry 0.0454471 0.0454471 0.0454471 0.0454471 0 1
[art]:TriggerResults:TriggerResultInserter 0.000145874 0.000145874 0.000145874 0.000145874 0 1
end_path:out1:RootOutput 3.3282e-05 3.3282e-05 3.3282e-05 3.3282e-05 0 1
end_path:out1:RootOutput(write) 5.37033 5.37033 5.37033 5.37033 0 1
==================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 8589.89 MB
Peak resident set size usage (VmHWM): 6705.83 MB
Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException: PostEndJob 02-Nov-2025 09:20:58 CET ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- ScheduleExecutionFailure BEGIN
Path: ProcessingStopped.
---- BadAlloc BEGIN
A bad_alloc exception was thrown while processing module WireCellToolkit/wclsdatavd run: 40270 subRun: 1 event: 115555
The job has probably exhausted the virtual memory available to the process.
---- BadAlloc END
Exception going through path produce
---- ScheduleExecutionFailure END
---- EventProcessorFailure END
%MSG
Art has completed and will exit with status 1.
Error in reco1 