Jobsub ID 40719.1@dunegpschedd02.fnal.gov
| Jobsub ID | 40719.1@dunegpschedd02.fnal.gov |
| Workflow ID | 2576 |
| Stage ID | 1 |
| User name | ykermaid@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 18000 (5 hours) |
| Submitted time | 2025-09-17 01:20:33 |
| Site | NL_NIKHEF |
| Entry | VIRGO_NL_NIKHEF_dissel |
| Last heartbeat | 2025-09-17 02:52:23 |
| From worker node | Hostname | wn-choc-034.farm.nikhef.nl |
| cpuinfo | Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 129600 (36 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | jobscript_error |
| Started | 2025-09-17 01:21:06 |
| Input files | vd-protodune:np02vd_raw_run039275_0146_df-s02-d1_dw_0_20250901T191605.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Max RSS bytes | 0 (0 MiB) |
| Outputting started | |
| Output files | |
| Finished | 2025-09-17 02:52:23 |
| Saved logs | justin-logs:40719.1-dunegpschedd02.fnal.gov.logs.tgz |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
17-Sep-2025 04:50:45 CEST run: 39275 subRun: 1 event: 52843
Error! unrecognized channel number -1. Ignoring pulse
%MSG
%MSG-e OpHitFinder: OpHitFinder:ophit@BeginModule 17-Sep-2025 04:50:45 CEST run: 39275 subRun: 1 event: 52843
Error! unrecognized channel number -1. Ignoring pulse
%MSG
%MSG-e OpHitFinder: OpHitFinder:ophit@BeginModule 17-Sep-2025 04:50:45 CEST run: 39275 subRun: 1 event: 52843
Error! unrecognized channel number -1. Ignoring pulse
%MSG
%MSG-e OpHitFinder: OpHitFinder:ophit@BeginModule 17-Sep-2025 04:50:45 CEST run: 39275 subRun: 1 event: 52843
Error! unrecognized channel number -1. Ignoring pulse
%MSG
%MSG-e OpHitFinder: OpHitFinder:ophit@BeginModule 17-Sep-2025 04:50:45 CEST run: 39275 subRun: 1 event: 52843
Error! unrecognized channel number -1. Ignoring pulse
%MSG
%MSG-e OpHitFinder: OpHitFinder:ophit@BeginModule 17-Sep-2025 04:50:45 CEST run: 39275 subRun: 1 event: 52843
Error! unrecognized channel number -1. Ignoring pulse
%MSG
%MSG-e OpHitFinder: OpHitFinder:ophit@BeginModule 17-Sep-2025 04:50:45 CEST run: 39275 subRun: 1 event: 52843
Error! unrecognized channel number -1. Ignoring pulse
%MSG
%MSG-e OpHitFinder: OpHitFinder:ophit@BeginModule 17-Sep-2025 04:50:45 CEST run: 39275 subRun: 1 event: 52843
Error! unrecognized channel number -1. Ignoring pulse
%MSG
%MSG-e OpHitFinder: OpHitFinder:ophit@BeginModule 17-Sep-2025 04:50:45 CEST run: 39275 subRun: 1 event: 52843
Error! unrecognized channel number -1. Ignoring pulse
%MSG
%MSG-e OpHitFinder: OpHitFinder:ophit@BeginModule 17-Sep-2025 04:50:45 CEST run: 39275 subRun: 1 event: 52843
Error! unrecognized channel number -1. Ignoring pulse
%MSG
%MSG-e OpHitFinder: OpHitFinder:ophit@BeginModule 17-Sep-2025 04:50:45 CEST run: 39275 subRun: 1 event: 52843
Error! unrecognized channel number -1. Ignoring pulse
%MSG
%MSG-e OpHitFinder: OpHitFinder:ophit@BeginModule 17-Sep-2025 04:50:45 CEST run: 39275 subRun: 1 event: 52843
Error! unrecognized channel number -1. Ignoring pulse
%MSG
%MSG-e OpHitFinder: OpHitFinder:ophit@BeginModule 17-Sep-2025 04:50:45 CEST run: 39275 subRun: 1 event: 52843
Error! unrecognized channel number -1. Ignoring pulse
%MSG
%MSG-e OpHitFinder: OpHitFinder:ophit@BeginModule 17-Sep-2025 04:50:45 CEST run: 39275 subRun: 1 event: 52843
Error! unrecognized channel number -1. Ignoring pulse
%MSG
%MSG-e OpHitFinder: OpHitFinder:ophit@BeginModule 17-Sep-2025 04:50:45 CEST run: 39275 subRun: 1 event: 52843
Error! unrecognized channel number -1. Ignoring pulse
%MSG
%MSG-e OpHitFinder: OpHitFinder:ophit@BeginModule 17-Sep-2025 04:50:45 CEST run: 39275 subRun: 1 event: 52843
Error! unrecognized channel number -1. Ignoring pulse
%MSG
%MSG-e OpHitFinder: OpHitFinder:ophit@BeginModule 17-Sep-2025 04:50:45 CEST run: 39275 subRun: 1 event: 52843
Error! unrecognized channel number -1. Ignoring pulse
%MSG
%MSG-e OpHitFinder: OpHitFinder:ophit@BeginModule 17-Sep-2025 04:50:45 CEST run: 39275 subRun: 1 event: 52843
Error! unrecognized channel number -1. Ignoring pulse
%MSG
%MSG-e OpHitFinder: OpHitFinder:ophit@BeginModule 17-Sep-2025 04:50:45 CEST run: 39275 subRun: 1 event: 52843
Error! unrecognized channel number -1. Ignoring pulse
%MSG
%MSG-e OpHitFinder: OpHitFinder:ophit@BeginModule 17-Sep-2025 04:50:45 CEST run: 39275 subRun: 1 event: 52843
Error! unrecognized channel number -1. Ignoring pulse
%MSG
RawFrameSource: got 12288 raw::RawDigit objects
input nticks=10048 keeping as is
[04:50:45.613] D [ main ] executing 1 apps, thread limit 0:
[04:50:45.613] D [ main ] executing 1 apps, thread limit 0:
[04:50:45.613] D [ main ] executing app: "Pgrapher"
[04:50:45.613] D [ pgraph ] <Pgrapher:> executing graph
[04:50:45.613] D [ pgraph ] executing with 26 nodes
[04:50:45.616] D [ glue ] <FrameFanout:nfsp> call=28: input: frame: ident=52843 time=340 tick=512 with 12288 traces. frame tags:[ "orig" ] 0 tagged trace sets:[ ] cmm:[ ] output 0: frame: ident=52843 time=340 tick=512 with 12288 traces. frame tags:[ "orig0" ] 0 tagged trace sets:[ ] cmm:[ ] output 1: frame: ident=52843 time=340 tick=512 with 12288 traces. frame tags:[ "orig1" ] 0 tagged trace sets:[ ] cmm:[ ] output 2: frame: ident=52843 time=340 tick=512 with 12288 traces. frame tags:[ "orig2" ] 0 tagged trace sets:[ ] cmm:[ ] output 3: frame: ident=52843 time=340 tick=512 with 12288 traces. frame tags:[ "orig3" ] 0 tagged trace sets:[ ] cmm:[ ] output 4: frame: ident=52843 time=340 tick=512 with 12288 traces. frame tags:[ "orig4" ] 0 tagged trace sets:[ ] cmm:[ ] output 5: frame: ident=52843 time=340 tick=512 with 12288 traces. frame tags:[ "orig5" ] 0 tagged trace sets:[ ] cmm:[ ] output 6: frame: ident=52843 time=340 tick=512 with 12288 traces. frame tags:[ "orig6" ] 0 tagged trace sets:[ ] cmm:[ ] output 7: frame: ident=52843 time=340 tick=512 with 12288 traces. frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ]
[04:50:45.616] W [ glue ] <ChannelSelector:chsel7> Untagged summary not supported, summary will be dropped.
[04:50:45.617] D [ glue ] <ChannelSelector:chsel7> input frame: ident=52843 time=340 tick=512 with 12288 traces. frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=52843 time=340 tick=512 with 1536 traces. frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ]
[04:50:45.617] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=28 input frame: frame: ident=52843 time=340 tick=512 with 1536 traces. frame tags:[ "orig7" ] 0 tagged trace sets:[ ] cmm:[ ]
[04:50:45.617] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=28 init nticks=10048 tbinmin=0 tbinmax=10048
[04:50:45.683] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=28 load plane index: 0, ntraces=1536, input bad regions: 0
[04:50:50.095] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=28 load plane index: 1, ntraces=1536, input bad regions: 0
[04:50:53.866] D [sigproc ] <OmnibusSigProc:anode7sigproc7> call=28 load plane index: 2, ntraces=1536, input bad regions: 0
==================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
==================================================================================================================================
Full event 6.6143e-05 346.98 487.426 361.393 110.064 15
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read) 5.8222e-05 8.03699e-05 0.000230345 6.6221e-05 4.14018e-05 15
produce:tpcrawdecoder:PDVDTPCReader 78.9996 105.04 130.101 107.005 15.3271 15
produce:triggerrawdecoder:PDVDTriggerReader4 0.0347678 0.0375228 0.0500389 0.0349042 0.00463241 15
produce:pdvddaphne:DAPHNEReaderPDVD 13.0087 15.9343 19.2821 16.4262 2.14983 15
produce:ophit:OpHitFinder 0.0628339 0.0772542 0.0844816 0.0784475 0.00643591 15
produce:opflash:OpFlashFinderVerticalDrift 0.0145627 0.0195646 0.0309482 0.0189883 0.00382316 15
produce:wclsdatavd:WireCellToolkit 100.578 114.084 126.359 113.628 6.28589 14
produce:gaushit:GausHitFinder 1.2199 1.98822 2.94383 1.90964 0.458081 14
produce:nhitsfilter:NumberOfHitsFilter 0.000329666 0.000583448 0.000799823 0.000590666 0.000142492 14
produce:reco3d:SpacePointSolver 9.78696 19.2329 32.5086 19.8237 5.77849 14
produce:hitpdune:DisambigFromSpacePoints 0.170095 0.320026 0.595933 0.308731 0.117558 14
produce:pandora:StandardPandora 39.4633 105.102 223.698 97.8337 49.3834 14
produce:pandoraTrack:LArPandoraTrackCreation 1.56663 3.60396 5.62652 3.91534 1.28921 14
produce:pandoraGnocalo:GnocchiCalorimetry 0.0303355 0.0507131 0.0718036 0.0530756 0.011905 14
[art]:TriggerResults:TriggerResultInserter 2.6884e-05 5.66478e-05 0.000141548 5.56475e-05 2.65912e-05 14
end_path:out1:RootOutput 7.686e-06 1.34269e-05 4.103e-05 1.16845e-05 8.24579e-06 14
end_path:out1:RootOutput(write) 6.10729 6.48667 6.90797 6.53457 0.229604 14
==================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 8589.89 MB
Peak resident set size usage (VmHWM): 6706.23 MB
Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException: PostEndJob 17-Sep-2025 04:52:03 CEST ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- ScheduleExecutionFailure BEGIN
Path: ProcessingStopped.
---- BadAlloc BEGIN
A bad_alloc exception was thrown while processing module WireCellToolkit/wclsdatavd run: 39275 subRun: 1 event: 52843
The job has probably exhausted the virtual memory available to the process.
---- BadAlloc END
Exception going through path produce
---- ScheduleExecutionFailure END
---- EventProcessorFailure END
%MSG
Art has completed and will exit with status 1.
Error in reco1 