justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 39026.34@dunegpschedd01.fnal.gov

Jobsub ID39026.34@dunegpschedd01.fnal.gov
Workflow ID2183
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod.mcsim
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-09-02 15:26:50
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce509
Last heartbeat2025-09-02 16:03:40
From worker nodeHostnameb9p10p2979.cern.ch
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Started2025-09-02 15:27:58
Input filesvd-protodune:np02vd_raw_run038563_0023_df-s05-d2_dw_0_20250804T183130.hdf5
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-09-02 16:03:40
Saved logsjustin-logs:39026.34-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

[0] "mp3_roi2":1206 [0] "wiener2":5537 [5537] ] cmm:[ ] input 3: frame: ident=4748 time=79 tick=500 with 25515 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss3":7249 [0] "mp2_roi3":7320 [0] "mp3_roi3":2020 [0] "wiener3":8926 [8926] ] cmm:[ ] input 4: frame: ident=4748 time=79 tick=512 with 0 traces.  frame tags:[ ] 0 tagged trace sets:[ ] cmm:[ ] input 5: frame: ident=4748 time=79 tick=512 with 0 traces.  frame tags:[ ] 0 tagged trace sets:[ ] cmm:[ ] input 6: frame: ident=4748 time=79 tick=512 with 0 traces.  frame tags:[ ] 0 tagged trace sets:[ ] cmm:[ ] input 7: frame: ident=4748 time=79 tick=512 with 0 traces.  frame tags:[ ] 0 tagged trace sets:[ ] cmm:[ ] output: frame: ident=4748 time=79 tick=500 with 75861 traces.  frame tags:[ "framefanin" ] 8 tagged trace sets:[ "gauss0":5074 [0] "gauss1":5208 [0] "gauss2":4414 [0] "gauss3":7249 [0] "wiener0":6301 [6301] "wiener1":6406 [6406] "wiener2":5537 [5537] "wiener3":8926 [8926] ] cmm:[ ] 
Retagger: tagging trace set: wiener with 27170 traces, 27170 summary
Retagger: tagging trace set: gauss with 21945 traces, 0 summary
[18:02:54.319] D [  glue  ] sink frame: #4748 @7.9e-05 with 75861 traces, frame tags:[retagger], trace tags:[gauss, wiener]
[18:02:54.319] D [  glue  ] <FrameFanout:nfsp> call=65: see EOS 
[18:02:54.319] D [  glue  ] <ChannelSelector:chsel7> see EOS at call=32 
[18:02:54.319] D [sigproc ] <OmnibusSigProc:anode7sigproc7> EOS at call=65 anode=7 
[18:02:54.319] D [  glue  ] <ChannelSelector:chsel6> see EOS at call=32 
[18:02:54.319] D [sigproc ] <OmnibusSigProc:anode6sigproc6> EOS at call=65 anode=6 
[18:02:54.319] D [  glue  ] <ChannelSelector:chsel5> see EOS at call=32 
[18:02:54.319] D [sigproc ] <OmnibusSigProc:anode5sigproc5> EOS at call=65 anode=5 
[18:02:54.319] D [  glue  ] <ChannelSelector:chsel4> see EOS at call=32 
[18:02:54.320] D [sigproc ] <OmnibusSigProc:anode4sigproc4> EOS at call=65 anode=4 
[18:02:54.320] D [  glue  ] <ChannelSelector:chsel3> see EOS at call=32 
[18:02:54.320] D [  aux   ] <Resampler:resmp3> EOS at call=65 
[18:02:54.320] D [sigproc ] <OmnibusSigProc:anode3sigproc3> EOS at call=65 anode=3 
[18:02:54.320] D [  glue  ] <ChannelSelector:chsel2> see EOS at call=32 
[18:02:54.320] D [  aux   ] <Resampler:resmp2> EOS at call=65 
[18:02:54.320] D [sigproc ] <OmnibusSigProc:anode2sigproc2> EOS at call=65 anode=2 
[18:02:54.320] D [  glue  ] <ChannelSelector:chsel1> see EOS at call=32 
[18:02:54.320] D [  aux   ] <Resampler:resmp1> EOS at call=65 
[18:02:54.320] D [sigproc ] <OmnibusSigProc:anode1sigproc1> EOS at call=65 anode=1 
[18:02:54.320] D [  glue  ] <ChannelSelector:chsel0> see EOS at call=32 
[18:02:54.320] D [  aux   ] <Resampler:resmp0> EOS at call=65 
[18:02:54.320] D [sigproc ] <OmnibusSigProc:anode0sigproc0> EOS at call=65 anode=0 
[18:02:54.320] D [  glue  ] <FrameFanin:nfsp> EOS at call=65 with 8 
[18:02:54.320] D [  glue  ] frame sink sees EOS
[18:02:54.320] D [ pgraph ] <Pgrapher:> graph execution complete 
[18:02:54.320] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 7.82 sec
[18:02:54.320] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 7.75 sec
[18:02:54.320] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 7.53 sec
[18:02:54.320] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 7.33 sec
[18:02:54.320] I [ timer  ] Timer: WireCell::Aux::Resampler : 1.1 sec
[18:02:54.320] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.85 sec
[18:02:54.320] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.8 sec
[18:02:54.320] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.79 sec
[18:02:54.320] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.02 sec
[18:02:54.320] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.01 sec
[18:02:54.320] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 0 sec
[18:02:54.320] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[18:02:54.320] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[18:02:54.320] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 0 sec
[18:02:54.320] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[18:02:54.320] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[18:02:54.320] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[18:02:54.320] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[18:02:54.320] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 0 sec
[18:02:54.320] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[18:02:54.320] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[18:02:54.320] I [ timer  ] Timer: WireCell::Gen::Retagger : 0 sec
[18:02:54.320] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[18:02:54.320] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[18:02:54.320] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 0 sec
[18:02:54.320] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[18:02:54.320] I [ timer  ] Timer: Total node execution : 34.000000385567546 sec
wclsFrameSaver saving cooked to 10000 ticks
wclsFrameSaver: saving 21945 traces tagged "gauss"
FrameSaver: q=2.76346e+06 n=504768 tag=gauss
wclsFrameSaver: saving 27170 traces tagged "wiener"
FrameSaver: q=2.89632e+06 n=490559 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 783 XUs and 923 XVs -> 30 XUVs
C:0 T:1 764 XUs and 713 XVs -> 29 XUVs
C:0 T:2 1226 XUs and 1181 XVs -> 47 XUVs
C:0 T:3 634 XUs and 802 XVs -> 32 XUVs
C:0 T:4 26 XUs and 37 XVs -> 0 XUVs
C:0 T:5 1467 XUs and 1979 XVs -> 85 XUVs
C:0 T:6 1187 XUs and 1634 XVs -> 48 XUVs
C:0 T:7 1348 XUs and 1049 XVs -> 27 XUVs
298 XUVs total
270 collection wire objects
298 potential space points
Neighbour search...
1258 tests to find 600 neighbours
Iterating with no regularization...
Begin: 5.74921e+07
0 5.67251e+07
1 5.67225e+07
Now with regularization...
Begin: 5.54886e+07
0 5.54886e+07
TestBeamParticleCreationAlgorithm::SetupTestBeamVertex - Test beam particle has no initial vertex
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
02-Sep-2025 18:03:17 CEST  Closed output file "np02vd_raw_run038563_0023_df-s05-d2_dw_0_20250804T183130_reco_stage1_20250902T160317_keepup.root"

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                        47.251        62.914        84.848        63.0521       7.96164        33     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       5.206e-05    0.000153375   0.00160642    9.8661e-05    0.000263394      33     
produce:tpcrawdecoder:PDVDTPCReader               2.99312       3.41676       3.70124       3.41533      0.132018        33     
produce:triggerrawdecoder:PDVDTriggerReader4    0.00117193    0.00167239    0.00495207    0.00148789    0.000680352      33     
produce:pdvddaphne:DAPHNEReaderPDVD               3.86597       4.29875       4.70303       4.3275       0.242094        33     
produce:ophit:OpHitFinder                        0.0363231     0.0627325     0.0751207     0.0635151    0.00722092       33     
produce:opflash:OpFlashFinderVerticalDrift      0.00576529     0.0142278     0.0222247     0.014233     0.00365566       33     
produce:wclsdatavd:WireCellToolkit                33.0924       37.8027       44.1478       38.0362       2.16077        33     
produce:gaushit:GausHitFinder                    0.251613       0.56273       1.02831      0.548733      0.159656        33     
produce:nhitsfilter:NumberOfHitsFilter          8.1461e-05    0.000141498   0.000385083   0.000135192   5.03445e-05      33     
produce:reco3d:SpacePointSolver                   1.01504       4.19404       8.8169        4.21967       1.84606        33     
produce:hitpdune:DisambigFromSpacePoints        0.00804083     0.0365548     0.086358      0.0309083     0.0188758       33     
produce:pandora:StandardPandora                   1.62075       9.30041       22.9722       8.23744       4.99539        33     
produce:pandoraTrack:LArPandoraTrackCreation     0.0475231     0.168227      0.353722      0.163735      0.0717608       33     
produce:pandoraGnocalo:GnocchiCalorimetry       0.00658587     0.014394      0.0269036     0.0149951    0.00414561       33     
[art]:TriggerResults:TriggerResultInserter      1.9491e-05    2.81513e-05   6.4441e-05     2.501e-05    9.00809e-06      33     
end_path:out1:RootOutput                         4.06e-06     7.95415e-06    3.662e-05     7.73e-06     5.33921e-06      33     
end_path:out1:RootOutput(write)                   2.61619       3.02871       3.30867       3.02841      0.147427        33     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4061.75 MB
  Peak resident set size usage (VmHWM): 2112.4 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run038563_0023_df-s05-d2_dw_0_20250804T183130_reco_stage1_20250902T160317_keepup.root
\tHists: np02vd_raw_run038563_0023_df-s05-d2_dw_0_20250804T183130_reco_stage1_20250902T160317_keepup_hists.root
Forming reco metadata
/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/bin/python: No module named meta_maker
Error in reco metadata
justIN time: 2025-11-05 12:37:41 UTC       justIN version: 01.05.01