justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 32473.2@dunegpschedd02.fnal.gov

Jobsub ID32473.2@dunegpschedd02.fnal.gov
Workflow ID2135
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod.mcsim
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-09-02 21:47:08
SiteCZ_FZU
EntryNova_CZ_FZU_osgce1
Last heartbeat2025-09-02 23:07:44
From worker nodeHostnamelupin22.farm.particle.cz
cpuinfoAMD EPYC 7551 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Started2025-09-02 21:51:13
Input filesvd-protodune:np02vd_raw_run039275_0447_df-s01-d3_dw_0_20250902T052648.hdf5
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-09-02 23:07:44
Saved logsjustin-logs:32473.2-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

] cmm:[ ] input 7: frame: ident=161261 time=111 tick=512 with 13416 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss7":4112 [0] "mp2_roi7":2914 [0] "mp3_roi7":1232 [0] "wiener7":5158 [5158] ] cmm:[ ] output: frame: ident=161261 time=111 tick=500 with 146441 traces.  frame tags:[ "framefanin" ] 16 tagged trace sets:[ "gauss0":2032 [0] "gauss1":4055 [0] "gauss2":3589 [0] "gauss3":4141 [0] "gauss4":10099 [0] "gauss5":8952 [0] "gauss6":5062 [0] "gauss7":4112 [0] "wiener0":2372 [2372] "wiener1":5226 [5226] "wiener2":4475 [4475] "wiener3":4911 [4911] "wiener4":13142 [13142] "wiener5":11191 [11191] "wiener6":5995 [5995] "wiener7":5158 [5158] ] cmm:[ ] 
Retagger: tagging trace set: wiener with 52470 traces, 52470 summary
Retagger: tagging trace set: gauss with 42042 traces, 0 summary
[01:05:28.903] D [  glue  ] sink frame: #161261 @0.000111 with 146441 traces, frame tags:[retagger], trace tags:[gauss, wiener]
[01:05:28.903] D [  glue  ] <FrameFanout:nfsp> call=35: see EOS 
[01:05:28.903] D [  glue  ] <ChannelSelector:chsel7> see EOS at call=17 
[01:05:28.903] D [sigproc ] <OmnibusSigProc:anode7sigproc7> EOS at call=35 anode=7 
[01:05:28.903] D [  glue  ] <ChannelSelector:chsel6> see EOS at call=17 
[01:05:28.903] D [sigproc ] <OmnibusSigProc:anode6sigproc6> EOS at call=35 anode=6 
[01:05:28.903] D [  glue  ] <ChannelSelector:chsel5> see EOS at call=17 
[01:05:28.903] D [sigproc ] <OmnibusSigProc:anode5sigproc5> EOS at call=35 anode=5 
[01:05:28.903] D [  glue  ] <ChannelSelector:chsel4> see EOS at call=17 
[01:05:28.903] D [sigproc ] <OmnibusSigProc:anode4sigproc4> EOS at call=35 anode=4 
[01:05:28.903] D [  glue  ] <ChannelSelector:chsel3> see EOS at call=17 
[01:05:28.903] D [  aux   ] <Resampler:resmp3> EOS at call=35 
[01:05:28.903] D [sigproc ] <OmnibusSigProc:anode3sigproc3> EOS at call=35 anode=3 
[01:05:28.903] D [  glue  ] <ChannelSelector:chsel2> see EOS at call=17 
[01:05:28.903] D [  aux   ] <Resampler:resmp2> EOS at call=35 
[01:05:28.903] D [sigproc ] <OmnibusSigProc:anode2sigproc2> EOS at call=35 anode=2 
[01:05:28.903] D [  glue  ] <ChannelSelector:chsel1> see EOS at call=17 
[01:05:28.903] D [  aux   ] <Resampler:resmp1> EOS at call=35 
[01:05:28.903] D [sigproc ] <OmnibusSigProc:anode1sigproc1> EOS at call=35 anode=1 
[01:05:28.903] D [  glue  ] <ChannelSelector:chsel0> see EOS at call=17 
[01:05:28.903] D [  aux   ] <Resampler:resmp0> EOS at call=35 
[01:05:28.903] D [sigproc ] <OmnibusSigProc:anode0sigproc0> EOS at call=35 anode=0 
[01:05:28.903] D [  glue  ] <FrameFanin:nfsp> EOS at call=35 with 8 
[01:05:28.903] D [  glue  ] frame sink sees EOS
[01:05:28.903] D [ pgraph ] <Pgrapher:> graph execution complete 
[01:05:28.904] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 12.15 sec
[01:05:28.904] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 9.75 sec
[01:05:28.904] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 8.39 sec
[01:05:28.904] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 7.87 sec
[01:05:28.904] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 7.85 sec
[01:05:28.904] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 7.83 sec
[01:05:28.904] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 7.68 sec
[01:05:28.904] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 7.55 sec
[01:05:28.904] I [ timer  ] Timer: WireCell::Aux::Resampler : 1.15 sec
[01:05:28.904] I [ timer  ] Timer: WireCell::Aux::Resampler : 1.14 sec
[01:05:28.904] I [ timer  ] Timer: WireCell::Aux::Resampler : 1.14 sec
[01:05:28.904] I [ timer  ] Timer: WireCell::Aux::Resampler : 1.11 sec
[01:05:28.904] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.02 sec
[01:05:28.904] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.02 sec
[01:05:28.904] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[01:05:28.904] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[01:05:28.904] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[01:05:28.904] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[01:05:28.904] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[01:05:28.904] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[01:05:28.904] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[01:05:28.904] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[01:05:28.904] I [ timer  ] Timer: WireCell::Gen::Retagger : 0 sec
[01:05:28.904] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[01:05:28.904] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[01:05:28.904] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[01:05:28.904] I [ timer  ] Timer: Total node execution : 73.65999965555966 sec
wclsFrameSaver saving cooked to 10000 ticks
wclsFrameSaver: saving 42042 traces tagged "gauss"
FrameSaver: q=6.99038e+06 n=883167 tag=gauss
wclsFrameSaver: saving 52470 traces tagged "wiener"
FrameSaver: q=7.53095e+06 n=843208 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 143 XUs and 144 XVs -> 12 XUVs
C:0 T:1 103 XUs and 116 XVs -> 2 XUVs
C:0 T:2 970 XUs and 1107 XVs -> 37 XUVs
C:0 T:3 361 XUs and 350 XVs -> 16 XUVs
C:0 T:4 370 XUs and 495 XVs -> 33 XUVs
C:0 T:5 588 XUs and 699 XVs -> 30 XUVs
C:0 T:6 82 XUs and 77 XVs -> 1 XUVs
C:0 T:7 1129 XUs and 1064 XVs -> 57 XUVs
C:0 T:8 8709 XUs and 14025 XVs -> 2167 XUVs
C:0 T:9 18505 XUs and 32983 XVs -> 7136 XUVs
C:0 T:10 13713 XUs and 35636 XVs -> 5444 XUVs
C:0 T:11 980 XUs and 786 XVs -> 38 XUVs
C:0 T:12 5740 XUs and 4691 XVs -> 1087 XUVs
C:0 T:13 1055 XUs and 1234 XVs -> 55 XUVs
C:0 T:14 375 XUs and 467 XVs -> 40 XUVs
C:0 T:15 1171 XUs and 1691 XVs -> 318 XUVs
16473 XUVs total
1717 collection wire objects
16473 potential space points
Neighbour search...
2665155 tests to find 1092166 neighbours
Iterating with no regularization...
Begin: 2.63946e+08
0 2.24754e+08
1 2.21677e+08
2 2.21089e+08
3 2.20875e+08
Now with regularization...
Begin: 2.13321e+08
0 2.1311e+08
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
03-Sep-2025 01:07:21 CEST  Closed output file "np02vd_raw_run039275_0447_df-s01-d3_dw_0_20250902T052648_reco_stage1_20250902T230721_keepup.root"

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                        135.835       250.861       368.925       236.811       59.8868        18     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                      7.1451e-05    0.000113341   0.000282281   0.00010403    4.58203e-05      18     
produce:tpcrawdecoder:PDVDTPCReader               16.7129       18.1828       20.8656       17.4475       1.34888        18     
produce:triggerrawdecoder:PDVDTriggerReader4     0.0320167     0.0375086     0.0967097     0.0326483     0.0147124       18     
produce:pdvddaphne:DAPHNEReaderPDVD               5.64514       6.54778       8.06606       6.20667      0.787387        18     
produce:ophit:OpHitFinder                        0.0611897     0.0789845     0.0913263     0.0816151    0.00752279       18     
produce:opflash:OpFlashFinderVerticalDrift      0.00894876     0.020831      0.0358897     0.0187041    0.00728832       18     
produce:wclsdatavd:WireCellToolkit                72.5421       86.3841       102.324       83.9252       8.94568        18     
produce:gaushit:GausHitFinder                    0.775629       1.79521       2.56877       1.87341      0.439614        18     
produce:nhitsfilter:NumberOfHitsFilter          0.000273071   0.000476487   0.000731563   0.000449306   0.000109773      18     
produce:reco3d:SpacePointSolver                   8.44484       22.1577       33.9422       23.329        6.66013        18     
produce:hitpdune:DisambigFromSpacePoints         0.130139      0.352497      0.550539       0.35822      0.114205        18     
produce:pandora:StandardPandora                   20.4905       106.213       224.698       87.6118       49.5726        18     
produce:pandoraTrack:LArPandoraTrackCreation     0.691391       2.11343       3.66723       1.87209      0.823759        18     
produce:pandoraGnocalo:GnocchiCalorimetry        0.0293634     0.0470589     0.0644462     0.045998     0.00937318       18     
[art]:TriggerResults:TriggerResultInserter       3.163e-05    5.63806e-05   0.000174521   4.7135e-05    3.26514e-05      18     
end_path:out1:RootOutput                         5.15e-06     1.48406e-05    5.541e-05    1.1275e-05    1.09507e-05      18     
end_path:out1:RootOutput(write)                   6.10735       6.87405       7.61292       6.87575      0.441123        18     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5082.79 MB
  Peak resident set size usage (VmHWM): 3161.12 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run039275_0447_df-s01-d3_dw_0_20250902T052648_reco_stage1_20250902T230721_keepup.root
\tHists: np02vd_raw_run039275_0447_df-s01-d3_dw_0_20250902T052648_reco_stage1_20250902T230721_keepup_hists.root
Forming reco metadata
/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/bin/python: No module named meta_maker
Error in reco metadata
justIN time: 2025-11-05 14:43:54 UTC       justIN version: 01.05.01