justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 32472.4@dunegpschedd02.fnal.gov

Jobsub ID32472.4@dunegpschedd02.fnal.gov
Workflow ID2135
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod.mcsim
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-09-02 21:43:08
SiteES_PIC
EntryDUNE_T1_ES_PIC_ce16-multicore
Last heartbeat2025-09-02 23:37:16
From worker nodeHostnametdm005.pic.es
cpuinfoAMD EPYC 9654 96-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit216000 (60 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Started2025-09-02 21:50:02
Input filesvd-protodune:np02vd_raw_run039275_0444_df-s04-d2_dw_0_20250902T051939.hdf5
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-09-02 23:37:16
Saved logsjustin-logs:32472.4-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ed trace sets:[ "gauss7":7620 [0] "mp2_roi7":7071 [0] "mp3_roi7":2224 [0] "wiener7":9079 [9079] ] cmm:[ ] output: frame: ident=160192 time=140 tick=500 with 207186 traces.  frame tags:[ "framefanin" ] 16 tagged trace sets:[ "gauss0":4097 [0] "gauss1":6576 [0] "gauss2":3217 [0] "gauss3":5953 [0] "gauss4":10865 [0] "gauss5":9775 [0] "gauss6":9250 [0] "gauss7":7620 [0] "wiener0":5100 [5100] "wiener1":8063 [8063] "wiener2":3986 [3986] "wiener3":7214 [7214] "wiener4":13791 [13791] "wiener5":12225 [12225] "wiener6":12223 [12223] "wiener7":9079 [9079] ] cmm:[ ] 
Retagger: tagging trace set: wiener with 71681 traces, 71681 summary
Retagger: tagging trace set: gauss with 57353 traces, 0 summary
[01:33:31.255] D [  glue  ] sink frame: #160192 @0.00014 with 207186 traces, frame tags:[retagger], trace tags:[gauss, wiener]
[01:33:31.255] D [  glue  ] <FrameFanout:nfsp> call=35: see EOS 
[01:33:31.255] D [  glue  ] <ChannelSelector:chsel7> see EOS at call=17 
[01:33:31.255] D [sigproc ] <OmnibusSigProc:anode7sigproc7> EOS at call=35 anode=7 
[01:33:31.255] D [  glue  ] <ChannelSelector:chsel6> see EOS at call=17 
[01:33:31.255] D [sigproc ] <OmnibusSigProc:anode6sigproc6> EOS at call=35 anode=6 
[01:33:31.255] D [  glue  ] <ChannelSelector:chsel5> see EOS at call=17 
[01:33:31.255] D [sigproc ] <OmnibusSigProc:anode5sigproc5> EOS at call=35 anode=5 
[01:33:31.255] D [  glue  ] <ChannelSelector:chsel4> see EOS at call=17 
[01:33:31.255] D [sigproc ] <OmnibusSigProc:anode4sigproc4> EOS at call=35 anode=4 
[01:33:31.255] D [  glue  ] <ChannelSelector:chsel3> see EOS at call=17 
[01:33:31.255] D [  aux   ] <Resampler:resmp3> EOS at call=35 
[01:33:31.255] D [sigproc ] <OmnibusSigProc:anode3sigproc3> EOS at call=35 anode=3 
[01:33:31.255] D [  glue  ] <ChannelSelector:chsel2> see EOS at call=17 
[01:33:31.255] D [  aux   ] <Resampler:resmp2> EOS at call=35 
[01:33:31.255] D [sigproc ] <OmnibusSigProc:anode2sigproc2> EOS at call=35 anode=2 
[01:33:31.255] D [  glue  ] <ChannelSelector:chsel1> see EOS at call=17 
[01:33:31.255] D [  aux   ] <Resampler:resmp1> EOS at call=35 
[01:33:31.255] D [sigproc ] <OmnibusSigProc:anode1sigproc1> EOS at call=35 anode=1 
[01:33:31.255] D [  glue  ] <ChannelSelector:chsel0> see EOS at call=17 
[01:33:31.255] D [  aux   ] <Resampler:resmp0> EOS at call=35 
[01:33:31.255] D [sigproc ] <OmnibusSigProc:anode0sigproc0> EOS at call=35 anode=0 
[01:33:31.256] D [  glue  ] <FrameFanin:nfsp> EOS at call=35 with 8 
[01:33:31.256] D [  glue  ] frame sink sees EOS
[01:33:31.256] D [ pgraph ] <Pgrapher:> graph execution complete 
[01:33:31.256] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 14.14 sec
[01:33:31.256] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 12.2 sec
[01:33:31.256] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 11.72 sec
[01:33:31.256] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 11.43 sec
[01:33:31.256] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 10.21 sec
[01:33:31.256] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 10.21 sec
[01:33:31.256] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 9.73 sec
[01:33:31.256] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 8.45 sec
[01:33:31.256] I [ timer  ] Timer: WireCell::Aux::Resampler : 1.53 sec
[01:33:31.256] I [ timer  ] Timer: WireCell::Aux::Resampler : 1.14 sec
[01:33:31.256] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.98 sec
[01:33:31.256] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.73 sec
[01:33:31.256] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.03 sec
[01:33:31.256] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.02 sec
[01:33:31.256] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[01:33:31.256] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[01:33:31.256] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[01:33:31.256] I [ timer  ] Timer: WireCell::Gen::Retagger : 0.01 sec
[01:33:31.256] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0.01 sec
[01:33:31.256] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[01:33:31.256] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[01:33:31.256] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[01:33:31.256] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[01:33:31.256] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[01:33:31.256] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[01:33:31.256] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[01:33:31.256] I [ timer  ] Timer: Total node execution : 92.57000014558434 sec
wclsFrameSaver saving cooked to 10000 ticks
wclsFrameSaver: saving 57353 traces tagged "gauss"
FrameSaver: q=1.03995e+07 n=1250653 tag=gauss
wclsFrameSaver: saving 71681 traces tagged "wiener"
FrameSaver: q=1.10694e+07 n=1196388 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 501 XUs and 759 XVs -> 34 XUVs
C:0 T:1 805 XUs and 915 XVs -> 29 XUVs
C:0 T:2 1789 XUs and 2439 XVs -> 94 XUVs
C:0 T:3 214 XUs and 395 XVs -> 19 XUVs
C:0 T:4 256 XUs and 309 XVs -> 23 XUVs
C:0 T:5 573 XUs and 595 XVs -> 24 XUVs
C:0 T:6 1952 XUs and 1722 XVs -> 70 XUVs
C:0 T:7 487 XUs and 847 XVs -> 25 XUVs
C:0 T:8 17741 XUs and 34464 XVs -> 6148 XUVs
C:0 T:9 4729 XUs and 6595 XVs -> 913 XUVs
C:0 T:10 1759 XUs and 2198 XVs -> 176 XUVs
C:0 T:11 12695 XUs and 20766 XVs -> 3364 XUVs
C:0 T:12 5713 XUs and 8598 XVs -> 1557 XUVs
C:0 T:13 8135 XUs and 10101 XVs -> 1464 XUVs
C:0 T:14 10188 XUs and 15271 XVs -> 2846 XUVs
C:0 T:15 785 XUs and 658 XVs -> 39 XUVs
16825 XUVs total
2479 collection wire objects
16825 potential space points
Neighbour search...
2370085 tests to find 1020648 neighbours
Iterating with no regularization...
Begin: 6.56296e+08
0 5.9298e+08
1 5.89957e+08
2 5.89365e+08
3 5.89143e+08
Now with regularization...
Begin: 5.68897e+08
0 5.68632e+08
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
03-Sep-2025 01:36:21 CEST  Closed output file "np02vd_raw_run039275_0444_df-s04-d2_dw_0_20250902T051939_reco_stage1_20250902T233621_keepup.root"

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                        221.848       350.114       567.371       306.754       98.1992        18     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                      0.000137936   0.000252078   0.00043972    0.000220761   9.32212e-05      18     
produce:tpcrawdecoder:PDVDTPCReader               21.2765       26.4656       29.0547       26.7431       1.4686         18     
produce:triggerrawdecoder:PDVDTriggerReader4     0.0599601     0.061287      0.0644704     0.0609208    0.00123604       18     
produce:pdvddaphne:DAPHNEReaderPDVD               6.11311       7.25091       8.40253       7.09642      0.614726        18     
produce:ophit:OpHitFinder                        0.0748947     0.0931439     0.141333      0.0896659     0.0147692       18     
produce:opflash:OpFlashFinderVerticalDrift       0.0192275     0.0281487     0.0378444     0.0269298    0.00535577       18     
produce:wclsdatavd:WireCellToolkit                95.2046       123.596       156.486       123.976       12.7136        18     
produce:gaushit:GausHitFinder                     1.7926        2.61171       3.99761       2.42606      0.534228        18     
produce:nhitsfilter:NumberOfHitsFilter          0.00111536    0.00270301    0.00597676    0.00244137    0.00108461       18     
produce:reco3d:SpacePointSolver                   15.1672       29.0753       42.815        26.7645       7.27201        18     
produce:hitpdune:DisambigFromSpacePoints         0.254893      0.516982       1.09823      0.469149       0.19979        18     
produce:pandora:StandardPandora                   44.6689       145.546       352.525       117.887       84.9055        18     
produce:pandoraTrack:LArPandoraTrackCreation      2.7244        6.60528       12.6982       6.00812       2.52706        18     
produce:pandoraGnocalo:GnocchiCalorimetry        0.0386576     0.0853562     0.145124      0.0841105     0.0241067       18     
[art]:TriggerResults:TriggerResultInserter      8.4102e-05    0.000154909   0.000383181   0.000114349   8.74895e-05      18     
end_path:out1:RootOutput                        1.3201e-05    2.52201e-05   7.4788e-05    2.15435e-05   1.34162e-05      18     
end_path:out1:RootOutput(write)                   6.85124       7.98924       8.95212       7.9871       0.562124        18     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5067.1 MB
  Peak resident set size usage (VmHWM): 3053.51 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run039275_0444_df-s04-d2_dw_0_20250902T051939_reco_stage1_20250902T233621_keepup.root
\tHists: np02vd_raw_run039275_0444_df-s04-d2_dw_0_20250902T051939_reco_stage1_20250902T233621_keepup_hists.root
Forming reco metadata
/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/bin/python: No module named meta_maker
Error in reco metadata
justIN time: 2025-11-05 14:39:34 UTC       justIN version: 01.05.01