justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 32023.20@dunegpschedd02.fnal.gov

Jobsub ID32023.20@dunegpschedd02.fnal.gov
Workflow ID2131
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod.mcsim
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-09-02 07:52:28
SiteFR_CCIN2P3
EntryDUNE_FR_CCIN2P3_cccondorce03
Last heartbeat2025-09-02 09:10:30
From worker nodeHostnameccwcondor0051
cpuinfoAMD EPYC 9334 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit106200 (29 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Started2025-09-02 07:54:15
Input filesvd-protodune:np02vd_raw_run039275_0102_df-s05-d1_dw_0_20250901T182029.hdf5
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-09-02 09:10:30
Saved logsjustin-logs:32023.20-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

k=512 with 18294 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss7":5596 [0] "mp2_roi7":4882 [0] "mp3_roi7":1368 [0] "wiener7":6448 [6448] ] cmm:[ ] output: frame: ident=37075 time=72 tick=500 with 146219 traces.  frame tags:[ "framefanin" ] 16 tagged trace sets:[ "gauss0":2553 [0] "gauss1":4645 [0] "gauss2":5094 [0] "gauss3":5373 [0] "gauss4":6867 [0] "gauss5":4558 [0] "gauss6":7222 [0] "gauss7":5596 [0] "wiener0":3123 [3123] "wiener1":5819 [5819] "wiener2":6401 [6401] "wiener3":6951 [6951] "wiener4":8709 [8709] "wiener5":5381 [5381] "wiener6":9045 [9045] "wiener7":6448 [6448] ] cmm:[ ] 
Retagger: tagging trace set: wiener with 51877 traces, 51877 summary
Retagger: tagging trace set: gauss with 41908 traces, 0 summary
[11:09:09.923] D [  glue  ] sink frame: #37075 @7.2e-05 with 146219 traces, frame tags:[retagger], trace tags:[gauss, wiener]
[11:09:09.923] D [  glue  ] <FrameFanout:nfsp> call=35: see EOS 
[11:09:09.923] D [  glue  ] <ChannelSelector:chsel7> see EOS at call=17 
[11:09:09.923] D [sigproc ] <OmnibusSigProc:anode7sigproc7> EOS at call=35 anode=7 
[11:09:09.923] D [  glue  ] <ChannelSelector:chsel6> see EOS at call=17 
[11:09:09.923] D [sigproc ] <OmnibusSigProc:anode6sigproc6> EOS at call=35 anode=6 
[11:09:09.923] D [  glue  ] <ChannelSelector:chsel5> see EOS at call=17 
[11:09:09.923] D [sigproc ] <OmnibusSigProc:anode5sigproc5> EOS at call=35 anode=5 
[11:09:09.923] D [  glue  ] <ChannelSelector:chsel4> see EOS at call=17 
[11:09:09.923] D [sigproc ] <OmnibusSigProc:anode4sigproc4> EOS at call=35 anode=4 
[11:09:09.923] D [  glue  ] <ChannelSelector:chsel3> see EOS at call=17 
[11:09:09.923] D [  aux   ] <Resampler:resmp3> EOS at call=35 
[11:09:09.923] D [sigproc ] <OmnibusSigProc:anode3sigproc3> EOS at call=35 anode=3 
[11:09:09.923] D [  glue  ] <ChannelSelector:chsel2> see EOS at call=17 
[11:09:09.923] D [  aux   ] <Resampler:resmp2> EOS at call=35 
[11:09:09.923] D [sigproc ] <OmnibusSigProc:anode2sigproc2> EOS at call=35 anode=2 
[11:09:09.923] D [  glue  ] <ChannelSelector:chsel1> see EOS at call=17 
[11:09:09.924] D [  aux   ] <Resampler:resmp1> EOS at call=35 
[11:09:09.924] D [sigproc ] <OmnibusSigProc:anode1sigproc1> EOS at call=35 anode=1 
[11:09:09.924] D [  glue  ] <ChannelSelector:chsel0> see EOS at call=17 
[11:09:09.924] D [  aux   ] <Resampler:resmp0> EOS at call=35 
[11:09:09.924] D [sigproc ] <OmnibusSigProc:anode0sigproc0> EOS at call=35 anode=0 
[11:09:09.924] D [  glue  ] <FrameFanin:nfsp> EOS at call=35 with 8 
[11:09:09.924] D [  glue  ] frame sink sees EOS
[11:09:09.924] D [ pgraph ] <Pgrapher:> graph execution complete 
[11:09:09.924] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 8.93 sec
[11:09:09.924] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 8.33 sec
[11:09:09.924] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 8.29 sec
[11:09:09.924] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 8.21 sec
[11:09:09.924] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 7.76 sec
[11:09:09.924] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 7.68 sec
[11:09:09.924] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 7.36 sec
[11:09:09.924] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 6.81 sec
[11:09:09.924] I [ timer  ] Timer: WireCell::Aux::Resampler : 1.03 sec
[11:09:09.924] I [ timer  ] Timer: WireCell::Aux::Resampler : 1 sec
[11:09:09.924] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.95 sec
[11:09:09.924] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.74 sec
[11:09:09.924] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.03 sec
[11:09:09.924] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.02 sec
[11:09:09.924] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[11:09:09.924] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[11:09:09.924] I [ timer  ] Timer: WireCell::Gen::Retagger : 0.01 sec
[11:09:09.924] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[11:09:09.924] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[11:09:09.924] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[11:09:09.924] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[11:09:09.924] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[11:09:09.924] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[11:09:09.924] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[11:09:09.924] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[11:09:09.924] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[11:09:09.924] I [ timer  ] Timer: Total node execution : 67.17000032961369 sec
wclsFrameSaver saving cooked to 10000 ticks
wclsFrameSaver: saving 41908 traces tagged "gauss"
FrameSaver: q=6.18762e+06 n=829992 tag=gauss
wclsFrameSaver: saving 51877 traces tagged "wiener"
FrameSaver: q=6.57406e+06 n=793992 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 355 XUs and 485 XVs -> 11 XUVs
C:0 T:1 116 XUs and 118 XVs -> 9 XUVs
C:0 T:2 1023 XUs and 1525 XVs -> 68 XUVs
C:0 T:3 406 XUs and 551 XVs -> 20 XUVs
C:0 T:4 497 XUs and 781 XVs -> 40 XUVs
C:0 T:5 1365 XUs and 1679 XVs -> 76 XUVs
C:0 T:6 1175 XUs and 1077 XVs -> 60 XUVs
C:0 T:7 1383 XUs and 1559 XVs -> 73 XUVs
C:0 T:8 2847 XUs and 5153 XVs -> 535 XUVs
C:0 T:9 4388 XUs and 7014 XVs -> 664 XUVs
C:0 T:10 1388 XUs and 1567 XVs -> 101 XUVs
C:0 T:11 653 XUs and 834 XVs -> 50 XUVs
C:0 T:12 1358 XUs and 2049 XVs -> 156 XUVs
C:0 T:13 2382 XUs and 3692 XVs -> 507 XUVs
C:0 T:14 551 XUs and 534 XVs -> 36 XUVs
C:0 T:15 1539 XUs and 1943 XVs -> 90 XUVs
2496 XUVs total
1258 collection wire objects
2496 potential space points
Neighbour search...
76846 tests to find 42590 neighbours
Iterating with no regularization...
Begin: 3.38656e+08
0 3.21008e+08
1 3.20062e+08
2 3.19922e+08
Now with regularization...
Begin: 3.12281e+08
0 3.12224e+08
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
02-Sep-2025 11:10:04 CEST  Closed output file "np02vd_raw_run039275_0102_df-s05-d1_dw_0_20250901T182029_reco_stage1_20250902T091004_keepup.root"

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                        139.376       249.897       358.097       241.859       56.763         18     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                      9.1198e-05    0.000141573   0.000388659   0.000119856   6.98993e-05      18     
produce:tpcrawdecoder:PDVDTPCReader               8.14156       10.1072       14.1457       9.93532       1.38642        18     
produce:triggerrawdecoder:PDVDTriggerReader4    0.00580838     0.0115032     0.0253055     0.0089369    0.00654157       18     
produce:pdvddaphne:DAPHNEReaderPDVD               4.21669       4.83234       5.69719       4.8498       0.394935        18     
produce:ophit:OpHitFinder                        0.0527131      0.06494      0.0761347     0.063594     0.00615803       18     
produce:opflash:OpFlashFinderVerticalDrift       0.0115095     0.0171776     0.0253586     0.0167134    0.00371797       18     
produce:wclsdatavd:WireCellToolkit                69.2447       92.8682       112.661       92.367        11.5034        18     
produce:gaushit:GausHitFinder                     1.2385        2.03956       2.49614       2.03522      0.344983        18     
produce:nhitsfilter:NumberOfHitsFilter          0.000286895   0.000700463   0.00215926    0.000551527   0.000451327      18     
produce:reco3d:SpacePointSolver                   10.9313       22.4182       33.4256       21.673        5.4366         18     
produce:hitpdune:DisambigFromSpacePoints         0.217649      0.406725      0.705719      0.402031       0.12118        18     
produce:pandora:StandardPandora                   33.7774       107.654       182.401       94.0913       45.2676        18     
produce:pandoraTrack:LArPandoraTrackCreation      0.84295       2.51288       4.17616       2.54807      0.856641        18     
produce:pandoraGnocalo:GnocchiCalorimetry        0.0313442     0.0497637     0.0657891     0.0496675    0.00759555       18     
[art]:TriggerResults:TriggerResultInserter       2.678e-05    4.87243e-05   0.000142216   4.30405e-05   2.45949e-05      18     
end_path:out1:RootOutput                         6.14e-06     1.39167e-05   5.0717e-05    1.13425e-05   9.66301e-06      18     
end_path:out1:RootOutput(write)                   6.24188       6.80087       7.48927       6.70954       0.38368        18     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5304.47 MB
  Peak resident set size usage (VmHWM): 3387.32 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run039275_0102_df-s05-d1_dw_0_20250901T182029_reco_stage1_20250902T091004_keepup.root
\tHists: np02vd_raw_run039275_0102_df-s05-d1_dw_0_20250901T182029_reco_stage1_20250902T091004_keepup_hists.root
Forming reco metadata
/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/bin/python: No module named meta_maker
Error in reco metadata
justIN time: 2025-09-19 04:25:12 UTC       justIN version: 01.05.00