justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 40871.22@dunegpschedd02.fnal.gov

Jobsub ID40871.22@dunegpschedd02.fnal.gov
Workflow ID2741
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-09-17 08:51:02
SiteNL_NIKHEF
EntryVIRGO_NL_NIKHEF_dissel
Last heartbeat2025-09-17 10:26:39
From worker nodeHostnamewn-sate-051.farm.nikhef.nl
cpuinfoAMD EPYC 7551P 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Started2025-09-17 09:20:49
Input filesvd-protodune:np02vd_raw_run039388_0753_df-s05-d0_dw_0_20250917T050258.hdf5
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-09-17 10:26:39
Saved logsjustin-logs:40871.22-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

time=72 tick=512 with 24265 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss7":7327 [0] "mp2_roi7":6561 [0] "mp3_roi7":1886 [0] "wiener7":8491 [8491] ] cmm:[ ] output: frame: ident=58076 time=72 tick=500 with 182029 traces.  frame tags:[ "framefanin" ] 16 tagged trace sets:[ "gauss0":6952 [0] "gauss1":4564 [0] "gauss2":3852 [0] "gauss3":9914 [0] "gauss4":9195 [0] "gauss5":6804 [0] "gauss6":4268 [0] "gauss7":7327 [0] "wiener0":8705 [8705] "wiener1":5707 [5707] "wiener2":4878 [4878] "wiener3":12288 [12288] "wiener4":10966 [10966] "wiener5":8049 [8049] "wiener6":4956 [4956] "wiener7":8491 [8491] ] cmm:[ ] 
Retagger: tagging trace set: wiener with 64040 traces, 64040 summary
Retagger: tagging trace set: gauss with 52876 traces, 0 summary
[12:24:45.838] D [  glue  ] sink frame: #58076 @7.2e-05 with 182029 traces, frame tags:[retagger], trace tags:[gauss, wiener]
[12:24:45.838] D [  glue  ] <FrameFanout:nfsp> call=37: see EOS 
[12:24:45.838] D [  glue  ] <ChannelSelector:chsel7> see EOS at call=18 
[12:24:45.839] D [sigproc ] <OmnibusSigProc:anode7sigproc7> EOS at call=37 anode=7 
[12:24:45.839] D [  glue  ] <ChannelSelector:chsel6> see EOS at call=18 
[12:24:45.839] D [sigproc ] <OmnibusSigProc:anode6sigproc6> EOS at call=37 anode=6 
[12:24:45.839] D [  glue  ] <ChannelSelector:chsel5> see EOS at call=18 
[12:24:45.839] D [sigproc ] <OmnibusSigProc:anode5sigproc5> EOS at call=37 anode=5 
[12:24:45.839] D [  glue  ] <ChannelSelector:chsel4> see EOS at call=18 
[12:24:45.839] D [sigproc ] <OmnibusSigProc:anode4sigproc4> EOS at call=37 anode=4 
[12:24:45.839] D [  glue  ] <ChannelSelector:chsel3> see EOS at call=18 
[12:24:45.839] D [  aux   ] <Resampler:resmp3> EOS at call=37 
[12:24:45.839] D [sigproc ] <OmnibusSigProc:anode3sigproc3> EOS at call=37 anode=3 
[12:24:45.839] D [  glue  ] <ChannelSelector:chsel2> see EOS at call=18 
[12:24:45.839] D [  aux   ] <Resampler:resmp2> EOS at call=37 
[12:24:45.839] D [sigproc ] <OmnibusSigProc:anode2sigproc2> EOS at call=37 anode=2 
[12:24:45.839] D [  glue  ] <ChannelSelector:chsel1> see EOS at call=18 
[12:24:45.839] D [  aux   ] <Resampler:resmp1> EOS at call=37 
[12:24:45.839] D [sigproc ] <OmnibusSigProc:anode1sigproc1> EOS at call=37 anode=1 
[12:24:45.839] D [  glue  ] <ChannelSelector:chsel0> see EOS at call=18 
[12:24:45.839] D [  aux   ] <Resampler:resmp0> EOS at call=37 
[12:24:45.839] D [sigproc ] <OmnibusSigProc:anode0sigproc0> EOS at call=37 anode=0 
[12:24:45.839] D [  glue  ] <FrameFanin:nfsp> EOS at call=37 with 8 
[12:24:45.839] D [  glue  ] frame sink sees EOS
[12:24:45.839] D [ pgraph ] <Pgrapher:> graph execution complete 
[12:24:45.839] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 9.72 sec
[12:24:45.839] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 9.72 sec
[12:24:45.839] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 9.28 sec
[12:24:45.839] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 8.99 sec
[12:24:45.839] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 8.83 sec
[12:24:45.839] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 8.4 sec
[12:24:45.839] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 8.35 sec
[12:24:45.839] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 7.94 sec
[12:24:45.839] I [ timer  ] Timer: WireCell::Aux::Resampler : 1.04 sec
[12:24:45.839] I [ timer  ] Timer: WireCell::Aux::Resampler : 1.03 sec
[12:24:45.839] I [ timer  ] Timer: WireCell::Aux::Resampler : 1.02 sec
[12:24:45.839] I [ timer  ] Timer: WireCell::Aux::Resampler : 1.01 sec
[12:24:45.839] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.02 sec
[12:24:45.839] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[12:24:45.839] I [ timer  ] Timer: WireCell::Gen::Retagger : 0.01 sec
[12:24:45.839] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0.01 sec
[12:24:45.839] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[12:24:45.839] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[12:24:45.839] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[12:24:45.839] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[12:24:45.839] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[12:24:45.839] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[12:24:45.839] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[12:24:45.839] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[12:24:45.839] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[12:24:45.839] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[12:24:45.839] I [ timer  ] Timer: Total node execution : 75.37999992258847 sec
wclsFrameSaver saving cooked to 10000 ticks
wclsFrameSaver: saving 52876 traces tagged "gauss"
FrameSaver: q=7.78375e+06 n=1043124 tag=gauss
wclsFrameSaver: saving 64040 traces tagged "wiener"
FrameSaver: q=8.12333e+06 n=999777 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 1256 XUs and 1368 XVs -> 53 XUVs
C:0 T:1 1069 XUs and 1573 XVs -> 49 XUVs
C:0 T:2 676 XUs and 492 XVs -> 25 XUVs
C:0 T:3 714 XUs and 1059 XVs -> 20 XUVs
C:0 T:4 618 XUs and 775 XVs -> 36 XUVs
C:0 T:5 481 XUs and 717 XVs -> 10 XUVs
C:0 T:6 2442 XUs and 1812 XVs -> 210 XUVs
C:0 T:7 3991 XUs and 3877 XVs -> 128 XUVs
C:0 T:8 2418 XUs and 2610 XVs -> 159 XUVs
C:0 T:9 3268 XUs and 3917 XVs -> 120 XUVs
C:0 T:10 923 XUs and 1698 XVs -> 74 XUVs
C:0 T:11 1855 XUs and 1576 XVs -> 94 XUVs
C:0 T:12 488 XUs and 534 XVs -> 37 XUVs
C:0 T:13 816 XUs and 1235 XVs -> 57 XUVs
C:0 T:14 2259 XUs and 1739 XVs -> 103 XUVs
C:0 T:15 1292 XUs and 1310 XVs -> 92 XUVs
1267 XUVs total
1037 collection wire objects
1267 potential space points
Neighbour search...
18581 tests to find 10684 neighbours
Iterating with no regularization...
Begin: 2.63006e+08
0 2.58196e+08
1 2.58124e+08
Now with regularization...
Begin: 2.54018e+08
0 2.54014e+08
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
17-Sep-2025 12:26:21 CEST  Closed output file "np02vd_raw_run039388_0753_df-s05-d0_dw_0_20250917T050258_reco_stage1_20250917T102621_keepup.root"

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                        146.523       204.658       235.395       206.463       21.6946        19     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                      6.6875e-05    8.04251e-05   0.000186119   7.1574e-05    2.62038e-05      19     
produce:tpcrawdecoder:PDVDTPCReader               14.6756       17.4098       22.1017       17.6846       2.14924        19     
produce:triggerrawdecoder:PDVDTriggerReader4     0.291785       0.29896      0.319574       0.29447     0.00833749       19     
produce:pdvddaphne:DAPHNEReaderPDVD             0.000320872   0.000358476   0.000729849   0.000340759   8.85247e-05      19     
produce:ophit:OpHitFinder                       0.000122379   0.000174822   0.000783039   0.00013365    0.000144652      19     
produce:opflash:OpFlashFinderVerticalDrift      5.2909e-05    7.06347e-05   0.000336481   5.5354e-05    6.26945e-05      19     
produce:wclsdatavd:WireCellToolkit                70.5582       81.3783       94.3015       79.337        6.34582        19     
produce:gaushit:GausHitFinder                     1.06391       1.56402       1.79942       1.61789      0.170265        19     
produce:nhitsfilter:NumberOfHitsFilter          0.000311534   0.00039949    0.000632306   0.000380414   8.12297e-05      19     
produce:reco3d:SpacePointSolver                   12.4665       18.5529       22.8826       18.2424       2.40563        19     
produce:hitpdune:DisambigFromSpacePoints         0.164718      0.249202      0.365966      0.242502      0.0432723       19     
produce:pandora:StandardPandora                   33.1971       77.6063       108.641       76.7865       17.0196        19     
produce:pandoraTrack:LArPandoraTrackCreation     0.915357       1.6336        2.0438        1.70867      0.316438        19     
produce:pandoraGnocalo:GnocchiCalorimetry        0.0294317     0.0393858     0.0536542     0.0377672    0.00638107       19     
[art]:TriggerResults:TriggerResultInserter      1.8354e-05     3.727e-05    0.000135945   3.0488e-05    2.56238e-05      19     
end_path:out1:RootOutput                         6.753e-06    9.99811e-06   2.8263e-05     7.705e-06    5.14384e-06      19     
end_path:out1:RootOutput(write)                   5.48376       5.88038       7.51243       5.74435       0.41417        19     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4797.48 MB
  Peak resident set size usage (VmHWM): 2915 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run039388_0753_df-s05-d0_dw_0_20250917T050258_reco_stage1_20250917T102621_keepup.root
\tHists: np02vd_raw_run039388_0753_df-s05-d0_dw_0_20250917T050258_reco_stage1_20250917T102621_keepup_hists.root
Forming reco metadata
/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/bin/python: No module named meta_maker
Error in reco metadata
justIN time: 2025-09-18 19:29:19 UTC       justIN version: 01.05.00