justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 38892.99@dunegpschedd01.fnal.gov

Jobsub ID38892.99@dunegpschedd01.fnal.gov
Workflow ID2131
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod.mcsim
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-09-02 11:16:38
SiteFR_CCIN2P3
EntryDUNE_FR_CCIN2P3_cccondorce01
Last heartbeat2025-09-02 12:45:11
From worker nodeHostnameccwcondor0564
cpuinfoAMD EPYC 7453 28-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit106200 (29 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Started2025-09-02 11:21:31
Input filesvd-protodune:np02vd_raw_run039275_0112_df-s04-d2_dw_0_20250901T183226.hdf5
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-09-02 12:45:11
Saved logsjustin-logs:38892.99-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

s.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss7":7780 [0] "mp2_roi7":8470 [0] "mp3_roi7":1999 [0] "wiener7":9386 [9386] ] cmm:[ ] output: frame: ident=40672 time=73 tick=500 with 221480 traces.  frame tags:[ "framefanin" ] 16 tagged trace sets:[ "gauss0":8302 [0] "gauss1":3716 [0] "gauss2":6786 [0] "gauss3":7809 [0] "gauss4":11923 [0] "gauss5":4241 [0] "gauss6":10567 [0] "gauss7":7780 [0] "wiener0":10450 [10450] "wiener1":4565 [4565] "wiener2":8424 [8424] "wiener3":9591 [9591] "wiener4":14444 [14444] "wiener5":4868 [4868] "wiener6":12547 [12547] "wiener7":9386 [9386] ] cmm:[ ] 
Retagger: tagging trace set: wiener with 74275 traces, 74275 summary
Retagger: tagging trace set: gauss with 61124 traces, 0 summary
[14:40:52.390] D [  glue  ] sink frame: #40672 @7.3e-05 with 221480 traces, frame tags:[retagger], trace tags:[gauss, wiener]
[14:40:52.390] D [  glue  ] <FrameFanout:nfsp> call=35: see EOS 
[14:40:52.390] D [  glue  ] <ChannelSelector:chsel7> see EOS at call=17 
[14:40:52.390] D [sigproc ] <OmnibusSigProc:anode7sigproc7> EOS at call=35 anode=7 
[14:40:52.390] D [  glue  ] <ChannelSelector:chsel6> see EOS at call=17 
[14:40:52.390] D [sigproc ] <OmnibusSigProc:anode6sigproc6> EOS at call=35 anode=6 
[14:40:52.390] D [  glue  ] <ChannelSelector:chsel5> see EOS at call=17 
[14:40:52.390] D [sigproc ] <OmnibusSigProc:anode5sigproc5> EOS at call=35 anode=5 
[14:40:52.390] D [  glue  ] <ChannelSelector:chsel4> see EOS at call=17 
[14:40:52.390] D [sigproc ] <OmnibusSigProc:anode4sigproc4> EOS at call=35 anode=4 
[14:40:52.390] D [  glue  ] <ChannelSelector:chsel3> see EOS at call=17 
[14:40:52.390] D [  aux   ] <Resampler:resmp3> EOS at call=35 
[14:40:52.390] D [sigproc ] <OmnibusSigProc:anode3sigproc3> EOS at call=35 anode=3 
[14:40:52.390] D [  glue  ] <ChannelSelector:chsel2> see EOS at call=17 
[14:40:52.390] D [  aux   ] <Resampler:resmp2> EOS at call=35 
[14:40:52.390] D [sigproc ] <OmnibusSigProc:anode2sigproc2> EOS at call=35 anode=2 
[14:40:52.390] D [  glue  ] <ChannelSelector:chsel1> see EOS at call=17 
[14:40:52.390] D [  aux   ] <Resampler:resmp1> EOS at call=35 
[14:40:52.390] D [sigproc ] <OmnibusSigProc:anode1sigproc1> EOS at call=35 anode=1 
[14:40:52.390] D [  glue  ] <ChannelSelector:chsel0> see EOS at call=17 
[14:40:52.390] D [  aux   ] <Resampler:resmp0> EOS at call=35 
[14:40:52.391] D [sigproc ] <OmnibusSigProc:anode0sigproc0> EOS at call=35 anode=0 
[14:40:52.391] D [  glue  ] <FrameFanin:nfsp> EOS at call=35 with 8 
[14:40:52.391] D [  glue  ] frame sink sees EOS
[14:40:52.391] D [ pgraph ] <Pgrapher:> graph execution complete 
[14:40:52.391] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 12.68 sec
[14:40:52.391] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 11.8 sec
[14:40:52.391] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 11.33 sec
[14:40:52.391] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 11.21 sec
[14:40:52.391] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 10.46 sec
[14:40:52.391] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 10.4 sec
[14:40:52.391] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 10 sec
[14:40:52.391] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 9.97 sec
[14:40:52.391] I [ timer  ] Timer: WireCell::Aux::Resampler : 1.11 sec
[14:40:52.391] I [ timer  ] Timer: WireCell::Aux::Resampler : 1.09 sec
[14:40:52.391] I [ timer  ] Timer: WireCell::Aux::Resampler : 1.07 sec
[14:40:52.391] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.94 sec
[14:40:52.391] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.05 sec
[14:40:52.391] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[14:40:52.391] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[14:40:52.391] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[14:40:52.391] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[14:40:52.391] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[14:40:52.391] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[14:40:52.391] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[14:40:52.391] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[14:40:52.391] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[14:40:52.391] I [ timer  ] Timer: WireCell::Gen::Retagger : 0 sec
[14:40:52.391] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[14:40:52.391] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[14:40:52.391] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[14:40:52.391] I [ timer  ] Timer: Total node execution : 92.16000047884881 sec
wclsFrameSaver saving cooked to 10000 ticks
wclsFrameSaver: saving 61124 traces tagged "gauss"
FrameSaver: q=9.51111e+06 n=1323717 tag=gauss
wclsFrameSaver: saving 74275 traces tagged "wiener"
FrameSaver: q=9.93147e+06 n=1285171 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 2971 XUs and 2618 XVs -> 99 XUVs
C:0 T:1 657 XUs and 1001 XVs -> 29 XUVs
C:0 T:2 909 XUs and 763 XVs -> 38 XUVs
C:0 T:3 181 XUs and 290 XVs -> 2 XUVs
C:0 T:4 871 XUs and 1149 XVs -> 21 XUVs
C:0 T:5 2505 XUs and 1542 XVs -> 164 XUVs
C:0 T:6 923 XUs and 1046 XVs -> 38 XUVs
C:0 T:7 3268 XUs and 3206 XVs -> 276 XUVs
C:0 T:8 3726 XUs and 4323 XVs -> 739 XUVs
C:0 T:9 7269 XUs and 6688 XVs -> 228 XUVs
C:0 T:10 439 XUs and 420 XVs -> 33 XUVs
C:0 T:11 498 XUs and 589 XVs -> 34 XUVs
C:0 T:12 2828 XUs and 3012 XVs -> 161 XUVs
C:0 T:13 2297 XUs and 2811 XVs -> 138 XUVs
C:0 T:14 1988 XUs and 2315 XVs -> 134 XUVs
C:0 T:15 2510 XUs and 2464 XVs -> 186 XUVs
2320 XUVs total
1530 collection wire objects
2320 potential space points
Neighbour search...
62226 tests to find 34650 neighbours
Iterating with no regularization...
Begin: 3.83124e+08
0 3.70731e+08
1 3.70285e+08
2 3.70201e+08
Now with regularization...
Begin: 3.64406e+08
0 3.64373e+08
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
02-Sep-2025 14:44:28 CEST  Closed output file "np02vd_raw_run039275_0112_df-s04-d2_dw_0_20250901T183226_reco_stage1_20250902T124428_keepup.root"

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                        186.096       272.961       457.075       232.403       81.3949        18     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                      8.5951e-05    0.000130523   0.000247162   0.000122298   3.58128e-05      18     
produce:tpcrawdecoder:PDVDTPCReader               8.40358       9.20491       9.88564       9.23837      0.437573        18     
produce:triggerrawdecoder:PDVDTriggerReader4    0.00788481     0.0119971     0.0252163     0.0110395    0.00441098       18     
produce:pdvddaphne:DAPHNEReaderPDVD               5.74202       6.21561       6.98253       6.11637      0.388183        18     
produce:ophit:OpHitFinder                         0.05372      0.0786321     0.0911926     0.0788552    0.00855088       18     
produce:opflash:OpFlashFinderVerticalDrift       0.0120644     0.0203313     0.0306781     0.0200659    0.00496567       18     
produce:wclsdatavd:WireCellToolkit                83.5421       95.5105       114.451       97.1973       8.57912        18     
produce:gaushit:GausHitFinder                     1.38247       2.05871       2.79998       1.90858      0.402462        18     
produce:nhitsfilter:NumberOfHitsFilter          0.000450402   0.00122679     0.0023215    0.00120009    0.000547315      18     
produce:reco3d:SpacePointSolver                   13.3065       22.311        36.3368       21.0389       6.49333        18     
produce:hitpdune:DisambigFromSpacePoints         0.228279      0.439683       1.03135      0.377065      0.179089        18     
produce:pandora:StandardPandora                   58.7361       126.561       272.952       91.2673       68.2225        18     
produce:pandoraTrack:LArPandoraTrackCreation      1.90967       3.46548       6.80658       3.18092       1.23182        18     
produce:pandoraGnocalo:GnocchiCalorimetry        0.0386539     0.0570959     0.0777517     0.055498      0.012294        18     
[art]:TriggerResults:TriggerResultInserter      4.2599e-05    7.76964e-05   0.00024012    6.71405e-05   4.23625e-05      18     
end_path:out1:RootOutput                         8.245e-06    2.35128e-05   6.2878e-05    2.2502e-05    1.08968e-05      18     
end_path:out1:RootOutput(write)                   6.5082        6.91745       7.33796       6.86544      0.236887        18     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5081.96 MB
  Peak resident set size usage (VmHWM): 3164.78 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run039275_0112_df-s04-d2_dw_0_20250901T183226_reco_stage1_20250902T124428_keepup.root
\tHists: np02vd_raw_run039275_0112_df-s04-d2_dw_0_20250901T183226_reco_stage1_20250902T124428_keepup_hists.root
Forming reco metadata
/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/bin/python: No module named meta_maker
Error in reco metadata
justIN time: 2025-09-19 19:07:40 UTC       justIN version: 01.05.00