justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 32336.87@dunegpschedd02.fnal.gov

Jobsub ID32336.87@dunegpschedd02.fnal.gov
Workflow ID2178
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod.mcsim
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-09-02 17:24:57
SiteFR_CCIN2P3
EntryDUNE_FR_CCIN2P3_cccondorce01
Last heartbeat2025-09-02 18:46:59
From worker nodeHostnameccwcondor0564
cpuinfoAMD EPYC 7453 28-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit106200 (29 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Started2025-09-02 17:45:42
Input filesvd-protodune:np02vd_raw_run039275_0263_df-s05-d4_dw_0_20250901T234242.hdf5
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-09-02 18:46:59
Saved logsjustin-logs:32336.87-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

058 time=144 tick=512 with 34748 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss7":9742 [0] "mp2_roi7":10847 [0] "mp3_roi7":2568 [0] "wiener7":11591 [11591] ] cmm:[ ] output: frame: ident=95058 time=144 tick=500 with 195356 traces.  frame tags:[ "framefanin" ] 16 tagged trace sets:[ "gauss0":2698 [0] "gauss1":4719 [0] "gauss2":5882 [0] "gauss3":2877 [0] "gauss4":11339 [0] "gauss5":9456 [0] "gauss6":7282 [0] "gauss7":9742 [0] "wiener0":3536 [3536] "wiener1":5689 [5689] "wiener2":7275 [7275] "wiener3":3504 [3504] "wiener4":14915 [14915] "wiener5":11502 [11502] "wiener6":9046 [9046] "wiener7":11591 [11591] ] cmm:[ ] 
Retagger: tagging trace set: wiener with 67058 traces, 67058 summary
Retagger: tagging trace set: gauss with 53995 traces, 0 summary
[20:44:26.454] D [  glue  ] sink frame: #95058 @0.000144 with 195356 traces, frame tags:[retagger], trace tags:[gauss, wiener]
[20:44:26.454] D [  glue  ] <FrameFanout:nfsp> call=35: see EOS 
[20:44:26.454] D [  glue  ] <ChannelSelector:chsel7> see EOS at call=17 
[20:44:26.454] D [sigproc ] <OmnibusSigProc:anode7sigproc7> EOS at call=35 anode=7 
[20:44:26.454] D [  glue  ] <ChannelSelector:chsel6> see EOS at call=17 
[20:44:26.454] D [sigproc ] <OmnibusSigProc:anode6sigproc6> EOS at call=35 anode=6 
[20:44:26.454] D [  glue  ] <ChannelSelector:chsel5> see EOS at call=17 
[20:44:26.454] D [sigproc ] <OmnibusSigProc:anode5sigproc5> EOS at call=35 anode=5 
[20:44:26.454] D [  glue  ] <ChannelSelector:chsel4> see EOS at call=17 
[20:44:26.455] D [sigproc ] <OmnibusSigProc:anode4sigproc4> EOS at call=35 anode=4 
[20:44:26.455] D [  glue  ] <ChannelSelector:chsel3> see EOS at call=17 
[20:44:26.455] D [  aux   ] <Resampler:resmp3> EOS at call=35 
[20:44:26.455] D [sigproc ] <OmnibusSigProc:anode3sigproc3> EOS at call=35 anode=3 
[20:44:26.455] D [  glue  ] <ChannelSelector:chsel2> see EOS at call=17 
[20:44:26.455] D [  aux   ] <Resampler:resmp2> EOS at call=35 
[20:44:26.455] D [sigproc ] <OmnibusSigProc:anode2sigproc2> EOS at call=35 anode=2 
[20:44:26.455] D [  glue  ] <ChannelSelector:chsel1> see EOS at call=17 
[20:44:26.455] D [  aux   ] <Resampler:resmp1> EOS at call=35 
[20:44:26.455] D [sigproc ] <OmnibusSigProc:anode1sigproc1> EOS at call=35 anode=1 
[20:44:26.455] D [  glue  ] <ChannelSelector:chsel0> see EOS at call=17 
[20:44:26.455] D [  aux   ] <Resampler:resmp0> EOS at call=35 
[20:44:26.455] D [sigproc ] <OmnibusSigProc:anode0sigproc0> EOS at call=35 anode=0 
[20:44:26.455] D [  glue  ] <FrameFanin:nfsp> EOS at call=35 with 8 
[20:44:26.455] D [  glue  ] frame sink sees EOS
[20:44:26.455] D [ pgraph ] <Pgrapher:> graph execution complete 
[20:44:26.455] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 11.49 sec
[20:44:26.455] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 9.42 sec
[20:44:26.455] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 8.19 sec
[20:44:26.455] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 7.27 sec
[20:44:26.455] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 6.79 sec
[20:44:26.455] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 6.58 sec
[20:44:26.455] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 5.79 sec
[20:44:26.455] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 5.36 sec
[20:44:26.455] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.71 sec
[20:44:26.455] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.68 sec
[20:44:26.455] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.63 sec
[20:44:26.455] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.6 sec
[20:44:26.455] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.02 sec
[20:44:26.455] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.02 sec
[20:44:26.455] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[20:44:26.455] I [ timer  ] Timer: WireCell::Gen::Retagger : 0.01 sec
[20:44:26.455] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[20:44:26.455] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[20:44:26.455] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[20:44:26.455] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[20:44:26.455] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[20:44:26.455] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[20:44:26.455] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[20:44:26.455] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[20:44:26.455] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[20:44:26.455] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[20:44:26.455] I [ timer  ] Timer: Total node execution : 63.569999393075705 sec
wclsFrameSaver saving cooked to 10000 ticks
wclsFrameSaver: saving 53995 traces tagged "gauss"
FrameSaver: q=9.59271e+06 n=1172700 tag=gauss
wclsFrameSaver: saving 67058 traces tagged "wiener"
FrameSaver: q=1.03968e+07 n=1122978 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 191 XUs and 66 XVs -> 3 XUVs
C:0 T:1 1214 XUs and 1291 XVs -> 72 XUVs
C:0 T:2 341 XUs and 247 XVs -> 14 XUVs
C:0 T:3 1065 XUs and 915 XVs -> 26 XUVs
C:0 T:4 828 XUs and 1151 XVs -> 36 XUVs
C:0 T:5 860 XUs and 940 XVs -> 40 XUVs
C:0 T:6 294 XUs and 354 XVs -> 17 XUVs
C:0 T:7 317 XUs and 613 XVs -> 16 XUVs
C:0 T:8 21844 XUs and 45421 XVs -> 7729 XUVs
C:0 T:9 18381 XUs and 42422 XVs -> 7270 XUVs
C:0 T:10 16694 XUs and 33943 XVs -> 5951 XUVs
C:0 T:11 1722 XUs and 1300 XVs -> 63 XUVs
C:0 T:12 1635 XUs and 1924 XVs -> 154 XUVs
C:0 T:13 2602 XUs and 4039 XVs -> 322 XUVs
C:0 T:14 1623 XUs and 1654 XVs -> 127 XUVs
C:0 T:15 3760 XUs and 4010 XVs -> 185 XUVs
22025 XUVs total
2168 collection wire objects
22025 potential space points
Neighbour search...
3765667 tests to find 1505810 neighbours
Iterating with no regularization...
Begin: 4.87868e+08
0 4.33044e+08
1 4.29468e+08
2 4.28795e+08
3 4.28559e+08
Now with regularization...
Begin: 4.19275e+08
0 4.18993e+08
PcaShowerParticleBuildingAlgorithm::OpeningAngle - principal eigenvalue less than or equal to 0.
02-Sep-2025 20:46:35 CEST  Closed output file "np02vd_raw_run039275_0263_df-s05-d4_dw_0_20250901T234242_reco_stage1_20250902T184635_keepup.root"

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                        143.26        199.038       268.047       199.368       34.7673        18     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                      7.2034e-05    0.000132241   0.00026235    0.000130665   5.15882e-05      18     
produce:tpcrawdecoder:PDVDTPCReader               7.88626       9.03951       22.1077       8.30484       3.17613        18     
produce:triggerrawdecoder:PDVDTriggerReader4    0.00601321    0.00662992     0.0104546    0.00640009    0.000982318      18     
produce:pdvddaphne:DAPHNEReaderPDVD               3.93648       4.89877       5.76071       5.01593      0.458545        18     
produce:ophit:OpHitFinder                        0.0422477     0.0668399     0.0849633     0.0683698     0.0101291       18     
produce:opflash:OpFlashFinderVerticalDrift      0.00752381     0.0166155     0.0291624     0.0162319    0.00459145       18     
produce:wclsdatavd:WireCellToolkit                58.9769       77.4485       92.3749       78.5504       8.57025        18     
produce:gaushit:GausHitFinder                    0.816825       1.6307        2.23445       1.61101      0.452349        18     
produce:nhitsfilter:NumberOfHitsFilter          0.000240209   0.000384829   0.00069486    0.000379019   9.47086e-05      18     
produce:reco3d:SpacePointSolver                   8.24336       17.3762       23.2541       17.2401       4.21403        18     
produce:hitpdune:DisambigFromSpacePoints         0.150059      0.311819      0.501861      0.304158      0.101462        18     
produce:pandora:StandardPandora                   28.2857       80.1059       140.754       77.0466       28.5239        18     
produce:pandoraTrack:LArPandoraTrackCreation     0.970358       1.67925       2.84794       1.61571      0.490884        18     
produce:pandoraGnocalo:GnocchiCalorimetry        0.0192147     0.0429088     0.0566786     0.0415967     0.0106578       18     
[art]:TriggerResults:TriggerResultInserter      2.3594e-05    4.68084e-05   0.000153447   4.3887e-05    2.86289e-05      18     
end_path:out1:RootOutput                         3.907e-06    1.09286e-05   4.0926e-05    9.7885e-06    8.25471e-06      18     
end_path:out1:RootOutput(write)                   4.58194       6.3481        7.62288       6.77472      0.985644        18     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5090.43 MB
  Peak resident set size usage (VmHWM): 3159.49 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run039275_0263_df-s05-d4_dw_0_20250901T234242_reco_stage1_20250902T184635_keepup.root
\tHists: np02vd_raw_run039275_0263_df-s05-d4_dw_0_20250901T234242_reco_stage1_20250902T184636_keepup_hists.root
Forming reco metadata
/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/bin/python: No module named meta_maker
Error in reco metadata
justIN time: 2025-09-19 04:25:50 UTC       justIN version: 01.05.00