justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 32112.3@dunegpschedd02.fnal.gov

Jobsub ID32112.3@dunegpschedd02.fnal.gov
Workflow ID2128
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod.mcsim
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-09-02 10:30:35
SiteFR_CCIN2P3
EntryDUNE_FR_CCIN2P3_cccondorce03
Last heartbeat2025-09-02 11:47:51
From worker nodeHostnameccwcondor0051
cpuinfoAMD EPYC 9334 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit106200 (29 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Started2025-09-02 10:31:58
Input filesvd-protodune:np02vd_raw_run039273_1518_df-s04-d3_dw_0_20250901T064020.hdf5
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-09-02 11:47:51
Saved logsjustin-logs:32112.3-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

mm:[ ] input 7: frame: ident=546833 time=40 tick=512 with 23042 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss7":6413 [0] "mp2_roi7":6238 [0] "mp3_roi7":2073 [0] "wiener7":8318 [8318] ] cmm:[ ] output: frame: ident=546833 time=40 tick=500 with 159241 traces.  frame tags:[ "framefanin" ] 16 tagged trace sets:[ "gauss0":5061 [0] "gauss1":2044 [0] "gauss2":3011 [0] "gauss3":1271 [0] "gauss4":8190 [0] "gauss5":5251 [0] "gauss6":12736 [0] "gauss7":6413 [0] "wiener0":6092 [6092] "wiener1":2538 [2538] "wiener2":3848 [3848] "wiener3":1597 [1597] "wiener4":10197 [10197] "wiener5":6415 [6415] "wiener6":15944 [15944] "wiener7":8318 [8318] ] cmm:[ ] 
Retagger: tagging trace set: wiener with 54949 traces, 54949 summary
Retagger: tagging trace set: gauss with 43977 traces, 0 summary
[13:45:50.028] D [  glue  ] sink frame: #546833 @4e-05 with 159241 traces, frame tags:[retagger], trace tags:[gauss, wiener]
[13:45:50.028] D [  glue  ] <FrameFanout:nfsp> call=35: see EOS 
[13:45:50.028] D [  glue  ] <ChannelSelector:chsel7> see EOS at call=17 
[13:45:50.028] D [sigproc ] <OmnibusSigProc:anode7sigproc7> EOS at call=35 anode=7 
[13:45:50.028] D [  glue  ] <ChannelSelector:chsel6> see EOS at call=17 
[13:45:50.028] D [sigproc ] <OmnibusSigProc:anode6sigproc6> EOS at call=35 anode=6 
[13:45:50.028] D [  glue  ] <ChannelSelector:chsel5> see EOS at call=17 
[13:45:50.028] D [sigproc ] <OmnibusSigProc:anode5sigproc5> EOS at call=35 anode=5 
[13:45:50.028] D [  glue  ] <ChannelSelector:chsel4> see EOS at call=17 
[13:45:50.028] D [sigproc ] <OmnibusSigProc:anode4sigproc4> EOS at call=35 anode=4 
[13:45:50.028] D [  glue  ] <ChannelSelector:chsel3> see EOS at call=17 
[13:45:50.028] D [  aux   ] <Resampler:resmp3> EOS at call=35 
[13:45:50.028] D [sigproc ] <OmnibusSigProc:anode3sigproc3> EOS at call=35 anode=3 
[13:45:50.028] D [  glue  ] <ChannelSelector:chsel2> see EOS at call=17 
[13:45:50.028] D [  aux   ] <Resampler:resmp2> EOS at call=35 
[13:45:50.028] D [sigproc ] <OmnibusSigProc:anode2sigproc2> EOS at call=35 anode=2 
[13:45:50.028] D [  glue  ] <ChannelSelector:chsel1> see EOS at call=17 
[13:45:50.028] D [  aux   ] <Resampler:resmp1> EOS at call=35 
[13:45:50.028] D [sigproc ] <OmnibusSigProc:anode1sigproc1> EOS at call=35 anode=1 
[13:45:50.028] D [  glue  ] <ChannelSelector:chsel0> see EOS at call=17 
[13:45:50.028] D [  aux   ] <Resampler:resmp0> EOS at call=35 
[13:45:50.028] D [sigproc ] <OmnibusSigProc:anode0sigproc0> EOS at call=35 anode=0 
[13:45:50.029] D [  glue  ] <FrameFanin:nfsp> EOS at call=35 with 8 
[13:45:50.029] D [  glue  ] frame sink sees EOS
[13:45:50.029] D [ pgraph ] <Pgrapher:> graph execution complete 
[13:45:50.029] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 12.05 sec
[13:45:50.029] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 8.52 sec
[13:45:50.029] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 8.42 sec
[13:45:50.029] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 8.18 sec
[13:45:50.029] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 8.05 sec
[13:45:50.029] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 7.25 sec
[13:45:50.029] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 6.84 sec
[13:45:50.029] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 6.74 sec
[13:45:50.029] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.99 sec
[13:45:50.029] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.95 sec
[13:45:50.029] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.94 sec
[13:45:50.029] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.93 sec
[13:45:50.029] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.03 sec
[13:45:50.029] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.02 sec
[13:45:50.029] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[13:45:50.029] I [ timer  ] Timer: WireCell::Gen::Retagger : 0.01 sec
[13:45:50.029] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[13:45:50.029] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[13:45:50.029] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[13:45:50.029] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[13:45:50.029] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[13:45:50.029] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[13:45:50.029] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[13:45:50.029] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[13:45:50.029] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[13:45:50.029] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[13:45:50.029] I [ timer  ] Timer: Total node execution : 69.93000114522874 sec
wclsFrameSaver saving cooked to 10000 ticks
wclsFrameSaver: saving 43977 traces tagged "gauss"
FrameSaver: q=8.13163e+06 n=949159 tag=gauss
wclsFrameSaver: saving 54949 traces tagged "wiener"
FrameSaver: q=8.78838e+06 n=903577 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 1090 XUs and 1054 XVs -> 18 XUVs
C:0 T:1 238 XUs and 318 XVs -> 8 XUVs
C:0 T:2 159 XUs and 198 XVs -> 12 XUVs
C:0 T:3 234 XUs and 386 XVs -> 28 XUVs
C:0 T:4 158 XUs and 175 XVs -> 18 XUVs
C:0 T:5 351 XUs and 950 XVs -> 25 XUVs
C:0 T:6 187 XUs and 262 XVs -> 13 XUVs
C:0 T:7 59 XUs and 118 XVs -> 3 XUVs
C:0 T:8 9469 XUs and 11109 XVs -> 1339 XUVs
C:0 T:9 1839 XUs and 1844 XVs -> 174 XUVs
C:0 T:10 692 XUs and 459 XVs -> 34 XUVs
C:0 T:11 10389 XUs and 27546 XVs -> 3005 XUVs
C:0 T:12 8901 XUs and 18634 XVs -> 3885 XUVs
C:0 T:13 18950 XUs and 33534 XVs -> 4201 XUVs
C:0 T:14 7762 XUs and 12342 XVs -> 2286 XUVs
C:0 T:15 4715 XUs and 4112 XVs -> 516 XUVs
15565 XUVs total
1976 collection wire objects
15565 potential space points
Neighbour search...
2109473 tests to find 854062 neighbours
Iterating with no regularization...
Begin: 9.81579e+08
0 8.95495e+08
1 8.90086e+08
2 8.8918e+08
3 8.88859e+08
Now with regularization...
Begin: 8.79064e+08
0 8.78792e+08
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
02-Sep-2025 13:47:26 CEST  Closed output file "np02vd_raw_run039273_1518_df-s04-d3_dw_0_20250901T064020_reco_stage1_20250902T114726_keepup.root"

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                        146.421       248.47        458.838       213.885       87.3354        18     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                      8.1764e-05    0.00014486    0.000407898   0.000123918   8.02581e-05      18     
produce:tpcrawdecoder:PDVDTPCReader               7.48419       8.78429       10.3925       8.64247      0.887972        18     
produce:triggerrawdecoder:PDVDTriggerReader4    0.00768626     0.0151453     0.0289783     0.0146466    0.00654413       18     
produce:pdvddaphne:DAPHNEReaderPDVD               4.05232       4.54763       5.21105       4.47188      0.336578        18     
produce:ophit:OpHitFinder                        0.042804      0.0541701     0.0626197      0.05479     0.00420367       18     
produce:opflash:OpFlashFinderVerticalDrift      0.00851894     0.0153844     0.0243776     0.0144952    0.00397582       18     
produce:wclsdatavd:WireCellToolkit                71.6547       81.0009       101.423       80.021        7.50566        18     
produce:gaushit:GausHitFinder                     1.29556       1.99131       2.87756       2.02518      0.450638        18     
produce:nhitsfilter:NumberOfHitsFilter          0.000282768   0.000468419   0.000986666   0.000413397    0.000175        18     
produce:reco3d:SpacePointSolver                   12.0752       23.1428       36.3145       22.3722       6.89494        18     
produce:hitpdune:DisambigFromSpacePoints         0.190641      0.443233      0.785381      0.438421      0.157902        18     
produce:pandora:StandardPandora                   38.6598       119.673       308.519       87.8743       74.4652        18     
produce:pandoraTrack:LArPandoraTrackCreation      1.15007       2.38745       4.59724       2.22983      0.982321        18     
produce:pandoraGnocalo:GnocchiCalorimetry        0.0293837     0.0521241     0.0826912     0.0517709     0.0128736       18     
[art]:TriggerResults:TriggerResultInserter      3.5484e-05    5.66708e-05   0.00015811    5.01655e-05   2.63145e-05      18     
end_path:out1:RootOutput                         5.579e-06    1.21157e-05   4.9335e-05    1.0331e-05    9.28313e-06      18     
end_path:out1:RootOutput(write)                   5.74672       6.27776       6.86452       6.23635      0.332208        18     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5318.33 MB
  Peak resident set size usage (VmHWM): 3377.57 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run039273_1518_df-s04-d3_dw_0_20250901T064020_reco_stage1_20250902T114726_keepup.root
\tHists: np02vd_raw_run039273_1518_df-s04-d3_dw_0_20250901T064020_reco_stage1_20250902T114726_keepup_hists.root
Forming reco metadata
/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/bin/python: No module named meta_maker
Error in reco metadata
justIN time: 2025-09-19 04:15:33 UTC       justIN version: 01.05.00