justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 228689.0@dunegpschedd02.fnal.gov

Jobsub ID228689.0@dunegpschedd02.fnal.gov
Workflow ID9003
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-10-15 08:59:10
SiteUK_Bristol
EntryCMSHTPC_T2_UK_SGrid_Bristol_lcgce02
Last heartbeat2025-10-15 12:01:30
From worker nodeHostnamehd76.dice.priv
cpuinfoAMD EPYC 7551P 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit259200 (72 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Started2025-10-15 09:11:18
Input filesvd-protodune:np02vd_raw_run040073_0232_df-s03-d0_dw_0_20251015T030749.hdf5
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-10-15 12:01:30
Saved logsjustin-logs:228689.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ime=108 tick=500 with 236259 traces.  frame tags:[ "framefanin" ] 16 tagged trace sets:[ "gauss0":10065 [0] "gauss1":8036 [0] "gauss2":3449 [0] "gauss3":4738 [0] "gauss4":8451 [0] "gauss5":13817 [0] "gauss6":5920 [0] "gauss7":7776 [0] "wiener0":12329 [12329] "wiener1":10159 [10159] "wiener2":4108 [4108] "wiener3":5859 [5859] "wiener4":10180 [10180] "wiener5":16838 [16838] "wiener6":7446 [7446] "wiener7":9617 [9617] ] cmm:[ ] 
Retagger: tagging trace set: wiener with 76536 traces, 76536 summary
Retagger: tagging trace set: gauss with 62252 traces, 0 summary
[11:57:52.587] D [  glue  ] sink frame: #40712 @0.000108 with 236259 traces, frame tags:[retagger], trace tags:[gauss, wiener]
[11:57:52.588] D [  glue  ] <FrameFanout:nfsp> call=43: see EOS 
[11:57:52.588] D [  glue  ] <ChannelSelector:chsel7> see EOS at call=21 
[11:57:52.588] D [sigproc ] <OmnibusSigProc:anode7sigproc7> EOS at call=43 anode=7 
[11:57:52.588] D [  glue  ] <ChannelSelector:chsel6> see EOS at call=21 
[11:57:52.588] D [sigproc ] <OmnibusSigProc:anode6sigproc6> EOS at call=43 anode=6 
[11:57:52.588] D [  glue  ] <ChannelSelector:chsel5> see EOS at call=21 
[11:57:52.588] D [sigproc ] <OmnibusSigProc:anode5sigproc5> EOS at call=43 anode=5 
[11:57:52.588] D [  glue  ] <ChannelSelector:chsel4> see EOS at call=21 
[11:57:52.588] D [sigproc ] <OmnibusSigProc:anode4sigproc4> EOS at call=43 anode=4 
[11:57:52.588] D [  glue  ] <ChannelSelector:chsel3> see EOS at call=21 
[11:57:52.589] D [  aux   ] <Resampler:resmp3> EOS at call=43 
[11:57:52.589] D [sigproc ] <OmnibusSigProc:anode3sigproc3> EOS at call=43 anode=3 
[11:57:52.589] D [  glue  ] <ChannelSelector:chsel2> see EOS at call=21 
[11:57:52.589] D [  aux   ] <Resampler:resmp2> EOS at call=43 
[11:57:52.589] D [sigproc ] <OmnibusSigProc:anode2sigproc2> EOS at call=43 anode=2 
[11:57:52.589] D [  glue  ] <ChannelSelector:chsel1> see EOS at call=21 
[11:57:52.589] D [  aux   ] <Resampler:resmp1> EOS at call=43 
[11:57:52.589] D [sigproc ] <OmnibusSigProc:anode1sigproc1> EOS at call=43 anode=1 
[11:57:52.589] D [  glue  ] <ChannelSelector:chsel0> see EOS at call=21 
[11:57:52.589] D [  aux   ] <Resampler:resmp0> EOS at call=43 
[11:57:52.589] D [sigproc ] <OmnibusSigProc:anode0sigproc0> EOS at call=43 anode=0 
[11:57:52.589] D [  glue  ] <FrameFanin:nfsp> EOS at call=43 with 8 
[11:57:52.589] D [  glue  ] frame sink sees EOS
[11:57:52.589] D [ pgraph ] <Pgrapher:> graph execution complete 
[11:57:52.589] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 12.36 sec
[11:57:52.589] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 11.15 sec
[11:57:52.589] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 10.94 sec
[11:57:52.590] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 10.92 sec
[11:57:52.590] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 10.79 sec
[11:57:52.590] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 9.58 sec
[11:57:52.590] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 9.47 sec
[11:57:52.590] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 9.27 sec
[11:57:52.590] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.34 sec
[11:57:52.590] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.34 sec
[11:57:52.590] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.33 sec
[11:57:52.590] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.32 sec
[11:57:52.590] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.04 sec
[11:57:52.590] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.03 sec
[11:57:52.590] I [ timer  ] Timer: WireCell::Gen::Retagger : 0.02 sec
[11:57:52.590] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[11:57:52.590] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[11:57:52.590] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[11:57:52.590] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[11:57:52.590] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[11:57:52.590] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[11:57:52.590] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[11:57:52.590] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[11:57:52.590] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[11:57:52.590] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[11:57:52.590] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[11:57:52.590] I [ timer  ] Timer: Total node execution : 85.93999955244362 sec
wclsFrameSaver saving cooked to 10000 ticks
wclsFrameSaver: saving 62252 traces tagged "gauss"
FrameSaver: q=1.5531e+07 n=1272754 tag=gauss
wclsFrameSaver: saving 76536 traces tagged "wiener"
FrameSaver: q=1.63437e+07 n=1204602 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 2236 XUs and 2706 XVs -> 101 XUVs
C:0 T:1 3972 XUs and 4004 XVs -> 143 XUVs
C:0 T:2 2195 XUs and 1997 XVs -> 94 XUVs
C:0 T:3 7029 XUs and 9277 XVs -> 527 XUVs
C:0 T:4 440 XUs and 465 XVs -> 23 XUVs
C:0 T:5 771 XUs and 753 XVs -> 36 XUVs
C:0 T:6 744 XUs and 806 XVs -> 26 XUVs
C:0 T:7 748 XUs and 1428 XVs -> 36 XUVs
C:0 T:8 5045 XUs and 4052 XVs -> 194 XUVs
C:0 T:9 1369 XUs and 1812 XVs -> 70 XUVs
C:0 T:10 7194 XUs and 7003 XVs -> 325 XUVs
C:0 T:11 5783 XUs and 6228 XVs -> 250 XUVs
C:0 T:12 2408 XUs and 4406 XVs -> 316 XUVs
C:0 T:13 804 XUs and 926 XVs -> 42 XUVs
C:0 T:14 3401 XUs and 5198 XVs -> 242 XUVs
C:0 T:15 1598 XUs and 2468 XVs -> 148 XUVs
2573 XUVs total
2050 collection wire objects
2573 potential space points
Neighbour search...
26225 tests to find 10864 neighbours
Iterating with no regularization...
Begin: 1.30131e+10
0 1.28154e+10
1 1.2814e+10
Now with regularization...
Begin: 1.27237e+10
0 1.27236e+10
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
15-Oct-2025 12:01:03 UTC  Closed output file "np02vd_raw_run040073_0232_df-s03-d0_dw_0_20251015T030749_reco_stage1_20251015T120103_keepup.root"

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                        234.905       479.286       2523.65       348.782       471.888        21     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                      7.7866e-05    0.000139902   0.000416301   0.000112321   7.02042e-05      22     
produce:tpcrawdecoder:PDVDTPCReader               15.105        112.743       237.452       110.107       40.7324        22     
produce:triggerrawdecoder:PDVDTriggerReader4     0.333613      0.454654      0.674452      0.427545      0.0957257       22     
produce:pdvddaphne:DAPHNEReaderPDVD             0.000432471     7.9578        14.4203       7.85037       2.66941        22     
produce:ophit:OpHitFinder                       0.000183284    0.0432736     0.0556116     0.0448778     0.0105979       22     
produce:opflash:OpFlashFinderVerticalDrift      6.4241e-05    0.00609906     0.018934     0.00494083    0.00450365       22     
produce:wclsdatavd:WireCellToolkit                2.99326       102.929       308.413       86.8577       53.1305        22     
produce:gaushit:GausHitFinder                    0.0454411      2.16079       4.25575       2.18272      0.734112        22     
produce:nhitsfilter:NumberOfHitsFilter          5.6255e-05    0.00046183    0.000707416   0.000436238   0.000134527      22     
produce:reco3d:SpacePointSolver                  0.0214906      20.759        53.2432       19.0895       9.48548        22     
produce:hitpdune:DisambigFromSpacePoints        0.00201298      0.41689       2.30262      0.324372      0.426721        22     
produce:pandora:StandardPandora                  0.077619       200.375       2130.16       108.582       423.895        22     
produce:pandoraTrack:LArPandoraTrackCreation    0.00463757      2.99856       15.2802       2.4748        2.83355        22     
produce:pandoraGnocalo:GnocchiCalorimetry       0.00140662     0.0489096     0.0880611     0.0471115     0.016092        22     
[art]:TriggerResults:TriggerResultInserter      1.8394e-05    4.9219e-05    0.000112351   4.60815e-05   2.11194e-05      22     
end_path:out1:RootOutput                         4.217e-06    1.02657e-05   2.9004e-05    9.7535e-06    5.35191e-06      22     
end_path:out1:RootOutput(write)                  0.276924       6.51741       12.5917       6.25517       2.00043        22     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6823.15 MB
  Peak resident set size usage (VmHWM): 4857.24 MB
  Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException:  PostEndJob 15-Oct-2025 12:01:03 UTC ModuleEndJob
---- EventRangeOverlap BEGIN
  Attempt to merge event ranges that both contain one or more of the same events
   Run: 40073
    SubRun: 1 Event range: [40576,40577)  vs.
    SubRun: 1 Event range: [40576,40577)
---- EventRangeOverlap END
---- EventRangeOverlap BEGIN
  Attempt to merge event ranges that both contain one or more of the same events
   Run: 40073
    SubRun: 1 Event range: [40576,40577)  vs.
    SubRun: 1 Event range: [40576,40577)
---- EventRangeOverlap END
%MSG
Art has completed and will exit with status 1.
Error in reco1
justIN time: 2025-11-05 04:51:38 UTC       justIN version: 01.05.01