justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 262919.2@dunegpschedd01.fnal.gov

Jobsub ID262919.2@dunegpschedd01.fnal.gov
Workflow ID10978
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors2
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-12-03 21:37:45
SiteUK_Lancaster
EntryUBoone_UK_Lancaster_HEC_grendel_ce02
Last heartbeat2025-12-04 00:47:17
From worker nodeHostnamecomp20-01
cpuinfoIntel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors2
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job stateaborted
Started2025-12-03 21:41:06
Input filesvd-protodune:np02vd_raw_run041106_0882_df-s04-d0_dw_0_20251203T212535.hdf5
Outputting started2025-12-04 00:29:53
Output filesvd-protodune-det-reco:np02vd_raw_run041106_0882_df-s04-d0_dw_0_20251203T212535_reco_stage1_20251204T002931_keepup.root
Finished2025-12-04 00:47:17
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

aux   ] <Resampler:resmp0> EOS at call=97 
[00:29:06.506] D [sigproc ] <OmnibusSigProc:anode0sigproc0> EOS at call=97 anode=0 
[00:29:06.506] D [  glue  ] <FrameFanin:nfsp> EOS at call=97 with 8 
[00:29:06.506] D [  glue  ] frame sink sees EOS
[00:29:06.506] D [ pgraph ] <Pgrapher:> graph execution complete 
[00:29:06.506] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 12.65 sec
[00:29:06.506] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 10.26 sec
[00:29:06.506] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 10.16 sec
[00:29:06.506] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 10 sec
[00:29:06.506] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.22 sec
[00:29:06.506] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.22 sec
[00:29:06.506] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.21 sec
[00:29:06.506] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.21 sec
[00:29:06.506] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.01 sec
[00:29:06.506] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 0.01 sec
[00:29:06.506] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 0 sec
[00:29:06.506] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[00:29:06.506] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[00:29:06.506] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 0 sec
[00:29:06.506] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[00:29:06.506] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[00:29:06.506] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[00:29:06.506] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[00:29:06.506] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[00:29:06.506] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[00:29:06.506] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[00:29:06.506] I [ timer  ] Timer: WireCell::Gen::Retagger : 0 sec
[00:29:06.506] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[00:29:06.506] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[00:29:06.506] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 0 sec
[00:29:06.506] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[00:29:06.506] I [ timer  ] Timer: Total node execution : 43.949999678879976 sec
wclsFrameSaver saving cooked to 10000 ticks
wclsFrameSaver: saving 19760 traces tagged "gauss"
FrameSaver: q=7.08711e+06 n=481814 tag=gauss
wclsFrameSaver: saving 25142 traces tagged "wiener"
FrameSaver: q=7.61964e+06 n=461871 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 1250 XUs and 1285 XVs -> 47 XUVs
C:0 T:1 383 XUs and 311 XVs -> 14 XUVs
C:0 T:2 248 XUs and 364 XVs -> 6 XUVs
C:0 T:3 655 XUs and 913 XVs -> 21 XUVs
C:0 T:4 173 XUs and 164 XVs -> 2 XUVs
C:0 T:5 272 XUs and 342 XVs -> 8 XUVs
C:0 T:6 1669 XUs and 2196 XVs -> 60 XUVs
C:0 T:7 10144 XUs and 14063 XVs -> 718 XUVs
876 XUVs total
607 collection wire objects
876 potential space points
Neighbour search...
19220 tests to find 4328 neighbours
Iterating with no regularization...
Begin: 8.19478e+10
0 8.18558e+10
1 8.18509e+10
Now with regularization...
Begin: 7.99417e+10
0 7.99415e+10
04-Dec-2025 00:29:31 GMT  Closed output file "np02vd_raw_run041106_0882_df-s04-d0_dw_0_20251203T212535_reco_stage1_20251204T002931_keepup.root"

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                        140.852       204.752       397.715       190.162       54.635         49     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                      4.6925e-05    5.9211e-05    0.000115625   5.4948e-05    1.31694e-05      49     
produce:tpcrawdecoder:PDVDTPCReader               73.9323       123.161       211.541       106.573       39.693         49     
produce:triggerrawdecoder:PDVDTriggerReader4      0.4555       0.860941       2.32236       0.76711      0.322703        49     
produce:pdvddaphne:DAPHNEReaderPDVD             0.000313112   0.000358281   0.000791243   0.000338703   7.58752e-05      49     
produce:ophit:OpHitFinder                       4.3693e-05    6.9917e-05    0.000459264   5.2608e-05    6.26473e-05      49     
produce:opflash:OpFlashFinderVerticalDrift       3.25e-05     4.98475e-05   0.000254858   4.0441e-05    4.18835e-05      49     
produce:wclsdatavd:WireCellToolkit                38.775        45.5134       81.1202       43.6676       7.29529        49     
produce:gaushit:GausHitFinder                    0.431345      0.656017       1.24846      0.612876       0.1683         49     
produce:nhitsfilter:NumberOfHitsFilter           9.256e-05    0.00017829    0.000333463   0.000164477   5.51585e-05      49     
produce:reco3d:SpacePointSolver                   2.83759       6.18339       13.7423       5.45524       2.44995        49     
produce:hitpdune:DisambigFromSpacePoints         0.0320715     0.093889      0.412835      0.0790464     0.0736087       49     
produce:pandora:StandardPandora                   6.70898       26.0084       234.367       16.0388       37.6247        49     
produce:pandoraTrack:LArPandoraTrackCreation     0.0719608     0.328314       1.03965      0.274408      0.196404        49     
produce:pandoraGnocalo:GnocchiCalorimetry        0.007632      0.0123477     0.0213558     0.0116898    0.00300309       49     
[art]:TriggerResults:TriggerResultInserter       1.447e-05    1.97559e-05   4.3908e-05    1.8504e-05    4.61788e-06      49     
end_path:out1:RootOutput                         5.751e-06    7.42555e-06   1.7997e-05     6.925e-06    2.23386e-06      49     
end_path:out1:RootOutput(write)                   1.64955       1.85719       2.78528       1.79535      0.221708        49     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5241.72 MB
  Peak resident set size usage (VmHWM): 3212.56 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run041106_0882_df-s04-d0_dw_0_20251203T212535_reco_stage1_20251204T002931_keepup.root
\tHists: np02vd_raw_run041106_0882_df-s04-d0_dw_0_20251203T212535_reco_stage1_20251204T002945_keepup_hists.root
Forming reco metadata
Successfully opened file np02vd_raw_run041106_0882_df-s04-d0_dw_0_20251203T212535_reco_stage1_20251204T002931_keepup.root
Ran successfully
{
  "name": "np02vd_raw_run041106_0882_df-s04-d0_dw_0_20251203T212535_reco_stage1_20251204T002931_keepup.root",
  "namespace": "vd-protodune-det-reco",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_13_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage1_protodunevd_keepup.fcl",
    "dune.campaign": "vd-protodune-reco-keepup-v0",
    "core.start_time": 1764808185.0,
    "core.end_time": 1764808185.0,
    "core.events": [
      43076,
      43077,
      43078,
      43079,
      43080,
      43081,
      43082,
      43083,
      43084,
      43085,
      43086,
      43087,
      43088,
      43089,
      43090,
      43091,
      43092,
      43093,
      43094,
      43095,
      43096,
      43097,
      43098,
      43099,
      43100,
      43101,
      43102,
      43103,
      43104,
      43105,
      43106,
      43107,
      43108,
      43109,
      43110,
      43111,
      43112,
      43113,
      43114,
      43115,
      43116,
      43117,
      43118,
      43119,
      43120,
      43121,
      43122,
      43123,
      43124
    ],
    "core.event_count": 49,
    "core.first_event_number": 43076,
    "core.last_event_number": 43124,
    "core.data_stream": "cosmics",
    "core.file_content_status": "good",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      41106
    ],
    "core.runs_subruns": [
      4110600001
    ],
    "dune.daq_test": true,
    "retention.status": "active",
    "retention.class": "physics"
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run041106_0882_df-s04-d0_dw_0_20251203T212535.hdf5"
    }
  ]
}Forming hist metadata
formed
{
  "name": "np02vd_raw_run041106_0882_df-s04-d0_dw_0_20251203T212535_reco_stage1_20251204T002945_keepup_hists.root",
  "namespace": "vd-protodune-det-reco",
  "metadata": {
    "core.file_format": "root",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_13_00d00",
    "core.data_tier": "root-tuple-virtual",
    "dune.config_file": "standard_reco_stage1_protodunevd_keepup.fcl",
    "dune.campaign": "vd-protodune-reco-keepup-v0",
    "core.start_time": 1764808185.0,
    "core.end_time": 1764808185.0,
    "core.data_stream": "cosmics",
    "core.file_content_status": "good",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      41106
    ],
    "core.runs_subruns": [
      4110600001
    ],
    "dune.daq_test": true,
    "retention.status": "active",
    "retention.class": "physics"
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run041106_0882_df-s04-d0_dw_0_20251203T212535.hdf5"
    }
  ]
}
justIN time: 2025-12-19 02:04:24 UTC       justIN version: 01.05.03