justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 242278.155@dunegpschedd01.fnal.gov

Jobsub ID242278.155@dunegpschedd01.fnal.gov
Workflow ID9538
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors2
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-11-03 10:40:44
SiteUK_Lancaster
EntryUBoone_UK_Lancaster_HEC_grendel_ce02
Last heartbeat2025-11-03 12:27:14
From worker nodeHostnamecomp19-16
cpuinfoIntel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors2
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-11-03 10:47:02
Input filesvd-protodune:np02vd_raw_run040346_3286_df-s04-d1_dw_0_20251103T100554.hdf5
JobscriptExit code0
Real time1h (5868s)
CPU time55m (3353s = 57%)
Max RSS bytes3898576896 (3717 MiB)
Outputting started2025-11-03 12:24:51
Output filesvd-protodune-det-reco:np02vd_raw_run040346_3286_df-s04-d1_dw_0_20251103T100554_reco_stage1_20251103T122435_keepup.root
Finished2025-11-03 12:27:14
Saved logsjustin-logs:242278.155-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

usSigProc:anode0sigproc0> EOS at call=45 anode=0 
[12:17:15.259] D [  glue  ] <FrameFanin:nfsp> EOS at call=45 with 8 
[12:17:15.259] D [  glue  ] frame sink sees EOS
[12:17:15.259] D [ pgraph ] <Pgrapher:> graph execution complete 
[12:17:15.259] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 20.14 sec
[12:17:15.259] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 13.21 sec
[12:17:15.259] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 12.5 sec
[12:17:15.259] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 12.46 sec
[12:17:15.259] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 12.41 sec
[12:17:15.259] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 12.39 sec
[12:17:15.259] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 11.63 sec
[12:17:15.259] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 11.55 sec
[12:17:15.259] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.46 sec
[12:17:15.259] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.44 sec
[12:17:15.259] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.43 sec
[12:17:15.259] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.43 sec
[12:17:15.259] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.06 sec
[12:17:15.259] I [ timer  ] Timer: WireCell::Gen::Retagger : 0.02 sec
[12:17:15.259] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[12:17:15.259] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[12:17:15.259] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[12:17:15.259] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[12:17:15.259] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[12:17:15.259] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[12:17:15.259] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[12:17:15.259] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[12:17:15.259] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[12:17:15.259] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[12:17:15.259] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[12:17:15.259] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[12:17:15.259] I [ timer  ] Timer: Total node execution : 108.14999997988343 sec
wclsFrameSaver saving cooked to 10000 ticks
wclsFrameSaver: saving 70108 traces tagged "gauss"
FrameSaver: q=4.75705e+07 n=1705334 tag=gauss
wclsFrameSaver: saving 89459 traces tagged "wiener"
FrameSaver: q=5.02238e+07 n=1635353 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 959 XUs and 1081 XVs -> 70 XUVs
C:0 T:1 32562 XUs and 50029 XVs -> 2382 XUVs
C:0 T:2 916 XUs and 1144 XVs -> 36 XUVs
C:0 T:3 2417 XUs and 3884 XVs -> 103 XUVs
C:0 T:4 28888 XUs and 36945 XVs -> 2648 XUVs
C:0 T:5 2829 XUs and 3235 XVs -> 242 XUVs
C:0 T:6 688 XUs and 901 XVs -> 21 XUVs
C:0 T:7 2205 XUs and 1927 XVs -> 75 XUVs
C:0 T:8 3141 XUs and 2866 XVs -> 193 XUVs
C:0 T:9 3086 XUs and 2496 XVs -> 411 XUVs
C:0 T:10 403 XUs and 469 XVs -> 19 XUVs
C:0 T:11 1458 XUs and 1460 XVs -> 60 XUVs
C:0 T:12 2494 XUs and 2072 XVs -> 382 XUVs
C:0 T:13 1881 XUs and 1414 XVs -> 417 XUVs
C:0 T:14 1193 XUs and 1462 XVs -> 45 XUVs
C:0 T:15 691 XUs and 676 XVs -> 37 XUVs
7141 XUVs total
3544 collection wire objects
7141 potential space points
Neighbour search...
250703 tests to find 70078 neighbours
Iterating with no regularization...
Begin: 1.39356e+12
0 1.38421e+12
1 1.38378e+12
Now with regularization...
Begin: 1.3811e+12
0 1.38109e+12
03-Nov-2025 12:24:35 GMT  Closed output file "np02vd_raw_run040346_3286_df-s04-d1_dw_0_20251103T100554_reco_stage1_20251103T122435_keepup.root"

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                        166.608       252.574       650.527       226.516       101.583        23     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       5.484e-05    7.58763e-05   0.000175999   7.0677e-05    2.26105e-05      23     
produce:tpcrawdecoder:PDVDTPCReader               68.5854       111.081       219.311       100.656       36.5733        23     
produce:triggerrawdecoder:PDVDTriggerReader4     0.744798       1.00796       1.71675      0.839763      0.271551        23     
produce:pdvddaphne:DAPHNEReaderPDVD             0.000362907   0.000398553   0.000695593   0.000387547   6.41852e-05      23     
produce:ophit:OpHitFinder                       5.3722e-05    8.23467e-05   0.000502706   5.7411e-05    9.07882e-05      23     
produce:opflash:OpFlashFinderVerticalDrift      3.9962e-05    5.68558e-05   0.000292401   4.5602e-05    5.04505e-05      23     
produce:wclsdatavd:WireCellToolkit                54.7282       81.0541       122.169       69.5714       20.1749        23     
produce:gaushit:GausHitFinder                    0.770995       1.19147       1.95734       1.10845      0.313517        23     
produce:nhitsfilter:NumberOfHitsFilter          0.000266587   0.000364972   0.000604094   0.000336263   8.3922e-05       23     
produce:reco3d:SpacePointSolver                   5.63113       9.83246       23.1319       8.66694       3.67174        23     
produce:hitpdune:DisambigFromSpacePoints         0.107547      0.188158      0.544518      0.169191      0.0929418       23     
produce:pandora:StandardPandora                   12.9479       42.8644       401.585       21.5256       77.8411        23     
produce:pandoraTrack:LArPandoraTrackCreation      0.55858       1.23103       6.43149      0.860955       1.19886        23     
produce:pandoraGnocalo:GnocchiCalorimetry        0.0172933     0.0267808     0.0480296     0.0248513    0.00637037       23     
[art]:TriggerResults:TriggerResultInserter      2.7093e-05    3.76246e-05   7.8976e-05    3.5451e-05    9.56504e-06      23     
end_path:out1:RootOutput                         8.359e-06    1.0134e-05    2.6405e-05     9.416e-06    3.52141e-06      23     
end_path:out1:RootOutput(write)                   3.5844        4.04844       5.56264       3.84949      0.482488        23     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5943.61 MB
  Peak resident set size usage (VmHWM): 3898.58 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run040346_3286_df-s04-d1_dw_0_20251103T100554_reco_stage1_20251103T122435_keepup.root
\tHists: np02vd_raw_run040346_3286_df-s04-d1_dw_0_20251103T100554_reco_stage1_20251103T122442_keepup_hists.root
Forming reco metadata
Successfully opened file np02vd_raw_run040346_3286_df-s04-d1_dw_0_20251103T100554_reco_stage1_20251103T122435_keepup.root
Ran successfully
{
  "name": "np02vd_raw_run040346_3286_df-s04-d1_dw_0_20251103T100554_reco_stage1_20251103T122435_keepup.root",
  "namespace": "vd-protodune-det-reco",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_11_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
    "dune.campaign": "vd-protodune-reco-keepup-v0",
    "core.start_time": 1762172683.0,
    "core.end_time": 1762172683.0,
    "core.events": [
      303445,
      303449,
      303453,
      303457,
      303461,
      303465,
      303469,
      303473,
      303477,
      303481,
      303485,
      303489,
      303493,
      303497,
      303501,
      303505,
      303509,
      303513,
      303517,
      303521,
      303525,
      303529,
      303533
    ],
    "core.event_count": 23,
    "core.first_event_number": 303445,
    "core.last_event_number": 303533,
    "core.data_stream": "cosmics",
    "core.file_content_status": "good",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      40346
    ],
    "core.runs_subruns": [
      4034600001
    ],
    "dune.daq_test": false,
    "retention.status": "active",
    "retention.class": "physics"
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run040346_3286_df-s04-d1_dw_0_20251103T100554.hdf5"
    }
  ]
}Forming hist metadata
formed
{
  "name": "np02vd_raw_run040346_3286_df-s04-d1_dw_0_20251103T100554_reco_stage1_20251103T122442_keepup_hists.root",
  "namespace": "vd-protodune-det-reco",
  "metadata": {
    "core.file_format": "root",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_11_00d00",
    "core.data_tier": "root-tuple-virtual",
    "dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
    "dune.campaign": "vd-protodune-reco-keepup-v0",
    "core.start_time": 1762172683.0,
    "core.end_time": 1762172683.0,
    "core.data_stream": "cosmics",
    "core.file_content_status": "good",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      40346
    ],
    "core.runs_subruns": [
      4034600001
    ],
    "dune.daq_test": false,
    "retention.status": "active",
    "retention.class": "physics"
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run040346_3286_df-s04-d1_dw_0_20251103T100554.hdf5"
    }
  ]
}
justIN time: 2025-11-03 18:43:38 UTC       justIN version: 01.05.01