justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 242287.3@dunegpschedd01.fnal.gov

Jobsub ID242287.3@dunegpschedd01.fnal.gov
Workflow ID9538
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors2
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-11-03 11:20:47
SiteES_PIC
EntryDUNE_T1_ES_PIC_ce16-multicore
Last heartbeat2025-11-03 13:04:21
From worker nodeHostnamehnode79.pic.es
cpuinfoAMD EPYC 7443P 24-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors2
RSS bytes4194304000 (4000 MiB)
Wall seconds limit216000 (60 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-11-03 11:22:03
Input filesvd-protodune:np02vd_raw_run040346_3053_df-s04-d0_dw_0_20251103T064630.hdf5
JobscriptExit code0
Real time1h (6023s)
CPU time1h (4597s = 76%)
Max RSS bytes14608490496 (13931 MiB)
Outputting started2025-11-03 13:02:27
Output filesvd-protodune-det-reco:np02vd_raw_run040346_3053_df-s04-d0_dw_0_20251103T064630_reco_stage1_20251103T130217_keepup.root
Finished2025-11-03 13:04:21
Saved logsjustin-logs:242287.3-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ue  ] frame sink sees EOS
[14:01:17.513] D [ pgraph ] <Pgrapher:> graph execution complete 
[14:01:17.513] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 12.65 sec
[14:01:17.513] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 11.34 sec
[14:01:17.513] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 11.16 sec
[14:01:17.513] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 9.52 sec
[14:01:17.513] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 8.8 sec
[14:01:17.513] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 7.56 sec
[14:01:17.514] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 7.47 sec
[14:01:17.514] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 7.02 sec
[14:01:17.514] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.49 sec
[14:01:17.514] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.47 sec
[14:01:17.514] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.47 sec
[14:01:17.514] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.28 sec
[14:01:17.514] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.02 sec
[14:01:17.514] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[14:01:17.514] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[14:01:17.514] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[14:01:17.514] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[14:01:17.514] I [ timer  ] Timer: WireCell::Gen::Retagger : 0.01 sec
[14:01:17.514] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[14:01:17.514] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[14:01:17.514] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[14:01:17.514] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[14:01:17.514] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[14:01:17.514] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[14:01:17.514] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[14:01:17.514] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[14:01:17.514] I [ timer  ] Timer: Total node execution : 77.29999998770654 sec
wclsFrameSaver saving cooked to 10000 ticks
wclsFrameSaver: saving 52473 traces tagged "gauss"
FrameSaver: q=1.45746e+07 n=1145998 tag=gauss
wclsFrameSaver: saving 64179 traces tagged "wiener"
FrameSaver: q=1.56215e+07 n=1103758 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 18909 XUs and 17683 XVs -> 1223 XUVs
C:0 T:1 701 XUs and 654 XVs -> 36 XUVs
C:0 T:2 2120 XUs and 1847 XVs -> 90 XUVs
C:0 T:3 929 XUs and 1134 XVs -> 44 XUVs
C:0 T:4 5346 XUs and 4663 XVs -> 281 XUVs
C:0 T:5 241 XUs and 237 XVs -> 14 XUVs
C:0 T:6 2314 XUs and 2570 XVs -> 84 XUVs
C:0 T:7 1061 XUs and 1098 XVs -> 57 XUVs
C:0 T:8 161 XUs and 208 XVs -> 14 XUVs
C:0 T:9 701 XUs and 796 XVs -> 49 XUVs
C:0 T:10 1186 XUs and 1531 XVs -> 75 XUVs
C:0 T:11 1816 XUs and 2067 XVs -> 81 XUVs
C:0 T:12 843 XUs and 1281 XVs -> 65 XUVs
C:0 T:13 763 XUs and 1241 XVs -> 47 XUVs
C:0 T:14 483 XUs and 688 XVs -> 28 XUVs
C:0 T:15 154 XUs and 166 XVs -> 5 XUVs
2193 XUVs total
1509 collection wire objects
2193 potential space points
Neighbour search...
55547 tests to find 14036 neighbours
Iterating with no regularization...
Begin: 1.89005e+10
0 1.84882e+10
1 1.84545e+10
2 1.84539e+10
Now with regularization...
Begin: 1.82955e+10
0 1.82953e+10
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
03-Nov-2025 14:02:17 CET  Closed output file "np02vd_raw_run040346_3053_df-s04-d0_dw_0_20251103T064630_reco_stage1_20251103T130217_keepup.root"

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                        84.643        271.135       852.141       139.621       231.122        22     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                      4.6741e-05    6.48423e-05   0.000117202   6.4866e-05    1.5019e-05       22     
produce:tpcrawdecoder:PDVDTPCReader               19.4352       40.1916       273.206       26.8269       51.1251        22     
produce:triggerrawdecoder:PDVDTriggerReader4     0.998566       1.1168        1.83713       1.04683      0.201028        22     
produce:pdvddaphne:DAPHNEReaderPDVD             0.000254154   0.000338479   0.000682543   0.000321711   7.95186e-05      22     
produce:ophit:OpHitFinder                       5.4631e-05    8.49011e-05   0.00052381    6.1991e-05    9.59511e-05      22     
produce:opflash:OpFlashFinderVerticalDrift      4.1571e-05    6.61008e-05   0.000307685   5.2221e-05    5.38269e-05      22     
produce:wclsdatavd:WireCellToolkit                37.9856       121.347       791.971       67.1063       162.147        22     
produce:gaushit:GausHitFinder                    0.604669       2.0874        13.4778       1.22665       2.82144        22     
produce:nhitsfilter:NumberOfHitsFilter          0.000158073   0.000404624   0.00217312    0.000253705   0.00045239       22     
produce:reco3d:SpacePointSolver                   5.94304       25.3088       149.274       11.7518       38.3048        20     
produce:hitpdune:DisambigFromSpacePoints         0.0761389     0.250407      0.876749      0.171396      0.200687        20     
produce:pandora:StandardPandora                   8.71453       85.0596       617.223       22.6968       148.702        20     
produce:pandoraTrack:LArPandoraTrackCreation     0.289513       1.17473       4.63825      0.779731       1.03506        20     
produce:pandoraGnocalo:GnocchiCalorimetry        0.0155187     0.0295732     0.0541806     0.0269304     0.0121159       20     
[art]:TriggerResults:TriggerResultInserter      1.0991e-05    1.82689e-05   3.3551e-05     1.724e-05    6.14793e-06      22     
end_path:out1:RootOutput                         2.711e-06     4.076e-06     1.348e-05     3.66e-06     2.11952e-06      22     
end_path:out1:RootOutput(write)                   2.76475       4.69878       11.418        4.68389       1.74219        22     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 16661.8 MB
  Peak resident set size usage (VmHWM): 14608.5 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run040346_3053_df-s04-d0_dw_0_20251103T064630_reco_stage1_20251103T130217_keepup.root
\tHists: np02vd_raw_run040346_3053_df-s04-d0_dw_0_20251103T064630_reco_stage1_20251103T130217_keepup_hists.root
Forming reco metadata
Successfully opened file np02vd_raw_run040346_3053_df-s04-d0_dw_0_20251103T064630_reco_stage1_20251103T130217_keepup.root
Ran successfully
{
  "name": "np02vd_raw_run040346_3053_df-s04-d0_dw_0_20251103T064630_reco_stage1_20251103T130217_keepup.root",
  "namespace": "vd-protodune-det-reco",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_11_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
    "dune.campaign": "vd-protodune-reco-keepup-v0",
    "core.start_time": 1762174939.0,
    "core.end_time": 1762174939.0,
    "core.events": [
      281812,
      281816,
      281820,
      281824,
      281828,
      281832,
      281836,
      281840,
      281844,
      281848,
      281852,
      281856,
      281860,
      281864,
      281868,
      281872,
      281876,
      281880,
      281884,
      281888,
      281892,
      281896
    ],
    "core.event_count": 22,
    "core.first_event_number": 281812,
    "core.last_event_number": 281896,
    "core.data_stream": "cosmics",
    "core.file_content_status": "good",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      40346
    ],
    "core.runs_subruns": [
      4034600001
    ],
    "dune.daq_test": false,
    "retention.status": "active",
    "retention.class": "physics"
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run040346_3053_df-s04-d0_dw_0_20251103T064630.hdf5"
    }
  ]
}Forming hist metadata
formed
{
  "name": "np02vd_raw_run040346_3053_df-s04-d0_dw_0_20251103T064630_reco_stage1_20251103T130217_keepup_hists.root",
  "namespace": "vd-protodune-det-reco",
  "metadata": {
    "core.file_format": "root",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_11_00d00",
    "core.data_tier": "root-tuple-virtual",
    "dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
    "dune.campaign": "vd-protodune-reco-keepup-v0",
    "core.start_time": 1762174939.0,
    "core.end_time": 1762174939.0,
    "core.data_stream": "cosmics",
    "core.file_content_status": "good",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      40346
    ],
    "core.runs_subruns": [
      4034600001
    ],
    "dune.daq_test": false,
    "retention.status": "active",
    "retention.class": "physics"
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run040346_3053_df-s04-d0_dw_0_20251103T064630.hdf5"
    }
  ]
}
justIN time: 2025-11-05 04:55:14 UTC       justIN version: 01.05.01