justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 236601.82@dunegpschedd02.fnal.gov

Jobsub ID236601.82@dunegpschedd02.fnal.gov
Workflow ID9538
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors2
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-11-03 10:42:44
SiteNL_SURFsara
EntryDUNE_SurfSARA_arc01
Last heartbeat2025-11-03 12:40:53
From worker nodeHostnamewn-da-06.gina.surf.nl
cpuinfoAMD EPYC 7702P 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors2
RSS bytes4194304000 (4000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-11-03 10:57:28
Input filesvd-protodune:np02vd_raw_run040346_3314_df-s04-d2_dw_0_20251103T102900.hdf5
JobscriptExit code0
Real time1h (6062s)
CPU time1h (4038s = 66%)
Max RSS bytes3614838784 (3447 MiB)
Outputting started2025-11-03 12:38:31
Output filesvd-protodune-det-reco:np02vd_raw_run040346_3314_df-s04-d2_dw_0_20251103T102900_reco_stage1_20251103T123820_keepup.root
Finished2025-11-03 12:40:53
Saved logsjustin-logs:236601.82-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

fsp> EOS at call=45 with 8 
[13:35:16.643] D [  glue  ] frame sink sees EOS
[13:35:16.643] D [ pgraph ] <Pgrapher:> graph execution complete 
[13:35:16.643] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 32.62 sec
[13:35:16.643] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 19.39 sec
[13:35:16.643] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 19.04 sec
[13:35:16.643] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 18.71 sec
[13:35:16.643] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 18.06 sec
[13:35:16.643] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 18.05 sec
[13:35:16.643] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 17.99 sec
[13:35:16.643] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 17.66 sec
[13:35:16.643] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.72 sec
[13:35:16.643] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.71 sec
[13:35:16.643] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.71 sec
[13:35:16.643] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.7 sec
[13:35:16.644] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.03 sec
[13:35:16.644] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[13:35:16.644] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[13:35:16.644] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0.01 sec
[13:35:16.644] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[13:35:16.644] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[13:35:16.644] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[13:35:16.644] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[13:35:16.644] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[13:35:16.644] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[13:35:16.644] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[13:35:16.644] I [ timer  ] Timer: WireCell::Gen::Retagger : 0 sec
[13:35:16.644] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[13:35:16.644] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[13:35:16.644] I [ timer  ] Timer: Total node execution : 164.41999661549926 sec
wclsFrameSaver saving cooked to 10000 ticks
wclsFrameSaver: saving 63246 traces tagged "gauss"
FrameSaver: q=3.95942e+07 n=1572434 tag=gauss
wclsFrameSaver: saving 81075 traces tagged "wiener"
FrameSaver: q=4.21709e+07 n=1515337 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 28009 XUs and 48811 XVs -> 2689 XUVs
C:0 T:1 19965 XUs and 22031 XVs -> 1747 XUVs
C:0 T:2 1759 XUs and 2228 XVs -> 65 XUVs
C:0 T:3 3838 XUs and 4712 XVs -> 195 XUVs
C:0 T:4 303 XUs and 285 XVs -> 12 XUVs
C:0 T:5 217 XUs and 256 XVs -> 7 XUVs
C:0 T:6 753 XUs and 866 XVs -> 79 XUVs
C:0 T:7 1218 XUs and 1453 XVs -> 46 XUVs
C:0 T:8 1492 XUs and 1736 XVs -> 422 XUVs
C:0 T:9 5614 XUs and 5104 XVs -> 557 XUVs
C:0 T:10 1049 XUs and 956 XVs -> 49 XUVs
C:0 T:11 1968 XUs and 1947 XVs -> 193 XUVs
C:0 T:12 258 XUs and 401 XVs -> 31 XUVs
C:0 T:13 3207 XUs and 3487 XVs -> 278 XUVs
C:0 T:14 2752 XUs and 2394 XVs -> 530 XUVs
C:0 T:15 1577 XUs and 1298 XVs -> 308 XUVs
7208 XUVs total
3185 collection wire objects
7208 potential space points
Neighbour search...
250240 tests to find 81402 neighbours
Iterating with no regularization...
Begin: 4.71157e+11
0 4.49244e+11
1 4.48282e+11
2 4.48272e+11
Now with regularization...
Begin: 4.4534e+11
0 4.4533e+11
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
03-Nov-2025 13:38:20 CET  Closed output file "np02vd_raw_run040346_3314_df-s04-d2_dw_0_20251103T102900_reco_stage1_20251103T123820_keepup.root"

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                        181.631       260.975       568.988       242.767       77.0417        23     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                      6.0404e-05    7.70068e-05   0.000191683   6.8842e-05    2.6219e-05       23     
produce:tpcrawdecoder:PDVDTPCReader               49.369        90.7888       220.148        86.25        32.8943        23     
produce:triggerrawdecoder:PDVDTriggerReader4     0.539516      0.991946       1.71461      0.960654      0.261229        23     
produce:pdvddaphne:DAPHNEReaderPDVD             0.000276727   0.000328032   0.000755227   0.000305491   9.31376e-05      23     
produce:ophit:OpHitFinder                       5.1599e-05    7.48957e-05   0.00045183    5.6988e-05    8.04689e-05      23     
produce:opflash:OpFlashFinderVerticalDrift      3.8303e-05    5.89494e-05   0.000300621   4.0767e-05    5.84307e-05      23     
produce:wclsdatavd:WireCellToolkit                66.3735       115.304       164.837       114.392       21.3161        23     
produce:gaushit:GausHitFinder                    0.781046       1.1914        1.72297       1.19733      0.237232        23     
produce:nhitsfilter:NumberOfHitsFilter          0.000211101   0.000377737   0.000537131   0.000380955   8.35681e-05      23     
produce:reco3d:SpacePointSolver                   6.56992       11.8211       21.3517       11.0029       3.77822        23     
produce:hitpdune:DisambigFromSpacePoints         0.0888442      0.18962      0.437604      0.177118      0.0874342       23     
produce:pandora:StandardPandora                   13.1556       35.3657       150.581       25.2948       30.2703        23     
produce:pandoraTrack:LArPandoraTrackCreation     0.480839       1.22919       3.56311       1.01693      0.723522        23     
produce:pandoraGnocalo:GnocchiCalorimetry        0.0198031     0.0319282     0.0578279     0.0300359    0.00792263       23     
[art]:TriggerResults:TriggerResultInserter       1.567e-05    2.91302e-05   5.9323e-05    2.3956e-05    1.23053e-05      23     
end_path:out1:RootOutput                         4.219e-06    6.35452e-06   2.1159e-05     4.939e-06    3.7229e-06       23     
end_path:out1:RootOutput(write)                   3.62108       4.00383       4.7744        3.95212      0.272173        23     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5633.85 MB
  Peak resident set size usage (VmHWM): 3614.84 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run040346_3314_df-s04-d2_dw_0_20251103T102900_reco_stage1_20251103T123820_keepup.root
\tHists: np02vd_raw_run040346_3314_df-s04-d2_dw_0_20251103T102900_reco_stage1_20251103T123820_keepup_hists.root
Forming reco metadata
Successfully opened file np02vd_raw_run040346_3314_df-s04-d2_dw_0_20251103T102900_reco_stage1_20251103T123820_keepup.root
Ran successfully
{
  "name": "np02vd_raw_run040346_3314_df-s04-d2_dw_0_20251103T102900_reco_stage1_20251103T123820_keepup.root",
  "namespace": "vd-protodune-det-reco",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_11_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
    "dune.campaign": "vd-protodune-reco-keepup-v0",
    "core.start_time": 1762173501.0,
    "core.end_time": 1762173501.0,
    "core.events": [
      305954,
      305958,
      305962,
      305966,
      305970,
      305974,
      305978,
      305982,
      305986,
      305990,
      305994,
      305998,
      306002,
      306006,
      306010,
      306014,
      306018,
      306022,
      306026,
      306030,
      306034,
      306038,
      306042
    ],
    "core.event_count": 23,
    "core.first_event_number": 305954,
    "core.last_event_number": 306042,
    "core.data_stream": "cosmics",
    "core.file_content_status": "good",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      40346
    ],
    "core.runs_subruns": [
      4034600001
    ],
    "dune.daq_test": false,
    "retention.status": "active",
    "retention.class": "physics"
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run040346_3314_df-s04-d2_dw_0_20251103T102900.hdf5"
    }
  ]
}Forming hist metadata
formed
{
  "name": "np02vd_raw_run040346_3314_df-s04-d2_dw_0_20251103T102900_reco_stage1_20251103T123820_keepup_hists.root",
  "namespace": "vd-protodune-det-reco",
  "metadata": {
    "core.file_format": "root",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_11_00d00",
    "core.data_tier": "root-tuple-virtual",
    "dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
    "dune.campaign": "vd-protodune-reco-keepup-v0",
    "core.start_time": 1762173501.0,
    "core.end_time": 1762173501.0,
    "core.data_stream": "cosmics",
    "core.file_content_status": "good",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      40346
    ],
    "core.runs_subruns": [
      4034600001
    ],
    "dune.daq_test": false,
    "retention.status": "active",
    "retention.class": "physics"
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run040346_3314_df-s04-d2_dw_0_20251103T102900.hdf5"
    }
  ]
}
justIN time: 2025-11-04 01:29:13 UTC       justIN version: 01.05.01