justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 242104.135@dunegpschedd01.fnal.gov

Jobsub ID242104.135@dunegpschedd01.fnal.gov
Workflow ID9531
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors2
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-11-02 22:38:06
SiteNL_NIKHEF
EntryVIRGO_NL_NIKHEF_klomp_multicore
Last heartbeat2025-11-03 00:18:00
From worker nodeHostnamewn-sate-002.farm.nikhef.nl
cpuinfoAMD EPYC 7551P 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors2
RSS bytes4194304000 (4000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-11-02 22:39:05
Input filesvd-protodune:np02vd_raw_run040346_2400_df-s04-d3_dw_0_20251102T213959.hdf5
JobscriptExit code0
Real time1h (5800s)
CPU time1h (4458s = 76%)
Max RSS bytes5636214784 (5375 MiB)
Outputting started2025-11-03 00:15:45
Output filesvd-protodune-det-reco:np02vd_raw_run040346_2400_df-s04-d3_dw_0_20251102T213959_reco_stage1_20251103T001536_keepup.root
Finished2025-11-03 00:18:00
Saved logsjustin-logs:242104.135-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

igproc0> EOS at call=47 anode=0 
[01:05:18.829] D [  glue  ] <FrameFanin:nfsp> EOS at call=47 with 8 
[01:05:18.829] D [  glue  ] frame sink sees EOS
[01:05:18.829] D [ pgraph ] <Pgrapher:> graph execution complete 
[01:05:18.829] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 58.78 sec
[01:05:18.829] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 26.21 sec
[01:05:18.829] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 19.28 sec
[01:05:18.829] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 15.28 sec
[01:05:18.829] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 15.1 sec
[01:05:18.829] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 14.88 sec
[01:05:18.829] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 14.8 sec
[01:05:18.829] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 14.52 sec
[01:05:18.829] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.52 sec
[01:05:18.829] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.5 sec
[01:05:18.829] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.49 sec
[01:05:18.829] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.49 sec
[01:05:18.829] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.04 sec
[01:05:18.829] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.02 sec
[01:05:18.829] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[01:05:18.829] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[01:05:18.829] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[01:05:18.829] I [ timer  ] Timer: WireCell::Gen::Retagger : 0.01 sec
[01:05:18.829] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[01:05:18.829] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[01:05:18.829] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[01:05:18.829] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[01:05:18.829] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[01:05:18.829] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[01:05:18.829] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[01:05:18.829] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[01:05:18.829] I [ timer  ] Timer: Total node execution : 180.94999942556024 sec
wclsFrameSaver saving cooked to 10000 ticks
wclsFrameSaver: saving 70117 traces tagged "gauss"
FrameSaver: q=5.34025e+07 n=2237550 tag=gauss
wclsFrameSaver: saving 90037 traces tagged "wiener"
FrameSaver: q=5.61656e+07 n=2154967 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 735 XUs and 1019 XVs -> 15 XUVs
C:0 T:1 52370 XUs and 66456 XVs -> 4697 XUVs
C:0 T:2 830 XUs and 938 XVs -> 28 XUVs
C:0 T:3 290 XUs and 323 XVs -> 9 XUVs
C:0 T:4 1735 XUs and 1609 XVs -> 150 XUVs
C:0 T:5 652 XUs and 509 XVs -> 25 XUVs
C:0 T:6 330 XUs and 521 XVs -> 42 XUVs
C:0 T:7 81 XUs and 119 XVs -> 3 XUVs
C:0 T:8 5557 XUs and 4317 XVs -> 265 XUVs
C:0 T:9 88933 XUs and 119190 XVs -> 6862 XUVs
C:0 T:10 2268 XUs and 1802 XVs -> 186 XUVs
C:0 T:11 1101 XUs and 1281 XVs -> 73 XUVs
C:0 T:12 55149 XUs and 66834 XVs -> 4952 XUVs
C:0 T:13 2757 XUs and 1382 XVs -> 90 XUVs
C:0 T:14 3049 XUs and 3288 XVs -> 225 XUVs
C:0 T:15 573 XUs and 634 XVs -> 49 XUVs
17671 XUVs total
6771 collection wire objects
17671 potential space points
Neighbour search...
782809 tests to find 179612 neighbours
Iterating with no regularization...
Begin: 6.75814e+11
0 6.66906e+11
1 6.66372e+11
Now with regularization...
Begin: 6.65077e+11
0 6.65063e+11
03-Nov-2025 01:15:36 CET  Closed output file "np02vd_raw_run040346_2400_df-s04-d3_dw_0_20251102T213959_reco_stage1_20251103T001536_keepup.root"

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                        147.324       239.26        856.758       198.702       140.498        24     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                      5.7759e-05    8.72953e-05   0.000249178   7.37135e-05   3.97972e-05      24     
produce:tpcrawdecoder:PDVDTPCReader               42.3059       58.0871       77.9685       57.7145       9.55612        24     
produce:triggerrawdecoder:PDVDTriggerReader4     0.512858      0.600927       0.80824      0.569481      0.0877035       24     
produce:pdvddaphne:DAPHNEReaderPDVD             0.000376939   0.000434284   0.000818167   0.000399781   9.05353e-05      24     
produce:ophit:OpHitFinder                       6.5653e-05    0.000111788   0.000699364   7.64435e-05   0.00012512       24     
produce:opflash:OpFlashFinderVerticalDrift      5.3571e-05    7.57185e-05   0.000364645   5.67515e-05   6.25822e-05      24     
produce:wclsdatavd:WireCellToolkit                67.9724       89.3011       182.322       76.5018       26.0078        24     
produce:gaushit:GausHitFinder                    0.929104       1.52363       4.2197        1.32338      0.725658        24     
produce:nhitsfilter:NumberOfHitsFilter          0.000249428   0.000366399   0.000728349   0.000343916   0.000119577      24     
produce:reco3d:SpacePointSolver                   7.48863       15.0605       39.9022       12.8995       7.19416        24     
produce:hitpdune:DisambigFromSpacePoints         0.103479      0.255975       1.03833      0.188388      0.189725        24     
produce:pandora:StandardPandora                   16.3683       67.3514       552.163       33.3598       107.601        24     
produce:pandoraTrack:LArPandoraTrackCreation      0.55418       1.84448       14.0418       1.10645       2.63753        24     
produce:pandoraGnocalo:GnocchiCalorimetry        0.0223735     0.0369107     0.0682118     0.0361672     0.0100726       24     
[art]:TriggerResults:TriggerResultInserter      2.1821e-05    3.56942e-05   0.000140263   2.8183e-05    2.42444e-05      24     
end_path:out1:RootOutput                         4.108e-06    8.49638e-06    3.758e-05    7.0385e-06    6.90417e-06      24     
end_path:out1:RootOutput(write)                   4.60325       5.13624       6.24867       5.06206      0.444355        24     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 7654.12 MB
  Peak resident set size usage (VmHWM): 5636.21 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run040346_2400_df-s04-d3_dw_0_20251102T213959_reco_stage1_20251103T001536_keepup.root
\tHists: np02vd_raw_run040346_2400_df-s04-d3_dw_0_20251102T213959_reco_stage1_20251103T001536_keepup_hists.root
Forming reco metadata
Successfully opened file np02vd_raw_run040346_2400_df-s04-d3_dw_0_20251102T213959_reco_stage1_20251103T001536_keepup.root
Ran successfully
{
  "name": "np02vd_raw_run040346_2400_df-s04-d3_dw_0_20251102T213959_reco_stage1_20251103T001536_keepup.root",
  "namespace": "vd-protodune-det-reco",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_11_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
    "dune.campaign": "vd-protodune-reco-keepup-v0",
    "core.start_time": 1762128937.0,
    "core.end_time": 1762128937.0,
    "core.events": [
      221531,
      221535,
      221539,
      221543,
      221547,
      221551,
      221555,
      221559,
      221563,
      221567,
      221571,
      221575,
      221579,
      221583,
      221587,
      221591,
      221595,
      221599,
      221603,
      221607,
      221611,
      221615,
      221619,
      221623
    ],
    "core.event_count": 24,
    "core.first_event_number": 221531,
    "core.last_event_number": 221623,
    "core.data_stream": "cosmics",
    "core.file_content_status": "good",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      40346
    ],
    "core.runs_subruns": [
      4034600001
    ],
    "dune.daq_test": false,
    "retention.status": "active",
    "retention.class": "physics"
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run040346_2400_df-s04-d3_dw_0_20251102T213959.hdf5"
    }
  ]
}Forming hist metadata
formed
{
  "name": "np02vd_raw_run040346_2400_df-s04-d3_dw_0_20251102T213959_reco_stage1_20251103T001536_keepup_hists.root",
  "namespace": "vd-protodune-det-reco",
  "metadata": {
    "core.file_format": "root",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_11_00d00",
    "core.data_tier": "root-tuple-virtual",
    "dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
    "dune.campaign": "vd-protodune-reco-keepup-v0",
    "core.start_time": 1762128937.0,
    "core.end_time": 1762128937.0,
    "core.data_stream": "cosmics",
    "core.file_content_status": "good",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      40346
    ],
    "core.runs_subruns": [
      4034600001
    ],
    "dune.daq_test": false,
    "retention.status": "active",
    "retention.class": "physics"
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run040346_2400_df-s04-d3_dw_0_20251102T213959.hdf5"
    }
  ]
}
justIN time: 2025-11-05 07:24:40 UTC       justIN version: 01.05.01