justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 41697.102@dunegpschedd02.fnal.gov

Jobsub ID41697.102@dunegpschedd02.fnal.gov
Workflow ID2867
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-09-18 06:46:09
SiteNL_NIKHEF
EntryVIRGO_NL_NIKHEF_klomp
Last heartbeat2025-09-18 10:33:22
From worker nodeHostnamewn-sate-050.farm.nikhef.nl
cpuinfoAMD EPYC 7551P 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-09-18 07:08:58
Input filesvd-protodune:np02vd_raw_run039433_0367_df-s05-d2_dw_0_20250918T062521.hdf5
JobscriptExit code0
Real time3h (12197s)
CPU time2h (9821s = 80%)
Max RSS bytes5588385792 (5329 MiB)
Outputting started2025-09-18 10:32:16
Output filesvd-protodune-det-reco:np02vd_raw_run039433_0367_df-s05-d2_dw_0_20250918T062521_reco_stage1_20250918T103207_keepup.root
Finished2025-09-18 10:33:22
Saved logsjustin-logs:41697.102-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

] D [sigproc ] <OmnibusSigProc:anode0sigproc0> EOS at call=93 anode=0 
[12:30:58.492] D [  glue  ] <FrameFanin:nfsp> EOS at call=93 with 8 
[12:30:58.492] D [  glue  ] frame sink sees EOS
[12:30:58.492] D [ pgraph ] <Pgrapher:> graph execution complete 
[12:30:58.492] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 10.94 sec
[12:30:58.492] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 7.25 sec
[12:30:58.492] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 6.17 sec
[12:30:58.492] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 5.93 sec
[12:30:58.492] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.35 sec
[12:30:58.492] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.34 sec
[12:30:58.492] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.34 sec
[12:30:58.492] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.34 sec
[12:30:58.492] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.01 sec
[12:30:58.492] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[12:30:58.492] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 0 sec
[12:30:58.492] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[12:30:58.492] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[12:30:58.492] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 0 sec
[12:30:58.492] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[12:30:58.492] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[12:30:58.492] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[12:30:58.492] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[12:30:58.492] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 0 sec
[12:30:58.492] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[12:30:58.492] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[12:30:58.492] I [ timer  ] Timer: WireCell::Gen::Retagger : 0 sec
[12:30:58.492] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[12:30:58.492] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[12:30:58.492] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 0 sec
[12:30:58.492] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[12:30:58.492] I [ timer  ] Timer: Total node execution : 31.679999489337206 sec
wclsFrameSaver saving cooked to 10000 ticks
wclsFrameSaver: saving 28135 traces tagged "gauss"
FrameSaver: q=8.58507e+06 n=689078 tag=gauss
wclsFrameSaver: saving 36299 traces tagged "wiener"
FrameSaver: q=9.27e+06 n=666412 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 841 XUs and 865 XVs -> 31 XUVs
C:0 T:1 324 XUs and 602 XVs -> 23 XUVs
C:0 T:2 745 XUs and 611 XVs -> 48 XUVs
C:0 T:3 2982 XUs and 3926 XVs -> 120 XUVs
C:0 T:4 28 XUs and 44 XVs -> 0 XUVs
C:0 T:5 488 XUs and 495 XVs -> 29 XUVs
C:0 T:6 17495 XUs and 17969 XVs -> 1459 XUVs
C:0 T:7 30892 XUs and 37345 XVs -> 3434 XUVs
5144 XUVs total
1678 collection wire objects
5144 potential space points
Neighbour search...
272218 tests to find 56638 neighbours
Iterating with no regularization...
Begin: 2.92398e+10
0 2.81333e+10
1 2.80343e+10
2 2.80333e+10
Now with regularization...
Begin: 2.76438e+10
0 2.76425e+10
18-Sep-2025 12:32:07 CEST  Closed output file "np02vd_raw_run039433_0367_df-s05-d2_dw_0_20250918T062521_reco_stage1_20250918T103207_keepup.root"

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                        68.4944       256.746       6515.42       103.177       925.168        47     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                      4.6207e-05    5.41117e-05   0.000137158   5.1717e-05    1.30458e-05      47     
produce:tpcrawdecoder:PDVDTPCReader               14.3026       40.9114       82.608        36.0126       15.8547        47     
produce:triggerrawdecoder:PDVDTriggerReader4     0.292101      0.388782      0.988481      0.343354      0.130546        47     
produce:pdvddaphne:DAPHNEReaderPDVD               5.47589       10.0354       18.9391       8.59083       3.6839         47     
produce:ophit:OpHitFinder                        0.0286079     0.0405625     0.0495008     0.0401369    0.00435965       47     
produce:opflash:OpFlashFinderVerticalDrift      0.00382883    0.00901207     0.0146776    0.00892231    0.00225362       47     
produce:wclsdatavd:WireCellToolkit                21.8688       33.9156       84.0022       27.7885       12.4638        47     
produce:gaushit:GausHitFinder                    0.294845      0.595031       2.00098      0.508008      0.286645        47     
produce:nhitsfilter:NumberOfHitsFilter           6.35e-05     0.000127026   0.000416723   0.000101471   6.56183e-05      47     
produce:reco3d:SpacePointSolver                   1.57729       6.28073       55.3589       4.24702       7.93497        47     
produce:hitpdune:DisambigFromSpacePoints         0.0180267     0.086265      0.886303      0.0468938     0.130815        47     
produce:pandora:StandardPandora                   2.96338       161.725       6282.05       10.4946       904.225        47     
produce:pandoraTrack:LArPandoraTrackCreation     0.0646403     0.387059       4.36588      0.207327      0.641949        47     
produce:pandoraGnocalo:GnocchiCalorimetry       0.00693147     0.014943      0.0716783     0.0124528    0.00969792       47     
[art]:TriggerResults:TriggerResultInserter      1.3325e-05    1.64829e-05   5.5554e-05    1.5138e-05    6.00103e-06      47     
end_path:out1:RootOutput                         3.586e-06    5.79474e-06   2.0218e-05     6.012e-06    2.43911e-06      47     
end_path:out1:RootOutput(write)                   1.97878       2.34142       3.30898       2.18933      0.336876        47     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 7465.85 MB
  Peak resident set size usage (VmHWM): 5588.39 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run039433_0367_df-s05-d2_dw_0_20250918T062521_reco_stage1_20250918T103207_keepup.root
\tHists: np02vd_raw_run039433_0367_df-s05-d2_dw_0_20250918T062521_reco_stage1_20250918T103207_keepup_hists.root
Forming reco metadata
Successfully opened file np02vd_raw_run039433_0367_df-s05-d2_dw_0_20250918T062521_reco_stage1_20250918T103207_keepup.root
Ran successfully
{
  "name": "np02vd_raw_run039433_0367_df-s05-d2_dw_0_20250918T062521_reco_stage1_20250918T103207_keepup.root",
  "namespace": "vd-protodune-det-reco",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_10_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
    "dune.campaign": "vd-protodune-reco-keepup-v0",
    "core.start_time": 1758191528.0,
    "core.end_time": 1758191528.0,
    "core.events": [
      105278,
      105284,
      105290,
      105296,
      105302,
      105308,
      105314,
      105320,
      105326,
      105332,
      105338,
      105344,
      105350,
      105356,
      105362,
      105368,
      105374,
      105380,
      105386,
      105392,
      105398,
      105404,
      105410,
      105416,
      105422,
      105428,
      105434,
      105440,
      105446,
      105452,
      105458,
      105464,
      105470,
      105476,
      105482,
      105488,
      105494,
      105500,
      105506,
      105512,
      105518,
      105524,
      105530,
      105536,
      105542,
      105548,
      105554
    ],
    "core.event_count": 47,
    "core.first_event_number": 105278,
    "core.last_event_number": 105554,
    "core.data_stream": "cosmics",
    "core.file_content_status": "good",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      39433
    ],
    "core.runs_subruns": [
      3943300001
    ],
    "dune.daq_test": false,
    "retention.status": "active",
    "retention.class": "physics"
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run039433_0367_df-s05-d2_dw_0_20250918T062521.hdf5"
    }
  ]
}Forming hist metadata
formed
{
  "name": "np02vd_raw_run039433_0367_df-s05-d2_dw_0_20250918T062521_reco_stage1_20250918T103207_keepup_hists.root",
  "namespace": "vd-protodune-det-reco",
  "metadata": {
    "core.file_format": "root",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_10_00d00",
    "core.data_tier": "root-tuple-virtual",
    "dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
    "dune.campaign": "vd-protodune-reco-keepup-v0",
    "core.start_time": 1758191528.0,
    "core.end_time": 1758191528.0,
    "core.data_stream": "cosmics",
    "core.file_content_status": "good",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      39433
    ],
    "core.runs_subruns": [
      3943300001
    ],
    "dune.daq_test": false,
    "retention.status": "active",
    "retention.class": "physics"
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run039433_0367_df-s05-d2_dw_0_20250918T062521.hdf5"
    }
  ]
}
justIN time: 2025-09-18 19:19:16 UTC       justIN version: 01.05.00