justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 41739.12@dunegpschedd02.fnal.gov

Jobsub ID41739.12@dunegpschedd02.fnal.gov
Workflow ID2867
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-09-18 07:48:12
SiteNL_SURFsara
EntryDUNE_SurfSARA_arc03
Last heartbeat2025-09-18 09:11:43
From worker nodeHostnamewn-la-22.gina.surf.nl
cpuinfoAMD EPYC 9754 128-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-09-18 07:49:07
Input filesvd-protodune:np02vd_raw_run039433_0374_df-s05-d2_dw_0_20250918T064252.hdf5
JobscriptExit code0
Real time1h (4799s)
CPU time46m (2793s = 58%)
Max RSS bytes2557452288 (2438 MiB)
Outputting started2025-09-18 09:09:07
Output filesvd-protodune-det-reco:np02vd_raw_run039433_0374_df-s05-d2_dw_0_20250918T064252_reco_stage1_20250918T090850_keepup.root
Finished2025-09-18 09:11:43
Saved logsjustin-logs:41739.12-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

[11:08:37.092] D [sigproc ] <OmnibusSigProc:anode0sigproc0> EOS at call=95 anode=0 
[11:08:37.092] D [  glue  ] <FrameFanin:nfsp> EOS at call=95 with 8 
[11:08:37.092] D [  glue  ] frame sink sees EOS
[11:08:37.092] D [ pgraph ] <Pgrapher:> graph execution complete 
[11:08:37.092] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 8.96 sec
[11:08:37.092] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 8.79 sec
[11:08:37.092] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 7.63 sec
[11:08:37.092] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 7.37 sec
[11:08:37.092] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.16 sec
[11:08:37.092] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.16 sec
[11:08:37.092] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.16 sec
[11:08:37.092] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.16 sec
[11:08:37.092] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.01 sec
[11:08:37.092] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 0 sec
[11:08:37.092] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[11:08:37.092] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[11:08:37.092] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 0 sec
[11:08:37.092] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[11:08:37.092] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[11:08:37.092] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[11:08:37.092] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[11:08:37.092] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[11:08:37.092] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 0 sec
[11:08:37.092] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[11:08:37.092] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[11:08:37.092] I [ timer  ] Timer: WireCell::Gen::Retagger : 0 sec
[11:08:37.092] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[11:08:37.092] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[11:08:37.092] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 0 sec
[11:08:37.092] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[11:08:37.092] I [ timer  ] Timer: Total node execution : 33.39999998547137 sec
wclsFrameSaver saving cooked to 10000 ticks
wclsFrameSaver: saving 15108 traces tagged "gauss"
FrameSaver: q=6.65222e+06 n=447296 tag=gauss
wclsFrameSaver: saving 18787 traces tagged "wiener"
FrameSaver: q=7.0957e+06 n=438331 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 277 XUs and 338 XVs -> 11 XUVs
C:0 T:1 95 XUs and 71 XVs -> 1 XUVs
C:0 T:2 1091 XUs and 929 XVs -> 59 XUVs
C:0 T:3 240 XUs and 234 XVs -> 21 XUVs
C:0 T:4 158 XUs and 178 XVs -> 11 XUVs
C:0 T:5 8623 XUs and 12713 XVs -> 838 XUVs
C:0 T:6 269 XUs and 203 XVs -> 11 XUVs
C:0 T:7 4118 XUs and 4136 XVs -> 295 XUVs
1247 XUVs total
741 collection wire objects
1247 potential space points
Neighbour search...
41435 tests to find 9966 neighbours
Iterating with no regularization...
Begin: 3.67789e+10
0 3.62731e+10
1 3.62615e+10
Now with regularization...
Begin: 3.57806e+10
0 3.57797e+10
18-Sep-2025 11:08:50 CEST  Closed output file "np02vd_raw_run039433_0374_df-s05-d2_dw_0_20250918T064252_reco_stage1_20250918T090850_keepup.root"

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                        62.2947       98.3433       223.362       95.0252       26.493         48     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       6.094e-05    8.45373e-05   0.000258757    7.366e-05    3.53829e-05      48     
produce:tpcrawdecoder:PDVDTPCReader               21.2073       36.2258       69.8765       34.9447       9.13308        48     
produce:triggerrawdecoder:PDVDTriggerReader4     0.315988      0.393529      0.581968      0.377912      0.0653771       48     
produce:pdvddaphne:DAPHNEReaderPDVD               5.49805        9.234        14.1348       9.20601       1.86012        48     
produce:ophit:OpHitFinder                        0.0287995     0.0405528     0.0618879     0.0393671    0.00694525       48     
produce:opflash:OpFlashFinderVerticalDrift      0.00421415    0.00711578     0.0135448    0.00654645    0.00200675       48     
produce:wclsdatavd:WireCellToolkit                19.4791       29.3728       63.1482       26.2176       8.50821        48     
produce:gaushit:GausHitFinder                    0.242693      0.464003      0.815911      0.452884      0.119958        48     
produce:nhitsfilter:NumberOfHitsFilter          7.5362e-05    0.000152951   0.000674429   0.000121402   9.68026e-05      48     
produce:reco3d:SpacePointSolver                   1.52606       4.55275       11.2982       4.14514       1.99784        48     
produce:hitpdune:DisambigFromSpacePoints         0.0157978     0.0525791      0.16613      0.0417269     0.0293127       48     
produce:pandora:StandardPandora                   3.28381       15.7682       127.951       9.92018       19.3518        48     
produce:pandoraTrack:LArPandoraTrackCreation     0.0769557     0.224725      0.662174      0.193017      0.119968        48     
produce:pandoraGnocalo:GnocchiCalorimetry       0.00616312     0.010612      0.0203185    0.00944671    0.00313483       48     
[art]:TriggerResults:TriggerResultInserter      1.4882e-05    2.59476e-05   5.8117e-05     2.383e-05    8.26549e-06      48     
end_path:out1:RootOutput                         4.066e-06    6.26119e-06    2.628e-05     5.308e-06    3.78173e-06      48     
end_path:out1:RootOutput(write)                   1.68006       1.96634       2.68092       1.89194      0.258686        48     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4445.49 MB
  Peak resident set size usage (VmHWM): 2557.45 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run039433_0374_df-s05-d2_dw_0_20250918T064252_reco_stage1_20250918T090850_keepup.root
\tHists: np02vd_raw_run039433_0374_df-s05-d2_dw_0_20250918T064252_reco_stage1_20250918T090850_keepup_hists.root
Forming reco metadata
Successfully opened file np02vd_raw_run039433_0374_df-s05-d2_dw_0_20250918T064252_reco_stage1_20250918T090850_keepup.root
Ran successfully
{
  "name": "np02vd_raw_run039433_0374_df-s05-d2_dw_0_20250918T064252_reco_stage1_20250918T090850_keepup.root",
  "namespace": "vd-protodune-det-reco",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_10_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
    "dune.campaign": "vd-protodune-reco-keepup-v0",
    "core.start_time": 1758186531.0,
    "core.end_time": 1758186531.0,
    "core.events": [
      107276,
      107282,
      107288,
      107294,
      107300,
      107306,
      107312,
      107318,
      107324,
      107330,
      107336,
      107342,
      107348,
      107354,
      107360,
      107366,
      107372,
      107378,
      107384,
      107390,
      107396,
      107402,
      107408,
      107414,
      107420,
      107426,
      107432,
      107438,
      107444,
      107450,
      107456,
      107462,
      107468,
      107474,
      107480,
      107486,
      107492,
      107498,
      107504,
      107510,
      107516,
      107522,
      107528,
      107534,
      107540,
      107546,
      107552,
      107558
    ],
    "core.event_count": 48,
    "core.first_event_number": 107276,
    "core.last_event_number": 107558,
    "core.data_stream": "cosmics",
    "core.file_content_status": "good",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      39433
    ],
    "core.runs_subruns": [
      3943300001
    ],
    "dune.daq_test": false,
    "retention.status": "active",
    "retention.class": "physics"
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run039433_0374_df-s05-d2_dw_0_20250918T064252.hdf5"
    }
  ]
}Forming hist metadata
formed
{
  "name": "np02vd_raw_run039433_0374_df-s05-d2_dw_0_20250918T064252_reco_stage1_20250918T090850_keepup_hists.root",
  "namespace": "vd-protodune-det-reco",
  "metadata": {
    "core.file_format": "root",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_10_00d00",
    "core.data_tier": "root-tuple-virtual",
    "dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
    "dune.campaign": "vd-protodune-reco-keepup-v0",
    "core.start_time": 1758186531.0,
    "core.end_time": 1758186531.0,
    "core.data_stream": "cosmics",
    "core.file_content_status": "good",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      39433
    ],
    "core.runs_subruns": [
      3943300001
    ],
    "dune.daq_test": false,
    "retention.status": "active",
    "retention.class": "physics"
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run039433_0374_df-s05-d2_dw_0_20250918T064252.hdf5"
    }
  ]
}
justIN time: 2025-09-19 01:29:17 UTC       justIN version: 01.05.00