justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 243389.92@dunegpschedd01.fnal.gov

Jobsub ID243389.92@dunegpschedd01.fnal.gov
Workflow ID9610
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-11-05 07:39:06
SiteNL_SURFsara
EntryDUNE_SurfSARA_arc03
Last heartbeat2025-11-05 11:26:34
From worker nodeHostnamewn-la-18.gina.surf.nl
cpuinfoAMD EPYC 9754 128-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-11-05 09:24:49
Input filesvd-protodune:np02vd_raw_run040361_0599_df-s04-d1_dw_0_20251105T051445.hdf5
JobscriptExit code0
Real time1h (7124s)
CPU time1h (4658s = 65%)
Max RSS bytes15285067776 (14576 MiB)
Outputting started2025-11-05 11:23:34
Output filesvd-protodune-det-reco:np02vd_raw_run040361_0599_df-s04-d1_dw_0_20251105T051445_reco_stage1_20251105T112318_keepup.root
Finished2025-11-05 11:26:34
Saved logsjustin-logs:243389.92-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

0sigproc0> EOS at call=45 anode=0 
[12:22:26.743] D [  glue  ] <FrameFanin:nfsp> EOS at call=45 with 8 
[12:22:26.743] D [  glue  ] frame sink sees EOS
[12:22:26.743] D [ pgraph ] <Pgrapher:> graph execution complete 
[12:22:26.743] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 12.29 sec
[12:22:26.743] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 11.32 sec
[12:22:26.743] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 10.69 sec
[12:22:26.743] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 10.53 sec
[12:22:26.743] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 10.5 sec
[12:22:26.743] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 10.43 sec
[12:22:26.743] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 10.05 sec
[12:22:26.743] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 9.33 sec
[12:22:26.743] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.26 sec
[12:22:26.743] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.25 sec
[12:22:26.743] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.25 sec
[12:22:26.743] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.25 sec
[12:22:26.743] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.02 sec
[12:22:26.743] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[12:22:26.743] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[12:22:26.743] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[12:22:26.743] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[12:22:26.743] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[12:22:26.743] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[12:22:26.743] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[12:22:26.743] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[12:22:26.743] I [ timer  ] Timer: WireCell::Gen::Retagger : 0 sec
[12:22:26.743] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[12:22:26.743] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[12:22:26.743] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[12:22:26.743] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[12:22:26.743] I [ timer  ] Timer: Total node execution : 86.17999937944114 sec
wclsFrameSaver saving cooked to 10000 ticks
wclsFrameSaver: saving 47307 traces tagged "gauss"
FrameSaver: q=1.24798e+07 n=990394 tag=gauss
wclsFrameSaver: saving 58887 traces tagged "wiener"
FrameSaver: q=1.34405e+07 n=947419 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 269 XUs and 401 XVs -> 18 XUVs
C:0 T:1 925 XUs and 1080 XVs -> 58 XUVs
C:0 T:2 66 XUs and 178 XVs -> 2 XUVs
C:0 T:3 78 XUs and 363 XVs -> 7 XUVs
C:0 T:4 544 XUs and 590 XVs -> 30 XUVs
C:0 T:5 15351 XUs and 18935 XVs -> 906 XUVs
C:0 T:6 173 XUs and 168 XVs -> 10 XUVs
C:0 T:7 6521 XUs and 8238 XVs -> 433 XUVs
C:0 T:8 66 XUs and 124 XVs -> 2 XUVs
C:0 T:9 1064 XUs and 1070 XVs -> 62 XUVs
C:0 T:10 341 XUs and 696 XVs -> 24 XUVs
C:0 T:11 878 XUs and 1104 XVs -> 59 XUVs
C:0 T:12 1550 XUs and 1852 XVs -> 69 XUVs
C:0 T:13 745 XUs and 930 XVs -> 55 XUVs
C:0 T:14 578 XUs and 548 XVs -> 41 XUVs
C:0 T:15 2459 XUs and 2570 XVs -> 153 XUVs
1929 XUVs total
1458 collection wire objects
1929 potential space points
Neighbour search...
37547 tests to find 10018 neighbours
Iterating with no regularization...
Begin: 2.34569e+10
0 2.33537e+10
1 2.33496e+10
Now with regularization...
Begin: 2.32675e+10
0 2.32674e+10
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
05-Nov-2025 12:23:18 CET  Closed output file "np02vd_raw_run040361_0599_df-s04-d1_dw_0_20251105T051445_reco_stage1_20251105T112318_keepup.root"

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                        188.423       306.58        1108.38       258.706       183.673        23     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                      6.2724e-05    8.39284e-05   0.000169062   8.0179e-05    2.08276e-05      23     
produce:tpcrawdecoder:PDVDTPCReader               80.4023        109.9        142.771       109.893       15.172         23     
produce:triggerrawdecoder:PDVDTriggerReader4     0.690071       2.24525       10.3875      0.818911       2.84897        23     
produce:pdvddaphne:DAPHNEReaderPDVD             0.000323302   0.000365006   0.000619613   0.000351694   5.65361e-05      23     
produce:ophit:OpHitFinder                       7.5682e-05    0.000109993   0.00067828    8.2433e-05    0.000121436      23     
produce:opflash:OpFlashFinderVerticalDrift      5.2488e-05    7.28992e-05   0.000277895   5.8749e-05    4.5035e-05       23     
produce:wclsdatavd:WireCellToolkit                48.6006       147.269       942.394       90.3725       181.209        23     
produce:gaushit:GausHitFinder                    0.883239       2.31369       16.9687       1.30103       3.53979        23     
produce:nhitsfilter:NumberOfHitsFilter          0.000211665   0.000497192    0.0028846    0.000271355   0.000653692      23     
produce:reco3d:SpacePointSolver                    8.84         13.9341       22.4971       14.1236       3.61413        21     
produce:hitpdune:DisambigFromSpacePoints         0.108555      0.198817      0.331857      0.193359      0.0570431       21     
produce:pandora:StandardPandora                   12.1266       27.235        59.8865       25.0681       11.9466        21     
produce:pandoraTrack:LArPandoraTrackCreation     0.473345       1.37023       2.43398       1.26128      0.537087        21     
produce:pandoraGnocalo:GnocchiCalorimetry        0.0205637     0.0341607     0.0558714     0.0320737     0.008232        21     
[art]:TriggerResults:TriggerResultInserter      2.9274e-05    4.80987e-05   0.000139028    4.025e-05    2.41265e-05      23     
end_path:out1:RootOutput                         5.939e-06    9.11687e-06   3.6885e-05     7.532e-06    6.05046e-06      23     
end_path:out1:RootOutput(write)                   3.92277       5.68981       11.1943       5.40617       1.43488        23     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 17209.9 MB
  Peak resident set size usage (VmHWM): 15285.1 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run040361_0599_df-s04-d1_dw_0_20251105T051445_reco_stage1_20251105T112318_keepup.root
\tHists: np02vd_raw_run040361_0599_df-s04-d1_dw_0_20251105T051445_reco_stage1_20251105T112319_keepup_hists.root
Forming reco metadata
Successfully opened file np02vd_raw_run040361_0599_df-s04-d1_dw_0_20251105T051445_reco_stage1_20251105T112318_keepup.root
Ran successfully
{
  "name": "np02vd_raw_run040361_0599_df-s04-d1_dw_0_20251105T051445_reco_stage1_20251105T112318_keepup.root",
  "namespace": "vd-protodune-det-reco",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_11_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
    "dune.campaign": "vd-protodune-reco-keepup-v0",
    "core.start_time": 1762341801.0,
    "core.end_time": 1762341801.0,
    "core.events": [
      55457,
      55461,
      55465,
      55469,
      55473,
      55477,
      55481,
      55485,
      55489,
      55493,
      55497,
      55501,
      55505,
      55509,
      55513,
      55517,
      55521,
      55525,
      55529,
      55533,
      55537,
      55541,
      55545
    ],
    "core.event_count": 23,
    "core.first_event_number": 55457,
    "core.last_event_number": 55545,
    "core.data_stream": "cosmics",
    "core.file_content_status": "good",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      40361
    ],
    "core.runs_subruns": [
      4036100001
    ],
    "dune.daq_test": true,
    "retention.status": "active",
    "retention.class": "physics"
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run040361_0599_df-s04-d1_dw_0_20251105T051445.hdf5"
    }
  ]
}Forming hist metadata
formed
{
  "name": "np02vd_raw_run040361_0599_df-s04-d1_dw_0_20251105T051445_reco_stage1_20251105T112319_keepup_hists.root",
  "namespace": "vd-protodune-det-reco",
  "metadata": {
    "core.file_format": "root",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_11_00d00",
    "core.data_tier": "root-tuple-virtual",
    "dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
    "dune.campaign": "vd-protodune-reco-keepup-v0",
    "core.start_time": 1762341801.0,
    "core.end_time": 1762341801.0,
    "core.data_stream": "cosmics",
    "core.file_content_status": "good",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      40361
    ],
    "core.runs_subruns": [
      4036100001
    ],
    "dune.daq_test": true,
    "retention.status": "active",
    "retention.class": "physics"
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run040361_0599_df-s04-d1_dw_0_20251105T051445.hdf5"
    }
  ]
}
justIN time: 2025-11-05 13:03:50 UTC       justIN version: 01.05.01