justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303742.2@dunegpschedd01.fnal.gov

Jobsub ID303742.2@dunegpschedd01.fnal.gov
Workflow ID12717
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-04 01:32:30
SiteNL_NIKHEF
EntryVIRGO_NL_NIKHEF_dissel
Last heartbeat2026-02-04 03:23:19
From worker nodeHostnamewn-pijl-023.farm.nikhef.nl
cpuinfoAMD EPYC 9754 128-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-04 01:33:30
Input filesvd-protodune:np02vd_raw_run042421_0086_df-s02-d0_dw_0_20260203T141731.hdf5
JobscriptExit code0
Real time1h (6447s)
CPU time48m (2897s = 44%)
Max RSS bytes2611351552 (2490 MiB)
Outputting started2026-02-04 03:20:59
Output files
Finished2026-02-04 03:23:19
Saved logsjustin-logs:303742.2-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

exception in looking up space point ptr... 0x468e67f0, 0x468ee180
Caught exception in looking up space point ptr... 0x468ee180, 0x5e0bdeb0
Caught exception in looking up space point ptr... 0x3320a2b0, 0x2500d720
Caught exception in looking up space point ptr... 0x2500d720, 0x3b09dbe0
Caught exception in looking up space point ptr... 0x4f243a30, 0x2500d720
Caught exception in looking up space point ptr... 0x2500d720, 0x1a698530
Caught exception in looking up space point ptr... 0x1a698530, 0x3320a2b0
Caught exception in looking up space point ptr... 0x5f0794a0, 0x5f079350
Caught exception in looking up space point ptr... 0x303cc3c0, 0x5f0794a0
Caught exception in looking up space point ptr... 0x4344cc90, 0x524e92f0
Caught exception in looking up space point ptr... 0x524e4700, 0x4344cc90
Caught exception in looking up space point ptr... 0x251aa090, 0x251a9e70
Caught exception in looking up space point ptr... 0x251a9e70, 0x251abe80
Caught exception in looking up space point ptr... 0x3fc6cb60, 0x1877ee00
Caught exception in looking up space point ptr... 0x25e26b40, 0x3fc6cb60
Caught exception in looking up space point ptr... 0x526dec50, 0x526dbe80
Caught exception in looking up space point ptr... 0x526dbe80, 0x526decf0
Caught exception in looking up space point ptr... 0x466d2b20, 0x304bafd0
Caught exception in looking up space point ptr... 0x304bafd0, 0x466d2b20
Caught exception in looking up space point ptr... 0x466d2b20, 0x47c66120
Caught exception in looking up space point ptr... 0x47c66120, 0x466d2b20
Caught exception in looking up space point ptr... 0x24dfa9e0, 0x365c9040
Caught exception in looking up space point ptr... 0x365c9040, 0x1dd69fa0
Caught exception in looking up space point ptr... 0x5f8c1da0, 0x2455c5e0
Caught exception in looking up space point ptr... 0x2455c5e0, 0x5f8c1da0
Caught exception in looking up space point ptr... 0x5f8c1da0, 0x18f26cc0
Caught exception in looking up space point ptr... 0x18f26cc0, 0x5f8c1da0
Caught exception in looking up space point ptr... 0x4359f040, 0x4f69d800
Caught exception in looking up space point ptr... 0x4f69d800, 0x4359f040
Caught exception in looking up space point ptr... 0x524e2de0, 0x3d29fb00
Caught exception in looking up space point ptr... 0x3d29fb00, 0x524e2de0
Caught exception in looking up space point ptr... 0x672dd2e0, 0x232d1d30
Caught exception in looking up space point ptr... 0x232d1d30, 0x672dd2e0
Caught exception in looking up space point ptr... 0x672dd2e0, 0x5d5b49c0
Caught exception in looking up space point ptr... 0x5d5b49c0, 0x672dd2e0
Caught exception in looking up space point ptr... 0x35c31070, 0x35c31390
Caught exception in looking up space point ptr... 0x35c31390, 0x35c31070
Caught exception in looking up space point ptr... 0x3fb96940, 0x3fb98c20
Caught exception in looking up space point ptr... 0x3fb98c20, 0x3fb91500
Caught exception in looking up space point ptr... 0x1be99e50, 0x4359f150
Caught exception in looking up space point ptr... 0x4359f150, 0x528dbe40
Caught exception in looking up space point ptr... 0x5f2b7df0, 0x5f2b8160
Caught exception in looking up space point ptr... 0x5f2b8160, 0x240e2640
Caught exception in looking up space point ptr... 0x62c02ca0, 0x5c058360
Caught exception in looking up space point ptr... 0x5c05e300, 0x62c02ca0
Caught exception in looking up space point ptr... 0x4c6857d0, 0x4a634d70
Caught exception in looking up space point ptr... 0x4a634d70, 0x4c6857d0
Caught exception in looking up space point ptr... 0x6a44e650, 0x437f3c00
Caught exception in looking up space point ptr... 0x437f3c00, 0x6a44e650
Caught exception in looking up space point ptr... 0x4f0fb3d0, 0x5f1d12d0
Caught exception in looking up space point ptr... 0x5f1d12d0, 0x238e6590
Caught exception in looking up space point ptr... 0x365c7f10, 0x3acc5a70
Caught exception in looking up space point ptr... 0x3acc5a70, 0x330b6860
Caught exception in looking up space point ptr... 0x24de1fd0, 0x24de2100
Caught exception in looking up space point ptr... 0x24de2100, 0x24de1ea0
Caught exception in looking up space point ptr... 0x5d712350, 0x5d712480
Caught exception in looking up space point ptr... 0x5d712480, 0x5d712220
Caught exception in looking up space point ptr... 0x3d1afd10, 0x3d1abca0
Caught exception in looking up space point ptr... 0x3d1abca0, 0x3d1afd10
Caught exception in looking up space point ptr... 0x447b7530, 0x42bf6bd0
Caught exception in looking up space point ptr... 0x42bf6bd0, 0x447b7530
Caught exception in looking up space point ptr... 0x3a247b60, 0x2021e530
Caught exception in looking up space point ptr... 0x2021e530, 0x3a247b60
Caught exception in looking up space point ptr... 0x60091be0, 0x5905fcf0
Caught exception in looking up space point ptr... 0x5905fcf0, 0x60091be0
++++>>>> total num hits: 70786, num free: 68953
04-Feb-2026 04:20:49 CET  Closed output file "np02vd_raw_run042421_0086_df-s02-d0_dw_0_20260203T141731_reco_stage0_20260204T032049_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         196.719       235.094       273.502       234.736       20.2164        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       8.0772e-05    9.99871e-05   0.000295536   8.8473e-05    4.12952e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                76.3282       122.196       146.218       126.626       16.7175        27     
produce:triggerrawdecoder:PDVDTriggerReader4      0.593177      0.778029       1.38118      0.707713      0.197851        27     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0176164     0.0204741     0.0839447     0.0179538     0.0124594       27     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0874382     0.0950016     0.140141      0.0884921     0.0114685       27     
produce:beamevent:BeamEvent                      0.000104057   0.000166387   0.00113973    0.000130597   0.000191459      27     
produce:pdvddaphne:DAPHNEReaderPDVD                9.69151       15.5897       19.6451       16.1015       2.03037        27     
produce:ophit:OpHitFinder                         0.0356206     0.0470591     0.057593      0.0479156    0.00580658       27     
produce:wclsdatavd:WireCellToolkit                 64.6328       75.7717       86.534        76.8982       6.35424        27     
produce:gaushit:GausHitFinder                     0.380805       1.11237       2.12391       1.17431      0.364617        27     
produce:nhitsfilter:NumberOfHitsFilter           0.000137117   0.000310439   0.000630212   0.000296688   0.00010588       27     
produce:reco3d:SpacePointSolver                    4.23376       11.2264       26.736        9.73164       4.95356        27     
produce:hitpdune:DisambigFromSpacePoints          0.0525364     0.125674      0.281727      0.121592      0.0506415       27     
produce:cluster3d:Cluster3D                        1.0621        2.80802       7.52238       2.78054       1.39884        27     
[art]:TriggerResults:TriggerResultInserter       2.3285e-05    3.72948e-05   0.000110958   3.5103e-05    1.68089e-05      27     
end_path:out1:RootOutput                          3.605e-06    5.80867e-06    2.634e-05     4.977e-06    4.19353e-06      27     
end_path:out1:RootOutput(write)                    4.21691       5.28469       6.48557       5.25722      0.590095        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4575.87 MB
  Peak resident set size usage (VmHWM): 2611.35 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042421_0086_df-s02-d0_dw_0_20260203T141731_reco_stage0_20260204T032049_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042421_0086_df-s02-d0_dw_0_20260203T141731_reco_stage0_20260204T032049_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042421_0086_df-s02-d0_dw_0_20260203T141731_reco_stage0_20260204T032049_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770168824.0,
    "core.end_time": 1770175251.0,
    "core.events": [
      13992,
      13998,
      14004,
      14010,
      14016,
      14022,
      14028,
      14034,
      14040,
      14046,
      14052,
      14058,
      14064,
      14070,
      14076,
      14082,
      14088,
      14094,
      14100,
      14106,
      14112,
      14118,
      14124,
      14130,
      14136,
      14142,
      14148
    ],
    "core.event_count": 27,
    "core.first_event_number": 13992,
    "core.last_event_number": 14148,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42421
    ],
    "core.runs_subruns": [
      4242100001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042421_0086_df-s02-d0_dw_0_20260203T141731.hdf5"
    }
  ]
}
justIN time: 2026-02-04 04:47:02 UTC       justIN version: 01.06.00