justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297923.145@dunegpschedd02.fnal.gov

Jobsub ID297923.145@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 23:00:26
SiteNL_NIKHEF
EntryVIRGO_NL_NIKHEF_dissel_multicore
Last heartbeat2026-02-04 02:20:32
From worker nodeHostnamewn-pijl-015.farm.nikhef.nl
cpuinfoAMD EPYC 9754 128-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-04 00:07:19
Input filesvd-protodune:np02vd_raw_run042433_0167_df-s04-d0_dw_0_20260203T194536.hdf5
JobscriptExit code0
Real time2h (7835s)
CPU time1h (3973s = 50%)
Max RSS bytes3573379072 (3407 MiB)
Outputting started2026-02-04 02:17:57
Output files
Finished2026-02-04 02:20:32
Saved logsjustin-logs:297923.145-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

0x27447bf0, 0x27448b90
Caught exception in looking up space point ptr... 0x7abc5770, 0x7abd33c0
Caught exception in looking up space point ptr... 0x7abd3230, 0x7abc5770
Caught exception in looking up space point ptr... 0x33c1f770, 0x33c11670
Caught exception in looking up space point ptr... 0x284268d0, 0x33c1f770
Caught exception in looking up space point ptr... 0x31d96700, 0x7f85e760
Caught exception in looking up space point ptr... 0x7f85e760, 0x31d96700
Caught exception in looking up space point ptr... 0x7152fa30, 0x715411e0
Caught exception in looking up space point ptr... 0x715411e0, 0x71553480
Caught exception in looking up space point ptr... 0x4ebfa8d0, 0x4ec0a780
Caught exception in looking up space point ptr... 0x4ec0a780, 0x4ec1a310
Caught exception in looking up space point ptr... 0x718c8f80, 0x718b8f40
Caught exception in looking up space point ptr... 0x718b8f40, 0x718a7600
Caught exception in looking up space point ptr... 0xa0ad4ea0, 0xa0ac6120
Caught exception in looking up space point ptr... 0xa0ac6120, 0xa0ab76c0
Caught exception in looking up space point ptr... 0x2a331700, 0x2a32fc90
Caught exception in looking up space point ptr... 0x2a32fc90, 0x2a3314a0
Caught exception in looking up space point ptr... 0x7f39f3b0, 0x31e068c0
Caught exception in looking up space point ptr... 0x31e068c0, 0x7f39f3b0
Caught exception in looking up space point ptr... 0x50ebd110, 0x50ebd480
Caught exception in looking up space point ptr... 0x50ebd480, 0x50ebd110
Caught exception in looking up space point ptr... 0x31d883b0, 0x31d973e0
Caught exception in looking up space point ptr... 0x31d973e0, 0x31d883b0
Caught exception in looking up space point ptr... 0xa3e98ef0, 0xa3e987b0
Caught exception in looking up space point ptr... 0xa3e987b0, 0xa3e98ef0
Caught exception in looking up space point ptr... 0x590ff670, 0x59100440
Caught exception in looking up space point ptr... 0x59100440, 0x43d25530
Caught exception in looking up space point ptr... 0x629b8d40, 0x629b8bb0
Caught exception in looking up space point ptr... 0x629b8bb0, 0x629b8d40
Caught exception in looking up space point ptr... 0x67aec3b0, 0x261d2480
Caught exception in looking up space point ptr... 0x261d2480, 0x67aec3b0
Caught exception in looking up space point ptr... 0x526e5390, 0x5270cf80
Caught exception in looking up space point ptr... 0x5270cf80, 0x526e5390
Caught exception in looking up space point ptr... 0x7f8ad110, 0xa3e7a050
Caught exception in looking up space point ptr... 0xa3e7a050, 0xa3e77a90
Caught exception in looking up space point ptr... 0xa3e77a90, 0xa3e78b20
Caught exception in looking up space point ptr... 0xa3e78b20, 0x7f8a2b20
Caught exception in looking up space point ptr... 0x52ad0490, 0x52ad1400
Caught exception in looking up space point ptr... 0x52ad1400, 0x52ad0490
Caught exception in looking up space point ptr... 0x4b6769f0, 0x4b676860
Caught exception in looking up space point ptr... 0x4b676860, 0x4b6765d0
Caught exception in looking up space point ptr... 0x5c0d4ff0, 0x5c0d4320
Caught exception in looking up space point ptr... 0x5c0d4320, 0x5c0d4ff0
Caught exception in looking up space point ptr... 0x337aa890, 0x337ac690
Caught exception in looking up space point ptr... 0x337ac690, 0x337aa890
Caught exception in looking up space point ptr... 0x81269090, 0x9a3c5bf0
Caught exception in looking up space point ptr... 0x9a3c5bf0, 0x812682e0
Caught exception in looking up space point ptr... 0x812682e0, 0x812694f0
Caught exception in looking up space point ptr... 0x812694f0, 0x81267350
Caught exception in looking up space point ptr... 0x27424a60, 0x230f0200
Caught exception in looking up space point ptr... 0x230f0200, 0x27424a60
Caught exception in looking up space point ptr... 0x87e608f0, 0x87e60a80
Caught exception in looking up space point ptr... 0x7102d390, 0x87e608f0
Caught exception in looking up space point ptr... 0x5e9027a0, 0x21872770
Caught exception in looking up space point ptr... 0x21872770, 0x5e901ea0
Caught exception in looking up space point ptr... 0x9c0f17a0, 0x67494950
Caught exception in looking up space point ptr... 0x67494950, 0x2e317c40
Caught exception in looking up space point ptr... 0x50ebf2d0, 0x50ebf5f0
Caught exception in looking up space point ptr... 0x50ebf5f0, 0x50ebf2d0
Caught exception in looking up space point ptr... 0xa3ebff10, 0x2744b250
Caught exception in looking up space point ptr... 0x2744b250, 0xa3ebff10
Caught exception in looking up space point ptr... 0xa3ebff10, 0x2745a710
Caught exception in looking up space point ptr... 0x2745a710, 0xa3ebff10
Caught exception in looking up space point ptr... 0xa3ec7630, 0xa3ec7970
Caught exception in looking up space point ptr... 0xa3ec7970, 0xa3ec62e0
Caught exception in looking up space point ptr... 0x4f36e020, 0x4f36e410
Caught exception in looking up space point ptr... 0x4f36e410, 0x4f36e020
++++>>>> total num hits: 513891, num free: 504149
04-Feb-2026 03:17:46 CET  Closed output file "np02vd_raw_run042433_0167_df-s04-d0_dw_0_20260203T194536_reco_stage0_20260204T021746_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         307.643       407.012       499.031        408.5        50.9314        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       9.0517e-05    0.000121176   0.00047798    0.000101102   8.45384e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                142.184       195.728       266.145       187.623       33.6619        19     
produce:triggerrawdecoder:PDVDTriggerReader4       0.64948      0.959415       1.52235      0.943217       0.21667        19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0175518     0.0241769     0.0375202     0.0237052    0.00643195       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0175177     0.0178392     0.018361      0.0178083    0.000238889      19     
produce:beamevent:BeamEvent                      0.000133001   0.00022229    0.00129353     0.0001592    0.000253425      19     
produce:pdvddaphne:DAPHNEReaderPDVD                11.5084       16.5065       21.9834       16.1763       2.99193        19     
produce:ophit:OpHitFinder                         0.0383373     0.0476866     0.059898      0.0488208    0.00571295       19     
produce:wclsdatavd:WireCellToolkit                 104.185       125.482       150.515       123.567       11.5673        19     
produce:gaushit:GausHitFinder                     0.821239       2.09921       3.73707       2.14517      0.795724        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000374074   0.00105484    0.00307428    0.000971014   0.00063388       19     
produce:reco3d:SpacePointSolver                    10.1931       15.6835       23.4388       15.2219       3.61654        19     
produce:hitpdune:DisambigFromSpacePoints          0.132487      0.339667      0.616859      0.334022      0.140792        19     
produce:cluster3d:Cluster3D                        1.92011       42.3705       83.4406       62.791        33.686         19     
[art]:TriggerResults:TriggerResultInserter       3.1037e-05    5.32354e-05   0.000206421   4.4938e-05    3.70415e-05      19     
end_path:out1:RootOutput                          3.846e-06    2.42955e-05   0.000293182    6.26e-06      6.423e-05       19     
end_path:out1:RootOutput(write)                    5.5968        7.66501       9.89415       7.52337       1.30063        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5605.94 MB
  Peak resident set size usage (VmHWM): 3573.38 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0167_df-s04-d0_dw_0_20260203T194536_reco_stage0_20260204T021746_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0167_df-s04-d0_dw_0_20260203T194536_reco_stage0_20260204T021746_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0167_df-s04-d0_dw_0_20260203T194536_reco_stage0_20260204T021746_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770163652.0,
    "core.end_time": 1770171468.0,
    "core.events": [
      19048,
      19054,
      19060,
      19066,
      19072,
      19078,
      19084,
      19090,
      19096,
      19102,
      19108,
      19114,
      19120,
      19126,
      19132,
      19138,
      19144,
      19150,
      19156
    ],
    "core.event_count": 19,
    "core.first_event_number": 19048,
    "core.last_event_number": 19156,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0167_df-s04-d0_dw_0_20260203T194536.hdf5"
    }
  ]
}
justIN time: 2026-02-04 06:00:34 UTC       justIN version: 01.06.00