justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303656.54@dunegpschedd01.fnal.gov

Jobsub ID303656.54@dunegpschedd01.fnal.gov
Workflow ID12717
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:56:25
SiteUK_Imperial
EntryDUNE_T2_UK_London_IC_ceprod03
Last heartbeat2026-02-04 01:16:18
From worker nodeHostnamewl02.grid.hep.ph.ic.ac.uk
cpuinfoAMD EPYC 7742 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-04 00:12:02
Input filesvd-protodune:np02vd_raw_run042421_0087_df-s04-d1_dw_0_20260203T141805.hdf5
JobscriptExit code0
Real time1h (3684s)
CPU time52m (3160s = 85%)
Max RSS bytes2611380224 (2490 MiB)
Outputting started2026-02-04 01:13:28
Output files
Finished2026-02-04 01:16:18
Saved logsjustin-logs:303656.54-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

0x412f91a0
Caught exception in looking up space point ptr... 0x412f91a0, 0xdb19960
Caught exception in looking up space point ptr... 0xdb19960, 0x41309df0
Caught exception in looking up space point ptr... 0x41309df0, 0xdb19960
Caught exception in looking up space point ptr... 0x412ef000, 0x412eeec0
Caught exception in looking up space point ptr... 0x412eeec0, 0x412ef000
Caught exception in looking up space point ptr... 0x5622a2f0, 0x5622a1a0
Caught exception in looking up space point ptr... 0x5622a440, 0x5622a2f0
Caught exception in looking up space point ptr... 0x4586b1d0, 0x2aa47f60
Caught exception in looking up space point ptr... 0x2aa47f60, 0x4586b1d0
Caught exception in looking up space point ptr... 0x4586b1d0, 0x2aa47e90
Caught exception in looking up space point ptr... 0x2aa47e90, 0x4586b1d0
Caught exception in looking up space point ptr... 0x4586b1d0, 0x4586c7b0
Caught exception in looking up space point ptr... 0x4586c7b0, 0x4586b1d0
Caught exception in looking up space point ptr... 0x13b5fdb0, 0x13b5fcd0
Caught exception in looking up space point ptr... 0x13b5fcd0, 0x13b5fe90
Caught exception in looking up space point ptr... 0x3b296260, 0x3b295fc0
Caught exception in looking up space point ptr... 0x3b295fc0, 0x3b296260
Caught exception in looking up space point ptr... 0x7e39470, 0x8274820
Caught exception in looking up space point ptr... 0x8274820, 0x7e39470
Caught exception in looking up space point ptr... 0x7e39470, 0x82746f0
Caught exception in looking up space point ptr... 0x82746f0, 0x7e39470
Caught exception in looking up space point ptr... 0x4592e110, 0x45937a50
Caught exception in looking up space point ptr... 0x45937a50, 0x4592e110
Caught exception in looking up space point ptr... 0x2e830d70, 0x2e82ce90
Caught exception in looking up space point ptr... 0x2e82cfa0, 0x2e830d70
Caught exception in looking up space point ptr... 0x34f5a5a0, 0x34f406f0
Caught exception in looking up space point ptr... 0x34f406f0, 0x34f5a5a0
Caught exception in looking up space point ptr... 0x768ac50, 0x13992120
Caught exception in looking up space point ptr... 0x13992120, 0x768acf0
Caught exception in looking up space point ptr... 0x4d982580, 0x9af5830
Caught exception in looking up space point ptr... 0x9af5830, 0x4d982580
Caught exception in looking up space point ptr... 0x28ceaf70, 0x28cdb6e0
Caught exception in looking up space point ptr... 0x28cdb6e0, 0x28ceaf70
Caught exception in looking up space point ptr... 0x3c008180, 0x3c008370
Caught exception in looking up space point ptr... 0x3c008370, 0x3c008180
Caught exception in looking up space point ptr... 0x10191140, 0x10191320
Caught exception in looking up space point ptr... 0x10191320, 0x1018b6f0
Caught exception in looking up space point ptr... 0xcbbd020, 0x405ed0b0
Caught exception in looking up space point ptr... 0x405ed0b0, 0x405ed380
Caught exception in looking up space point ptr... 0x405ed380, 0x405d9820
Caught exception in looking up space point ptr... 0x3bda1730, 0x4131fac0
Caught exception in looking up space point ptr... 0x4131fac0, 0x3bda1730
Caught exception in looking up space point ptr... 0x3c00e030, 0x3c0061c0
Caught exception in looking up space point ptr... 0x3c0061c0, 0x3c00e030
Caught exception in looking up space point ptr... 0x42fe8a00, 0x56a47150
Caught exception in looking up space point ptr... 0x56a47150, 0x42fe8a00
Caught exception in looking up space point ptr... 0x42fe8a00, 0x56a472b0
Caught exception in looking up space point ptr... 0x56a472b0, 0x42fe8a00
Caught exception in looking up space point ptr... 0x3b7970e0, 0x10126b00
Caught exception in looking up space point ptr... 0x10126b00, 0xdb26c50
Caught exception in looking up space point ptr... 0x3c013e50, 0x3c013a00
Caught exception in looking up space point ptr... 0x3c013a00, 0x3c013e50
Caught exception in looking up space point ptr... 0x53cd0ea0, 0x560d9b70
Caught exception in looking up space point ptr... 0x455fdb10, 0x53cd0ea0
Caught exception in looking up space point ptr... 0x32c91450, 0x3fe04ae0
Caught exception in looking up space point ptr... 0x3fe04ae0, 0x32c91450
Caught exception in looking up space point ptr... 0x3c000280, 0x3c00a6b0
Caught exception in looking up space point ptr... 0x3c00a6b0, 0x3c00a800
Caught exception in looking up space point ptr... 0x47d6d940, 0x564d5e00
Caught exception in looking up space point ptr... 0x564d5e00, 0x564d5f90
Caught exception in looking up space point ptr... 0x24973ee0, 0x51671e10
Caught exception in looking up space point ptr... 0x51671e10, 0x490cabc0
Caught exception in looking up space point ptr... 0x490cabc0, 0x516728a0
Caught exception in looking up space point ptr... 0xdf334a0, 0x53de5f90
Caught exception in looking up space point ptr... 0x53de5f90, 0x2f6dc680
++++>>>> total num hits: 62650, num free: 60600
04-Feb-2026 01:13:18 GMT  Closed output file "np02vd_raw_run042421_0087_df-s04-d1_dw_0_20260203T141805_reco_stage0_20260204T011318_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         101.609       132.965       170.939       129.653       16.1259        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       6.9092e-05    0.000157128   0.000347651   0.000164946   6.89367e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                10.863        12.1083       21.5342       11.8237       1.90535        27     
produce:triggerrawdecoder:PDVDTriggerReader4      0.486838      0.510774      0.641843      0.505823      0.0299708       27     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0161996     0.0248595      0.23953      0.0166112     0.0421009       27     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0801766     0.0840708     0.0986478     0.0817966     0.0057828       27     
produce:beamevent:BeamEvent                      9.4373e-05    0.000207343   0.00181135    0.000134704   0.000316983      27     
produce:pdvddaphne:DAPHNEReaderPDVD                3.54844       4.31378       5.07379       4.41228      0.438255        27     
produce:ophit:OpHitFinder                         0.030326      0.0471939     0.0591157     0.0477094    0.00824043       27     
produce:wclsdatavd:WireCellToolkit                 71.9868       90.1169       107.825       87.9049       11.2706        27     
produce:gaushit:GausHitFinder                     0.765575       1.37653       2.28393       1.28844      0.377977        27     
produce:nhitsfilter:NumberOfHitsFilter           0.000219697   0.000394163    0.0010397    0.000374532   0.000154215      27     
produce:reco3d:SpacePointSolver                    5.63068       14.0177       30.5056       13.0721       5.35817        27     
produce:hitpdune:DisambigFromSpacePoints          0.0773408     0.148761      0.275139      0.145308      0.0444333       27     
produce:cluster3d:Cluster3D                        1.30505       3.27621       9.92564       2.89199       1.5472         27     
[art]:TriggerResults:TriggerResultInserter       2.5971e-05    4.59548e-05   0.000246557   3.4961e-05    4.28618e-05      27     
end_path:out1:RootOutput                          5.391e-06    1.39977e-05   7.0422e-05     1.12e-05     1.28876e-05      27     
end_path:out1:RootOutput(write)                    5.29772       6.91231       9.00734       6.9655        0.93619        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4515.12 MB
  Peak resident set size usage (VmHWM): 2611.38 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042421_0087_df-s04-d1_dw_0_20260203T141805_reco_stage0_20260204T011318_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042421_0087_df-s04-d1_dw_0_20260203T141805_reco_stage0_20260204T011318_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042421_0087_df-s04-d1_dw_0_20260203T141805_reco_stage0_20260204T011318_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770163936.0,
    "core.end_time": 1770167600.0,
    "core.events": [
      14159,
      14165,
      14171,
      14177,
      14183,
      14189,
      14195,
      14201,
      14207,
      14213,
      14219,
      14225,
      14231,
      14237,
      14243,
      14249,
      14255,
      14261,
      14267,
      14273,
      14279,
      14285,
      14291,
      14297,
      14303,
      14309,
      14315
    ],
    "core.event_count": 27,
    "core.first_event_number": 14159,
    "core.last_event_number": 14315,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42421
    ],
    "core.runs_subruns": [
      4242100001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042421_0087_df-s04-d1_dw_0_20260203T141805.hdf5"
    }
  ]
}
justIN time: 2026-02-04 06:06:28 UTC       justIN version: 01.06.00