justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297951.2@dunegpschedd02.fnal.gov

Jobsub ID297951.2@dunegpschedd02.fnal.gov
Workflow ID12718
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 23:58:26
SiteUK_Imperial
EntryDUNE_T2_UK_London_IC_ceprod03
Last heartbeat2026-02-04 01:16:46
From worker nodeHostnamewm00.grid.hep.ph.ic.ac.uk
cpuinfoAMD EPYC 7742 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-04 00:14:32
Input filesvd-protodune:np02vd_raw_run042422_0082_df-s03-d0_dw_0_20260203T155907.hdf5
JobscriptExit code0
Real time59m (3586s)
CPU time50m (3046s = 84%)
Max RSS bytes2566651904 (2447 MiB)
Outputting started2026-02-04 01:14:19
Output files
Finished2026-02-04 01:16:46
Saved logsjustin-logs:297951.2-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

98980
Caught exception in looking up space point ptr... 0x5f698980, 0x5f6988a0
Caught exception in looking up space point ptr... 0x3314e640, 0x3314dde0
Caught exception in looking up space point ptr... 0x3314dde0, 0x3314cf50
Caught exception in looking up space point ptr... 0x36685d40, 0x249722a0
Caught exception in looking up space point ptr... 0x59a1c1f0, 0x36685d40
Caught exception in looking up space point ptr... 0x3b870a80, 0x3b86e4e0
Caught exception in looking up space point ptr... 0x3b873f30, 0x3b870a80
Caught exception in looking up space point ptr... 0x158beb40, 0x158bea10
Caught exception in looking up space point ptr... 0x158bea10, 0x158beb40
Caught exception in looking up space point ptr... 0xd4eeb60, 0xd4ee840
Caught exception in looking up space point ptr... 0xd4ee840, 0xd4eeb60
Caught exception in looking up space point ptr... 0x35850b10, 0x35850970
Caught exception in looking up space point ptr... 0x35850970, 0x35850b10
Caught exception in looking up space point ptr... 0x177bd4b0, 0x177bd640
Caught exception in looking up space point ptr... 0x177bd640, 0x177bd320
Caught exception in looking up space point ptr... 0x11980190, 0x11988920
Caught exception in looking up space point ptr... 0x11988920, 0x11987990
Caught exception in looking up space point ptr... 0x3ea35bf0, 0x3ea35f30
Caught exception in looking up space point ptr... 0x3ea35f30, 0x3ea35bf0
Caught exception in looking up space point ptr... 0x62579680, 0x49b2b470
Caught exception in looking up space point ptr... 0x49b2b470, 0x8fd9880
Caught exception in looking up space point ptr... 0x8fd9880, 0x6257aa20
Caught exception in looking up space point ptr... 0x6257aa20, 0x62579680
Caught exception in looking up space point ptr... 0x52408a60, 0x52407a10
Caught exception in looking up space point ptr... 0x52407a10, 0x52408a60
Caught exception in looking up space point ptr... 0x177b3220, 0x2d105050
Caught exception in looking up space point ptr... 0x2d105050, 0x177b3220
Caught exception in looking up space point ptr... 0x10d48900, 0x464bab00
Caught exception in looking up space point ptr... 0x464bab00, 0x10d48900
Caught exception in looking up space point ptr... 0x3931e800, 0x3931e930
Caught exception in looking up space point ptr... 0x3931e930, 0x3931eb90
Caught exception in looking up space point ptr... 0x115f0b00, 0x115f0c30
Caught exception in looking up space point ptr... 0x5092b200, 0x115f0b00
Caught exception in looking up space point ptr... 0x241ff0f0, 0x2a92ce40
Caught exception in looking up space point ptr... 0x2a92ce40, 0x241ff0f0
Caught exception in looking up space point ptr... 0x39322610, 0x39320910
Caught exception in looking up space point ptr... 0x39320910, 0x39322610
Caught exception in looking up space point ptr... 0x39322610, 0x3931cec0
Caught exception in looking up space point ptr... 0x3931cec0, 0x39322610
Caught exception in looking up space point ptr... 0x158d5be0, 0x16b6a100
Caught exception in looking up space point ptr... 0x16b6a100, 0x16b6a290
Caught exception in looking up space point ptr... 0xace8890, 0xace8a70
Caught exception in looking up space point ptr... 0xace8a70, 0xace8890
Caught exception in looking up space point ptr... 0x8fcdfc0, 0x56f7b0e0
Caught exception in looking up space point ptr... 0x56f7b0e0, 0x8fcdfc0
Caught exception in looking up space point ptr... 0x3f80d0a0, 0x3f80cdb0
Caught exception in looking up space point ptr... 0x3f80cdb0, 0x3f80d0a0
Caught exception in looking up space point ptr... 0x2259a590, 0x225a4440
Caught exception in looking up space point ptr... 0x225a4440, 0x2259a590
Caught exception in looking up space point ptr... 0x4aafc280, 0x4093bf70
Caught exception in looking up space point ptr... 0x4093bf70, 0x4aafc280
Caught exception in looking up space point ptr... 0x23aad7c0, 0x23aad5f0
Caught exception in looking up space point ptr... 0x23abe770, 0x23aad7c0
Caught exception in looking up space point ptr... 0x5f69c780, 0x5f69d1b0
Caught exception in looking up space point ptr... 0x5f69d1b0, 0x5f69c780
Caught exception in looking up space point ptr... 0x23aba0b0, 0x23abd0b0
Caught exception in looking up space point ptr... 0x23ac2380, 0x23aba0b0
Caught exception in looking up space point ptr... 0x55fef6c0, 0x55fefee0
Caught exception in looking up space point ptr... 0x55fef960, 0x55fef6c0
Caught exception in looking up space point ptr... 0x2ab1ba90, 0x23fbe4d0
Caught exception in looking up space point ptr... 0x222f0bd0, 0x2ab1ba90
Caught exception in looking up space point ptr... 0x5f6a6710, 0x5f6a56c0
Caught exception in looking up space point ptr... 0x5f6a56c0, 0x5f6a6710
Caught exception in looking up space point ptr... 0x50dee150, 0x50de5ea0
Caught exception in looking up space point ptr... 0x50de5ea0, 0x50de6620
++++>>>> total num hits: 72906, num free: 70624
04-Feb-2026 01:14:09 GMT  Closed output file "np02vd_raw_run042422_0082_df-s03-d0_dw_0_20260203T155907_reco_stage0_20260204T011409_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         101.199       129.423       154.777       134.342       17.6334        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       5.3371e-05    0.000131391   0.000282835   0.000107974   5.90575e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                10.7049       13.6977       14.9625       13.8196      0.999555        27     
produce:triggerrawdecoder:PDVDTriggerReader4      0.480816      0.500277      0.517375      0.505715      0.012044        27     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0160052     0.0168394     0.0245632     0.0165428    0.00154036       27     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0808592     0.0880487     0.0991577     0.0828845    0.00747182       27     
produce:beamevent:BeamEvent                      6.0545e-05    0.000178952   0.00105067    0.000132941   0.000176952      27     
produce:pdvddaphne:DAPHNEReaderPDVD                3.22521       4.2859        5.4683        4.55971       0.60322        27     
produce:ophit:OpHitFinder                         0.0266255     0.0455787     0.0624548     0.046126     0.00973487       27     
produce:wclsdatavd:WireCellToolkit                 68.8124       87.368        106.134       93.268        14.8116        27     
produce:gaushit:GausHitFinder                     0.808556       1.32392       2.21494       1.22349      0.374495        27     
produce:nhitsfilter:NumberOfHitsFilter           0.000222611   0.000394496   0.00073482    0.000375571   0.00010504       27     
produce:reco3d:SpacePointSolver                    5.82772       11.9545       18.1249       12.1544       3.25876        27     
produce:hitpdune:DisambigFromSpacePoints          0.0699413      0.14089      0.321805      0.129928      0.0498256       27     
produce:cluster3d:Cluster3D                        1.1629        3.01885       7.54791       3.08071       1.3392         27     
[art]:TriggerResults:TriggerResultInserter        2.132e-05    3.29638e-05   8.4339e-05    3.2621e-05    1.12523e-05      27     
end_path:out1:RootOutput                          3.808e-06    1.52908e-05   0.000183427    8.847e-06    3.32908e-05      27     
end_path:out1:RootOutput(write)                    5.12024       6.95161       8.73544       7.29888       1.14023        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4480.24 MB
  Peak resident set size usage (VmHWM): 2566.65 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042422_0082_df-s03-d0_dw_0_20260203T155907_reco_stage0_20260204T011409_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042422_0082_df-s03-d0_dw_0_20260203T155907_reco_stage0_20260204T011409_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042422_0082_df-s03-d0_dw_0_20260203T155907_reco_stage0_20260204T011409_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770164084.0,
    "core.end_time": 1770167651.0,
    "core.events": [
      13298,
      13304,
      13310,
      13316,
      13322,
      13328,
      13334,
      13340,
      13346,
      13352,
      13358,
      13364,
      13370,
      13376,
      13382,
      13388,
      13394,
      13400,
      13406,
      13412,
      13418,
      13424,
      13430,
      13436,
      13442,
      13448,
      13454
    ],
    "core.event_count": 27,
    "core.first_event_number": 13298,
    "core.last_event_number": 13454,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42422
    ],
    "core.runs_subruns": [
      4242200001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042422_0082_df-s03-d0_dw_0_20260203T155907.hdf5"
    }
  ]
}
justIN time: 2026-02-04 10:29:47 UTC       justIN version: 01.06.00