justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297941.0@dunegpschedd02.fnal.gov

Jobsub ID297941.0@dunegpschedd02.fnal.gov
Workflow ID12718
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 23:42:26
SiteUK_Imperial
EntryDUNE_T2_UK_London_IC_ceprod02
Last heartbeat2026-02-04 01:10:33
From worker nodeHostnamewl05.grid.hep.ph.ic.ac.uk
cpuinfoAMD EPYC 7742 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-04 00:13:44
Input filesvd-protodune:np02vd_raw_run042422_0105_df-s03-d0_dw_0_20260203T161131.hdf5
JobscriptExit code0
Real time54m (3247s)
CPU time45m (2744s = 84%)
Max RSS bytes2601218048 (2480 MiB)
Outputting started2026-02-04 01:07:53
Output files
Finished2026-02-04 01:10:33
Saved logsjustin-logs:297941.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

x1fca4f90, 0xa71c930
Caught exception in looking up space point ptr... 0xa71c930, 0x1fca4f90
Caught exception in looking up space point ptr... 0x13b183a0, 0x3f7ab4f0
Caught exception in looking up space point ptr... 0x3f7ab4f0, 0x3f7ac910
Caught exception in looking up space point ptr... 0x11948440, 0x11948620
Caught exception in looking up space point ptr... 0x11948620, 0x119487b0
Caught exception in looking up space point ptr... 0x28d3a150, 0x3b568060
Caught exception in looking up space point ptr... 0x3b568060, 0x28d36160
Caught exception in looking up space point ptr... 0x28d364f0, 0x28d3a150
Caught exception in looking up space point ptr... 0xfb1b7a0, 0xfb0b170
Caught exception in looking up space point ptr... 0xfb0b170, 0xfb1b7a0
Caught exception in looking up space point ptr... 0x532dee20, 0x532de8d0
Caught exception in looking up space point ptr... 0x532de8d0, 0x532dee20
Caught exception in looking up space point ptr... 0x22099a90, 0x22099db0
Caught exception in looking up space point ptr... 0x22099db0, 0x22099c20
Caught exception in looking up space point ptr... 0x4172cc70, 0x47aaf210
Caught exception in looking up space point ptr... 0x47aaf210, 0x4054c460
Caught exception in looking up space point ptr... 0x4054c460, 0x334fba30
Caught exception in looking up space point ptr... 0x5262c110, 0x2d9f7b70
Caught exception in looking up space point ptr... 0x2d9f7b70, 0x5262c110
Caught exception in looking up space point ptr... 0x40b80d60, 0x350cebd0
Caught exception in looking up space point ptr... 0x350cebd0, 0x2b681500
Caught exception in looking up space point ptr... 0x22085510, 0x28d31f40
Caught exception in looking up space point ptr... 0x143e5bc0, 0x22085510
Caught exception in looking up space point ptr... 0x3f7cc890, 0x133cd7e0
Caught exception in looking up space point ptr... 0x133cd7e0, 0x3f7cc890
Caught exception in looking up space point ptr... 0x22cd4790, 0x22cd5de0
Caught exception in looking up space point ptr... 0x22cd5de0, 0x22cd7fc0
Caught exception in looking up space point ptr... 0x28efd010, 0x28efcbf0
Caught exception in looking up space point ptr... 0x28efd6f0, 0x28efd010
Caught exception in looking up space point ptr... 0x3b361d90, 0x3b361c00
Caught exception in looking up space point ptr... 0x3b361c00, 0x3b3620b0
Caught exception in looking up space point ptr... 0xa642400, 0x2a321010
Caught exception in looking up space point ptr... 0x2a321010, 0xa642400
Caught exception in looking up space point ptr... 0xa642400, 0x2cde9a80
Caught exception in looking up space point ptr... 0x2cde9a80, 0xa642400
Caught exception in looking up space point ptr... 0x31d72820, 0x3d664310
Caught exception in looking up space point ptr... 0x3d664310, 0x31d72820
Caught exception in looking up space point ptr... 0x12afc6b0, 0x12afc7b0
Caught exception in looking up space point ptr... 0x12afc7b0, 0x12afc6b0
Caught exception in looking up space point ptr... 0x12afc6b0, 0x2ce9ff50
Caught exception in looking up space point ptr... 0x2ce9ff50, 0x12afc6b0
Caught exception in looking up space point ptr... 0x514840f0, 0x51483fc0
Caught exception in looking up space point ptr... 0x51483fc0, 0x514840f0
Caught exception in looking up space point ptr... 0x514840f0, 0xececd50
Caught exception in looking up space point ptr... 0xececd50, 0x514840f0
Caught exception in looking up space point ptr... 0xe83fef0, 0xe83f0d0
Caught exception in looking up space point ptr... 0xe83f0d0, 0xe83fef0
Caught exception in looking up space point ptr... 0x4e4d8f90, 0x5d8a0ad0
Caught exception in looking up space point ptr... 0x5d8a0ad0, 0x4e4d8f90
Caught exception in looking up space point ptr... 0x532da7b0, 0x532da310
Caught exception in looking up space point ptr... 0x532da310, 0x532da7b0
Caught exception in looking up space point ptr... 0x31b4ac20, 0x7a488d0
Caught exception in looking up space point ptr... 0x7a488d0, 0x31b4ac20
Caught exception in looking up space point ptr... 0x4c10bfd0, 0x4abfec40
Caught exception in looking up space point ptr... 0x4abfec40, 0x4c10bfd0
Caught exception in looking up space point ptr... 0x3f7aaad0, 0x3f7aa7b0
Caught exception in looking up space point ptr... 0x3f7aa7b0, 0x3f7aaad0
Caught exception in looking up space point ptr... 0xd5ac0f0, 0x4805d960
Caught exception in looking up space point ptr... 0x4805d960, 0xd5ac0f0
Caught exception in looking up space point ptr... 0x28d2fcf0, 0x28d2fb60
Caught exception in looking up space point ptr... 0x28d2fb60, 0x28d2fcf0
Caught exception in looking up space point ptr... 0xff5f880, 0xd732210
Caught exception in looking up space point ptr... 0xd732210, 0xd7324d0
Caught exception in looking up space point ptr... 0xfb2a5e0, 0xfb2a2d0
Caught exception in looking up space point ptr... 0xfb2a2d0, 0xfb2a5e0
++++>>>> total num hits: 51726, num free: 50052
04-Feb-2026 01:07:42 GMT  Closed output file "np02vd_raw_run042422_0105_df-s03-d0_dw_0_20260203T161131_reco_stage0_20260204T010742_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         101.07        116.764       136.514        114.9        8.10044        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       5.9562e-05    0.000105195   0.000303019   8.1772e-05    5.25431e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                11.6684       12.2707       13.4267       12.1738      0.455694        27     
produce:triggerrawdecoder:PDVDTriggerReader4      0.484768      0.503131      0.532942      0.504101      0.0123241       27     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0163104     0.0165352     0.0172066     0.0165122    0.000185419      27     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0806223     0.0863736     0.098057      0.081787      0.0073215       27     
produce:beamevent:BeamEvent                      7.1152e-05    0.00013922     0.0009933    9.0702e-05    0.000171732      27     
produce:pdvddaphne:DAPHNEReaderPDVD                3.34062       3.73326       4.73817       3.67685      0.308953        27     
produce:ophit:OpHitFinder                         0.0263821     0.0387419     0.0482043     0.040199     0.00583333       27     
produce:wclsdatavd:WireCellToolkit                 72.3834       76.6449       99.5861       75.3539       5.35229        27     
produce:gaushit:GausHitFinder                     0.565962       1.12233       1.93767       1.06942      0.292522        27     
produce:nhitsfilter:NumberOfHitsFilter           0.000182365   0.000320822   0.000581927   0.00032323    8.23939e-05      27     
produce:reco3d:SpacePointSolver                    5.0884        12.8737       20.4822       12.1237       3.75189        27     
produce:hitpdune:DisambigFromSpacePoints          0.0490652     0.124432      0.209348       0.11982      0.0368753       27     
produce:cluster3d:Cluster3D                        1.00054       2.98717       5.81753       2.83116       1.13992        27     
[art]:TriggerResults:TriggerResultInserter        2.095e-05    2.93031e-05   7.3803e-05     2.693e-05    9.74723e-06      27     
end_path:out1:RootOutput                          4.09e-06     1.71941e-05   0.000238797    8.71e-06     4.36016e-05      27     
end_path:out1:RootOutput(write)                    5.16845        6.335        7.88254       6.39306      0.573807        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4512.52 MB
  Peak resident set size usage (VmHWM): 2601.22 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042422_0105_df-s03-d0_dw_0_20260203T161131_reco_stage0_20260204T010742_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042422_0105_df-s03-d0_dw_0_20260203T161131_reco_stage0_20260204T010742_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042422_0105_df-s03-d0_dw_0_20260203T161131_reco_stage0_20260204T010742_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770164037.0,
    "core.end_time": 1770167264.0,
    "core.events": [
      17024,
      17030,
      17036,
      17042,
      17048,
      17054,
      17060,
      17066,
      17072,
      17078,
      17084,
      17090,
      17096,
      17102,
      17108,
      17114,
      17120,
      17126,
      17132,
      17138,
      17144,
      17150,
      17156,
      17162,
      17168,
      17174,
      17180
    ],
    "core.event_count": 27,
    "core.first_event_number": 17024,
    "core.last_event_number": 17180,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42422
    ],
    "core.runs_subruns": [
      4242200001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042422_0105_df-s03-d0_dw_0_20260203T161131.hdf5"
    }
  ]
}
justIN time: 2026-02-04 04:27:11 UTC       justIN version: 01.06.00