justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303654.63@dunegpschedd01.fnal.gov

Jobsub ID303654.63@dunegpschedd01.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:52:26
SiteUK_QMUL
EntryDUNE_UK_London_QMUL_arcce03
Last heartbeat2026-02-04 02:07:14
From worker nodeHostnamecn017.htc.esc.qmul
cpuinfoIntel(R) Xeon(R) Gold 6248R CPU @ 3.00GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-04 00:00:58
Input filesvd-protodune:np02vd_raw_run042433_0553_df-s04-d0_dw_0_20260203T221218.hdf5
JobscriptExit code0
Real time2h (7290s)
CPU time1h (4083s = 56%)
Max RSS bytes3501830144 (3339 MiB)
Outputting started2026-02-04 02:02:33
Output files
Finished2026-02-04 02:07:14
Saved logsjustin-logs:303654.63-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

nt ptr... 0x2d6d7cf0, 0xd199b10
Caught exception in looking up space point ptr... 0x181ea5e0, 0x31a8d020
Caught exception in looking up space point ptr... 0x31a8d020, 0x181ea5e0
Caught exception in looking up space point ptr... 0x2f223b50, 0x56170900
Caught exception in looking up space point ptr... 0x56170900, 0x14bc4590
Caught exception in looking up space point ptr... 0x3bb08f50, 0x6540d330
Caught exception in looking up space point ptr... 0x6540d330, 0x3bb08f50
Caught exception in looking up space point ptr... 0x14682680, 0x548414e0
Caught exception in looking up space point ptr... 0x548414e0, 0x14682680
Caught exception in looking up space point ptr... 0x32239b20, 0xe9f7990
Caught exception in looking up space point ptr... 0xe9f7990, 0x32239b20
Caught exception in looking up space point ptr... 0x11479780, 0x7124ef00
Caught exception in looking up space point ptr... 0x7124ef00, 0x11479780
Caught exception in looking up space point ptr... 0x5b013c10, 0x84592740
Caught exception in looking up space point ptr... 0x84592740, 0x3394ca50
Caught exception in looking up space point ptr... 0x18e12210, 0x18e0ff10
Caught exception in looking up space point ptr... 0x18e11890, 0x18e12210
Caught exception in looking up space point ptr... 0x314dc510, 0x55e9f660
Caught exception in looking up space point ptr... 0x55e9f660, 0x2c365eb0
Caught exception in looking up space point ptr... 0x191ff2b0, 0x5ba33770
Caught exception in looking up space point ptr... 0x5ba33770, 0x191ff2b0
Caught exception in looking up space point ptr... 0x191ff2b0, 0x733ad180
Caught exception in looking up space point ptr... 0x733ad180, 0x191ff2b0
Caught exception in looking up space point ptr... 0x191ff2b0, 0x733ad4a0
Caught exception in looking up space point ptr... 0x733ad4a0, 0x191ff2b0
Caught exception in looking up space point ptr... 0x54f4e350, 0x34bec440
Caught exception in looking up space point ptr... 0x34bec440, 0x54f4e350
Caught exception in looking up space point ptr... 0x54f4e350, 0x54f4e200
Caught exception in looking up space point ptr... 0x54f4e200, 0x54f4e350
Caught exception in looking up space point ptr... 0x54f4e350, 0x73356a60
Caught exception in looking up space point ptr... 0x73356a60, 0x54f4e350
Caught exception in looking up space point ptr... 0x28bf4950, 0x11a47430
Caught exception in looking up space point ptr... 0x11a47430, 0x37897da0
Caught exception in looking up space point ptr... 0x4f9a5bf0, 0x2bfd5f20
Caught exception in looking up space point ptr... 0x175f3340, 0x4f9a5bf0
Caught exception in looking up space point ptr... 0xf5c7a40, 0x3e098900
Caught exception in looking up space point ptr... 0x3e098900, 0x8541dcf0
Caught exception in looking up space point ptr... 0x6ce4fd10, 0x3b2354a0
Caught exception in looking up space point ptr... 0x14bdcbe0, 0x6ce4fd10
Caught exception in looking up space point ptr... 0x5714a0a0, 0x57330bd0
Caught exception in looking up space point ptr... 0x57330bd0, 0x5714a0a0
Caught exception in looking up space point ptr... 0x6e82b920, 0x6e82bbd0
Caught exception in looking up space point ptr... 0x6e82bbd0, 0x6e82b920
Caught exception in looking up space point ptr... 0x14d4fdf0, 0x64c06100
Caught exception in looking up space point ptr... 0x64c06100, 0x14d4fdf0
Caught exception in looking up space point ptr... 0x14d4fdf0, 0x2d754530
Caught exception in looking up space point ptr... 0x2d754530, 0x14d4fdf0
Caught exception in looking up space point ptr... 0x14d4fdf0, 0x2da9fd00
Caught exception in looking up space point ptr... 0x2da9fd00, 0x14d4fdf0
Caught exception in looking up space point ptr... 0x14d4fdf0, 0x2a87b4b0
Caught exception in looking up space point ptr... 0x2a87b4b0, 0x14d4fdf0
Caught exception in looking up space point ptr... 0x81481ea0, 0x787ddc0
Caught exception in looking up space point ptr... 0x33773210, 0x81481ea0
Caught exception in looking up space point ptr... 0x3438f740, 0x58956990
Caught exception in looking up space point ptr... 0x58956990, 0x58956e50
Caught exception in looking up space point ptr... 0x7503990, 0x18f20110
Caught exception in looking up space point ptr... 0x18f20110, 0x3231c350
Caught exception in looking up space point ptr... 0x53af32f0, 0x72054bf0
Caught exception in looking up space point ptr... 0x72054bf0, 0xe0c7ed0
Caught exception in looking up space point ptr... 0x12aba020, 0x6cf50570
Caught exception in looking up space point ptr... 0x6cf50570, 0x114a2b20
Caught exception in looking up space point ptr... 0x5b9fe020, 0x5b9fd9e0
Caught exception in looking up space point ptr... 0x5b9fd9e0, 0x5b9fe1b0
Caught exception in looking up space point ptr... 0x6a4152e0, 0x36546640
Caught exception in looking up space point ptr... 0x36546640, 0xc824960
Caught exception in looking up space point ptr... 0xdde0c10, 0x166f5840
Caught exception in looking up space point ptr... 0x166f5840, 0xdde2020
++++>>>> total num hits: 726227, num free: 709776
04-Feb-2026 02:02:16 GMT  Closed output file "np02vd_raw_run042433_0553_df-s04-d0_dw_0_20260203T221218_reco_stage0_20260204T020216_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         307.556       376.606       503.405       362.436       52.7135        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       8.3898e-05    0.000124147   0.000329009   9.9115e-05    5.94395e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                119.303       160.071       215.471       152.955       27.9631        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.590698      0.834066       1.51794      0.812967      0.221969        19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0164173     0.0193412     0.0289755     0.018122     0.00359707       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0164362     0.0168575     0.0174671     0.0168448    0.000306478      19     
produce:beamevent:BeamEvent                      0.000125169   0.000238108    0.0014123    0.000175546   0.000278946      19     
produce:pdvddaphne:DAPHNEReaderPDVD                14.764        17.2423       22.5057       17.2098       1.91525        19     
produce:ophit:OpHitFinder                         0.0358336     0.0535121     0.0717017     0.0540116     0.0101343       19     
produce:wclsdatavd:WireCellToolkit                 118.571       154.274       188.621       158.398       22.2713        19     
produce:gaushit:GausHitFinder                      1.10427       1.77746       3.44662       1.68738      0.563971        19     
produce:nhitsfilter:NumberOfHitsFilter           0.00056354    0.000781454   0.00125357    0.000737806   0.000160931      19     
produce:reco3d:SpacePointSolver                    12.3601       20.0862       29.1495       19.474        4.61403        19     
produce:hitpdune:DisambigFromSpacePoints          0.155284      0.291946       0.70936      0.267124      0.131418        19     
produce:cluster3d:Cluster3D                        1.24659       15.8268       152.539       2.23477       39.6026        19     
[art]:TriggerResults:TriggerResultInserter       2.6081e-05    3.94435e-05    9.621e-05    3.5385e-05    1.53402e-05      19     
end_path:out1:RootOutput                          6.617e-06    9.60189e-06   2.7305e-05     8.668e-06    4.56736e-06      19     
end_path:out1:RootOutput(write)                    4.25969       6.05943       11.1892       6.04738       1.7871         19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5584.85 MB
  Peak resident set size usage (VmHWM): 3501.83 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0553_df-s04-d0_dw_0_20260203T221218_reco_stage0_20260204T020216_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0553_df-s04-d0_dw_0_20260203T221218_reco_stage0_20260204T020216_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0553_df-s04-d0_dw_0_20260203T221218_reco_stage0_20260204T020216_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770163271.0,
    "core.end_time": 1770170543.0,
    "core.events": [
      63052,
      63058,
      63064,
      63070,
      63076,
      63082,
      63088,
      63094,
      63100,
      63106,
      63112,
      63118,
      63124,
      63130,
      63136,
      63142,
      63148,
      63154,
      63160
    ],
    "core.event_count": 19,
    "core.first_event_number": 63052,
    "core.last_event_number": 63160,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0553_df-s04-d0_dw_0_20260203T221218.hdf5"
    }
  ]
}
justIN time: 2026-02-04 04:31:40 UTC       justIN version: 01.06.00