justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303654.26@dunegpschedd01.fnal.gov

Jobsub ID303654.26@dunegpschedd01.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:52:26
SiteUK_Imperial
EntryDUNE_T2_UK_London_IC_ceprod01
Last heartbeat2026-02-04 01:58:29
From worker nodeHostnamebf1770092109.novalocal
cpuinfoAMD EPYC-Genoa Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:51:36
Input filesvd-protodune:np02vd_raw_run042433_0182_df-s04-d0_dw_0_20260203T195119.hdf5
JobscriptExit code0
Real time2h (7307s)
CPU time1h (3797s = 51%)
Max RSS bytes3581997056 (3416 MiB)
Outputting started2026-02-04 01:53:26
Output files
Finished2026-02-04 01:58:29
Saved logsjustin-logs:303654.26-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

point ptr... 0x31c99520, 0x6f385200
Caught exception in looking up space point ptr... 0x16c476e0, 0x31c99520
Caught exception in looking up space point ptr... 0x16ea24d0, 0x18770110
Caught exception in looking up space point ptr... 0x18770110, 0x18770360
Caught exception in looking up space point ptr... 0x1873cdc0, 0x1330ae20
Caught exception in looking up space point ptr... 0x18760c00, 0x1873cdc0
Caught exception in looking up space point ptr... 0x13bd49c0, 0x13be42e0
Caught exception in looking up space point ptr... 0x13be42e0, 0x13bd49c0
Caught exception in looking up space point ptr... 0x905b8e0, 0x905bc50
Caught exception in looking up space point ptr... 0x905bc50, 0x905bfc0
Caught exception in looking up space point ptr... 0x385bd2c0, 0x74244530
Caught exception in looking up space point ptr... 0x74244530, 0x478af080
Caught exception in looking up space point ptr... 0x8b9f95f0, 0x8b9ed520
Caught exception in looking up space point ptr... 0x8b9ed520, 0x8b9ed200
Caught exception in looking up space point ptr... 0x440a8dd0, 0x15f7e050
Caught exception in looking up space point ptr... 0x15f7e050, 0x44167360
Caught exception in looking up space point ptr... 0x661ebad0, 0x40549180
Caught exception in looking up space point ptr... 0x40549180, 0x40541f80
Caught exception in looking up space point ptr... 0x6f35a6d0, 0x31c88df0
Caught exception in looking up space point ptr... 0x31c88df0, 0x31c88b90
Caught exception in looking up space point ptr... 0x6f355c10, 0x16ca5dc0
Caught exception in looking up space point ptr... 0x16ca5dc0, 0x16db06a0
Caught exception in looking up space point ptr... 0x40206420, 0x51778d20
Caught exception in looking up space point ptr... 0x402065b0, 0x40206420
Caught exception in looking up space point ptr... 0x11b03ec0, 0x11b03d30
Caught exception in looking up space point ptr... 0x11b03d30, 0x11ad0a50
Caught exception in looking up space point ptr... 0x61818250, 0x61817da0
Caught exception in looking up space point ptr... 0x61818570, 0x61818250
Caught exception in looking up space point ptr... 0x4f686c00, 0x4f6747d0
Caught exception in looking up space point ptr... 0x4f6747d0, 0x4f663980
Caught exception in looking up space point ptr... 0x1898ad90, 0x18964440
Caught exception in looking up space point ptr... 0x1898af20, 0x1898ad90
Caught exception in looking up space point ptr... 0x47e33140, 0x47e251d0
Caught exception in looking up space point ptr... 0x47e251d0, 0x47e332d0
Caught exception in looking up space point ptr... 0x15bc7a40, 0x185e89d0
Caught exception in looking up space point ptr... 0x185e89d0, 0x15bd4b70
Caught exception in looking up space point ptr... 0x814aa160, 0x814ba830
Caught exception in looking up space point ptr... 0x8149af30, 0x814aa160
Caught exception in looking up space point ptr... 0xdc8d260, 0x86374c80
Caught exception in looking up space point ptr... 0x86374c80, 0x5be9b010
Caught exception in looking up space point ptr... 0x5be9b010, 0x5beb4ea0
Caught exception in looking up space point ptr... 0x16dadb10, 0x16df37d0
Caught exception in looking up space point ptr... 0x16df3530, 0x16dadb10
Caught exception in looking up space point ptr... 0x38eda1b0, 0xd096500
Caught exception in looking up space point ptr... 0xd096500, 0x78a86290
Caught exception in looking up space point ptr... 0x81a45c80, 0x6626d7c0
Caught exception in looking up space point ptr... 0x6626d7c0, 0x81a6e650
Caught exception in looking up space point ptr... 0x6627e5b0, 0x81abc490
Caught exception in looking up space point ptr... 0x81abc490, 0x16834120
Caught exception in looking up space point ptr... 0x13bdac50, 0x13bb51d0
Caught exception in looking up space point ptr... 0x13bb51d0, 0x13bdac50
Caught exception in looking up space point ptr... 0x8b46b70, 0x8b2b1e0
Caught exception in looking up space point ptr... 0x8b2b1e0, 0x8b4fcf0
Caught exception in looking up space point ptr... 0x18ea11a0, 0x18ea1cb0
Caught exception in looking up space point ptr... 0x18ea1cb0, 0x18ea11a0
Caught exception in looking up space point ptr... 0x3c22d100, 0x355d1e30
Caught exception in looking up space point ptr... 0x355d1e30, 0x3c22d100
Caught exception in looking up space point ptr... 0x18e8adf0, 0x18e8af90
Caught exception in looking up space point ptr... 0x18e8af90, 0x18e8ac90
Caught exception in looking up space point ptr... 0x7fa59880, 0x7db2bba0
Caught exception in looking up space point ptr... 0x7db2bba0, 0x74255490
Caught exception in looking up space point ptr... 0x1025e070, 0x1025de90
Caught exception in looking up space point ptr... 0x1025de90, 0x1025dd00
Caught exception in looking up space point ptr... 0x37b72740, 0x7dbcff10
Caught exception in looking up space point ptr... 0x7dbcff10, 0x37b72740
Caught exception in looking up space point ptr... 0x18ea8a40, 0x18ea9a20
Caught exception in looking up space point ptr... 0x18ea9a20, 0x18ea8a40
++++>>>> total num hits: 531008, num free: 521719
04-Feb-2026 01:53:12 GMT  Closed output file "np02vd_raw_run042433_0182_df-s04-d0_dw_0_20260203T195119_reco_stage0_20260204T015312_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         281.705       379.725       463.439       378.118       39.8394        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       7.2028e-05    0.00017462    0.000684248    9.326e-05    0.000163028      19     
produce:tpcrawdecoder:PDVDTPCReader                 131.4        180.727       266.495       171.542       37.5946        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.661276      0.861229       1.16501      0.831271      0.135447        19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0182854     0.0234259     0.035484      0.0230772    0.00410061       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0180224     0.018852      0.0211833     0.0187256    0.000747705      19     
produce:beamevent:BeamEvent                      8.4807e-05    0.00022663     0.0013133    0.000124347   0.000270529      19     
produce:pdvddaphne:DAPHNEReaderPDVD                12.4524       16.0853        21.25        15.5273       2.71215        19     
produce:ophit:OpHitFinder                         0.0273701     0.0466256     0.0591564     0.0470606    0.00839861       19     
produce:wclsdatavd:WireCellToolkit                 108.302       116.138       124.574       115.145       4.71191        19     
produce:gaushit:GausHitFinder                      1.00449       1.92134       3.33446       2.03949      0.615009        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000202945   0.000319887   0.000622025   0.000290616   9.77509e-05      19     
produce:reco3d:SpacePointSolver                    8.85731       18.621        27.7054       19.3982       5.23324        19     
produce:hitpdune:DisambigFromSpacePoints           0.10123      0.273335       0.49282      0.290209      0.096998        19     
produce:cluster3d:Cluster3D                        1.41949       37.8165       90.192        50.9701       28.3371        19     
[art]:TriggerResults:TriggerResultInserter       2.2284e-05    2.94511e-05   7.2799e-05    2.5949e-05    1.10104e-05      19     
end_path:out1:RootOutput                          3.716e-06    3.9316e-05    0.000646952    4.878e-06    0.000143247      19     
end_path:out1:RootOutput(write)                    5.43234       7.13924       10.6768       7.12394       1.32723        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5544.35 MB
  Peak resident set size usage (VmHWM): 3582 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0182_df-s04-d0_dw_0_20260203T195119_reco_stage0_20260204T015312_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0182_df-s04-d0_dw_0_20260203T195119_reco_stage0_20260204T015312_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0182_df-s04-d0_dw_0_20260203T195119_reco_stage0_20260204T015312_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770162712.0,
    "core.end_time": 1770169995.0,
    "core.events": [
      20758,
      20764,
      20770,
      20776,
      20782,
      20788,
      20794,
      20800,
      20806,
      20812,
      20818,
      20824,
      20830,
      20836,
      20842,
      20848,
      20854,
      20860,
      20866
    ],
    "core.event_count": 19,
    "core.first_event_number": 20758,
    "core.last_event_number": 20866,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0182_df-s04-d0_dw_0_20260203T195119.hdf5"
    }
  ]
}
justIN time: 2026-02-04 06:06:43 UTC       justIN version: 01.06.00