justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303761.4@dunegpschedd01.fnal.gov

Jobsub ID303761.4@dunegpschedd01.fnal.gov
Workflow ID12717
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-04 01:56:31
SiteUK_Manchester
EntryUBoone_T2_UK_Manchester_ce01
Last heartbeat2026-02-04 04:10:20
From worker nodeHostnamewn2205251.tier2.hep.manchester.ac.uk
cpuinfoAMD EPYC 7513 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-04 01:57:48
Input filesvd-protodune:np02vd_raw_run042421_0162_df-s04-d1_dw_0_20260203T145838.hdf5
JobscriptExit code0
Real time2h (7785s)
CPU time51m (3067s = 39%)
Max RSS bytes2590375936 (2470 MiB)
Outputting started2026-02-04 04:07:35
Output files
Finished2026-02-04 04:10:20
Saved logsjustin-logs:303761.4-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

0x214f9ec0, 0x117d4d30
Caught exception in looking up space point ptr... 0x117d4d30, 0x214f9d70
Caught exception in looking up space point ptr... 0x214f9d70, 0x41d0a980
Caught exception in looking up space point ptr... 0x3aa6e470, 0x3aa6e2e0
Caught exception in looking up space point ptr... 0x3aa563a0, 0x3aa6e470
Caught exception in looking up space point ptr... 0x107ec780, 0x3643a450
Caught exception in looking up space point ptr... 0x3643a450, 0x107fb1d0
Caught exception in looking up space point ptr... 0x3643a350, 0x107fd060
Caught exception in looking up space point ptr... 0x107fd060, 0x107fb560
Caught exception in looking up space point ptr... 0x107fb560, 0x107ec780
Caught exception in looking up space point ptr... 0x2ac70e10, 0x2ac69600
Caught exception in looking up space point ptr... 0x2ac69600, 0x2ac69500
Caught exception in looking up space point ptr... 0x4625b910, 0x4f14a5f0
Caught exception in looking up space point ptr... 0x4f14a5f0, 0x3e0d7cc0
Caught exception in looking up space point ptr... 0x2c6ff140, 0x4be4f6a0
Caught exception in looking up space point ptr... 0x4be4f6a0, 0x6103c6e0
Caught exception in looking up space point ptr... 0xa426ab0, 0xa426810
Caught exception in looking up space point ptr... 0xa426810, 0xa426ab0
Caught exception in looking up space point ptr... 0x3196e440, 0x3196f400
Caught exception in looking up space point ptr... 0x2cc76860, 0x2cc62ff0
Caught exception in looking up space point ptr... 0x2cc62ff0, 0x3196e440
Caught exception in looking up space point ptr... 0x90b7830, 0x90b6470
Caught exception in looking up space point ptr... 0xb8d7d30, 0x90b7830
Caught exception in looking up space point ptr... 0x49a8a5a0, 0x4eb4f0b0
Caught exception in looking up space point ptr... 0x4eb4f0b0, 0x4eb522a0
Caught exception in looking up space point ptr... 0x2cc5ed10, 0x2cc60460
Caught exception in looking up space point ptr... 0x2cc60460, 0x2cc60ef0
Caught exception in looking up space point ptr... 0x14a8f040, 0x2cc5ed10
Caught exception in looking up space point ptr... 0x107dbaa0, 0x4bc6c920
Caught exception in looking up space point ptr... 0x4bc6c920, 0x107dbaa0
Caught exception in looking up space point ptr... 0xa40edf0, 0x216158b0
Caught exception in looking up space point ptr... 0x216158b0, 0xa41fb80
Caught exception in looking up space point ptr... 0x48d00530, 0x48d05c60
Caught exception in looking up space point ptr... 0x48d05a40, 0x48d00530
Caught exception in looking up space point ptr... 0x2fe9d480, 0x2fe9e1d0
Caught exception in looking up space point ptr... 0x2fe9e1d0, 0x2fe9d480
Caught exception in looking up space point ptr... 0x31f98eb0, 0x2ab20c30
Caught exception in looking up space point ptr... 0x2ab20c30, 0x97aeb70
Caught exception in looking up space point ptr... 0x21566d30, 0x1543e4c0
Caught exception in looking up space point ptr... 0x1543e4c0, 0x2155f210
Caught exception in looking up space point ptr... 0x35c84910, 0x35c85d80
Caught exception in looking up space point ptr... 0x35c86b20, 0x35c84910
Caught exception in looking up space point ptr... 0xa419a30, 0xa40f250
Caught exception in looking up space point ptr... 0xa40f250, 0xa419a30
Caught exception in looking up space point ptr... 0x35c742b0, 0x35c715c0
Caught exception in looking up space point ptr... 0x35c715c0, 0x35c742b0
Caught exception in looking up space point ptr... 0xb8a8bd0, 0xb8a2c70
Caught exception in looking up space point ptr... 0xb8a2c70, 0xb8a8d00
Caught exception in looking up space point ptr... 0xb8d9590, 0xb8d9400
Caught exception in looking up space point ptr... 0xb8d9400, 0xb8d98b0
Caught exception in looking up space point ptr... 0x4c67ff50, 0xb4f99c0
Caught exception in looking up space point ptr... 0xb4f99c0, 0x4c67ff50
Caught exception in looking up space point ptr... 0xa413bd0, 0xa4134a0
Caught exception in looking up space point ptr... 0xa4134a0, 0xa413240
Caught exception in looking up space point ptr... 0x3644b100, 0x36441810
Caught exception in looking up space point ptr... 0x36441810, 0x3644b100
Caught exception in looking up space point ptr... 0x3644b100, 0x3645bb00
Caught exception in looking up space point ptr... 0x3645bb00, 0x3644b100
Caught exception in looking up space point ptr... 0x3a51cb80, 0x14a92080
Caught exception in looking up space point ptr... 0x14a92080, 0x3a51cb80
Caught exception in looking up space point ptr... 0x45717b10, 0x62f81a90
Caught exception in looking up space point ptr... 0x41de0a20, 0x45717b10
Caught exception in looking up space point ptr... 0x39ed7870, 0x33a9f050
Caught exception in looking up space point ptr... 0x31daa580, 0x39ed7870
Caught exception in looking up space point ptr... 0x55e48690, 0x55e47780
Caught exception in looking up space point ptr... 0x55e47780, 0x55e48a20
++++>>>> total num hits: 55526, num free: 53691
04-Feb-2026 04:07:25 GMT  Closed output file "np02vd_raw_run042421_0162_df-s04-d1_dw_0_20260203T145838_reco_stage0_20260204T040725_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         225.447       283.972       339.332       288.835       29.4674        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       8.0972e-05    0.000126254   0.000276912   9.7143e-05    5.56186e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                109.662       158.149       196.034       161.773       25.4717        27     
produce:triggerrawdecoder:PDVDTriggerReader4      0.725361      0.981398       1.51078      0.985012      0.179798        27     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0218993     0.0279471     0.0604557     0.0229219    0.00867356       27     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.108113      0.110758      0.130796      0.109776     0.00423614       27     
produce:beamevent:BeamEvent                      0.000111049   0.000257703   0.00221903    0.000178036   0.000387435      27     
produce:pdvddaphne:DAPHNEReaderPDVD                10.0431       19.0888       23.9936       19.8946       3.47856        27     
produce:ophit:OpHitFinder                         0.0348508     0.042693      0.0478027     0.0436827    0.00354586       27     
produce:wclsdatavd:WireCellToolkit                 66.6875       85.2991       96.1544       85.4777       6.16339        27     
produce:gaushit:GausHitFinder                     0.590896       1.2758        1.81569       1.18726      0.298632        27     
produce:nhitsfilter:NumberOfHitsFilter           0.000151555   0.000313913   0.000699288   0.00030799    9.42339e-05      27     
produce:reco3d:SpacePointSolver                    5.41758       11.1845       18.5673       10.7701       3.30835        27     
produce:hitpdune:DisambigFromSpacePoints          0.0510842     0.125364      0.210966      0.128163      0.0392788       27     
produce:cluster3d:Cluster3D                        1.22747       2.6952        4.72625       2.6216       0.996616        27     
[art]:TriggerResults:TriggerResultInserter       1.8344e-05    3.35938e-05   0.000117281   3.1319e-05    1.77438e-05      27     
end_path:out1:RootOutput                          3.757e-06    1.13398e-05   3.2361e-05     9.788e-06    7.06906e-06      27     
end_path:out1:RootOutput(write)                    3.67636       4.96227       5.72919       5.01067      0.450054        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4542.72 MB
  Peak resident set size usage (VmHWM): 2590.38 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042421_0162_df-s04-d1_dw_0_20260203T145838_reco_stage0_20260204T040725_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042421_0162_df-s04-d1_dw_0_20260203T145838_reco_stage0_20260204T040725_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042421_0162_df-s04-d1_dw_0_20260203T145838_reco_stage0_20260204T040725_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770170283.0,
    "core.end_time": 1770178046.0,
    "core.events": [
      26327,
      26333,
      26339,
      26345,
      26351,
      26357,
      26363,
      26369,
      26375,
      26381,
      26387,
      26393,
      26399,
      26405,
      26411,
      26417,
      26423,
      26429,
      26435,
      26441,
      26447,
      26453,
      26459,
      26465,
      26471,
      26477,
      26483
    ],
    "core.event_count": 27,
    "core.first_event_number": 26327,
    "core.last_event_number": 26483,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42421
    ],
    "core.runs_subruns": [
      4242100001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042421_0162_df-s04-d1_dw_0_20260203T145838.hdf5"
    }
  ]
}
justIN time: 2026-02-04 04:32:08 UTC       justIN version: 01.06.00