justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297921.74@dunegpschedd02.fnal.gov

Jobsub ID297921.74@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:58:26
SiteUK_Manchester
EntryUBoone_T2_UK_Manchester_ce01
Last heartbeat2026-02-04 02:58:35
From worker nodeHostnamewn1907291.tier2.hep.manchester.ac.uk
cpuinfoIntel(R) Xeon(R) Gold 5118 CPU @ 2.30GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:51:14
Input filesvd-protodune:np02vd_raw_run042433_0231_df-s02-d0_dw_0_20260203T200954.hdf5
JobscriptExit code0
Real time3h (11058s)
CPU time1h (6076s = 54%)
Max RSS bytes3354558464 (3199 MiB)
Outputting started2026-02-04 02:55:33
Output files
Finished2026-02-04 02:58:35
Saved logsjustin-logs:297921.74-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

... 0x3a71d810, 0x414db670
Caught exception in looking up space point ptr... 0xcc62d90, 0xcc62a70
Caught exception in looking up space point ptr... 0xcc62a70, 0xcc62d90
Caught exception in looking up space point ptr... 0x7e254100, 0x7e25f1e0
Caught exception in looking up space point ptr... 0x7e270f50, 0x7e254100
Caught exception in looking up space point ptr... 0x81363860, 0x81364bb0
Caught exception in looking up space point ptr... 0x81364bb0, 0x81363860
Caught exception in looking up space point ptr... 0x7e260480, 0x7e2602a0
Caught exception in looking up space point ptr... 0x7e2602a0, 0x7e25c8f0
Caught exception in looking up space point ptr... 0x12a13770, 0x12a13590
Caught exception in looking up space point ptr... 0x12a13590, 0x12a13770
Caught exception in looking up space point ptr... 0x7ae52140, 0x7ae51fb0
Caught exception in looking up space point ptr... 0x7ae51fb0, 0x7ae52460
Caught exception in looking up space point ptr... 0x7ae52460, 0x7ae525f0
Caught exception in looking up space point ptr... 0x7ae525f0, 0x7ae522d0
Caught exception in looking up space point ptr... 0x7bb2e6d0, 0x7bb2e9f0
Caught exception in looking up space point ptr... 0x7bb2e9f0, 0x7bb2e860
Caught exception in looking up space point ptr... 0x7bb2e860, 0x7bb2e540
Caught exception in looking up space point ptr... 0x6b4898e0, 0x62d964c0
Caught exception in looking up space point ptr... 0x62d964c0, 0x6b4898e0
Caught exception in looking up space point ptr... 0x178ab300, 0x371fa890
Caught exception in looking up space point ptr... 0x371fa890, 0x178ab300
Caught exception in looking up space point ptr... 0x14f14660, 0x14f18120
Caught exception in looking up space point ptr... 0x14f18120, 0x10802110
Caught exception in looking up space point ptr... 0x7bb295b0, 0x7bb2d730
Caught exception in looking up space point ptr... 0x7bb2d730, 0x7bb295b0
Caught exception in looking up space point ptr... 0x7bb295b0, 0x568a6420
Caught exception in looking up space point ptr... 0x568a6420, 0x7bb295b0
Caught exception in looking up space point ptr... 0x7bb2f670, 0x7bb2f990
Caught exception in looking up space point ptr... 0x7bb2f990, 0x7bb2f670
Caught exception in looking up space point ptr... 0x6bfbf130, 0x5f8c5dd0
Caught exception in looking up space point ptr... 0x5f8c5dd0, 0x6bfbf130
Caught exception in looking up space point ptr... 0x3fb9efd0, 0x3fba4af0
Caught exception in looking up space point ptr... 0x3fba4af0, 0x3fb9efd0
Caught exception in looking up space point ptr... 0x3cb861e0, 0x3cb85e70
Caught exception in looking up space point ptr... 0x3cb85e70, 0x3cb861e0
Caught exception in looking up space point ptr... 0x321e74a0, 0x32102fa0
Caught exception in looking up space point ptr... 0x597e60a0, 0x321e74a0
Caught exception in looking up space point ptr... 0x2b68a090, 0x2b691b30
Caught exception in looking up space point ptr... 0x2b691b30, 0x2b68a270
Caught exception in looking up space point ptr... 0x51b40f20, 0x51b322c0
Caught exception in looking up space point ptr... 0x51b322c0, 0x51b40f20
Caught exception in looking up space point ptr... 0x178a6d90, 0x1789f1e0
Caught exception in looking up space point ptr... 0x1789f1e0, 0x178a4800
Caught exception in looking up space point ptr... 0x36c7d1c0, 0x7b7b2110
Caught exception in looking up space point ptr... 0x7b7b2110, 0x36c7d1c0
Caught exception in looking up space point ptr... 0x62c5d950, 0x62c60d00
Caught exception in looking up space point ptr... 0x62c60d00, 0x62c60950
Caught exception in looking up space point ptr... 0x62c60950, 0x62c5e6b0
Caught exception in looking up space point ptr... 0x371f7010, 0x371f6f30
Caught exception in looking up space point ptr... 0x371f6f30, 0x371f7010
Caught exception in looking up space point ptr... 0x5a846310, 0x5a8428c0
Caught exception in looking up space point ptr... 0x5a8428c0, 0x5a8433b0
Caught exception in looking up space point ptr... 0x7bb30070, 0x7bb30520
Caught exception in looking up space point ptr... 0x7bb30520, 0x7bb30070
Caught exception in looking up space point ptr... 0x7bb30070, 0x568a7f30
Caught exception in looking up space point ptr... 0x568a7f30, 0x7bb30070
Caught exception in looking up space point ptr... 0x185e8f30, 0x7e38cbc0
Caught exception in looking up space point ptr... 0x7e38cbc0, 0x185e8f30
Caught exception in looking up space point ptr... 0x87512240, 0x8750f0b0
Caught exception in looking up space point ptr... 0x8750f0b0, 0x87512240
Caught exception in looking up space point ptr... 0x371bc5f0, 0x371bc140
Caught exception in looking up space point ptr... 0x371bc140, 0x371bc5f0
Caught exception in looking up space point ptr... 0x371bc5f0, 0x371bc460
Caught exception in looking up space point ptr... 0x371bc460, 0x371bc5f0
Caught exception in looking up space point ptr... 0x371af740, 0x371b0360
Caught exception in looking up space point ptr... 0x371b0360, 0x371af740
++++>>>> total num hits: 116400, num free: 112449
04-Feb-2026 02:55:18 GMT  Closed output file "np02vd_raw_run042433_0231_df-s02-d0_dw_0_20260203T200954_reco_stage0_20260204T025518_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         503.186       571.949       724.749       572.161       46.3378        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       0.000127625   0.000265336   0.000618946   0.000265097   0.000117993      19     
produce:tpcrawdecoder:PDVDTPCReader                185.08        244.92        353.514       245.947       39.2072        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.831718       1.13996       1.65036       1.1008       0.218044        19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.023371      0.0303416     0.0525632     0.0274131    0.00756709       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.022182      0.0244878     0.0501391     0.0230068    0.00606866       19     
produce:beamevent:BeamEvent                      0.000213926   0.00034453    0.00136571    0.000289692   0.000243669      19     
produce:pdvddaphne:DAPHNEReaderPDVD                18.9867       24.4913       30.1084       24.7652       3.06148        19     
produce:ophit:OpHitFinder                         0.0466556     0.0781258     0.0981595     0.0802447     0.0116023       19     
produce:wclsdatavd:WireCellToolkit                 223.973       248.186       277.827       251.951       16.2907        19     
produce:gaushit:GausHitFinder                      1.94194       2.8292        4.02725       2.84116      0.498286        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000608283   0.000892711   0.00134803    0.000904195   0.000179886      19     
produce:reco3d:SpacePointSolver                    27.0523       34.3146       41.0792       33.9866       4.04869        19     
produce:hitpdune:DisambigFromSpacePoints          0.311359      0.477873      0.650036       0.46792      0.0895728       19     
produce:cluster3d:Cluster3D                        3.23931       5.21776       7.69148       5.43418       1.23249        19     
[art]:TriggerResults:TriggerResultInserter       4.2318e-05    6.12203e-05   0.000141344   5.1316e-05    2.31011e-05      19     
end_path:out1:RootOutput                          8.38e-06     1.39259e-05   3.7752e-05    1.1976e-05    6.23722e-06      19     
end_path:out1:RootOutput(write)                    8.72409       10.1486       11.0893       10.2249      0.659445        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5308.76 MB
  Peak resident set size usage (VmHWM): 3354.56 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0231_df-s02-d0_dw_0_20260203T200954_reco_stage0_20260204T025518_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0231_df-s02-d0_dw_0_20260203T200954_reco_stage0_20260204T025518_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0231_df-s02-d0_dw_0_20260203T200954_reco_stage0_20260204T025518_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770162696.0,
    "core.end_time": 1770173723.0,
    "core.events": [
      26340,
      26346,
      26352,
      26358,
      26364,
      26370,
      26376,
      26382,
      26388,
      26394,
      26400,
      26406,
      26412,
      26418,
      26424,
      26430,
      26436,
      26442,
      26448
    ],
    "core.event_count": 19,
    "core.first_event_number": 26340,
    "core.last_event_number": 26448,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0231_df-s02-d0_dw_0_20260203T200954.hdf5"
    }
  ]
}
justIN time: 2026-02-04 04:27:58 UTC       justIN version: 01.06.00