justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297923.62@dunegpschedd02.fnal.gov

Jobsub ID297923.62@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 23:00:26
SiteUK_Oxford
EntryDUNE_UK_SGrid_Oxford_arc01
Last heartbeat2026-02-04 00:41:31
From worker nodeHostnamet2wn173.physics.ox.ac.uk
cpuinfoAMD EPYC 9655 96-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:58:59
Input filesvd-protodune:np02vd_raw_run042433_0552_df-s04-d0_dw_0_20260203T221155.hdf5
JobscriptExit code0
Real time39m (2390s)
CPU time36m (2174s = 90%)
Max RSS bytes3388993536 (3231 MiB)
Outputting started2026-02-04 00:38:50
Output files
Finished2026-02-04 00:41:31
Saved logsjustin-logs:297923.62-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

pace point ptr... 0x3e5920a0, 0x5ab911a0
Caught exception in looking up space point ptr... 0x6fb8d440, 0x42332e60
Caught exception in looking up space point ptr... 0xd188d50, 0x29725b50
Caught exception in looking up space point ptr... 0x29725b50, 0x457278c0
Caught exception in looking up space point ptr... 0x457278c0, 0x6fb8d440
Caught exception in looking up space point ptr... 0x3fc96490, 0x64bf9d70
Caught exception in looking up space point ptr... 0x64bf9d70, 0x31b512a0
Caught exception in looking up space point ptr... 0x6ef9b880, 0x6ef9b620
Caught exception in looking up space point ptr... 0x6ef9b620, 0x6ef9b880
Caught exception in looking up space point ptr... 0x3b7f2780, 0x6ed5e230
Caught exception in looking up space point ptr... 0x6ed5e230, 0x29d5cf70
Caught exception in looking up space point ptr... 0x3c7f0f40, 0x3c0544f0
Caught exception in looking up space point ptr... 0x3c0544f0, 0x3c054290
Caught exception in looking up space point ptr... 0x3c6b39d0, 0x8ced600
Caught exception in looking up space point ptr... 0x30a40250, 0x3c6b39d0
Caught exception in looking up space point ptr... 0x3847f260, 0x63285dd0
Caught exception in looking up space point ptr... 0x63285dd0, 0xd40aa00
Caught exception in looking up space point ptr... 0x4b745d20, 0x382cbe50
Caught exception in looking up space point ptr... 0x382cbe50, 0x291ec8f0
Caught exception in looking up space point ptr... 0x63286350, 0x29b68ae0
Caught exception in looking up space point ptr... 0x29b68ae0, 0x3f2eb410
Caught exception in looking up space point ptr... 0x3f55bb20, 0x302ba930
Caught exception in looking up space point ptr... 0x302ba930, 0x3f55bb20
Caught exception in looking up space point ptr... 0x28d202a0, 0x49d9e370
Caught exception in looking up space point ptr... 0x49d9e370, 0x28d202a0
Caught exception in looking up space point ptr... 0x468656a0, 0x38280d10
Caught exception in looking up space point ptr... 0x38280d10, 0x38615dc0
Caught exception in looking up space point ptr... 0x4b7617e0, 0x3ac94650
Caught exception in looking up space point ptr... 0x3ac94650, 0x38602f20
Caught exception in looking up space point ptr... 0x38602f20, 0x3ac94650
Caught exception in looking up space point ptr... 0x3ac94650, 0x4231c800
Caught exception in looking up space point ptr... 0x62277720, 0x622778b0
Caught exception in looking up space point ptr... 0x622778b0, 0x62277bd0
Caught exception in looking up space point ptr... 0x42439de0, 0x49d7ff40
Caught exception in looking up space point ptr... 0x49d7ff40, 0x42439de0
Caught exception in looking up space point ptr... 0x163d6450, 0x163d63b0
Caught exception in looking up space point ptr... 0x61ddbf10, 0x163d6450
Caught exception in looking up space point ptr... 0x49d77830, 0x45336d00
Caught exception in looking up space point ptr... 0x296a2ff0, 0x49d77830
Caught exception in looking up space point ptr... 0x7a4454f0, 0x7a4455f0
Caught exception in looking up space point ptr... 0x6b803b30, 0x7a4454f0
Caught exception in looking up space point ptr... 0x85f71700, 0x4b71d380
Caught exception in looking up space point ptr... 0x4b71d380, 0x85f71700
Caught exception in looking up space point ptr... 0x46287f60, 0x46287d60
Caught exception in looking up space point ptr... 0x46287d60, 0x46287f60
Caught exception in looking up space point ptr... 0x3ef53e40, 0x163d4ba0
Caught exception in looking up space point ptr... 0x163d4ba0, 0x3ef53e40
Caught exception in looking up space point ptr... 0x3f12e550, 0x769a210
Caught exception in looking up space point ptr... 0x769a210, 0x3fc8c190
Caught exception in looking up space point ptr... 0x66c83900, 0x454f61b0
Caught exception in looking up space point ptr... 0x454f61b0, 0x66c83900
Caught exception in looking up space point ptr... 0x39859cc0, 0x5c8e6050
Caught exception in looking up space point ptr... 0x5c8e6050, 0x39859cc0
Caught exception in looking up space point ptr... 0x98b0d00, 0x98af0d0
Caught exception in looking up space point ptr... 0x98af0d0, 0x98b0d00
Caught exception in looking up space point ptr... 0x3f0c5f20, 0x500f8ae0
Caught exception in looking up space point ptr... 0x29b713e0, 0x3f0c5f20
Caught exception in looking up space point ptr... 0x28c01890, 0x6c6f950
Caught exception in looking up space point ptr... 0x6c6f950, 0x6c739a0
Caught exception in looking up space point ptr... 0x4533f1c0, 0x632679c0
Caught exception in looking up space point ptr... 0x6c1969d0, 0x4533f1c0
Caught exception in looking up space point ptr... 0x7f888120, 0x7f8884c0
Caught exception in looking up space point ptr... 0x7f8884c0, 0x7f888120
Caught exception in looking up space point ptr... 0x448f65c0, 0x448f7030
Caught exception in looking up space point ptr... 0x448f7030, 0x448f65c0
Caught exception in looking up space point ptr... 0xa1e3550, 0x9808810
Caught exception in looking up space point ptr... 0x9808810, 0xa1e3550
++++>>>> total num hits: 44527, num free: 42698
04-Feb-2026 00:38:43 GMT  Closed output file "np02vd_raw_run042433_0552_df-s04-d0_dw_0_20260203T221155_reco_stage0_20260204T003843_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         108.194       122.437       135.615       123.479       7.73878        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                        6.615e-05    0.000101846   0.000269539   8.8404e-05    4.1724e-05       19     
produce:tpcrawdecoder:PDVDTPCReader                11.8152       12.816        15.3913       12.4217       0.88755        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.569056      0.576363      0.595658      0.574159     0.00706072       19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0191736     0.0194243     0.0203221     0.0193322    0.000253209      19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0191149     0.0193833     0.0197837     0.0193307    0.000191078      19     
produce:beamevent:BeamEvent                      0.000110819   0.000193419   0.00110214    0.000145431   0.000215199      19     
produce:pdvddaphne:DAPHNEReaderPDVD                2.41592       2.60897       2.86744       2.60412      0.122845        19     
produce:ophit:OpHitFinder                         0.0294041     0.0357626     0.0433024     0.0361049    0.00375636       19     
produce:wclsdatavd:WireCellToolkit                 73.176        83.8541       95.8091       82.205        6.8454         19     
produce:gaushit:GausHitFinder                     0.675857       1.21217       1.64968       1.17771      0.261407        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000182588   0.000264476   0.000408551   0.000261978   4.52655e-05      19     
produce:reco3d:SpacePointSolver                    9.07411       14.4114       21.0581       14.2487       2.87073        19     
produce:hitpdune:DisambigFromSpacePoints          0.0838244     0.165979      0.242482      0.177696      0.0475498       19     
produce:cluster3d:Cluster3D                        0.94624       2.15571       3.99638       2.23566      0.804084        19     
[art]:TriggerResults:TriggerResultInserter       2.8994e-05    4.06461e-05   8.6912e-05    3.7527e-05    1.26429e-05      19     
end_path:out1:RootOutput                          5.248e-06    9.75595e-06   2.3906e-05     7.492e-06    5.37367e-06      19     
end_path:out1:RootOutput(write)                    3.71314       4.50639       5.44085       4.6134       0.589192        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5574.41 MB
  Peak resident set size usage (VmHWM): 3388.99 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0552_df-s04-d0_dw_0_20260203T221155_reco_stage0_20260204T003843_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0552_df-s04-d0_dw_0_20260203T221155_reco_stage0_20260204T003843_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0552_df-s04-d0_dw_0_20260203T221155_reco_stage0_20260204T003843_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770163148.0,
    "core.end_time": 1770165524.0,
    "core.events": [
      62938,
      62944,
      62950,
      62956,
      62962,
      62968,
      62974,
      62980,
      62986,
      62992,
      62998,
      63004,
      63010,
      63016,
      63022,
      63028,
      63034,
      63040,
      63046
    ],
    "core.event_count": 19,
    "core.first_event_number": 62938,
    "core.last_event_number": 63046,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0552_df-s04-d0_dw_0_20260203T221155.hdf5"
    }
  ]
}
justIN time: 2026-02-04 04:32:55 UTC       justIN version: 01.06.00