justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297933.0@dunegpschedd02.fnal.gov

Jobsub ID297933.0@dunegpschedd02.fnal.gov
Workflow ID12718
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 23:26:25
SiteUK_Oxford
EntryDUNE_UK_SGrid_Oxford_arc01
Last heartbeat2026-02-04 00:56:46
From worker nodeHostnamet2wn175.physics.ox.ac.uk
cpuinfoAMD EPYC 9655 96-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-04 00:14:17
Input filesvd-protodune:np02vd_raw_run042422_0125_df-s03-d1_dw_0_20260203T162222.hdf5
JobscriptExit code0
Real time39m (2378s)
CPU time34m (2042s = 85%)
Max RSS bytes2539982848 (2422 MiB)
Outputting started2026-02-04 00:53:56
Output files
Finished2026-02-04 00:56:46
Saved logsjustin-logs:297933.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

x2e6b17f0
Caught exception in looking up space point ptr... 0x283d9f40, 0x8457300
Caught exception in looking up space point ptr... 0x8457300, 0x2e76be10
Caught exception in looking up space point ptr... 0x5c8c72a0, 0x5c8c7150
Caught exception in looking up space point ptr... 0x5c8c7150, 0x5c8c72a0
Caught exception in looking up space point ptr... 0x5c8c72a0, 0x38ea8770
Caught exception in looking up space point ptr... 0x38ea8770, 0x5c8c72a0
Caught exception in looking up space point ptr... 0x4a834500, 0x2d479e30
Caught exception in looking up space point ptr... 0x2d479e30, 0x4a834500
Caught exception in looking up space point ptr... 0x4bce7fa0, 0xf1b8260
Caught exception in looking up space point ptr... 0xf1b8260, 0xf1b67d0
Caught exception in looking up space point ptr... 0x2872ae20, 0x13af2230
Caught exception in looking up space point ptr... 0x13af2230, 0x2872ae20
Caught exception in looking up space point ptr... 0x2188fbb0, 0x2188fa20
Caught exception in looking up space point ptr... 0x2188fa20, 0x36fbca70
Caught exception in looking up space point ptr... 0x28f928b0, 0x46feb560
Caught exception in looking up space point ptr... 0x28f92bd0, 0x28f928b0
Caught exception in looking up space point ptr... 0x2bee4df0, 0xefb0b50
Caught exception in looking up space point ptr... 0x35b2c6b0, 0x2bee5510
Caught exception in looking up space point ptr... 0x2bee5510, 0x2bee4df0
Caught exception in looking up space point ptr... 0x250dacc0, 0x4543f760
Caught exception in looking up space point ptr... 0x4543f760, 0x268843f0
Caught exception in looking up space point ptr... 0x4f8dfd00, 0x47017970
Caught exception in looking up space point ptr... 0x47017970, 0x4f8dfd00
Caught exception in looking up space point ptr... 0x4f076290, 0x4f075ff0
Caught exception in looking up space point ptr... 0x4f075ff0, 0x4f076290
Caught exception in looking up space point ptr... 0x9c616b0, 0x63788ed0
Caught exception in looking up space point ptr... 0x63788ed0, 0x3c667030
Caught exception in looking up space point ptr... 0x2f88e030, 0x2f88dd10
Caught exception in looking up space point ptr... 0x2f88dd10, 0x2f88e030
Caught exception in looking up space point ptr... 0x24e0d1d0, 0x4068a760
Caught exception in looking up space point ptr... 0x4068a760, 0x24e0d1d0
Caught exception in looking up space point ptr... 0x24e21e30, 0x24e1d2d0
Caught exception in looking up space point ptr... 0x24e21f40, 0x24e21e30
Caught exception in looking up space point ptr... 0x3ddfb160, 0x3ddfda00
Caught exception in looking up space point ptr... 0x3ddfda00, 0x3ddff640
Caught exception in looking up space point ptr... 0x291a12a0, 0x24deda60
Caught exception in looking up space point ptr... 0x24deda60, 0x291a12a0
Caught exception in looking up space point ptr... 0x2d1b52c0, 0x2d1b73e0
Caught exception in looking up space point ptr... 0x2d1b73e0, 0x2d1b52c0
Caught exception in looking up space point ptr... 0xcc2b6f0, 0xcc2bf00
Caught exception in looking up space point ptr... 0xcc2bf00, 0xcc2b6f0
Caught exception in looking up space point ptr... 0x39afbef0, 0x2be20e30
Caught exception in looking up space point ptr... 0x2be20e30, 0x23bfd300
Caught exception in looking up space point ptr... 0x24deb530, 0x24ded900
Caught exception in looking up space point ptr... 0x24ded900, 0x2d144710
Caught exception in looking up space point ptr... 0x2abcae70, 0x2abcd610
Caught exception in looking up space point ptr... 0x2abcc8e0, 0x2abcae70
Caught exception in looking up space point ptr... 0x38fa7070, 0x38fa6d50
Caught exception in looking up space point ptr... 0x38fa6d50, 0x38fa7070
Caught exception in looking up space point ptr... 0x2d1b5b60, 0x2d1c30f0
Caught exception in looking up space point ptr... 0x2d1c30f0, 0x388238f0
Caught exception in looking up space point ptr... 0x4f081880, 0x4f0819e0
Caught exception in looking up space point ptr... 0x4f0819e0, 0x4f082840
Caught exception in looking up space point ptr... 0x38e94ce0, 0x5c8e14d0
Caught exception in looking up space point ptr... 0x5c8e14d0, 0x38e94ce0
Caught exception in looking up space point ptr... 0x38e94ce0, 0x38e94ac0
Caught exception in looking up space point ptr... 0x38e94ac0, 0x38e94ce0
Caught exception in looking up space point ptr... 0x325229e0, 0x81440b0
Caught exception in looking up space point ptr... 0x81440b0, 0x325229e0
Caught exception in looking up space point ptr... 0x20eebca0, 0x2ec70370
Caught exception in looking up space point ptr... 0x20eebd90, 0x20eebca0
Caught exception in looking up space point ptr... 0x354caf90, 0x354cb530
Caught exception in looking up space point ptr... 0x354cb530, 0x354cbe20
Caught exception in looking up space point ptr... 0x4e2c9b60, 0x8c93d90
Caught exception in looking up space point ptr... 0x8c93d90, 0x8c93e90
++++>>>> total num hits: 58246, num free: 55911
04-Feb-2026 00:53:37 GMT  Closed output file "np02vd_raw_run042422_0125_df-s03-d1_dw_0_20260203T162222_reco_stage0_20260204T005337_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         74.7204       84.8289       104.354       84.8547       7.42377        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       6.7262e-05    0.000133414   0.000736288   9.8269e-05    0.000124075      27     
produce:tpcrawdecoder:PDVDTPCReader                11.1935       12.0083       15.0308       11.5605       1.0134         27     
produce:triggerrawdecoder:PDVDTriggerReader4       0.57711      0.592841      0.609419      0.587754     0.00936006       27     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0192758     0.0197092     0.0201561     0.019693     0.000222072      27     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0954169     0.101534      0.118156      0.0968001    0.00841404       27     
produce:beamevent:BeamEvent                      0.000132511   0.000196315   0.000992967   0.00015777    0.000158982      27     
produce:pdvddaphne:DAPHNEReaderPDVD                2.51858       2.77372       3.0323        2.78421      0.131402        27     
produce:ophit:OpHitFinder                         0.0196109     0.0349899     0.0411664     0.0364705    0.00517033       27     
produce:wclsdatavd:WireCellToolkit                 49.7113       54.5272       61.5585       54.3518       3.41233        27     
produce:gaushit:GausHitFinder                     0.478098      0.830475       1.29016      0.827719      0.218795        27     
produce:nhitsfilter:NumberOfHitsFilter           0.000140523   0.000214526   0.000472717   0.000185001   6.74727e-05      27     
produce:reco3d:SpacePointSolver                    3.95405       9.01485       17.4436       7.95818       3.45627        27     
produce:hitpdune:DisambigFromSpacePoints          0.0420454     0.0972658     0.238387      0.0877727     0.0411835       27     
produce:cluster3d:Cluster3D                       0.621492       1.45092       2.84941       1.43032      0.546938        27     
[art]:TriggerResults:TriggerResultInserter       2.6119e-05    3.98404e-05   0.000119392   3.5123e-05    1.7704e-05       27     
end_path:out1:RootOutput                          5.208e-06     8.563e-06    2.9285e-05     7.501e-06    5.55291e-06      27     
end_path:out1:RootOutput(write)                    2.70577       3.34077       4.19144       3.34515      0.372608        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4706.11 MB
  Peak resident set size usage (VmHWM): 2539.98 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042422_0125_df-s03-d1_dw_0_20260203T162222_reco_stage0_20260204T005337_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042422_0125_df-s03-d1_dw_0_20260203T162222_reco_stage0_20260204T005337_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042422_0125_df-s03-d1_dw_0_20260203T162222_reco_stage0_20260204T005337_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770164067.0,
    "core.end_time": 1770166419.0,
    "core.events": [
      20271,
      20277,
      20283,
      20289,
      20295,
      20301,
      20307,
      20313,
      20319,
      20325,
      20331,
      20337,
      20343,
      20349,
      20355,
      20361,
      20367,
      20373,
      20379,
      20385,
      20391,
      20397,
      20403,
      20409,
      20415,
      20421,
      20427
    ],
    "core.event_count": 27,
    "core.first_event_number": 20271,
    "core.last_event_number": 20427,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42422
    ],
    "core.runs_subruns": [
      4242200001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042422_0125_df-s03-d1_dw_0_20260203T162222.hdf5"
    }
  ]
}
justIN time: 2026-02-04 07:23:56 UTC       justIN version: 01.06.00