justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297923.58@dunegpschedd02.fnal.gov

Jobsub ID297923.58@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 23:00:26
SiteUK_Oxford
EntryDUNE_UK_SGrid_Oxford_arc01
Last heartbeat2026-02-04 01:45:34
From worker nodeHostnamet2wn173.physics.ox.ac.uk
cpuinfoAMD EPYC 9655 96-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:59:14
Input filesvd-protodune:np02vd_raw_run042433_0302_df-s03-d1_dw_0_20260203T203654.hdf5
JobscriptExit code0
Real time1h (6228s)
CPU time36m (2188s = 35%)
Max RSS bytes3337240576 (3182 MiB)
Outputting started2026-02-04 01:43:03
Output files
Finished2026-02-04 01:45:34
Saved logsjustin-logs:297923.58-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

t ptr... 0x2c0a17a0, 0x7b5975c0
Caught exception in looking up space point ptr... 0x2c09db40, 0x2c09d840
Caught exception in looking up space point ptr... 0x2c09d840, 0x2c0a8f80
Caught exception in looking up space point ptr... 0x16e88dc0, 0x16e89060
Caught exception in looking up space point ptr... 0x16e89060, 0x16e88090
Caught exception in looking up space point ptr... 0x16e88090, 0x16e895a0
Caught exception in looking up space point ptr... 0x16e895a0, 0x16e89300
Caught exception in looking up space point ptr... 0x16e89300, 0x16e88dc0
Caught exception in looking up space point ptr... 0x6684bb50, 0x189a4df0
Caught exception in looking up space point ptr... 0x189a4df0, 0x55c71b40
Caught exception in looking up space point ptr... 0x61828f50, 0x5605ff80
Caught exception in looking up space point ptr... 0x5605ff80, 0x37b36a80
Caught exception in looking up space point ptr... 0x37b36a80, 0x3d9e34b0
Caught exception in looking up space point ptr... 0x3d9e34b0, 0x3a870d30
Caught exception in looking up space point ptr... 0x7b565bd0, 0x7b56cc50
Caught exception in looking up space point ptr... 0x7b56cc50, 0x7b57c9b0
Caught exception in looking up space point ptr... 0x183bd3e0, 0x58d57e10
Caught exception in looking up space point ptr... 0x58d57e10, 0x183bd3e0
Caught exception in looking up space point ptr... 0x5f5590b0, 0x5f558ba0
Caught exception in looking up space point ptr... 0x5f558ba0, 0x5f5590b0
Caught exception in looking up space point ptr... 0x1699c6d0, 0x1699d2b0
Caught exception in looking up space point ptr... 0x1699d2b0, 0x6860f060
Caught exception in looking up space point ptr... 0x6860f060, 0x68610260
Caught exception in looking up space point ptr... 0x68610260, 0x1699c6d0
Caught exception in looking up space point ptr... 0x7b6bfdf0, 0x7b6cf370
Caught exception in looking up space point ptr... 0x7b6cf370, 0x7b6c45f0
Caught exception in looking up space point ptr... 0x39e29fe0, 0x29c68550
Caught exception in looking up space point ptr... 0x29c68550, 0x39e29fe0
Caught exception in looking up space point ptr... 0x39e29fe0, 0x3d266220
Caught exception in looking up space point ptr... 0x3d266220, 0x39e29fe0
Caught exception in looking up space point ptr... 0x58e93530, 0x84bbd40
Caught exception in looking up space point ptr... 0x84bbd40, 0x58e93530
Caught exception in looking up space point ptr... 0x7d855a00, 0x7d855b60
Caught exception in looking up space point ptr... 0x7d855b60, 0x7d855a00
Caught exception in looking up space point ptr... 0x367f57e0, 0x5f4c2bc0
Caught exception in looking up space point ptr... 0x39282950, 0x367f57e0
Caught exception in looking up space point ptr... 0x56fb4ee0, 0xf7eb8b0
Caught exception in looking up space point ptr... 0xf7eb8b0, 0x5277c210
Caught exception in looking up space point ptr... 0x5277c210, 0x7b7959c0
Caught exception in looking up space point ptr... 0x387bbee0, 0x65b92ee0
Caught exception in looking up space point ptr... 0x65b92ee0, 0x387bbee0
Caught exception in looking up space point ptr... 0x3440ff80, 0x520babb0
Caught exception in looking up space point ptr... 0x520babb0, 0x3440ff80
Caught exception in looking up space point ptr... 0x4bec7c40, 0x615e0480
Caught exception in looking up space point ptr... 0x615e0480, 0xf7d4e10
Caught exception in looking up space point ptr... 0x7b557c40, 0x7b55cf60
Caught exception in looking up space point ptr... 0x7b55cf60, 0x2c09f060
Caught exception in looking up space point ptr... 0x5af64f40, 0x5af65100
Caught exception in looking up space point ptr... 0x5af65100, 0x5af65020
Caught exception in looking up space point ptr... 0x18c43b60, 0x18c43d60
Caught exception in looking up space point ptr... 0x18c43d60, 0x18c43b60
Caught exception in looking up space point ptr... 0x18739b10, 0x18739cf0
Caught exception in looking up space point ptr... 0x18739cf0, 0x55d2ff10
Caught exception in looking up space point ptr... 0x39ea69e0, 0x15edf310
Caught exception in looking up space point ptr... 0x15edf310, 0x15edf7e0
Caught exception in looking up space point ptr... 0x37b5a020, 0x448dc910
Caught exception in looking up space point ptr... 0x448dc910, 0x1294cf00
Caught exception in looking up space point ptr... 0x571185b0, 0x18da7680
Caught exception in looking up space point ptr... 0x18da7680, 0x623f3900
Caught exception in looking up space point ptr... 0x5f42ecc0, 0x5f42ee50
Caught exception in looking up space point ptr... 0x5f42ee50, 0x5f42eb30
Caught exception in looking up space point ptr... 0x410c0ef0, 0x410bf3c0
Caught exception in looking up space point ptr... 0x410bf3c0, 0x410c0ef0
Caught exception in looking up space point ptr... 0x6060a890, 0xc71e180
Caught exception in looking up space point ptr... 0xc71e180, 0x6060a890
Caught exception in looking up space point ptr... 0x6060a890, 0x2a1ef940
Caught exception in looking up space point ptr... 0x2a1ef940, 0x6060a890
++++>>>> total num hits: 56351, num free: 54519
04-Feb-2026 01:42:55 GMT  Closed output file "np02vd_raw_run042433_0302_df-s03-d1_dw_0_20260203T203654_reco_stage0_20260204T014255_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         256.448       324.354       418.028       310.377       42.3795        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       7.3692e-05    0.000101374   0.000168546   8.7904e-05    2.60304e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                142.798       200.841       296.468       188.285       38.9264        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.667743      0.990391       1.96211      0.894346      0.284358        19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0191445     0.0273621     0.0576873     0.0234012     0.0103037       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0191375     0.0218382     0.0649944     0.019439      0.0101736       19     
produce:beamevent:BeamEvent                      0.000116276   0.000189903   0.00108832    0.000139643   0.000212641      19     
produce:pdvddaphne:DAPHNEReaderPDVD                11.0935       16.196        20.8198       15.6257       2.75185        19     
produce:ophit:OpHitFinder                         0.0299742     0.0347673     0.038813      0.0351111    0.00270508       19     
produce:wclsdatavd:WireCellToolkit                 73.3048       82.1482       92.2255       81.718        5.70839        19     
produce:gaushit:GausHitFinder                      0.63253       1.2217        1.86354       1.24476      0.320348        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000202027   0.00027714    0.000405736   0.000270812   4.57057e-05      19     
produce:reco3d:SpacePointSolver                    8.02358       15.0915       28.7196       14.6292       4.66144        19     
produce:hitpdune:DisambigFromSpacePoints          0.0643911     0.169309      0.650593      0.144746      0.121931        19     
produce:cluster3d:Cluster3D                        1.03795       3.27368       25.9473       2.22813       5.38828        19     
[art]:TriggerResults:TriggerResultInserter       3.0316e-05    4.21626e-05   9.0648e-05    3.5384e-05    1.42546e-05      19     
end_path:out1:RootOutput                          4.426e-06    7.19389e-06   2.1763e-05     5.788e-06     3.723e-06       19     
end_path:out1:RootOutput(write)                    3.68243       4.28252       5.27914       4.0687       0.566226        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5681.09 MB
  Peak resident set size usage (VmHWM): 3337.24 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0302_df-s03-d1_dw_0_20260203T203654_reco_stage0_20260204T014255_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0302_df-s03-d1_dw_0_20260203T203654_reco_stage0_20260204T014255_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0302_df-s03-d1_dw_0_20260203T203654_reco_stage0_20260204T014255_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770163161.0,
    "core.end_time": 1770169377.0,
    "core.events": [
      34437,
      34443,
      34449,
      34455,
      34461,
      34467,
      34473,
      34479,
      34485,
      34491,
      34497,
      34503,
      34509,
      34515,
      34521,
      34527,
      34533,
      34539,
      34545
    ],
    "core.event_count": 19,
    "core.first_event_number": 34437,
    "core.last_event_number": 34545,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0302_df-s03-d1_dw_0_20260203T203654.hdf5"
    }
  ]
}
justIN time: 2026-02-04 04:32:53 UTC       justIN version: 01.06.00