justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297923.39@dunegpschedd02.fnal.gov

Jobsub ID297923.39@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 23:00:26
SiteUK_Oxford
EntryDUNE_UK_SGrid_Oxford_arc01
Last heartbeat2026-02-04 02:23:22
From worker nodeHostnamet2wn062.physics.ox.ac.uk
cpuinfoAMD EPYC 7763 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:59:21
Input filesvd-protodune:np02vd_raw_run042433_0198_df-s03-d0_dw_0_20260203T195721.hdf5
JobscriptExit code0
Real time2h (8406s)
CPU time59m (3575s = 42%)
Max RSS bytes3300515840 (3147 MiB)
Outputting started2026-02-04 02:19:28
Output files
Finished2026-02-04 02:23:22
Saved logsjustin-logs:297923.39-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ptr... 0x385e4610, 0x529b0e80
Caught exception in looking up space point ptr... 0x529b0e80, 0x385e3020
Caught exception in looking up space point ptr... 0x3bbcd420, 0x3a0cb890
Caught exception in looking up space point ptr... 0x3a0cb890, 0x3b0f2040
Caught exception in looking up space point ptr... 0x3b0f2040, 0x3a0a0160
Caught exception in looking up space point ptr... 0x3be2e5d0, 0x3a09f110
Caught exception in looking up space point ptr... 0x3a09f110, 0x3be2e5d0
Caught exception in looking up space point ptr... 0x14f9c560, 0x14f9c2c0
Caught exception in looking up space point ptr... 0x14f9c2c0, 0x14f9c560
Caught exception in looking up space point ptr... 0x71661e60, 0x71661bc0
Caught exception in looking up space point ptr... 0x71661bc0, 0x71661e60
Caught exception in looking up space point ptr... 0x2a70eb30, 0xc01e0c0
Caught exception in looking up space point ptr... 0xc01e0c0, 0x2a70eb30
Caught exception in looking up space point ptr... 0x174c63d0, 0x174c5f20
Caught exception in looking up space point ptr... 0x174c5f20, 0x174c60b0
Caught exception in looking up space point ptr... 0x72b9bb70, 0x72b9bfc0
Caught exception in looking up space point ptr... 0x72b9bfc0, 0x72b9b570
Caught exception in looking up space point ptr... 0x72b9b570, 0x72b9b390
Caught exception in looking up space point ptr... 0x72b9b390, 0x72b9b1b0
Caught exception in looking up space point ptr... 0x4f5754f0, 0x4f574f10
Caught exception in looking up space point ptr... 0x4f574f10, 0x4f5754f0
Caught exception in looking up space point ptr... 0x3140d5f0, 0x3140b0d0
Caught exception in looking up space point ptr... 0x3140b0d0, 0x3140d5f0
Caught exception in looking up space point ptr... 0x3140d5f0, 0x3140d460
Caught exception in looking up space point ptr... 0x3140d460, 0x3140d5f0
Caught exception in looking up space point ptr... 0x3627ccd0, 0x3627c7c0
Caught exception in looking up space point ptr... 0x3627c7c0, 0x3627ccd0
Caught exception in looking up space point ptr... 0x72b91830, 0x72b91630
Caught exception in looking up space point ptr... 0x72b91630, 0x72b91830
Caught exception in looking up space point ptr... 0x3627cba0, 0x672aee30
Caught exception in looking up space point ptr... 0x672aee30, 0x672aef30
Caught exception in looking up space point ptr... 0x72b84540, 0x72b876d0
Caught exception in looking up space point ptr... 0x72b82080, 0x72b84540
Caught exception in looking up space point ptr... 0x72b94860, 0x72b94430
Caught exception in looking up space point ptr... 0x72b94430, 0x72b94860
Caught exception in looking up space point ptr... 0x5aae7270, 0x18b98050
Caught exception in looking up space point ptr... 0x18b98050, 0x5aae94e0
Caught exception in looking up space point ptr... 0x5aae94e0, 0x18b98050
Caught exception in looking up space point ptr... 0x18b98050, 0x5aae8570
Caught exception in looking up space point ptr... 0x5aae8570, 0x18b98050
Caught exception in looking up space point ptr... 0x18b98050, 0x5aae7270
Caught exception in looking up space point ptr... 0x4fdc8f20, 0x5f7519d0
Caught exception in looking up space point ptr... 0x5f7519d0, 0x4fdc8f20
Caught exception in looking up space point ptr... 0x462336a0, 0x40d0a380
Caught exception in looking up space point ptr... 0x40d0a380, 0x462336a0
Caught exception in looking up space point ptr... 0x3bff1f40, 0x47467e50
Caught exception in looking up space point ptr... 0x47467e50, 0x3bff1f40
Caught exception in looking up space point ptr... 0x17e11570, 0x3cd903a0
Caught exception in looking up space point ptr... 0x3cd903a0, 0x17e11570
Caught exception in looking up space point ptr... 0x7269d340, 0x6fbb8760
Caught exception in looking up space point ptr... 0x33841660, 0x7269d340
Caught exception in looking up space point ptr... 0x3a0e3470, 0x9187890
Caught exception in looking up space point ptr... 0x9187890, 0x3a4267d0
Caught exception in looking up space point ptr... 0x44425470, 0x3cc73e90
Caught exception in looking up space point ptr... 0x3cc73e90, 0x370007c0
Caught exception in looking up space point ptr... 0x75ca8310, 0x180a4680
Caught exception in looking up space point ptr... 0x180a4680, 0x44425470
Caught exception in looking up space point ptr... 0x345f2360, 0x345f09f0
Caught exception in looking up space point ptr... 0x345f09f0, 0x345f1c80
Caught exception in looking up space point ptr... 0x4f595540, 0x4f5957a0
Caught exception in looking up space point ptr... 0x4f5957a0, 0x4f595b30
Caught exception in looking up space point ptr... 0x39d85ce0, 0x41b516e0
Caught exception in looking up space point ptr... 0x41b516e0, 0x4c470090
Caught exception in looking up space point ptr... 0x3a9d5260, 0x3a9d5110
Caught exception in looking up space point ptr... 0x3a9d5110, 0x3a9d53b0
Caught exception in looking up space point ptr... 0x3647dfe0, 0x46c1cab0
Caught exception in looking up space point ptr... 0x46c1cab0, 0x46f0bf30
++++>>>> total num hits: 67299, num free: 65066
04-Feb-2026 02:18:58 GMT  Closed output file "np02vd_raw_run042433_0198_df-s03-d0_dw_0_20260203T195721_reco_stage0_20260204T021858_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         346.407       435.546       536.389       436.31        45.2117        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       0.00010162    0.000149845   0.000377967   0.000118652   6.40736e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                161.27        236.468       336.975       226.23        44.5497        19     
produce:triggerrawdecoder:PDVDTriggerReader4       0.81277       1.16961       1.61148       1.17285      0.213659        19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0197842     0.0298262     0.121908      0.022501      0.0230424       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0196386     0.0341608     0.269628      0.0198779     0.0556963       19     
produce:beamevent:BeamEvent                      0.000129232   0.000260532   0.00159655    0.000189414   0.000315724      19     
produce:pdvddaphne:DAPHNEReaderPDVD                10.4584       19.3491       27.9011       19.5138       3.78533        19     
produce:ophit:OpHitFinder                         0.0265746     0.0435995     0.0589818     0.0428846    0.00790197       19     
produce:wclsdatavd:WireCellToolkit                  127.6        140.656       150.431       141.53        6.81573        19     
produce:gaushit:GausHitFinder                      1.27542       1.92791       2.95418       1.74733      0.453841        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000286997   0.000511895   0.00105657    0.000484537   0.000178549      19     
produce:reco3d:SpacePointSolver                    14.9597       23.9888       39.4789       21.9456       7.10961        19     
produce:hitpdune:DisambigFromSpacePoints          0.171209      0.883863       11.8811      0.262706       2.59313        19     
produce:cluster3d:Cluster3D                        2.19127       4.06799       6.81165       3.78692       1.4674         19     
[art]:TriggerResults:TriggerResultInserter       2.8894e-05    4.60562e-05   0.000134622   3.8703e-05    2.33512e-05      19     
end_path:out1:RootOutput                          6.432e-06    3.13987e-05   0.000258875    1.057e-05    6.10645e-05      19     
end_path:out1:RootOutput(write)                    5.70127       6.85957       8.37547       6.98353      0.636261        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5355.5 MB
  Peak resident set size usage (VmHWM): 3300.52 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0198_df-s03-d0_dw_0_20260203T195721_reco_stage0_20260204T021858_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0198_df-s03-d0_dw_0_20260203T195721_reco_stage0_20260204T021858_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0198_df-s03-d0_dw_0_20260203T195721_reco_stage0_20260204T021858_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770163178.0,
    "core.end_time": 1770171541.0,
    "core.events": [
      22580,
      22586,
      22592,
      22598,
      22604,
      22610,
      22616,
      22622,
      22628,
      22634,
      22640,
      22646,
      22652,
      22658,
      22664,
      22670,
      22676,
      22682,
      22688
    ],
    "core.event_count": 19,
    "core.first_event_number": 22580,
    "core.last_event_number": 22688,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0198_df-s03-d0_dw_0_20260203T195721.hdf5"
    }
  ]
}
justIN time: 2026-02-04 04:32:54 UTC       justIN version: 01.06.00