justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297920.126@dunegpschedd02.fnal.gov

Jobsub ID297920.126@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:54:27
SiteUK_Oxford
EntryDUNE_UK_SGrid_Oxford_arc01
Last heartbeat2026-02-04 00:52:47
From worker nodeHostnamet2wn064.physics.ox.ac.uk
cpuinfoAMD EPYC 7763 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:42:17
Input filesvd-protodune:np02vd_raw_run042433_0129_df-s03-d1_dw_0_20260203T193110.hdf5
JobscriptExit code0
Real time1h (4028s)
CPU time1h (3642s = 90%)
Max RSS bytes3301896192 (3148 MiB)
Outputting started2026-02-04 00:49:26
Output files
Finished2026-02-04 00:52:47
Saved logsjustin-logs:297920.126-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

point ptr... 0x4dfa26e0, 0x4dfa4c60
Caught exception in looking up space point ptr... 0xf3c40b0, 0xf3c50a0
Caught exception in looking up space point ptr... 0xf3c50a0, 0xf3c40b0
Caught exception in looking up space point ptr... 0x84f6ba00, 0x9a03bf0
Caught exception in looking up space point ptr... 0x9a03bf0, 0x84f6ba00
Caught exception in looking up space point ptr... 0x50296cb0, 0x50296b80
Caught exception in looking up space point ptr... 0x50296b80, 0x50291b20
Caught exception in looking up space point ptr... 0x40ad94e0, 0x40acc560
Caught exception in looking up space point ptr... 0x40acc560, 0x40ad94e0
Caught exception in looking up space point ptr... 0xd885820, 0xd884110
Caught exception in looking up space point ptr... 0xd884110, 0xd885820
Caught exception in looking up space point ptr... 0x3b203740, 0x3b2065c0
Caught exception in looking up space point ptr... 0x3b206070, 0x3b203740
Caught exception in looking up space point ptr... 0x458da7b0, 0x458da620
Caught exception in looking up space point ptr... 0x458da620, 0x458daad0
Caught exception in looking up space point ptr... 0x38cc77b0, 0x79efe5a0
Caught exception in looking up space point ptr... 0x79efe5a0, 0x3b65f330
Caught exception in looking up space point ptr... 0x5e52a2e0, 0x52a59d00
Caught exception in looking up space point ptr... 0x52a59d00, 0x5e52a2e0
Caught exception in looking up space point ptr... 0x5c6f2b30, 0x5c6f0630
Caught exception in looking up space point ptr... 0x5c6f0630, 0x5b3c0620
Caught exception in looking up space point ptr... 0x15669600, 0x38833f70
Caught exception in looking up space point ptr... 0x38833cd0, 0x15669600
Caught exception in looking up space point ptr... 0x4bc40e70, 0x4bc40b50
Caught exception in looking up space point ptr... 0x4bc40b50, 0x4bc40e70
Caught exception in looking up space point ptr... 0x4d411b00, 0x4d410300
Caught exception in looking up space point ptr... 0x4d410300, 0x4d411b00
Caught exception in looking up space point ptr... 0x12a51ba0, 0x12a67a30
Caught exception in looking up space point ptr... 0x12a67a30, 0x12a51ba0
Caught exception in looking up space point ptr... 0x139415a0, 0x139416d0
Caught exception in looking up space point ptr... 0x139416d0, 0x139415a0
Caught exception in looking up space point ptr... 0x142a96a0, 0x142a7100
Caught exception in looking up space point ptr... 0x142a7100, 0x142a96a0
Caught exception in looking up space point ptr... 0x1355c810, 0x1355c680
Caught exception in looking up space point ptr... 0x1355c680, 0x1355ccc0
Caught exception in looking up space point ptr... 0x68036ef0, 0x680372e0
Caught exception in looking up space point ptr... 0x680372e0, 0x68036ef0
Caught exception in looking up space point ptr... 0x436baf70, 0x436baa30
Caught exception in looking up space point ptr... 0x436baa30, 0x436baf70
Caught exception in looking up space point ptr... 0x40b05990, 0x4bc9d6f0
Caught exception in looking up space point ptr... 0x4bc9d6f0, 0x40b05990
Caught exception in looking up space point ptr... 0x3b65fa30, 0x3b65f930
Caught exception in looking up space point ptr... 0x3b65f930, 0x3b65f830
Caught exception in looking up space point ptr... 0x6b4e92c0, 0x6b4e9170
Caught exception in looking up space point ptr... 0x6b4e9170, 0x6b4e92c0
Caught exception in looking up space point ptr... 0x4bbbb970, 0x4bbb8120
Caught exception in looking up space point ptr... 0x4bbb8120, 0x4bbbb970
Caught exception in looking up space point ptr... 0x4bbbb970, 0x4bbbbdf0
Caught exception in looking up space point ptr... 0x4bbbbdf0, 0x4bbbb970
Caught exception in looking up space point ptr... 0x1418e7d0, 0x1418e960
Caught exception in looking up space point ptr... 0x1418e960, 0x1418e7d0
Caught exception in looking up space point ptr... 0x68034860, 0x68038760
Caught exception in looking up space point ptr... 0x68038760, 0x68034860
Caught exception in looking up space point ptr... 0x3c846380, 0x85e78be0
Caught exception in looking up space point ptr... 0x85e78be0, 0x3c846280
Caught exception in looking up space point ptr... 0x960c1a0, 0x55304920
Caught exception in looking up space point ptr... 0x55304920, 0x960c1a0
Caught exception in looking up space point ptr... 0x44d34370, 0x142ac8b0
Caught exception in looking up space point ptr... 0x142ac8b0, 0x44d3fa30
Caught exception in looking up space point ptr... 0x44d3fa30, 0x142ac8b0
Caught exception in looking up space point ptr... 0x142ac8b0, 0x142a9740
Caught exception in looking up space point ptr... 0x142a9740, 0x142ac8b0
Caught exception in looking up space point ptr... 0x142ac8b0, 0x44d34370
Caught exception in looking up space point ptr... 0x1a7529c0, 0x1a750d70
Caught exception in looking up space point ptr... 0x1a750d70, 0x1a751bc0
Caught exception in looking up space point ptr... 0x1821bc10, 0x72d0ab10
Caught exception in looking up space point ptr... 0x72d0ab10, 0x3ba72b30
++++>>>> total num hits: 124697, num free: 119671
04-Feb-2026 00:49:12 GMT  Closed output file "np02vd_raw_run042433_0129_df-s03-d1_dw_0_20260203T193110_reco_stage0_20260204T004912_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         174.967       206.508       223.236       205.34        11.645         19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       0.000105928   0.000186665   0.00047107    0.000132598   0.000102795      19     
produce:tpcrawdecoder:PDVDTPCReader                15.8158       19.2266       24.6467       19.2532       2.41853        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.589338      0.625393      0.715456      0.625693      0.0286631       19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0197085     0.0225746     0.0316263     0.0201108    0.00385586       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0195793     0.0200417     0.020982      0.0199678    0.000306298      19     
produce:beamevent:BeamEvent                      0.00014967    0.000299193   0.00166819     0.0001983    0.000330672      19     
produce:pdvddaphne:DAPHNEReaderPDVD                3.75909       4.12346       4.71546       4.07226       0.26174        19     
produce:ophit:OpHitFinder                         0.0342471     0.049217      0.0581245     0.049365     0.00477026       19     
produce:wclsdatavd:WireCellToolkit                 133.026       150.327       159.396       150.985       6.77248        19     
produce:gaushit:GausHitFinder                     0.816064       1.81285       2.78798       1.84967       0.46167        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000270826   0.00053229    0.00116862    0.000473215   0.000220659      19     
produce:reco3d:SpacePointSolver                    9.88759       19.2537       26.7826       20.2741       4.56239        19     
produce:hitpdune:DisambigFromSpacePoints          0.164617      0.267105      0.410288      0.262731       0.06827        19     
produce:cluster3d:Cluster3D                        1.35456       3.70338       6.52105       3.55255       1.37385        19     
[art]:TriggerResults:TriggerResultInserter       3.2691e-05    5.17456e-05   0.000146464   4.5064e-05    2.48158e-05      19     
end_path:out1:RootOutput                          5.48e-06     9.68479e-06   3.9584e-05     8.436e-06    7.13686e-06      19     
end_path:out1:RootOutput(write)                    5.06239       7.00493       7.96555       6.93792       0.61147        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5335.03 MB
  Peak resident set size usage (VmHWM): 3301.9 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0129_df-s03-d1_dw_0_20260203T193110_reco_stage0_20260204T004912_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0129_df-s03-d1_dw_0_20260203T193110_reco_stage0_20260204T004912_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0129_df-s03-d1_dw_0_20260203T193110_reco_stage0_20260204T004912_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770162153.0,
    "core.end_time": 1770166154.0,
    "core.events": [
      14715,
      14721,
      14727,
      14733,
      14739,
      14745,
      14751,
      14757,
      14763,
      14769,
      14775,
      14781,
      14787,
      14793,
      14799,
      14805,
      14811,
      14817,
      14823
    ],
    "core.event_count": 19,
    "core.first_event_number": 14715,
    "core.last_event_number": 14823,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0129_df-s03-d1_dw_0_20260203T193110.hdf5"
    }
  ]
}
justIN time: 2026-02-04 04:28:37 UTC       justIN version: 01.06.00