justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297923.91@dunegpschedd02.fnal.gov

Jobsub ID297923.91@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 23:00:26
SiteUK_RAL-PPD
EntryCMSHTPC_T2_UK_SGrid_RALPP_hep206
Last heartbeat2026-02-04 01:07:13
From worker nodeHostnameheplnc153.pp.rl.ac.uk
cpuinfoAMD EPYC 7763 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-04 00:00:36
Input filesvd-protodune:np02vd_raw_run042433_0300_df-s02-d1_dw_0_20260203T203609.hdf5
JobscriptExit code0
Real time1h (3849s)
CPU time57m (3464s = 89%)
Max RSS bytes3369893888 (3213 MiB)
Outputting started2026-02-04 01:04:46
Output files
Finished2026-02-04 01:07:13
Saved logsjustin-logs:297923.91-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

point ptr... 0x7ab9cc20, 0x7ad1fd30
Caught exception in looking up space point ptr... 0x7ad1fd30, 0x7ab9cc20
Caught exception in looking up space point ptr... 0x3bd31b50, 0xdde3b60
Caught exception in looking up space point ptr... 0xdde3b60, 0x3bd31b50
Caught exception in looking up space point ptr... 0x3bd31b50, 0x8cf5f20
Caught exception in looking up space point ptr... 0x8cf5f20, 0x3bd31b50
Caught exception in looking up space point ptr... 0x15ae9e90, 0x15ae9d00
Caught exception in looking up space point ptr... 0x15ae9d00, 0x15ae9b70
Caught exception in looking up space point ptr... 0x3e616770, 0x3e616610
Caught exception in looking up space point ptr... 0x3e616610, 0x3e616770
Caught exception in looking up space point ptr... 0xe078a80, 0x6b8ff060
Caught exception in looking up space point ptr... 0xe077e00, 0xe078a80
Caught exception in looking up space point ptr... 0x2d82f6a0, 0x2d82f380
Caught exception in looking up space point ptr... 0x2d82f380, 0x2d82f6a0
Caught exception in looking up space point ptr... 0x37be7f50, 0x37beeaf0
Caught exception in looking up space point ptr... 0x37beeaf0, 0x37be9400
Caught exception in looking up space point ptr... 0x37be9400, 0x37be8ad0
Caught exception in looking up space point ptr... 0x6b349f00, 0x396f5fe0
Caught exception in looking up space point ptr... 0x4c24ae40, 0x6b349f00
Caught exception in looking up space point ptr... 0x7adf6f20, 0x7adf65a0
Caught exception in looking up space point ptr... 0x7adf65a0, 0x7adf5e80
Caught exception in looking up space point ptr... 0x46c30a50, 0x46c30be0
Caught exception in looking up space point ptr... 0x46c30be0, 0x46c30f00
Caught exception in looking up space point ptr... 0x59a6dd40, 0x59a661d0
Caught exception in looking up space point ptr... 0x59a66460, 0x59a6dd40
Caught exception in looking up space point ptr... 0x3bcd18d0, 0x1488d730
Caught exception in looking up space point ptr... 0x1488d730, 0x3bcd18d0
Caught exception in looking up space point ptr... 0x6025a900, 0x3f738130
Caught exception in looking up space point ptr... 0x3f738130, 0x6025a900
Caught exception in looking up space point ptr... 0x582b5cf0, 0x510df990
Caught exception in looking up space point ptr... 0x510df990, 0x582b5cf0
Caught exception in looking up space point ptr... 0x582aa070, 0x582a9ee0
Caught exception in looking up space point ptr... 0x582a9ee0, 0x582aa070
Caught exception in looking up space point ptr... 0x582aa070, 0x582abcb0
Caught exception in looking up space point ptr... 0x582abcb0, 0x582aa070
Caught exception in looking up space point ptr... 0x37be8d30, 0x37be8590
Caught exception in looking up space point ptr... 0x37be8590, 0x37be5ce0
Caught exception in looking up space point ptr... 0x18da6610, 0x18da6510
Caught exception in looking up space point ptr... 0x18da6510, 0x18da6780
Caught exception in looking up space point ptr... 0x2c332110, 0x2c33bc10
Caught exception in looking up space point ptr... 0x2c33bc10, 0x2c332110
Caught exception in looking up space point ptr... 0x4a902e80, 0x4a902d80
Caught exception in looking up space point ptr... 0x4a902d80, 0x4a902e80
Caught exception in looking up space point ptr... 0x3b7b9370, 0x47dc03b0
Caught exception in looking up space point ptr... 0x47dc03b0, 0x3b7b9370
Caught exception in looking up space point ptr... 0x7dfc74b0, 0x7dfc7210
Caught exception in looking up space point ptr... 0x7dfc7210, 0x7dfc70c0
Caught exception in looking up space point ptr... 0x3e63aff0, 0x3e641f50
Caught exception in looking up space point ptr... 0x3e63e060, 0x3e63aff0
Caught exception in looking up space point ptr... 0x3a65d190, 0x3a5a8930
Caught exception in looking up space point ptr... 0x3a5a8930, 0x4dc21de0
Caught exception in looking up space point ptr... 0x467f4250, 0x467f4700
Caught exception in looking up space point ptr... 0x467f4700, 0x467f4a20
Caught exception in looking up space point ptr... 0x5131c750, 0x5131c9f0
Caught exception in looking up space point ptr... 0x5131c9f0, 0x5131c750
Caught exception in looking up space point ptr... 0x3d9ded80, 0x3d9ec2c0
Caught exception in looking up space point ptr... 0x3d9ec2c0, 0x3d9ded80
Caught exception in looking up space point ptr... 0x3d9ded80, 0x3d9d49c0
Caught exception in looking up space point ptr... 0x3d9d49c0, 0x3d9ded80
Caught exception in looking up space point ptr... 0x3d9ded80, 0xac74fe0
Caught exception in looking up space point ptr... 0xac74fe0, 0x3d9ded80
Caught exception in looking up space point ptr... 0x89d1070, 0x89d1390
Caught exception in looking up space point ptr... 0x89d1390, 0x89d1070
Caught exception in looking up space point ptr... 0x2c2a6770, 0x2c2a6630
Caught exception in looking up space point ptr... 0x2c2a6630, 0x2c2a6770
Caught exception in looking up space point ptr... 0x46c4c710, 0x62fa9c60
Caught exception in looking up space point ptr... 0x62fa9c60, 0x46c4c710
++++>>>> total num hits: 119031, num free: 114138
04-Feb-2026 01:04:38 GMT  Closed output file "np02vd_raw_run042433_0300_df-s02-d1_dw_0_20260203T203609_reco_stage0_20260204T010438_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         179.109       197.831       225.702       196.253       15.2632        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       0.000116538   0.000175771   0.000318616   0.000174977   4.60845e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                15.6505       17.5303       19.5835       17.3609       1.29943        19     
produce:triggerrawdecoder:PDVDTriggerReader4       0.56446      0.646863       0.81669      0.632893      0.0778426       19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0178058     0.0230572     0.0801369     0.0180968     0.0138302       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0178274     0.0180287     0.0186571     0.0180058    0.000198538      19     
produce:beamevent:BeamEvent                      0.00012277    0.000219743   0.00108366    0.000175298   0.000205163      19     
produce:pdvddaphne:DAPHNEReaderPDVD                3.28852       3.84363       4.28152       3.82132      0.238498        19     
produce:ophit:OpHitFinder                         0.0325385     0.0460841     0.0528598     0.0479366    0.00572375       19     
produce:wclsdatavd:WireCellToolkit                 124.134       139.286       157.451       138.033       8.83843        19     
produce:gaushit:GausHitFinder                      1.27584       1.87289       2.52411       1.73938       0.40239        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000306935   0.000599739   0.00227126    0.000442439   0.000465712      19     
produce:reco3d:SpacePointSolver                    13.7661       23.7029       45.0257       21.7272       8.55457        19     
produce:hitpdune:DisambigFromSpacePoints           0.16514      0.281026      0.409633      0.282634      0.0694196       19     
produce:cluster3d:Cluster3D                        2.39043       3.78264       5.55406       3.65958      0.983606        19     
[art]:TriggerResults:TriggerResultInserter       2.5117e-05    4.58369e-05   0.000113202   4.0005e-05    2.30236e-05      19     
end_path:out1:RootOutput                          6.342e-06    1.24481e-05   3.0818e-05     1.065e-05    5.71973e-06      19     
end_path:out1:RootOutput(write)                    5.08426       6.73683       8.08659       6.85485      0.916896        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5354.14 MB
  Peak resident set size usage (VmHWM): 3369.89 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0300_df-s02-d1_dw_0_20260203T203609_reco_stage0_20260204T010438_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0300_df-s02-d1_dw_0_20260203T203609_reco_stage0_20260204T010438_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0300_df-s02-d1_dw_0_20260203T203609_reco_stage0_20260204T010438_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770163250.0,
    "core.end_time": 1770167080.0,
    "core.events": [
      34207,
      34213,
      34219,
      34225,
      34231,
      34237,
      34243,
      34249,
      34255,
      34261,
      34267,
      34273,
      34279,
      34285,
      34291,
      34297,
      34303,
      34309,
      34315
    ],
    "core.event_count": 19,
    "core.first_event_number": 34207,
    "core.last_event_number": 34315,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0300_df-s02-d1_dw_0_20260203T203609.hdf5"
    }
  ]
}
justIN time: 2026-02-04 07:43:14 UTC       justIN version: 01.06.00