justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297923.65@dunegpschedd02.fnal.gov

Jobsub ID297923.65@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 23:00:26
SiteUK_RAL-PPD
EntryCMSHTPC_T2_UK_SGrid_RALPP_hep207
Last heartbeat2026-02-04 01:05:25
From worker nodeHostnameheplnc153.pp.rl.ac.uk
cpuinfoAMD EPYC 7763 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:59:35
Input filesvd-protodune:np02vd_raw_run042433_0233_df-s03-d0_dw_0_20260203T201041.hdf5
JobscriptExit code0
Real time1h (3801s)
CPU time57m (3431s = 90%)
Max RSS bytes3351584768 (3196 MiB)
Outputting started2026-02-04 01:02:57
Output files
Finished2026-02-04 01:05:25
Saved logsjustin-logs:297923.65-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

space point ptr... 0x615a2b70, 0x615a2d20
Caught exception in looking up space point ptr... 0x615a2d20, 0x615a2b70
Caught exception in looking up space point ptr... 0xa44a870, 0xa44ab80
Caught exception in looking up space point ptr... 0xa44ab80, 0xa4417d0
Caught exception in looking up space point ptr... 0x187784d0, 0x47c66990
Caught exception in looking up space point ptr... 0x47c66990, 0x187784d0
Caught exception in looking up space point ptr... 0xa6ff400, 0x708475e0
Caught exception in looking up space point ptr... 0x708475e0, 0xa6ff400
Caught exception in looking up space point ptr... 0x34f98f30, 0x34f989e0
Caught exception in looking up space point ptr... 0x34f989e0, 0x34f98f30
Caught exception in looking up space point ptr... 0x572941d0, 0x57293eb0
Caught exception in looking up space point ptr... 0x57293eb0, 0x57294040
Caught exception in looking up space point ptr... 0x57294040, 0x5728f2c0
Caught exception in looking up space point ptr... 0x2a996700, 0x490e09e0
Caught exception in looking up space point ptr... 0x490e09e0, 0x2a996700
Caught exception in looking up space point ptr... 0x5c3874b0, 0x5c384100
Caught exception in looking up space point ptr... 0x5c384100, 0x5c3874b0
Caught exception in looking up space point ptr... 0x152c4470, 0x152c42e0
Caught exception in looking up space point ptr... 0x152c42e0, 0x152c4600
Caught exception in looking up space point ptr... 0x2accbe40, 0x19902200
Caught exception in looking up space point ptr... 0x198fe4f0, 0x2accbe40
Caught exception in looking up space point ptr... 0xf5e72f0, 0xf5e75f0
Caught exception in looking up space point ptr... 0xf5e75f0, 0xf5e78d0
Caught exception in looking up space point ptr... 0x3baedcf0, 0x767066a0
Caught exception in looking up space point ptr... 0x767066a0, 0x11f3c810
Caught exception in looking up space point ptr... 0x47ca1c30, 0x47ca1eb0
Caught exception in looking up space point ptr... 0x47ca1eb0, 0x47ca1c30
Caught exception in looking up space point ptr... 0x62270480, 0x62273390
Caught exception in looking up space point ptr... 0x62273390, 0x62270480
Caught exception in looking up space point ptr... 0xf5377b0, 0x6677bae0
Caught exception in looking up space point ptr... 0x6677bae0, 0xf5377b0
Caught exception in looking up space point ptr... 0x1aee6ea0, 0x1aee6f90
Caught exception in looking up space point ptr... 0x1aee6f90, 0x1aee6db0
Caught exception in looking up space point ptr... 0xa457760, 0xa456ac0
Caught exception in looking up space point ptr... 0xa456ac0, 0xa457760
Caught exception in looking up space point ptr... 0x2cfd6750, 0x2cfd6850
Caught exception in looking up space point ptr... 0x2cfd6850, 0x2cfd69a0
Caught exception in looking up space point ptr... 0x4d003c90, 0x4d003a30
Caught exception in looking up space point ptr... 0x4d003a30, 0x4d003c90
Caught exception in looking up space point ptr... 0x77fd5800, 0x77fd0730
Caught exception in looking up space point ptr... 0x77fd0730, 0x77fd0830
Caught exception in looking up space point ptr... 0x5f3ead70, 0x5f3eab10
Caught exception in looking up space point ptr... 0x5f3eab10, 0x5f3e98d0
Caught exception in looking up space point ptr... 0x2aeae0d0, 0x2aeae260
Caught exception in looking up space point ptr... 0x2aeae260, 0x2aeae0d0
Caught exception in looking up space point ptr... 0x3c7250f0, 0x440758c0
Caught exception in looking up space point ptr... 0x440758c0, 0x30851860
Caught exception in looking up space point ptr... 0x30851860, 0xa6fde60
Caught exception in looking up space point ptr... 0xa6fde60, 0x3c7250f0
Caught exception in looking up space point ptr... 0x2aeb6680, 0x2aeb6200
Caught exception in looking up space point ptr... 0x2aeb6200, 0x2aeb6680
Caught exception in looking up space point ptr... 0x3bb47820, 0x4ce853e0
Caught exception in looking up space point ptr... 0x4ce853e0, 0x196f8150
Caught exception in looking up space point ptr... 0x368ab190, 0x66762560
Caught exception in looking up space point ptr... 0x66762560, 0x368ab190
Caught exception in looking up space point ptr... 0x368ab190, 0xf617800
Caught exception in looking up space point ptr... 0xf617800, 0x368ab190
Caught exception in looking up space point ptr... 0x46e9d450, 0x46e9d0e0
Caught exception in looking up space point ptr... 0x46e9d0e0, 0x46e9cf00
Caught exception in looking up space point ptr... 0x72d3b450, 0x72d3b770
Caught exception in looking up space point ptr... 0x72d3b770, 0x72d3b5e0
Caught exception in looking up space point ptr... 0x72d619c0, 0x72d62c20
Caught exception in looking up space point ptr... 0x72d62c20, 0x72d619c0
Caught exception in looking up space point ptr... 0x196af7c0, 0x806c7910
Caught exception in looking up space point ptr... 0x806c7910, 0x3a271690
Caught exception in looking up space point ptr... 0x42c87e80, 0x42c86f00
Caught exception in looking up space point ptr... 0x42c86f00, 0x42c88120
++++>>>> total num hits: 136773, num free: 131006
04-Feb-2026 01:02:49 GMT  Closed output file "np02vd_raw_run042433_0233_df-s03-d0_dw_0_20260203T201041_reco_stage0_20260204T010249_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         174.071       195.64        217.543       196.584       10.2591        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       8.7173e-05    0.00013387    0.000314399   0.000104957   5.49064e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                14.787        16.5413       22.671        15.9941       1.68203        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.556008      0.645574      0.746562      0.645711      0.0497082       19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.017669      0.0238119     0.0686171     0.0203208     0.0110277       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0176919     0.0192842     0.034349      0.017992     0.00397981       19     
produce:beamevent:BeamEvent                      0.000119513   0.000259138   0.00152997    0.000203802   0.000302938      19     
produce:pdvddaphne:DAPHNEReaderPDVD                3.56772       3.94439       5.78064       3.80022      0.463248        19     
produce:ophit:OpHitFinder                         0.0366617     0.047159      0.0536708     0.0477716    0.00427353       19     
produce:wclsdatavd:WireCellToolkit                 123.205       141.389       153.67        142.104       8.75329        19     
produce:gaushit:GausHitFinder                     0.927021       1.8237        2.65277       1.81898      0.458028        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000261099   0.000495679   0.00111902    0.000422992   0.00019098       19     
produce:reco3d:SpacePointSolver                    11.5054       20.736        36.9782       19.5157       5.7151         19     
produce:hitpdune:DisambigFromSpacePoints          0.145486      0.251915      0.406015      0.264981      0.0772692       19     
produce:cluster3d:Cluster3D                        2.1766        3.5143        6.18813       3.3272        1.01659        19     
[art]:TriggerResults:TriggerResultInserter       2.9204e-05    4.84023e-05   0.000152656   4.0786e-05    2.70248e-05      19     
end_path:out1:RootOutput                          4.519e-06    1.01384e-05   3.9534e-05     8.506e-06    7.36387e-06      19     
end_path:out1:RootOutput(write)                    5.02305       6.65262       7.79926       6.79417      0.926933        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5387.59 MB
  Peak resident set size usage (VmHWM): 3351.58 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0233_df-s03-d0_dw_0_20260203T201041_reco_stage0_20260204T010249_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0233_df-s03-d0_dw_0_20260203T201041_reco_stage0_20260204T010249_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0233_df-s03-d0_dw_0_20260203T201041_reco_stage0_20260204T010249_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770163189.0,
    "core.end_time": 1770166971.0,
    "core.events": [
      26570,
      26576,
      26582,
      26588,
      26594,
      26600,
      26606,
      26612,
      26618,
      26624,
      26630,
      26636,
      26642,
      26648,
      26654,
      26660,
      26666,
      26672,
      26678
    ],
    "core.event_count": 19,
    "core.first_event_number": 26570,
    "core.last_event_number": 26678,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0233_df-s03-d0_dw_0_20260203T201041.hdf5"
    }
  ]
}
justIN time: 2026-02-04 07:43:40 UTC       justIN version: 01.06.00