justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297921.28@dunegpschedd02.fnal.gov

Jobsub ID297921.28@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:58:26
SiteUK_RAL-PPD
EntryCMSHTPC_T2_UK_SGrid_RALPP_hep206
Last heartbeat2026-02-04 00:39:52
From worker nodeHostnameheplnc171.pp.rl.ac.uk
cpuinfoAMD EPYC 9534 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:45:29
Input filesvd-protodune:np02vd_raw_run042433_0005_df-s02-d0_dw_0_20260203T184359.hdf5
JobscriptExit code0
Real time51m (3110s)
CPU time47m (2831s = 91%)
Max RSS bytes3358187520 (3202 MiB)
Outputting started2026-02-04 00:37:20
Output files
Finished2026-02-04 00:39:52
Saved logsjustin-logs:297921.28-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

d90
Caught exception in looking up space point ptr... 0x3a29fde0, 0x3a29ff30
Caught exception in looking up space point ptr... 0x3a29ff30, 0x3a29fde0
Caught exception in looking up space point ptr... 0x1ad94ae0, 0x1ad8a340
Caught exception in looking up space point ptr... 0x1ad8aa20, 0x1ad94ae0
Caught exception in looking up space point ptr... 0x47591ac0, 0x47591930
Caught exception in looking up space point ptr... 0x475ac7c0, 0x475ac350
Caught exception in looking up space point ptr... 0x475ac350, 0x475ac630
Caught exception in looking up space point ptr... 0x475ac630, 0x47591ac0
Caught exception in looking up space point ptr... 0x198ba9f0, 0x198bad60
Caught exception in looking up space point ptr... 0x198bad60, 0x198bb080
Caught exception in looking up space point ptr... 0x3fd4eb80, 0x519f3640
Caught exception in looking up space point ptr... 0x519f3640, 0x3fd4eb80
Caught exception in looking up space point ptr... 0x7410d700, 0x5230c8a0
Caught exception in looking up space point ptr... 0x5230c8a0, 0x52313290
Caught exception in looking up space point ptr... 0x33157680, 0x33157780
Caught exception in looking up space point ptr... 0x33157780, 0x3315a4e0
Caught exception in looking up space point ptr... 0x3dbef6c0, 0x19c1f440
Caught exception in looking up space point ptr... 0x19c1f440, 0x3dbef960
Caught exception in looking up space point ptr... 0x36c3ed00, 0x2a483640
Caught exception in looking up space point ptr... 0x2a483640, 0x30eddc40
Caught exception in looking up space point ptr... 0x45d171a0, 0x45d174c0
Caught exception in looking up space point ptr... 0x45d174c0, 0x45d17330
Caught exception in looking up space point ptr... 0x33156720, 0x33156620
Caught exception in looking up space point ptr... 0x33156620, 0x33159ea0
Caught exception in looking up space point ptr... 0x4e0a2b90, 0x38b4ab30
Caught exception in looking up space point ptr... 0x5fd05cb0, 0x4e0a2b90
Caught exception in looking up space point ptr... 0xa2d9520, 0xa2eb590
Caught exception in looking up space point ptr... 0xa2eb590, 0xa2d9520
Caught exception in looking up space point ptr... 0x423cccb0, 0x423cc990
Caught exception in looking up space point ptr... 0x423cc990, 0x423cccb0
Caught exception in looking up space point ptr... 0x1987ec10, 0x1987f020
Caught exception in looking up space point ptr... 0x1987f020, 0x1987d820
Caught exception in looking up space point ptr... 0x12faa1e0, 0x3a25b2e0
Caught exception in looking up space point ptr... 0x3a25b2e0, 0x17cd9be0
Caught exception in looking up space point ptr... 0x466b93b0, 0x466b92b0
Caught exception in looking up space point ptr... 0x466c3d00, 0x466b93b0
Caught exception in looking up space point ptr... 0x42f9c360, 0x42f9c1d0
Caught exception in looking up space point ptr... 0x42f9b840, 0x42f9c360
Caught exception in looking up space point ptr... 0x3b6a8880, 0x3b6a83d0
Caught exception in looking up space point ptr... 0x3b6a83d0, 0x3b6a8560
Caught exception in looking up space point ptr... 0x37a8cda0, 0x9328530
Caught exception in looking up space point ptr... 0x9328530, 0x37a8cda0
Caught exception in looking up space point ptr... 0x130c5c10, 0x130c5d60
Caught exception in looking up space point ptr... 0x130c5d60, 0x130c5970
Caught exception in looking up space point ptr... 0x3cb54a70, 0x38395f40
Caught exception in looking up space point ptr... 0x38395f40, 0x75f86130
Caught exception in looking up space point ptr... 0x43aa5740, 0x43ab31c0
Caught exception in looking up space point ptr... 0x43ab31c0, 0x43aa5740
Caught exception in looking up space point ptr... 0x423a3890, 0x3361f4b0
Caught exception in looking up space point ptr... 0x3361f4b0, 0x423a48b0
Caught exception in looking up space point ptr... 0x313c8d90, 0x313c8f80
Caught exception in looking up space point ptr... 0x313c8f80, 0x313c8d90
Caught exception in looking up space point ptr... 0x129e3050, 0x3da33810
Caught exception in looking up space point ptr... 0x3da33810, 0x4856dbc0
Caught exception in looking up space point ptr... 0x4856dbc0, 0x43abb9e0
Caught exception in looking up space point ptr... 0x740e1ac0, 0x740e17a0
Caught exception in looking up space point ptr... 0x740e17a0, 0x740e1480
Caught exception in looking up space point ptr... 0x466941a0, 0x43a9e930
Caught exception in looking up space point ptr... 0x43a9e930, 0x466941a0
Caught exception in looking up space point ptr... 0x5c988870, 0x9365f60
Caught exception in looking up space point ptr... 0x9365f60, 0x5c964bc0
Caught exception in looking up space point ptr... 0x14696db0, 0x14697240
Caught exception in looking up space point ptr... 0x14697240, 0x14696db0
Caught exception in looking up space point ptr... 0x15314610, 0x9e04e50
Caught exception in looking up space point ptr... 0x9e04e50, 0x15314610
Caught exception in looking up space point ptr... 0x130c3750, 0x130c3b40
Caught exception in looking up space point ptr... 0x130c3b40, 0x130c39f0
++++>>>> total num hits: 81449, num free: 78136
04-Feb-2026 00:37:11 GMT  Closed output file "np02vd_raw_run042433_0005_df-s02-d0_dw_0_20260203T184359_reco_stage0_20260204T003711_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         133.838       159.538       177.322       160.131       10.5336        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       9.1302e-05    0.000121469   0.000231494   9.6461e-05    4.09781e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                12.0678       15.4535       21.2635       15.3383       2.47933        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.531823      0.649459       1.39827      0.612895       0.18563        19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0178315     0.0237028     0.0664924     0.0182101     0.012616        19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0177403     0.0181374     0.0187293     0.0180992    0.000229905      19     
produce:beamevent:BeamEvent                      0.000147162   0.000261115   0.00164601    0.000177003   0.000326961      19     
produce:pdvddaphne:DAPHNEReaderPDVD                2.97652       3.40602       3.75638       3.44139      0.213857        19     
produce:ophit:OpHitFinder                         0.0378049     0.0426008     0.0467851     0.0430868    0.00279102       19     
produce:wclsdatavd:WireCellToolkit                 98.9571       114.428       125.086       115.805       6.94873        19     
produce:gaushit:GausHitFinder                     0.772563       1.55261       2.59761       1.61198      0.424158        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000209163   0.000363249   0.000816631   0.000338825   0.000121632      19     
produce:reco3d:SpacePointSolver                    7.73792        15.28        26.0003       14.8179       4.5247         19     
produce:hitpdune:DisambigFromSpacePoints          0.114358      0.215953      0.356431      0.220205      0.0667124       19     
produce:cluster3d:Cluster3D                        1.23842       2.74804       4.1322        2.64199      0.834854        19     
[art]:TriggerResults:TriggerResultInserter        2.61e-05     4.24942e-05   0.000112412    3.977e-05    1.81777e-05      19     
end_path:out1:RootOutput                          4.94e-06      1.036e-05    2.8231e-05     1.03e-05     4.66327e-06      19     
end_path:out1:RootOutput(write)                    4.26211       5.6633        6.57049       5.96448      0.682171        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5351.27 MB
  Peak resident set size usage (VmHWM): 3358.19 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0005_df-s02-d0_dw_0_20260203T184359_reco_stage0_20260204T003711_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0005_df-s02-d0_dw_0_20260203T184359_reco_stage0_20260204T003711_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0005_df-s02-d0_dw_0_20260203T184359_reco_stage0_20260204T003711_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770162341.0,
    "core.end_time": 1770165433.0,
    "core.events": [
      576,
      582,
      588,
      594,
      600,
      606,
      612,
      618,
      624,
      630,
      636,
      642,
      648,
      654,
      660,
      666,
      672,
      678,
      684
    ],
    "core.event_count": 19,
    "core.first_event_number": 576,
    "core.last_event_number": 684,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0005_df-s02-d0_dw_0_20260203T184359.hdf5"
    }
  ]
}
justIN time: 2026-02-04 07:43:34 UTC       justIN version: 01.06.00