justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297923.94@dunegpschedd02.fnal.gov

Jobsub ID297923.94@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 23:00:26
SiteUK_RAL-PPD
EntryCMSHTPC_T2_UK_SGrid_RALPP_hep206
Last heartbeat2026-02-04 01:06:35
From worker nodeHostnameheplnc153.pp.rl.ac.uk
cpuinfoAMD EPYC 7763 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-04 00:00:29
Input filesvd-protodune:np02vd_raw_run042433_0123_df-s02-d1_dw_0_20260203T192851.hdf5
JobscriptExit code0
Real time1h (3810s)
CPU time57m (3454s = 90%)
Max RSS bytes3380269056 (3223 MiB)
Outputting started2026-02-04 01:04:00
Output files
Finished2026-02-04 01:06:35
Saved logsjustin-logs:297923.94-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

space point ptr... 0xcc730d0, 0xcc73220
Caught exception in looking up space point ptr... 0x4ac98ea0, 0x556937a0
Caught exception in looking up space point ptr... 0x4c834b10, 0x4ac98ea0
Caught exception in looking up space point ptr... 0x79c76030, 0x46e630e0
Caught exception in looking up space point ptr... 0x46e630e0, 0x79c76030
Caught exception in looking up space point ptr... 0x114c5cd0, 0x30ee4e40
Caught exception in looking up space point ptr... 0x30ee4e40, 0x39f814b0
Caught exception in looking up space point ptr... 0x39f814b0, 0x301b4760
Caught exception in looking up space point ptr... 0x301b4760, 0x114c5cd0
Caught exception in looking up space point ptr... 0x15781cf0, 0x6ec260f0
Caught exception in looking up space point ptr... 0x6ec260f0, 0x15781cf0
Caught exception in looking up space point ptr... 0x37528370, 0x7e8c42b0
Caught exception in looking up space point ptr... 0x622848a0, 0x37528370
Caught exception in looking up space point ptr... 0xd8ff410, 0x6c68c1c0
Caught exception in looking up space point ptr... 0x6c68c1c0, 0x6cb02f40
Caught exception in looking up space point ptr... 0x310f78f0, 0x3941c990
Caught exception in looking up space point ptr... 0x3941c990, 0x310f78f0
Caught exception in looking up space point ptr... 0x310f78f0, 0x574def10
Caught exception in looking up space point ptr... 0x574def10, 0x310f78f0
Caught exception in looking up space point ptr... 0x1073bc20, 0x45caab30
Caught exception in looking up space point ptr... 0x45caab30, 0x1073bc20
Caught exception in looking up space point ptr... 0x41a773c0, 0x15a9cb30
Caught exception in looking up space point ptr... 0x15a9cb30, 0x15a9a6c0
Caught exception in looking up space point ptr... 0x10733b20, 0x10729a20
Caught exception in looking up space point ptr... 0x10729a20, 0x10733b20
Caught exception in looking up space point ptr... 0x6c685ce0, 0x6c689c80
Caught exception in looking up space point ptr... 0x6c689c80, 0x46c2dac0
Caught exception in looking up space point ptr... 0x134113d0, 0x13411280
Caught exception in looking up space point ptr... 0x13411280, 0x134113d0
Caught exception in looking up space point ptr... 0x3a3dc360, 0x508dafb0
Caught exception in looking up space point ptr... 0x508dafb0, 0x16cc2b10
Caught exception in looking up space point ptr... 0x38995f10, 0x6eb61190
Caught exception in looking up space point ptr... 0x6eb61190, 0x38995f10
Caught exception in looking up space point ptr... 0x2a64ef20, 0x2a64edd0
Caught exception in looking up space point ptr... 0x2a64edd0, 0x2a64ef20
Caught exception in looking up space point ptr... 0x1999cfb0, 0x1999d300
Caught exception in looking up space point ptr... 0x1999d300, 0x1999d170
Caught exception in looking up space point ptr... 0x4f116120, 0x4f115f90
Caught exception in looking up space point ptr... 0x4f115f90, 0x4f116120
Caught exception in looking up space point ptr... 0x1074b4e0, 0x1074ab80
Caught exception in looking up space point ptr... 0x1074ab80, 0x10740320
Caught exception in looking up space point ptr... 0x45cab730, 0x45cac4f0
Caught exception in looking up space point ptr... 0x45cac4f0, 0x45cb0510
Caught exception in looking up space point ptr... 0x62bae0e0, 0x62baf280
Caught exception in looking up space point ptr... 0x62baf280, 0x62bae0e0
Caught exception in looking up space point ptr... 0x61948660, 0x619487f0
Caught exception in looking up space point ptr... 0x619487f0, 0x61948660
Caught exception in looking up space point ptr... 0xdc78af0, 0x6cab7fe0
Caught exception in looking up space point ptr... 0x6cab7fe0, 0xdc78c30
Caught exception in looking up space point ptr... 0x6cb06e50, 0x6cb07500
Caught exception in looking up space point ptr... 0x6cb07500, 0x6cb05ff0
Caught exception in looking up space point ptr... 0x456db140, 0x456e19c0
Caught exception in looking up space point ptr... 0x456e19c0, 0xe387640
Caught exception in looking up space point ptr... 0xe387640, 0x456e19c0
Caught exception in looking up space point ptr... 0x456e19c0, 0x456e13a0
Caught exception in looking up space point ptr... 0x456e13a0, 0x456e19c0
Caught exception in looking up space point ptr... 0x456e19c0, 0x456d95f0
Caught exception in looking up space point ptr... 0x61d5e750, 0x598c92e0
Caught exception in looking up space point ptr... 0x598c92e0, 0x598c94b0
Caught exception in looking up space point ptr... 0x15aa6c70, 0x15aa6f60
Caught exception in looking up space point ptr... 0x15aa6f60, 0x15aa6c70
Caught exception in looking up space point ptr... 0xf01f7a0, 0xf01f610
Caught exception in looking up space point ptr... 0xf01f610, 0xf01f7a0
Caught exception in looking up space point ptr... 0xfff2ff0, 0xfff0e00
Caught exception in looking up space point ptr... 0xfff0e00, 0xfff2ff0
Caught exception in looking up space point ptr... 0x4f136a40, 0x4f1368b0
Caught exception in looking up space point ptr... 0x4f1368b0, 0x4f136a40
++++>>>> total num hits: 71596, num free: 68280
04-Feb-2026 01:03:51 GMT  Closed output file "np02vd_raw_run042433_0123_df-s02-d1_dw_0_20260203T192851_reco_stage0_20260204T010351_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         170.041       195.807       212.712       197.467       12.5634        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       9.1231e-05    0.000127638   0.000273051   9.9566e-05    4.97795e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                15.357        15.8862       16.5885       15.8032      0.354724        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.540126      0.577583      0.636679      0.574355      0.0282371       19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0177118     0.0213776     0.0518095     0.0181755    0.00786967       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0176088     0.0179703     0.0187318     0.0179434    0.00026886       19     
produce:beamevent:BeamEvent                      0.000109605   0.000255723    0.0016581    0.000194986   0.000332745      19     
produce:pdvddaphne:DAPHNEReaderPDVD                2.97717       3.81129       4.28628       3.86463      0.304396        19     
produce:ophit:OpHitFinder                         0.0362698     0.0466268     0.0530247     0.0468929    0.00376219       19     
produce:wclsdatavd:WireCellToolkit                 126.834       141.089       153.948       139.446       8.62842        19     
produce:gaushit:GausHitFinder                     0.855196       1.82687       2.49088       1.99632      0.475323        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000249768   0.000444487   0.000936594   0.000434042   0.000145922      19     
produce:reco3d:SpacePointSolver                    9.73098       21.5638       34.0827       21.7385       6.62329        19     
produce:hitpdune:DisambigFromSpacePoints          0.107807      0.287837      0.423973      0.312053      0.0861698       19     
produce:cluster3d:Cluster3D                        1.35243       3.84614       8.11563       3.33746       1.61294        19     
[art]:TriggerResults:TriggerResultInserter       2.7611e-05    4.86416e-05   0.000192781   3.9123e-05    3.51704e-05      19     
end_path:out1:RootOutput                            5e-06      1.04923e-05   5.5353e-05     8.145e-06    1.07842e-05      19     
end_path:out1:RootOutput(write)                    5.25378       6.77448       7.75113       7.07224      0.815058        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5420 MB
  Peak resident set size usage (VmHWM): 3380.27 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0123_df-s02-d1_dw_0_20260203T192851_reco_stage0_20260204T010351_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0123_df-s02-d1_dw_0_20260203T192851_reco_stage0_20260204T010351_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0123_df-s02-d1_dw_0_20260203T192851_reco_stage0_20260204T010351_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770163243.0,
    "core.end_time": 1770167033.0,
    "core.events": [
      14029,
      14035,
      14041,
      14047,
      14053,
      14059,
      14065,
      14071,
      14077,
      14083,
      14089,
      14095,
      14101,
      14107,
      14113,
      14119,
      14125,
      14131,
      14137
    ],
    "core.event_count": 19,
    "core.first_event_number": 14029,
    "core.last_event_number": 14137,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0123_df-s02-d1_dw_0_20260203T192851.hdf5"
    }
  ]
}
justIN time: 2026-02-04 07:43:35 UTC       justIN version: 01.06.00