justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297921.172@dunegpschedd02.fnal.gov

Jobsub ID297921.172@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:58:26
SiteUK_RAL-PPD
EntryCMSHTPC_T2_UK_SGrid_RALPP_hep207
Last heartbeat2026-02-04 01:20:01
From worker nodeHostnameheplnc167.pp.rl.ac.uk
cpuinfoAMD EPYC 7763 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:56:21
Input filesvd-protodune:np02vd_raw_run042433_0475_df-s02-d1_dw_0_20260203T214238.hdf5
JobscriptExit code0
Real time1h (4838s)
CPU time1h (4508s = 93%)
Max RSS bytes6160203776 (5874 MiB)
Outputting started2026-02-04 01:17:01
Output files
Finished2026-02-04 01:20:01
Saved logsjustin-logs:297921.172-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

... 0x10914ac0, 0x109146d0
Caught exception in looking up space point ptr... 0x5b660700, 0x5b660570
Caught exception in looking up space point ptr... 0x5b660570, 0x605db440
Caught exception in looking up space point ptr... 0x5ad7b7b0, 0x6e57e0d0
Caught exception in looking up space point ptr... 0x5ad671a0, 0x5ad7b7b0
Caught exception in looking up space point ptr... 0x3573c0e0, 0x3574dbb0
Caught exception in looking up space point ptr... 0x7615e460, 0x3573c0e0
Caught exception in looking up space point ptr... 0x5e083640, 0x5b056900
Caught exception in looking up space point ptr... 0x5b056900, 0x5dcbe690
Caught exception in looking up space point ptr... 0x5d97d040, 0x61b7da00
Caught exception in looking up space point ptr... 0x61b7da00, 0x5d97ceb0
Caught exception in looking up space point ptr... 0x7a675030, 0x7a662c00
Caught exception in looking up space point ptr... 0x7a662c00, 0x7a6560e0
Caught exception in looking up space point ptr... 0x16ae6520, 0x5b03cf50
Caught exception in looking up space point ptr... 0x5b03cf50, 0x9a36290
Caught exception in looking up space point ptr... 0x58826050, 0x588188b0
Caught exception in looking up space point ptr... 0x58833b10, 0x58826050
Caught exception in looking up space point ptr... 0x64fd0780, 0x6018d860
Caught exception in looking up space point ptr... 0x6018d860, 0x6017b2a0
Caught exception in looking up space point ptr... 0x549e8900, 0x49224d30
Caught exception in looking up space point ptr... 0x49224d30, 0x660e8c00
Caught exception in looking up space point ptr... 0x4abc4140, 0x49556890
Caught exception in looking up space point ptr... 0x49556890, 0x4abc4140
Caught exception in looking up space point ptr... 0x4741f810, 0x1d0cc6e0
Caught exception in looking up space point ptr... 0x1d0cc6e0, 0x4741f810
Caught exception in looking up space point ptr... 0x4741f810, 0x510cfdc0
Caught exception in looking up space point ptr... 0x510cfdc0, 0x4741f810
Caught exception in looking up space point ptr... 0x4bf27580, 0x53ac3180
Caught exception in looking up space point ptr... 0x53ac3180, 0x53ac38b0
Caught exception in looking up space point ptr... 0x53ac38b0, 0x51f5b880
Caught exception in looking up space point ptr... 0x51f5b880, 0x53ac4660
Caught exception in looking up space point ptr... 0x5ec96080, 0x5ec99b10
Caught exception in looking up space point ptr... 0x5ec99b10, 0x5ec97bc0
Caught exception in looking up space point ptr... 0x48403b60, 0x54b90bc0
Caught exception in looking up space point ptr... 0x54b90bc0, 0x48403b60
Caught exception in looking up space point ptr... 0x497dead0, 0x4a9111b0
Caught exception in looking up space point ptr... 0x497decb0, 0x497dead0
Caught exception in looking up space point ptr... 0x493fa910, 0x493f8b10
Caught exception in looking up space point ptr... 0x493f8b10, 0x476bd290
Caught exception in looking up space point ptr... 0x4a809630, 0x19b57ab0
Caught exception in looking up space point ptr... 0x19b57ab0, 0x4a809630
Caught exception in looking up space point ptr... 0x5d621140, 0x66580580
Caught exception in looking up space point ptr... 0x534a1e50, 0x5d621140
Caught exception in looking up space point ptr... 0x37fc48d0, 0x37fdebf0
Caught exception in looking up space point ptr... 0x37fdebf0, 0x37fc48d0
Caught exception in looking up space point ptr... 0x37fc48d0, 0x37fdeab0
Caught exception in looking up space point ptr... 0x37fdeab0, 0x37fc48d0
Caught exception in looking up space point ptr... 0x419af000, 0x419c2280
Caught exception in looking up space point ptr... 0x419c2280, 0x419c2cb0
Caught exception in looking up space point ptr... 0x491736e0, 0x425d1010
Caught exception in looking up space point ptr... 0x425d1010, 0x549e73c0
Caught exception in looking up space point ptr... 0x64752eb0, 0x64752cb0
Caught exception in looking up space point ptr... 0x64752cb0, 0x64752bb0
Caught exception in looking up space point ptr... 0x49be9410, 0x509e7490
Caught exception in looking up space point ptr... 0x509e7490, 0x49be9410
Caught exception in looking up space point ptr... 0x5a1e44b0, 0x5e1f5a20
Caught exception in looking up space point ptr... 0x5e1f5a20, 0x5a1e44b0
Caught exception in looking up space point ptr... 0x53e681a0, 0x4bda2430
Caught exception in looking up space point ptr... 0x4bda2430, 0x53e681a0
Caught exception in looking up space point ptr... 0x1bf2e910, 0x1cbecc90
Caught exception in looking up space point ptr... 0x1cbecc90, 0x1cbecb00
Caught exception in looking up space point ptr... 0x19d1e730, 0x19d1e910
Caught exception in looking up space point ptr... 0x19d1e910, 0x19d1ec80
Caught exception in looking up space point ptr... 0x37ac4470, 0x37ac4670
Caught exception in looking up space point ptr... 0x37ac4670, 0x37ac4470
Caught exception in looking up space point ptr... 0x1b7fa0c0, 0x739ff90
Caught exception in looking up space point ptr... 0x739ff90, 0x10027550
++++>>>> total num hits: 408109, num free: 399479
04-Feb-2026 01:16:51 GMT  Closed output file "np02vd_raw_run042433_0475_df-s02-d1_dw_0_20260203T214238_reco_stage0_20260204T011651_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         170.533       250.242       698.568       224.964       108.843        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       9.1391e-05    0.000141155   0.000252332   0.000115426   4.64877e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                14.0742       14.8052       18.9937       14.4676       1.05286        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.513752      0.543855      0.594944      0.540583      0.0183306       19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.017608      0.0178903     0.0180771     0.0179048    0.000117696      19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0175566     0.0178555     0.0182113     0.0178978    0.00014433       19     
produce:beamevent:BeamEvent                      0.00011739    0.000235048   0.00158841    0.000153167   0.000320344      19     
produce:pdvddaphne:DAPHNEReaderPDVD                3.36302       3.69095       3.90805       3.70649      0.150445        19     
produce:ophit:OpHitFinder                         0.0319642     0.0453428     0.0549233     0.0473706    0.00605858       19     
produce:wclsdatavd:WireCellToolkit                 112.979       140.212       157.99        139.252       13.8067        19     
produce:gaushit:GausHitFinder                      1.58441       2.78918       7.4572        2.44472       1.32435        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000321752   0.000509029    0.0011562    0.000508622   0.000180694      19     
produce:reco3d:SpacePointSolver                    14.888        25.007        59.0774       24.4792       9.88978        19     
produce:hitpdune:DisambigFromSpacePoints           0.19236      0.420789       2.19518      0.295918      0.429657        19     
produce:cluster3d:Cluster3D                        2.73983       54.7609       452.959       34.5067       94.9332        19     
[art]:TriggerResults:TriggerResultInserter       2.4426e-05    4.27046e-05   0.000184746   3.5236e-05    3.43911e-05      19     
end_path:out1:RootOutput                          3.316e-06    9.86521e-06   6.3478e-05     6.642e-06    1.30626e-05      19     
end_path:out1:RootOutput(write)                    5.61475       7.8689        20.0308       7.21775       3.17114        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8335.23 MB
  Peak resident set size usage (VmHWM): 6160.2 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0475_df-s02-d1_dw_0_20260203T214238_reco_stage0_20260204T011651_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0475_df-s02-d1_dw_0_20260203T214238_reco_stage0_20260204T011651_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0475_df-s02-d1_dw_0_20260203T214238_reco_stage0_20260204T011651_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770162993.0,
    "core.end_time": 1770167813.0,
    "core.events": [
      54157,
      54163,
      54169,
      54175,
      54181,
      54187,
      54193,
      54199,
      54205,
      54211,
      54217,
      54223,
      54229,
      54235,
      54241,
      54247,
      54253,
      54259,
      54265
    ],
    "core.event_count": 19,
    "core.first_event_number": 54157,
    "core.last_event_number": 54265,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0475_df-s02-d1_dw_0_20260203T214238.hdf5"
    }
  ]
}
justIN time: 2026-02-04 09:17:49 UTC       justIN version: 01.06.00