justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303654.45@dunegpschedd01.fnal.gov

Jobsub ID303654.45@dunegpschedd01.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:52:26
SiteUK_RAL-PPD
EntryCMSHTPC_T2_UK_SGrid_RALPP_hep206
Last heartbeat2026-02-04 01:20:09
From worker nodeHostnameheplnc170.pp.rl.ac.uk
cpuinfoAMD EPYC 7763 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:51:10
Input filesvd-protodune:np02vd_raw_run042433_0513_df-s04-d1_dw_0_20260203T215707.hdf5
JobscriptExit code0
Real time1h (5156s)
CPU time1h (4766s = 92%)
Max RSS bytes3617542144 (3449 MiB)
Outputting started2026-02-04 01:17:08
Output files
Finished2026-02-04 01:20:09
Saved logsjustin-logs:303654.45-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

up space point ptr... 0x61f14480, 0x17607650
Caught exception in looking up space point ptr... 0x59e111b0, 0x59e10f50
Caught exception in looking up space point ptr... 0x59e10f50, 0xe55c090
Caught exception in looking up space point ptr... 0xe55c090, 0xe55b120
Caught exception in looking up space point ptr... 0xe55b120, 0x59e111b0
Caught exception in looking up space point ptr... 0xee59200, 0xfaa7720
Caught exception in looking up space point ptr... 0xfaa7720, 0xfaa7680
Caught exception in looking up space point ptr... 0x506df550, 0x2f0a87f0
Caught exception in looking up space point ptr... 0x2f0a87f0, 0x506df550
Caught exception in looking up space point ptr... 0x2b17f3a0, 0x59dd7090
Caught exception in looking up space point ptr... 0x59dd7090, 0x59dd7400
Caught exception in looking up space point ptr... 0xb5843f0, 0x59dcc100
Caught exception in looking up space point ptr... 0x59dcc100, 0xb5843f0
Caught exception in looking up space point ptr... 0xb5843f0, 0xb584540
Caught exception in looking up space point ptr... 0xb584540, 0xb5843f0
Caught exception in looking up space point ptr... 0xb5843f0, 0x59dcbeb0
Caught exception in looking up space point ptr... 0x59dcbeb0, 0xb5843f0
Caught exception in looking up space point ptr... 0x46657730, 0x32f72480
Caught exception in looking up space point ptr... 0x32f72480, 0x46657730
Caught exception in looking up space point ptr... 0x44316fa0, 0x4ab932d0
Caught exception in looking up space point ptr... 0xd3d4630, 0x44316fa0
Caught exception in looking up space point ptr... 0x497c0800, 0x497c0350
Caught exception in looking up space point ptr... 0x15dc8380, 0x497c0800
Caught exception in looking up space point ptr... 0x1a3854f0, 0x2c4ead20
Caught exception in looking up space point ptr... 0x2c4ead20, 0x385e2090
Caught exception in looking up space point ptr... 0x45b0b8b0, 0x45b0be00
Caught exception in looking up space point ptr... 0x45b0be00, 0x45b0b8b0
Caught exception in looking up space point ptr... 0x15e3ed00, 0x15e3ea60
Caught exception in looking up space point ptr... 0x15e3ea60, 0x15e3ed00
Caught exception in looking up space point ptr... 0x92260d0, 0x4ef9d5a0
Caught exception in looking up space point ptr... 0x4ef9d5a0, 0x92260d0
Caught exception in looking up space point ptr... 0x791f96f0, 0x39fd2bf0
Caught exception in looking up space point ptr... 0x39fd2bf0, 0x5c3dba70
Caught exception in looking up space point ptr... 0x652280a0, 0x65227da0
Caught exception in looking up space point ptr... 0x65227da0, 0x59e3ea10
Caught exception in looking up space point ptr... 0x8000de60, 0x5288c0c0
Caught exception in looking up space point ptr... 0x5288c0c0, 0x8000de60
Caught exception in looking up space point ptr... 0x3818b760, 0x310d23c0
Caught exception in looking up space point ptr... 0x310d23c0, 0x310d07f0
Caught exception in looking up space point ptr... 0x51ea35d0, 0x56fa1c00
Caught exception in looking up space point ptr... 0x56fa1c00, 0x51ea35d0
Caught exception in looking up space point ptr... 0x66552790, 0x39f7f390
Caught exception in looking up space point ptr... 0x39f7f390, 0x1692e760
Caught exception in looking up space point ptr... 0x6d31edd0, 0x6d322c60
Caught exception in looking up space point ptr... 0x6d322c60, 0x6d31edd0
Caught exception in looking up space point ptr... 0x16b298f0, 0x7cb43de0
Caught exception in looking up space point ptr... 0x7cb43de0, 0x16b298f0
Caught exception in looking up space point ptr... 0x60de9490, 0x4c22c960
Caught exception in looking up space point ptr... 0x4c22c960, 0x33df0640
Caught exception in looking up space point ptr... 0x17530a90, 0x858330d0
Caught exception in looking up space point ptr... 0x858330d0, 0x7c7e5a70
Caught exception in looking up space point ptr... 0x49920d80, 0x499202f0
Caught exception in looking up space point ptr... 0x499202f0, 0x49920d80
Caught exception in looking up space point ptr... 0x32b78e40, 0x32b78d00
Caught exception in looking up space point ptr... 0x32b78d00, 0x32b785c0
Caught exception in looking up space point ptr... 0x62836110, 0xb57fb60
Caught exception in looking up space point ptr... 0x3a3ec6d0, 0x62836110
Caught exception in looking up space point ptr... 0x66de73f0, 0x66de86d0
Caught exception in looking up space point ptr... 0x66de86d0, 0x66de9800
Caught exception in looking up space point ptr... 0x528b8060, 0x528b7ef0
Caught exception in looking up space point ptr... 0x528b7ef0, 0x528b8060
Caught exception in looking up space point ptr... 0x7c8818a0, 0x7c881b40
Caught exception in looking up space point ptr... 0x7c881b40, 0x7c8818a0
Caught exception in looking up space point ptr... 0x52c52850, 0x52c50180
Caught exception in looking up space point ptr... 0x52c50180, 0x52c52850
Caught exception in looking up space point ptr... 0x52c52850, 0x3b1cb9d0
Caught exception in looking up space point ptr... 0x3b1cb9d0, 0x52c52850
++++>>>> total num hits: 93765, num free: 90432
04-Feb-2026 01:16:57 GMT  Closed output file "np02vd_raw_run042433_0513_df-s04-d1_dw_0_20260203T215707_reco_stage0_20260204T011657_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         188.364       266.658       308.576       277.912       35.1286        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       9.0138e-05     0.0001501    0.000271808   0.000123861   4.83925e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                15.0744       15.6318       19.9232       15.3865        1.033         19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.526146      0.554553      0.578493      0.554334      0.0139837       19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0177006     0.018072      0.0206972     0.017897     0.000648976      19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0176081     0.0178935     0.018617      0.0178422    0.00026425       19     
produce:beamevent:BeamEvent                      9.8053e-05    0.000221895   0.00127497    0.000160981   0.000250271      19     
produce:pdvddaphne:DAPHNEReaderPDVD                3.27704       3.90102       4.4101        3.95321      0.287698        19     
produce:ophit:OpHitFinder                         0.0401834     0.0489883     0.0575285     0.0500047    0.00406195       19     
produce:wclsdatavd:WireCellToolkit                 133.013       152.29        160.913       153.936       6.47943        19     
produce:gaushit:GausHitFinder                      1.72992       3.2567        4.22885       3.56646      0.803363        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000328553   0.000499082   0.000756904   0.000496067   0.000107676      19     
produce:reco3d:SpacePointSolver                    14.6455       23.3927       39.2947       23.0895       6.2903         19     
produce:hitpdune:DisambigFromSpacePoints          0.190593       0.46626      0.673601      0.515962      0.140026        19     
produce:cluster3d:Cluster3D                        2.60946       58.4857       82.3322       70.0805       28.8241        19     
[art]:TriggerResults:TriggerResultInserter       3.3332e-05    5.01726e-05   0.000105567   5.0203e-05    1.50718e-05      19     
end_path:out1:RootOutput                          6.302e-06    1.35356e-05    3.769e-05     1.119e-05    8.06833e-06      19     
end_path:out1:RootOutput(write)                    6.22932       8.5462        9.93441       8.7975        1.15109        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5608.68 MB
  Peak resident set size usage (VmHWM): 3617.54 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0513_df-s04-d1_dw_0_20260203T215707_reco_stage0_20260204T011657_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0513_df-s04-d1_dw_0_20260203T215707_reco_stage0_20260204T011657_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0513_df-s04-d1_dw_0_20260203T215707_reco_stage0_20260204T011657_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770162684.0,
    "core.end_time": 1770167819.0,
    "core.events": [
      58493,
      58499,
      58505,
      58511,
      58517,
      58523,
      58529,
      58535,
      58541,
      58547,
      58553,
      58559,
      58565,
      58571,
      58577,
      58583,
      58589,
      58595,
      58601
    ],
    "core.event_count": 19,
    "core.first_event_number": 58493,
    "core.last_event_number": 58601,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0513_df-s04-d1_dw_0_20260203T215707.hdf5"
    }
  ]
}
justIN time: 2026-02-04 06:06:16 UTC       justIN version: 01.06.00