justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303656.61@dunegpschedd01.fnal.gov

Jobsub ID303656.61@dunegpschedd01.fnal.gov
Workflow ID12717
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:56:25
SiteUK_RAL-PPD
EntryCMSHTPC_T2_UK_SGrid_RALPP_hep206
Last heartbeat2026-02-04 04:05:57
From worker nodeHostnameheplnc147.pp.rl.ac.uk
cpuinfoAMD EPYC 7452 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-04 00:11:57
Input filesvd-protodune:np02vd_raw_run042421_0095_df-s04-d1_dw_0_20260203T142223.hdf5
JobscriptExit code0
Real time3h (13758s)
CPU time3h (13008s = 94%)
Max RSS bytes15896592384 (15160 MiB)
Outputting started2026-02-04 04:01:22
Output files
Finished2026-02-04 04:05:57
Saved logsjustin-logs:303656.61-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

looking up space point ptr... 0x4dbf1340, 0xeda992c0
Caught exception in looking up space point ptr... 0x2cca2170, 0x2a3c1050
Caught exception in looking up space point ptr... 0x5c3ef570, 0xbc4f8f00
Caught exception in looking up space point ptr... 0xbc4f8f00, 0x268d5500
Caught exception in looking up space point ptr... 0x5e106b90, 0x4e1a8e10
Caught exception in looking up space point ptr... 0x5e113b60, 0x5e106b90
Caught exception in looking up space point ptr... 0x110c9a540, 0x4576a040
Caught exception in looking up space point ptr... 0x110c9a680, 0x110c9a540
Caught exception in looking up space point ptr... 0xa0a8f450, 0x4e189f20
Caught exception in looking up space point ptr... 0x29375d90, 0xa0a8f450
Caught exception in looking up space point ptr... 0x37144840, 0x37144480
Caught exception in looking up space point ptr... 0x37144480, 0x3bfde2f0
Caught exception in looking up space point ptr... 0xa1452c30, 0x14edefd0
Caught exception in looking up space point ptr... 0xa14530e0, 0xa1452c30
Caught exception in looking up space point ptr... 0xe248ddb0, 0x60717b50
Caught exception in looking up space point ptr... 0x60717b50, 0x99ad7d80
Caught exception in looking up space point ptr... 0x26136500, 0x2d1545b0
Caught exception in looking up space point ptr... 0x2d1545b0, 0x26136500
Caught exception in looking up space point ptr... 0x11782b810, 0x117819a20
Caught exception in looking up space point ptr... 0x117819a20, 0x9de03be0
Caught exception in looking up space point ptr... 0x20ae82a0, 0x52de0c40
Caught exception in looking up space point ptr... 0x52de0c40, 0x52de0d40
Caught exception in looking up space point ptr... 0xe1c3c60, 0xe1b2af0
Caught exception in looking up space point ptr... 0xe1b2af0, 0xe1c3ad0
Caught exception in looking up space point ptr... 0x37456300, 0x374441f0
Caught exception in looking up space point ptr... 0x2a273d50, 0x37456300
Caught exception in looking up space point ptr... 0x10e65720, 0x1b1dda990
Caught exception in looking up space point ptr... 0x1b1dda990, 0x4c8997c0
Caught exception in looking up space point ptr... 0x3cee99a0, 0x39bd5a10
Caught exception in looking up space point ptr... 0x3cee9f00, 0x3cee99a0
Caught exception in looking up space point ptr... 0x46ed54f0, 0x46ed5290
Caught exception in looking up space point ptr... 0x46ed5290, 0x46ed4f90
Caught exception in looking up space point ptr... 0x46ed4870, 0x46ed4490
Caught exception in looking up space point ptr... 0x46ed4490, 0x46ed4870
Caught exception in looking up space point ptr... 0x46ed4870, 0x46ed4610
Caught exception in looking up space point ptr... 0x46ed4610, 0x46ed4870
Caught exception in looking up space point ptr... 0x512e55c0, 0x6ecfa700
Caught exception in looking up space point ptr... 0x6ecfa700, 0x1a6a3f140
Caught exception in looking up space point ptr... 0x28a918e0, 0x114f1300
Caught exception in looking up space point ptr... 0x114f1300, 0x28a918e0
Caught exception in looking up space point ptr... 0x115fba60, 0x2cc66bb0
Caught exception in looking up space point ptr... 0x2ce434a0, 0x115fb7c0
Caught exception in looking up space point ptr... 0x115fb7c0, 0x115fba60
Caught exception in looking up space point ptr... 0x20ae7f60, 0x3da7eda0
Caught exception in looking up space point ptr... 0x3da7eda0, 0x14e0f930
Caught exception in looking up space point ptr... 0x23374ad0, 0x685bdd10
Caught exception in looking up space point ptr... 0x685bdd10, 0x23374900
Caught exception in looking up space point ptr... 0xcf3cef0, 0x31069250
Caught exception in looking up space point ptr... 0x31069250, 0xcf3cef0
Caught exception in looking up space point ptr... 0xe3aeb7d0, 0x288071e0
Caught exception in looking up space point ptr... 0x288071e0, 0x2cf94da0
Caught exception in looking up space point ptr... 0x2bcc4d30, 0x2bcc4e90
Caught exception in looking up space point ptr... 0x2bcc4e90, 0x2bcc0010
Caught exception in looking up space point ptr... 0xeac6180, 0xeac5c10
Caught exception in looking up space point ptr... 0xeac5c10, 0xeac6180
Caught exception in looking up space point ptr... 0x28890cb0, 0x37d5b9a0
Caught exception in looking up space point ptr... 0x37d5b9a0, 0x28890cb0
Caught exception in looking up space point ptr... 0x287c46e0, 0x530c00b0
Caught exception in looking up space point ptr... 0x530c00b0, 0xe446d90
Caught exception in looking up space point ptr... 0x4db7cf80, 0x31b01780
Caught exception in looking up space point ptr... 0x31b01780, 0x166e1e50
Caught exception in looking up space point ptr... 0x2cc66f30, 0x2cff2f00
Caught exception in looking up space point ptr... 0x2cff2f00, 0x2cc66f30
Caught exception in looking up space point ptr... 0x22e41640, 0x5ebbf1f0
Caught exception in looking up space point ptr... 0x5ebbf1f0, 0x22e41640
++++>>>> total num hits: 319536, num free: 312675
04-Feb-2026 04:01:07 GMT  Closed output file "np02vd_raw_run042421_0095_df-s04-d1_dw_0_20260203T142223_reco_stage0_20260204T040107_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         99.5689       487.743       1608.38       161.802       502.438        28     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       7.9881e-05    0.000159712   0.000599371   0.000124335   9.61273e-05      28     
produce:tpcrawdecoder:PDVDTPCReader                11.1302       12.9929       15.0964       12.4713       1.34966        28     
produce:triggerrawdecoder:PDVDTriggerReader4      0.517023      0.540462      0.564227      0.541581      0.0115653       28     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0173279     0.0178735     0.018378      0.0179204    0.00022832       28     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0857735     0.0888879     0.104487      0.0878662     0.0042393       28     
produce:beamevent:BeamEvent                      0.000126388    0.0002302    0.00142618    0.000176458   0.000236683      28     
produce:pdvddaphne:DAPHNEReaderPDVD                3.41914       4.19844       5.16624       4.06424      0.484328        28     
produce:ophit:OpHitFinder                         0.022021      0.0405748     0.0511181     0.0428209    0.00701988       28     
produce:wclsdatavd:WireCellToolkit                 68.1701       85.8951       101.295       85.9952       9.75822        28     
produce:gaushit:GausHitFinder                     0.706122       3.42432       16.294        1.5382        3.43511        28     
produce:nhitsfilter:NumberOfHitsFilter           0.000319112   0.000783186    0.0018142    0.000466861   0.000485733      28     
produce:reco3d:SpacePointSolver                    4.86514       45.1633       141.33        11.465        49.4124        28     
produce:hitpdune:DisambigFromSpacePoints          0.0758962      1.94152       7.50017      0.258291       2.48332        28     
produce:cluster3d:Cluster3D                        1.19696       322.174       1322.04       42.1148       438.047        28     
[art]:TriggerResults:TriggerResultInserter        2.636e-05    4.84851e-05   0.000119326   3.62785e-05   2.56969e-05      28     
end_path:out1:RootOutput                          8.176e-06    1.2798e-05    3.3112e-05    1.03845e-05   6.30715e-06      28     
end_path:out1:RootOutput(write)                    3.36518       11.2006       31.7295       5.56288       9.05054        28     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 18480.2 MB
  Peak resident set size usage (VmHWM): 15896.6 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042421_0095_df-s04-d1_dw_0_20260203T142223_reco_stage0_20260204T040107_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042421_0095_df-s04-d1_dw_0_20260203T142223_reco_stage0_20260204T040107_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042421_0095_df-s04-d1_dw_0_20260203T142223_reco_stage0_20260204T040107_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770163931.0,
    "core.end_time": 1770177670.0,
    "core.events": [
      15455,
      15461,
      15467,
      15473,
      15479,
      15485,
      15491,
      15497,
      15503,
      15509,
      15515,
      15521,
      15527,
      15533,
      15539,
      15545,
      15551,
      15557,
      15563,
      15569,
      15575,
      15581,
      15587,
      15593,
      15599,
      15605,
      15611,
      15617
    ],
    "core.event_count": 28,
    "core.first_event_number": 15455,
    "core.last_event_number": 15617,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42421
    ],
    "core.runs_subruns": [
      4242100001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042421_0095_df-s04-d1_dw_0_20260203T142223.hdf5"
    }
  ]
}
justIN time: 2026-02-04 04:28:17 UTC       justIN version: 01.06.00