justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303656.16@dunegpschedd01.fnal.gov

Jobsub ID303656.16@dunegpschedd01.fnal.gov
Workflow ID12717
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:56:25
SiteUK_RAL-PPD
EntryCMSHTPC_T2_UK_SGrid_RALPP_hep208
Last heartbeat2026-02-04 02:02:26
From worker nodeHostnameheplnc164.pp.rl.ac.uk
cpuinfoAMD EPYC 7763 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-04 00:11:39
Input filesvd-protodune:np02vd_raw_run042421_0008_df-s03-d0_dw_0_20260203T133515.hdf5
JobscriptExit code0
Real time1h (6436s)
CPU time1h (5953s = 92%)
Max RSS bytes12692684800 (12104 MiB)
Outputting started2026-02-04 01:58:59
Output files
Finished2026-02-04 02:02:26
Saved logsjustin-logs:303656.16-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

tr... 0xa5a2ab0, 0x2e8b0120
Caught exception in looking up space point ptr... 0x2e8b0120, 0xa5a2f80
Caught exception in looking up space point ptr... 0x15c50d30, 0x15c50e70
Caught exception in looking up space point ptr... 0x15c50e70, 0x15c51190
Caught exception in looking up space point ptr... 0x34238320, 0x2c542360
Caught exception in looking up space point ptr... 0x2c542360, 0x34238320
Caught exception in looking up space point ptr... 0x173375150, 0x39b2d6c0
Caught exception in looking up space point ptr... 0x39b2d170, 0x173375150
Caught exception in looking up space point ptr... 0x498e5430, 0x668a0520
Caught exception in looking up space point ptr... 0x668a0520, 0x13af9340
Caught exception in looking up space point ptr... 0x258337b0, 0x47e6db30
Caught exception in looking up space point ptr... 0x41447710, 0x414475c0
Caught exception in looking up space point ptr... 0x414475c0, 0xf789cf0
Caught exception in looking up space point ptr... 0xf789cf0, 0x258337b0
Caught exception in looking up space point ptr... 0xdba9c10, 0x3e7a32b0
Caught exception in looking up space point ptr... 0x3e7a32b0, 0xdba9c10
Caught exception in looking up space point ptr... 0x4ad825e0, 0xee89d70
Caught exception in looking up space point ptr... 0xee89d70, 0x30824660
Caught exception in looking up space point ptr... 0x5e247a80, 0x5e247780
Caught exception in looking up space point ptr... 0x5e247780, 0x5e247a80
Caught exception in looking up space point ptr... 0x2e7aab50, 0x5d923940
Caught exception in looking up space point ptr... 0x5d923940, 0x2e7aab50
Caught exception in looking up space point ptr... 0x4c0f7a10, 0xd137a70
Caught exception in looking up space point ptr... 0xd137a70, 0x35cfa980
Caught exception in looking up space point ptr... 0x5a115410, 0x3e7781b0
Caught exception in looking up space point ptr... 0x3e7781b0, 0x5a115410
Caught exception in looking up space point ptr... 0x35e3c930, 0x35e3ce80
Caught exception in looking up space point ptr... 0x35e3ce80, 0x35e3c930
Caught exception in looking up space point ptr... 0x3193ab40, 0x3193acd0
Caught exception in looking up space point ptr... 0x3193acd0, 0x3193ae60
Caught exception in looking up space point ptr... 0x3cb17b70, 0x50bcb940
Caught exception in looking up space point ptr... 0x50bcb940, 0x3cb17b70
Caught exception in looking up space point ptr... 0x35cc00c0, 0x35cbffc0
Caught exception in looking up space point ptr... 0x35cbffc0, 0x35cc00c0
Caught exception in looking up space point ptr... 0xdbe6d50, 0x667f0730
Caught exception in looking up space point ptr... 0x667f0730, 0x667f0fd0
Caught exception in looking up space point ptr... 0x23215680, 0x4be65290
Caught exception in looking up space point ptr... 0x4be65290, 0x23215680
Caught exception in looking up space point ptr... 0x2994cf40, 0x3fe307b0
Caught exception in looking up space point ptr... 0x3fe307b0, 0x2994cf40
Caught exception in looking up space point ptr... 0x3675ef70, 0x3675ed20
Caught exception in looking up space point ptr... 0x3675ed20, 0x3675ef70
Caught exception in looking up space point ptr... 0x4d8c0100, 0x8896050
Caught exception in looking up space point ptr... 0x8896050, 0x2e7de790
Caught exception in looking up space point ptr... 0x4710dfe0, 0x32347d50
Caught exception in looking up space point ptr... 0x32347d50, 0x4710dfe0
Caught exception in looking up space point ptr... 0x35cc5a90, 0x35cc5d90
Caught exception in looking up space point ptr... 0x35cc5d90, 0x35cc5a90
Caught exception in looking up space point ptr... 0x5c50f580, 0x5e239860
Caught exception in looking up space point ptr... 0x5e239860, 0x5c50f580
Caught exception in looking up space point ptr... 0x2e7953f0, 0x2e795580
Caught exception in looking up space point ptr... 0x2e795580, 0x2e7953f0
Caught exception in looking up space point ptr... 0x500f1ac0, 0x3c059820
Caught exception in looking up space point ptr... 0x3c059820, 0x715c550
Caught exception in looking up space point ptr... 0xa5b00b0, 0xa5aff20
Caught exception in looking up space point ptr... 0xa5aff20, 0xa5b00b0
Caught exception in looking up space point ptr... 0x66841330, 0x66841650
Caught exception in looking up space point ptr... 0x66841650, 0x66841970
Caught exception in looking up space point ptr... 0x34e22800, 0x35d410e0
Caught exception in looking up space point ptr... 0x35d410e0, 0x34e22800
Caught exception in looking up space point ptr... 0x13c44050, 0x13c441e0
Caught exception in looking up space point ptr... 0x13c441e0, 0x5994e260
Caught exception in looking up space point ptr... 0xdc28220, 0x5c2fbc30
Caught exception in looking up space point ptr... 0x5c2fbc30, 0xdc28220
Caught exception in looking up space point ptr... 0x13aa6860, 0x5e2d3ac0
Caught exception in looking up space point ptr... 0x5e2d3ac0, 0x13aa6860
++++>>>> total num hits: 67047, num free: 63145
04-Feb-2026 01:58:45 GMT  Closed output file "np02vd_raw_run042421_0008_df-s03-d0_dw_0_20260203T133515_reco_stage0_20260204T015845_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         89.9534       226.44        1063.73       112.385       264.191        28     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       8.8997e-05    0.000133963   0.000370747   0.000105383   6.83128e-05      28     
produce:tpcrawdecoder:PDVDTPCReader                12.2136       12.7662       17.1764       12.6611      0.872924        28     
produce:triggerrawdecoder:PDVDTriggerReader4      0.524029      0.551614      0.605795      0.545788      0.0207648       28     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.017515      0.0178132     0.0183712     0.0177998    0.000191044      28     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0870657     0.0911508     0.106137      0.0880848     0.0065605       28     
produce:beamevent:BeamEvent                      9.4798e-05    0.000217485   0.00100499    0.000169619   0.00020374       28     
produce:pdvddaphne:DAPHNEReaderPDVD                3.37684       3.92137       4.81942       3.85937      0.344151        28     
produce:ophit:OpHitFinder                         0.0381721     0.0456021     0.0507749     0.0463075     0.002981        28     
produce:wclsdatavd:WireCellToolkit                 57.5182       74.6341       92.7254       72.9975       9.26852        28     
produce:gaushit:GausHitFinder                     0.616068       2.32091       12.4154       1.20594       2.79006        28     
produce:nhitsfilter:NumberOfHitsFilter           0.000182223   0.000654604   0.00348635    0.00031898    0.000753426      28     
produce:reco3d:SpacePointSolver                    5.96657       32.1869       179.929       12.3587       46.6442        28     
produce:hitpdune:DisambigFromSpacePoints          0.0564288      1.54462       9.50229      0.111157       3.1164         28     
produce:cluster3d:Cluster3D                       0.930326       91.9358       751.055       2.1576        202.983        28     
[art]:TriggerResults:TriggerResultInserter       2.2622e-05    3.65315e-05   0.000110718   3.13885e-05   2.0595e-05       28     
end_path:out1:RootOutput                          3.637e-06    1.40503e-05   0.000199145    5.24e-06     3.60441e-05      28     
end_path:out1:RootOutput(write)                    3.28419       6.37368       19.6318       4.20057       5.12287        28     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 15433.4 MB
  Peak resident set size usage (VmHWM): 12692.7 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042421_0008_df-s03-d0_dw_0_20260203T133515_reco_stage0_20260204T015845_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042421_0008_df-s03-d0_dw_0_20260203T133515_reco_stage0_20260204T015845_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042421_0008_df-s03-d0_dw_0_20260203T133515_reco_stage0_20260204T015845_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770163912.0,
    "core.end_time": 1770170327.0,
    "core.events": [
      1310,
      1316,
      1322,
      1328,
      1334,
      1340,
      1346,
      1352,
      1358,
      1364,
      1370,
      1376,
      1382,
      1388,
      1394,
      1400,
      1406,
      1412,
      1418,
      1424,
      1430,
      1436,
      1442,
      1448,
      1454,
      1460,
      1466,
      1472
    ],
    "core.event_count": 28,
    "core.first_event_number": 1310,
    "core.last_event_number": 1472,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42421
    ],
    "core.runs_subruns": [
      4242100001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042421_0008_df-s03-d0_dw_0_20260203T133515.hdf5"
    }
  ]
}
justIN time: 2026-02-04 06:06:28 UTC       justIN version: 01.06.00