justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303656.21@dunegpschedd01.fnal.gov

Jobsub ID303656.21@dunegpschedd01.fnal.gov
Workflow ID12717
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:56:25
SiteUK_RAL-PPD
EntryCMSHTPC_T2_UK_SGrid_RALPP_hep206
Last heartbeat2026-02-04 01:07:53
From worker nodeHostnameheplnc147.pp.rl.ac.uk
cpuinfoAMD EPYC 7452 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-04 00:12:01
Input filesvd-protodune:np02vd_raw_run042421_0087_df-s04-d0_dw_0_20260203T141805.hdf5
JobscriptExit code0
Real time53m (3207s)
CPU time45m (2734s = 85%)
Max RSS bytes2583379968 (2463 MiB)
Outputting started2026-02-04 01:05:30
Output files
Finished2026-02-04 01:07:53
Saved logsjustin-logs:303656.21-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

04000
Caught exception in looking up space point ptr... 0x2f6ef650, 0x2bf8e810
Caught exception in looking up space point ptr... 0x2bf8e810, 0x3de87fe0
Caught exception in looking up space point ptr... 0x246c0cc0, 0x246c0ba0
Caught exception in looking up space point ptr... 0x246c0ba0, 0x246c0cc0
Caught exception in looking up space point ptr... 0x67af10a0, 0x67af0dd0
Caught exception in looking up space point ptr... 0x67af0dd0, 0x67af0690
Caught exception in looking up space point ptr... 0x42f28f80, 0x965b130
Caught exception in looking up space point ptr... 0x965b130, 0x2d175190
Caught exception in looking up space point ptr... 0x12aa3880, 0x187cc6f0
Caught exception in looking up space point ptr... 0x187cc6f0, 0x12aa36a0
Caught exception in looking up space point ptr... 0x13d3f5f0, 0x1601a4a0
Caught exception in looking up space point ptr... 0x1601a4a0, 0x1601b070
Caught exception in looking up space point ptr... 0x1601a6d0, 0x13d3f5f0
Caught exception in looking up space point ptr... 0x67aed480, 0x67adf290
Caught exception in looking up space point ptr... 0x67adf290, 0x67aed480
Caught exception in looking up space point ptr... 0x67aed480, 0x67aeb450
Caught exception in looking up space point ptr... 0x67aeb450, 0x67aed480
Caught exception in looking up space point ptr... 0xf9c9410, 0xf9cdf30
Caught exception in looking up space point ptr... 0xf9cdf30, 0xf9ce1d0
Caught exception in looking up space point ptr... 0xf9ce1d0, 0xf9ce080
Caught exception in looking up space point ptr... 0xf9ce080, 0xf9cde30
Caught exception in looking up space point ptr... 0x2e41b670, 0x2e419ee0
Caught exception in looking up space point ptr... 0x2e419c90, 0x2e41b670
Caught exception in looking up space point ptr... 0x66310c50, 0x662feb70
Caught exception in looking up space point ptr... 0x352e3e10, 0x66310c50
Caught exception in looking up space point ptr... 0xd379220, 0x59e22c70
Caught exception in looking up space point ptr... 0x59e22c70, 0xf77b990
Caught exception in looking up space point ptr... 0x2ad11530, 0x2ad0e370
Caught exception in looking up space point ptr... 0x2ad0e370, 0x2ad06760
Caught exception in looking up space point ptr... 0x2ad17e40, 0x2ad11530
Caught exception in looking up space point ptr... 0x3de81330, 0x3de81530
Caught exception in looking up space point ptr... 0x3de81530, 0x3de81430
Caught exception in looking up space point ptr... 0x3de81430, 0x3de779b0
Caught exception in looking up space point ptr... 0x4a545470, 0x22c449d0
Caught exception in looking up space point ptr... 0x22c449d0, 0x4a545470
Caught exception in looking up space point ptr... 0x151afa90, 0x4febd1d0
Caught exception in looking up space point ptr... 0x4febd1d0, 0x151afa90
Caught exception in looking up space point ptr... 0x664ada20, 0x664ad720
Caught exception in looking up space point ptr... 0x664ad720, 0x664ada20
Caught exception in looking up space point ptr... 0x664bae20, 0x664b2e90
Caught exception in looking up space point ptr... 0x664b2e90, 0x664acee0
Caught exception in looking up space point ptr... 0x2d56f8a0, 0x248eef70
Caught exception in looking up space point ptr... 0x248eef70, 0x2d56f8a0
Caught exception in looking up space point ptr... 0x67ae2860, 0x4a6b58a0
Caught exception in looking up space point ptr... 0x4a6b58a0, 0x67ae1f60
Caught exception in looking up space point ptr... 0x24682b60, 0x14d5d540
Caught exception in looking up space point ptr... 0x14d5d540, 0x24682b60
Caught exception in looking up space point ptr... 0x6a141aa0, 0x4e88fe80
Caught exception in looking up space point ptr... 0x4e88fe80, 0x6a141580
Caught exception in looking up space point ptr... 0x31c68390, 0x306fc740
Caught exception in looking up space point ptr... 0x306fc740, 0x31c68390
Caught exception in looking up space point ptr... 0x26c42310, 0x30e7b880
Caught exception in looking up space point ptr... 0x30e7b880, 0x26c42310
Caught exception in looking up space point ptr... 0x30800c90, 0x308009f0
Caught exception in looking up space point ptr... 0x308009f0, 0x30800c90
Caught exception in looking up space point ptr... 0x2bf8cf20, 0x2bf8c440
Caught exception in looking up space point ptr... 0x2bf8c440, 0x2bf8ce20
Caught exception in looking up space point ptr... 0x25f2ff90, 0x25f2fae0
Caught exception in looking up space point ptr... 0x25f2fae0, 0x25f2ff90
Caught exception in looking up space point ptr... 0x352fc0b0, 0x352fbe30
Caught exception in looking up space point ptr... 0x352fbe30, 0x352ffe70
Caught exception in looking up space point ptr... 0x2dae96c0, 0x4e88f580
Caught exception in looking up space point ptr... 0x4e88f580, 0x2dae96c0
Caught exception in looking up space point ptr... 0x2acf61b0, 0x2acf6990
Caught exception in looking up space point ptr... 0x2acf6990, 0x2acf6b60
++++>>>> total num hits: 54545, num free: 52660
04-Feb-2026 01:05:20 GMT  Closed output file "np02vd_raw_run042421_0087_df-s04-d0_dw_0_20260203T141805_reco_stage0_20260204T010520_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         95.5956       114.693       134.641       115.259       10.5759        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       8.3858e-05    0.000130067   0.000303482   0.000105248   4.93926e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                13.2654       15.5493       18.2951       15.6114       1.03313        27     
produce:triggerrawdecoder:PDVDTriggerReader4      0.524025      0.547592      0.598741      0.543891      0.0211215       27     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0177828     0.018236      0.0255171     0.0179415    0.00143285       27     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0872575     0.0935262     0.106646      0.0887853    0.00786165       27     
produce:beamevent:BeamEvent                      0.000105369   0.000197163   0.00137815    0.000151085   0.00023334       27     
produce:pdvddaphne:DAPHNEReaderPDVD                3.60821       3.99383       4.29558       3.98798      0.178685        27     
produce:ophit:OpHitFinder                         0.0333034     0.0434316     0.0497536     0.0450895    0.00402707       27     
produce:wclsdatavd:WireCellToolkit                 60.9828       75.4108       86.1394       75.514        7.34913        27     
produce:gaushit:GausHitFinder                     0.608109       1.10964       1.75476       1.08422      0.326611        27     
produce:nhitsfilter:NumberOfHitsFilter           0.000195169   0.000337312   0.000908044   0.000305517   0.000136498      27     
produce:reco3d:SpacePointSolver                    5.60432       11.5295       21.6784       11.0691       4.27296        27     
produce:hitpdune:DisambigFromSpacePoints          0.0615856      0.12953      0.238032      0.129075      0.0480064       27     
produce:cluster3d:Cluster3D                       0.906838       2.13707       4.30771       2.12323      0.807292        27     
[art]:TriggerResults:TriggerResultInserter       2.2592e-05    3.48126e-05   0.000124395   2.9396e-05    2.0671e-05       27     
end_path:out1:RootOutput                          3.507e-06    7.95348e-06   2.9205e-05     7.464e-06    4.89849e-06      27     
end_path:out1:RootOutput(write)                    3.35423       4.09972       5.37602       4.00877      0.603209        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4546.12 MB
  Peak resident set size usage (VmHWM): 2583.38 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042421_0087_df-s04-d0_dw_0_20260203T141805_reco_stage0_20260204T010520_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042421_0087_df-s04-d0_dw_0_20260203T141805_reco_stage0_20260204T010520_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042421_0087_df-s04-d0_dw_0_20260203T141805_reco_stage0_20260204T010520_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770163936.0,
    "core.end_time": 1770167122.0,
    "core.events": [
      14152,
      14158,
      14164,
      14170,
      14176,
      14182,
      14188,
      14194,
      14200,
      14206,
      14212,
      14218,
      14224,
      14230,
      14236,
      14242,
      14248,
      14254,
      14260,
      14266,
      14272,
      14278,
      14284,
      14290,
      14296,
      14302,
      14308
    ],
    "core.event_count": 27,
    "core.first_event_number": 14152,
    "core.last_event_number": 14308,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42421
    ],
    "core.runs_subruns": [
      4242100001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042421_0087_df-s04-d0_dw_0_20260203T141805.hdf5"
    }
  ]
}
justIN time: 2026-02-04 04:28:02 UTC       justIN version: 01.06.00