justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303646.174@dunegpschedd01.fnal.gov

Jobsub ID303646.174@dunegpschedd01.fnal.gov
Workflow ID12718
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:44:24
SiteUK_Lancaster
EntryUBoone_UK_Lancaster_HEC_grendel_ce02
Last heartbeat2026-02-04 01:41:11
From worker nodeHostnamecomp08-00
cpuinfoIntel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 22:46:04
Input filesvd-protodune:np02vd_raw_run042422_0092_df-s03-d0_dw_0_20260203T160431.hdf5
JobscriptExit code0
Real time2h (10290s)
CPU time43m (2581s = 25%)
Max RSS bytes2943057920 (2806 MiB)
Outputting started2026-02-04 01:37:36
Output files
Finished2026-02-04 01:41:11
Saved logsjustin-logs:303646.174-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

f0, 0x59e32c40
Caught exception in looking up space point ptr... 0x59e32c40, 0x59e32f60
Caught exception in looking up space point ptr... 0x59e32f60, 0x59e32dd0
Caught exception in looking up space point ptr... 0x43797170, 0x2a1733c0
Caught exception in looking up space point ptr... 0x290eea20, 0x5e380830
Caught exception in looking up space point ptr... 0x5e380830, 0x43797170
Caught exception in looking up space point ptr... 0x33329140, 0x33328e20
Caught exception in looking up space point ptr... 0x33324db0, 0x33329140
Caught exception in looking up space point ptr... 0x4f6713d0, 0x4f6712d0
Caught exception in looking up space point ptr... 0x4f6712d0, 0x4f6711d0
Caught exception in looking up space point ptr... 0x5822aef0, 0x58245f70
Caught exception in looking up space point ptr... 0x58245f70, 0x5822aef0
Caught exception in looking up space point ptr... 0x3a4fb300, 0x4183ca60
Caught exception in looking up space point ptr... 0x4183ca60, 0x3a4fb300
Caught exception in looking up space point ptr... 0x4975f1a0, 0x497646c0
Caught exception in looking up space point ptr... 0x497646c0, 0x49754790
Caught exception in looking up space point ptr... 0x49741c20, 0x33312120
Caught exception in looking up space point ptr... 0x33312120, 0x33312320
Caught exception in looking up space point ptr... 0x121af120, 0x2a16fa00
Caught exception in looking up space point ptr... 0x447b8010, 0x121af120
Caught exception in looking up space point ptr... 0xd6c31a0, 0x142c7730
Caught exception in looking up space point ptr... 0x142c7730, 0xd6c31a0
Caught exception in looking up space point ptr... 0x9da86f0, 0x9da85f0
Caught exception in looking up space point ptr... 0x9da85f0, 0x9da86f0
Caught exception in looking up space point ptr... 0x28528510, 0x4d92fca0
Caught exception in looking up space point ptr... 0x4d92fca0, 0x28528a70
Caught exception in looking up space point ptr... 0x57625f80, 0x57624f10
Caught exception in looking up space point ptr... 0x57624f10, 0x576251a0
Caught exception in looking up space point ptr... 0x249b89e0, 0x249ab910
Caught exception in looking up space point ptr... 0x249ab7c0, 0x249b89e0
Caught exception in looking up space point ptr... 0x49d1ca70, 0x49d1c8e0
Caught exception in looking up space point ptr... 0x2a0dd7e0, 0x49d1ca70
Caught exception in looking up space point ptr... 0x5c108760, 0x5c1090f0
Caught exception in looking up space point ptr... 0x5c1090f0, 0x5c108760
Caught exception in looking up space point ptr... 0x3a4c7db0, 0x39762dd0
Caught exception in looking up space point ptr... 0x39762dd0, 0x3a4c7db0
Caught exception in looking up space point ptr... 0x5761f0f0, 0x5761edd0
Caught exception in looking up space point ptr... 0x5761edd0, 0x5761f0f0
Caught exception in looking up space point ptr... 0x3e167d60, 0x28a05c90
Caught exception in looking up space point ptr... 0x4b2633f0, 0x3e167d60
Caught exception in looking up space point ptr... 0x5a53d720, 0x26c63b30
Caught exception in looking up space point ptr... 0x26c63b30, 0x5e054ca0
Caught exception in looking up space point ptr... 0x28c5f910, 0xfb7d440
Caught exception in looking up space point ptr... 0xfb7d440, 0x29fc5e90
Caught exception in looking up space point ptr... 0x4f739d70, 0x285298f0
Caught exception in looking up space point ptr... 0x285298f0, 0x4f739d70
Caught exception in looking up space point ptr... 0x4f7c5050, 0x4f788cd0
Caught exception in looking up space point ptr... 0x4f788cd0, 0x4f788dd0
Caught exception in looking up space point ptr... 0x3bd2d3b0, 0x7d16ae0
Caught exception in looking up space point ptr... 0x7d16ae0, 0x4f63ed50
Caught exception in looking up space point ptr... 0x27e31750, 0x3b487800
Caught exception in looking up space point ptr... 0x3b487800, 0x27e31750
Caught exception in looking up space point ptr... 0x7cefb50, 0x7cefca0
Caught exception in looking up space point ptr... 0x7cefca0, 0x7cefb50
Caught exception in looking up space point ptr... 0x7a49f00, 0x7a48d60
Caught exception in looking up space point ptr... 0x7a48d60, 0x7a37fd0
Caught exception in looking up space point ptr... 0x235d5d10, 0x48aefd60
Caught exception in looking up space point ptr... 0x48aefd60, 0x235d5d10
Caught exception in looking up space point ptr... 0x2c62cee0, 0x48bab8e0
Caught exception in looking up space point ptr... 0x48bab8e0, 0x48baba30
Caught exception in looking up space point ptr... 0x3bb51c10, 0x3bb51d90
Caught exception in looking up space point ptr... 0x43ace240, 0x3bb51c10
Caught exception in looking up space point ptr... 0xe06ebc0, 0xe026e60
Caught exception in looking up space point ptr... 0xe026e60, 0x3b321ba0
Caught exception in looking up space point ptr... 0x249b58a0, 0x54895670
Caught exception in looking up space point ptr... 0x54895670, 0x54897350
++++>>>> total num hits: 78440, num free: 75729
04-Feb-2026 01:35:22 GMT  Closed output file "np02vd_raw_run042422_0092_df-s03-d0_dw_0_20260203T160431_reco_stage0_20260204T013522_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         248.725       371.563       790.585       346.767        122.9         27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       4.4894e-05    6.01469e-05   0.000131358   5.3952e-05    1.96018e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                119.888       212.518       357.322       182.465       66.3046        27     
produce:triggerrawdecoder:PDVDTriggerReader4      0.750734       1.16343       1.86553       1.15088      0.300456        27     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0230314     0.0307108     0.0407623     0.0318318    0.00517355       27     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.113901      0.140842      0.209532       0.14153      0.0237175       27     
produce:beamevent:BeamEvent                      4.7735e-05    9.40644e-05   0.000764149   6.5526e-05    0.000132608      27     
produce:pdvddaphne:DAPHNEReaderPDVD                14.4349       41.7932       284.55        27.7856       51.7385        27     
produce:ophit:OpHitFinder                         0.0267323     0.0332137     0.0694668     0.0323159    0.00736118       27     
produce:wclsdatavd:WireCellToolkit                 64.5705       72.6196       107.239       67.7987       10.8227        27     
produce:gaushit:GausHitFinder                      0.61569       1.0024        1.79411      0.910188      0.332351        27     
produce:nhitsfilter:NumberOfHitsFilter           0.000166576   0.000294189   0.000502897   0.000271257   8.62804e-05      27     
produce:reco3d:SpacePointSolver                    4.92511       8.31069       15.7802       7.58282       2.51419        27     
produce:hitpdune:DisambigFromSpacePoints          0.0590109      0.12882      0.352088      0.0931567     0.0852225       27     
produce:cluster3d:Cluster3D                       0.857838       7.07357       63.3922       1.44387       13.8429        27     
[art]:TriggerResults:TriggerResultInserter       1.6332e-05    2.27026e-05   6.8599e-05     2.031e-05    9.71027e-06      27     
end_path:out1:RootOutput                          6.616e-06    8.9813e-06    2.1971e-05     7.68e-06     3.20972e-06      27     
end_path:out1:RootOutput(write)                    2.99608       26.7287       445.692       4.11538       86.0089        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4903.72 MB
  Peak resident set size usage (VmHWM): 2943.06 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042422_0092_df-s03-d0_dw_0_20260203T160431_reco_stage0_20260204T013522_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042422_0092_df-s03-d0_dw_0_20260203T160431_reco_stage0_20260204T013522_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042422_0092_df-s03-d0_dw_0_20260203T160431_reco_stage0_20260204T013522_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770158773.0,
    "core.end_time": 1770169047.0,
    "core.events": [
      14918,
      14924,
      14930,
      14936,
      14942,
      14948,
      14954,
      14960,
      14966,
      14972,
      14978,
      14984,
      14990,
      14996,
      15002,
      15008,
      15014,
      15020,
      15026,
      15032,
      15038,
      15044,
      15050,
      15056,
      15062,
      15068,
      15074
    ],
    "core.event_count": 27,
    "core.first_event_number": 14918,
    "core.last_event_number": 15074,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42422
    ],
    "core.runs_subruns": [
      4242200001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042422_0092_df-s03-d0_dw_0_20260203T160431.hdf5"
    }
  ]
}
justIN time: 2026-02-04 07:04:25 UTC       justIN version: 01.06.00