justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297920.44@dunegpschedd02.fnal.gov

Jobsub ID297920.44@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:54:27
SiteUK_Imperial
EntryDUNE_T2_UK_London_IC_ceprod03
Last heartbeat2026-02-04 01:03:02
From worker nodeHostnamewm14.grid.hep.ph.ic.ac.uk
cpuinfoAMD EPYC 7742 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:38:11
Input filesvd-protodune:np02vd_raw_run042433_0401_df-s04-d0_dw_0_20260203T211433.hdf5
JobscriptExit code0
Real time1h (4924s)
CPU time1h (4342s = 88%)
Max RSS bytes3423641600 (3265 MiB)
Outputting started2026-02-04 01:00:17
Output files
Finished2026-02-04 01:03:02
Saved logsjustin-logs:297920.44-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

.. 0x4749d950, 0x469b6890
Caught exception in looking up space point ptr... 0x69df62b0, 0x69df6050
Caught exception in looking up space point ptr... 0x69df3ab0, 0x69df62b0
Caught exception in looking up space point ptr... 0x8151380, 0x7ec7e9c0
Caught exception in looking up space point ptr... 0x7ec7e9c0, 0x8151380
Caught exception in looking up space point ptr... 0x384081c0, 0x38408310
Caught exception in looking up space point ptr... 0x38408310, 0x384081c0
Caught exception in looking up space point ptr... 0x15e1a430, 0x38623fd0
Caught exception in looking up space point ptr... 0x38623fd0, 0x4c860730
Caught exception in looking up space point ptr... 0x47a0ae20, 0x47a27460
Caught exception in looking up space point ptr... 0x47a09eb0, 0x47a0ae20
Caught exception in looking up space point ptr... 0x4dd08d00, 0x4dd08170
Caught exception in looking up space point ptr... 0x4dd08170, 0x4dd08d00
Caught exception in looking up space point ptr... 0x5d70f510, 0x5d710440
Caught exception in looking up space point ptr... 0x5d710440, 0x5d70f510
Caught exception in looking up space point ptr... 0x59e6f970, 0x4791da80
Caught exception in looking up space point ptr... 0x4791da80, 0x59e6f970
Caught exception in looking up space point ptr... 0x6cde8210, 0x6cde83a0
Caught exception in looking up space point ptr... 0x6cde83a0, 0x508ab170
Caught exception in looking up space point ptr... 0x38406340, 0x2ec44ac0
Caught exception in looking up space point ptr... 0x2ec44ac0, 0x5e8b1eb0
Caught exception in looking up space point ptr... 0x4dd19110, 0x69def3c0
Caught exception in looking up space point ptr... 0x69def3c0, 0x4a98fa60
Caught exception in looking up space point ptr... 0x372cde70, 0x36b1da20
Caught exception in looking up space point ptr... 0x36b1da20, 0x372cdff0
Caught exception in looking up space point ptr... 0x3039bca0, 0x3039bb70
Caught exception in looking up space point ptr... 0x3039bb70, 0x3039bca0
Caught exception in looking up space point ptr... 0x3a07da00, 0x5e43a810
Caught exception in looking up space point ptr... 0x5e43a810, 0x3a07da00
Caught exception in looking up space point ptr... 0x56a9bab0, 0x57d2fc00
Caught exception in looking up space point ptr... 0x57d2fc00, 0x56a9bab0
Caught exception in looking up space point ptr... 0x7b2604c0, 0x7b260760
Caught exception in looking up space point ptr... 0x7b260760, 0x7b260610
Caught exception in looking up space point ptr... 0x3250d580, 0x131ecc30
Caught exception in looking up space point ptr... 0x131ecc30, 0x3250d580
Caught exception in looking up space point ptr... 0x69de4f90, 0x69def160
Caught exception in looking up space point ptr... 0x69def160, 0x69de4f90
Caught exception in looking up space point ptr... 0x40cff6e0, 0x40e1cb60
Caught exception in looking up space point ptr... 0x40e1cb60, 0x40cff6e0
Caught exception in looking up space point ptr... 0x7b32dca0, 0x451b2f30
Caught exception in looking up space point ptr... 0x451b2f30, 0x5ff9a300
Caught exception in looking up space point ptr... 0x3773ffa0, 0x68e26740
Caught exception in looking up space point ptr... 0x68e26740, 0x6e86a140
Caught exception in looking up space point ptr... 0x29f67f70, 0x115eca60
Caught exception in looking up space point ptr... 0x115eca60, 0x29f67f70
Caught exception in looking up space point ptr... 0x580ed9e0, 0x7ee47640
Caught exception in looking up space point ptr... 0x7ee47640, 0x580ed9e0
Caught exception in looking up space point ptr... 0x49544740, 0x5fbcc6c0
Caught exception in looking up space point ptr... 0x5fbcc6c0, 0x62198430
Caught exception in looking up space point ptr... 0x4e970860, 0x17818540
Caught exception in looking up space point ptr... 0x17818540, 0x4e970860
Caught exception in looking up space point ptr... 0x6a2403b0, 0x473721e0
Caught exception in looking up space point ptr... 0x4e733ce0, 0x6a2403b0
Caught exception in looking up space point ptr... 0x38baeb40, 0x7e6745d0
Caught exception in looking up space point ptr... 0x7e6745d0, 0x38baeb40
Caught exception in looking up space point ptr... 0x582a3be0, 0x447cef50
Caught exception in looking up space point ptr... 0x447cef50, 0x582a3be0
Caught exception in looking up space point ptr... 0x40a2b440, 0x3b3a2d60
Caught exception in looking up space point ptr... 0x3b3a2d60, 0x40a2b440
Caught exception in looking up space point ptr... 0x31993450, 0x31990af0
Caught exception in looking up space point ptr... 0x31990af0, 0x31993450
Caught exception in looking up space point ptr... 0x7f57a0a0, 0x628f6af0
Caught exception in looking up space point ptr... 0x628f6af0, 0x38302560
Caught exception in looking up space point ptr... 0x52b6be20, 0x52b6bfb0
Caught exception in looking up space point ptr... 0x52b6bfb0, 0x52b6c140
Caught exception in looking up space point ptr... 0x2debae80, 0x1461bbd0
Caught exception in looking up space point ptr... 0x1461bbd0, 0x2debae80
++++>>>> total num hits: 102812, num free: 98259
04-Feb-2026 01:00:07 GMT  Closed output file "np02vd_raw_run042433_0401_df-s04-d0_dw_0_20260203T211433_reco_stage0_20260204T010007_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         198.926       253.594       287.357       256.238       23.0892        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                        7.47e-05     0.000146695   0.00030781    0.00010589    7.29732e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                15.1575       17.6972       20.5182       18.1412       1.54587        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.479395      0.493211      0.520515      0.490417     0.00873247       19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0162572     0.0165398     0.0168849     0.016574     0.000216046      19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0162157     0.0165085     0.0169402     0.0164708    0.000214402      19     
produce:beamevent:BeamEvent                      7.1335e-05    0.000199141    0.0011795    0.000149712   0.000235321      19     
produce:pdvddaphne:DAPHNEReaderPDVD                3.69134       4.68651       5.5934        4.67739      0.427729        19     
produce:ophit:OpHitFinder                         0.037134      0.0514443     0.0779397     0.0510792    0.00805905       19     
produce:wclsdatavd:WireCellToolkit                 126.121       176.008       192.284       184.228       18.6232        19     
produce:gaushit:GausHitFinder                      1.62592       2.27188       3.38815       2.14254      0.472829        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000533526   0.000676476   0.000958738   0.000661197   9.80584e-05      19     
produce:reco3d:SpacePointSolver                    28.1205       36.1239       47.9382       35.2649       5.70603        19     
produce:hitpdune:DisambigFromSpacePoints          0.223182      0.338098      0.539465       0.31538      0.0842766       19     
produce:cluster3d:Cluster3D                        3.7419        5.64776       12.5263       5.15713       2.02844        19     
[art]:TriggerResults:TriggerResultInserter       2.4255e-05    4.08455e-05   0.000116931   3.7751e-05    1.92307e-05      19     
end_path:out1:RootOutput                          5.952e-06    2.47072e-05   0.000244601    9.097e-06    5.42233e-05      19     
end_path:out1:RootOutput(write)                    7.40801       10.1997       11.6039       10.5647       1.18152        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5338.55 MB
  Peak resident set size usage (VmHWM): 3423.64 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0401_df-s04-d0_dw_0_20260203T211433_reco_stage0_20260204T010007_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0401_df-s04-d0_dw_0_20260203T211433_reco_stage0_20260204T010007_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0401_df-s04-d0_dw_0_20260203T211433_reco_stage0_20260204T010007_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770161909.0,
    "core.end_time": 1770166809.0,
    "core.events": [
      45724,
      45730,
      45736,
      45742,
      45748,
      45754,
      45760,
      45766,
      45772,
      45778,
      45784,
      45790,
      45796,
      45802,
      45808,
      45814,
      45820,
      45826,
      45832
    ],
    "core.event_count": 19,
    "core.first_event_number": 45724,
    "core.last_event_number": 45832,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0401_df-s04-d0_dw_0_20260203T211433.hdf5"
    }
  ]
}
justIN time: 2026-02-04 04:28:16 UTC       justIN version: 01.06.00