justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297921.11@dunegpschedd02.fnal.gov

Jobsub ID297921.11@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:58:26
SiteUK_RAL-Tier1
EntryLIGO_UK_RAL_arc_ce01
Last heartbeat2026-02-04 01:05:34
From worker nodeHostnamedune001-1527158.0-lcg2545.gridpp.rl.ac.uk
cpuinfoAMD EPYC 7452 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit216000 (60 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:45:45
Input filesvd-protodune:np02vd_raw_run042433_0471_df-s02-d1_dw_0_20260203T214107.hdf5
JobscriptExit code0
Real time1h (4663s)
CPU time1h (4350s = 93%)
Max RSS bytes3387453440 (3230 MiB)
Outputting started2026-02-04 01:03:30
Output files
Finished2026-02-04 01:05:34
Saved logsjustin-logs:297921.11-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

0x8ba3d6d0, 0x8ba3bc20
Caught exception in looking up space point ptr... 0x2b300a50, 0x2b2f97f0
Caught exception in looking up space point ptr... 0x6115fcf0, 0x2b300a50
Caught exception in looking up space point ptr... 0x63fa17b0, 0x63f9e0b0
Caught exception in looking up space point ptr... 0x63f9e0b0, 0x63fa17b0
Caught exception in looking up space point ptr... 0x4c438230, 0x4c437f10
Caught exception in looking up space point ptr... 0x4c438550, 0x4c438230
Caught exception in looking up space point ptr... 0x59bf9130, 0x73dd2040
Caught exception in looking up space point ptr... 0x73dd2040, 0x59bf9130
Caught exception in looking up space point ptr... 0x46932130, 0x4dc3aa30
Caught exception in looking up space point ptr... 0x4dc3aa30, 0x46932130
Caught exception in looking up space point ptr... 0x4f7b6e80, 0x6148d810
Caught exception in looking up space point ptr... 0x6148d810, 0x4f7b6e80
Caught exception in looking up space point ptr... 0x2425d5d0, 0x2425d8f0
Caught exception in looking up space point ptr... 0x2425d8f0, 0x6642dcf0
Caught exception in looking up space point ptr... 0x4bbd1710, 0x4bbd0880
Caught exception in looking up space point ptr... 0x4bbd0880, 0x4bbd1710
Caught exception in looking up space point ptr... 0x62e11da0, 0x62e12090
Caught exception in looking up space point ptr... 0x62e12090, 0x62e11da0
Caught exception in looking up space point ptr... 0x62e11da0, 0x62e11ef0
Caught exception in looking up space point ptr... 0x62e11ef0, 0x62e11da0
Caught exception in looking up space point ptr... 0x4d520b50, 0x4d520830
Caught exception in looking up space point ptr... 0x4d520830, 0x4d520b50
Caught exception in looking up space point ptr... 0x4d520b50, 0x4d520220
Caught exception in looking up space point ptr... 0x4d520220, 0x4d520b50
Caught exception in looking up space point ptr... 0x4d520b50, 0x4d51fcd0
Caught exception in looking up space point ptr... 0x4d51fcd0, 0x4d520b50
Caught exception in looking up space point ptr... 0x6b81bf90, 0x6b81be40
Caught exception in looking up space point ptr... 0x6b81be40, 0x6b81bf90
Caught exception in looking up space point ptr... 0x26905210, 0x26904e20
Caught exception in looking up space point ptr... 0x26904e20, 0x26905210
Caught exception in looking up space point ptr... 0x709dedf0, 0x709df110
Caught exception in looking up space point ptr... 0x709df110, 0x709dedf0
Caught exception in looking up space point ptr... 0x5f35b0f0, 0x5f3575a0
Caught exception in looking up space point ptr... 0x5f3575a0, 0x5f35b0f0
Caught exception in looking up space point ptr... 0x49596020, 0x49595ef0
Caught exception in looking up space point ptr... 0x49595ef0, 0x49596020
Caught exception in looking up space point ptr... 0x56ddf1a0, 0x56ddcae0
Caught exception in looking up space point ptr... 0x56ddf510, 0x56ddf1a0
Caught exception in looking up space point ptr... 0x22f722f0, 0x4f796280
Caught exception in looking up space point ptr... 0x4f796280, 0x22f722f0
Caught exception in looking up space point ptr... 0x2b96ce00, 0x2b96cd00
Caught exception in looking up space point ptr... 0x2b96cd00, 0x2b96ce00
Caught exception in looking up space point ptr... 0x4ddec520, 0x4ddec720
Caught exception in looking up space point ptr... 0x4ddec720, 0x4ddec520
Caught exception in looking up space point ptr... 0x5062ceb0, 0x5062d4f0
Caught exception in looking up space point ptr... 0x5062d4f0, 0x5062d360
Caught exception in looking up space point ptr... 0x27647d00, 0x2764c1a0
Caught exception in looking up space point ptr... 0x2764c1a0, 0x27647d00
Caught exception in looking up space point ptr... 0x6fc8fe90, 0x6fc8fff0
Caught exception in looking up space point ptr... 0x6fc8fff0, 0x2ee4aa30
Caught exception in looking up space point ptr... 0x4d525ee0, 0x4d5251d0
Caught exception in looking up space point ptr... 0x4d5251d0, 0x4d525a60
Caught exception in looking up space point ptr... 0x7761b660, 0x7761b340
Caught exception in looking up space point ptr... 0x7761b340, 0x7761b660
Caught exception in looking up space point ptr... 0x4d6857d0, 0x54da7130
Caught exception in looking up space point ptr... 0x54da7130, 0x4d6856d0
Caught exception in looking up space point ptr... 0x6ae189a0, 0x5930b0c0
Caught exception in looking up space point ptr... 0x5930b0c0, 0x7d5ff2c0
Caught exception in looking up space point ptr... 0x1edd52d0, 0x54109300
Caught exception in looking up space point ptr... 0x54109300, 0x1edd52d0
Caught exception in looking up space point ptr... 0x833bb430, 0x833bb0c0
Caught exception in looking up space point ptr... 0x833bb0c0, 0x833bb430
Caught exception in looking up space point ptr... 0x67254030, 0x67254180
Caught exception in looking up space point ptr... 0x67254180, 0x56c89d10
Caught exception in looking up space point ptr... 0x9d6a1630, 0x595dc730
Caught exception in looking up space point ptr... 0x595dc730, 0x9d6a1630
++++>>>> total num hits: 155042, num free: 148985
04-Feb-2026 01:03:15 UTC  Closed output file "np02vd_raw_run042433_0471_df-s02-d1_dw_0_20260203T214107_reco_stage0_20260204T010315_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         206.158       238.513       266.98        239.598       16.2241        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       0.00010117    0.00015716    0.000345332   0.00012601     7.543e-05       19     
produce:tpcrawdecoder:PDVDTPCReader                17.9471       18.4121       19.2563       18.3566      0.395791        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.687845      0.701737      0.731726      0.699593      0.0122767       19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0230931     0.0237652     0.0310677     0.0233142    0.00173176       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0230434     0.0233322     0.0238595     0.0233068    0.000165766      19     
produce:beamevent:BeamEvent                      0.000152111   0.000316681   0.00138953    0.000252531   0.000262767      19     
produce:pdvddaphne:DAPHNEReaderPDVD                3.98064        4.609        5.17278       4.67404      0.295974        19     
produce:ophit:OpHitFinder                         0.0394621     0.0549089     0.0687266     0.0555313    0.00690693       19     
produce:wclsdatavd:WireCellToolkit                 143.016       165.442       181.474       166.657       11.3885        19     
produce:gaushit:GausHitFinder                      1.39999       2.22638       3.43351       2.23919      0.493627        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000428203   0.000696214   0.00144379    0.000648463   0.000202202      19     
produce:reco3d:SpacePointSolver                    22.303        33.5284       59.0993       31.4252       7.99493        19     
produce:hitpdune:DisambigFromSpacePoints          0.226916      0.348571      0.605393      0.343185      0.0844488       19     
produce:cluster3d:Cluster3D                        3.02658       4.89284       9.9678        4.1253        1.92791        19     
[art]:TriggerResults:TriggerResultInserter        2.41e-05     4.62587e-05   0.000128571    3.491e-05    2.92748e-05      19     
end_path:out1:RootOutput                          4.46e-06     3.76286e-05   0.000494682    1.088e-05    0.000107961      19     
end_path:out1:RootOutput(write)                    6.33701       8.16681       9.66639       8.48693      0.946654        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5549.22 MB
  Peak resident set size usage (VmHWM): 3387.45 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0471_df-s02-d1_dw_0_20260203T214107_reco_stage0_20260204T010315_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0471_df-s02-d1_dw_0_20260203T214107_reco_stage0_20260204T010315_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0471_df-s02-d1_dw_0_20260203T214107_reco_stage0_20260204T010315_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770162362.0,
    "core.end_time": 1770166998.0,
    "core.events": [
      53701,
      53707,
      53713,
      53719,
      53725,
      53731,
      53737,
      53743,
      53749,
      53755,
      53761,
      53767,
      53773,
      53779,
      53785,
      53791,
      53797,
      53803,
      53809
    ],
    "core.event_count": 19,
    "core.first_event_number": 53701,
    "core.last_event_number": 53809,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0471_df-s02-d1_dw_0_20260203T214107.hdf5"
    }
  ]
}
justIN time: 2026-02-04 10:40:37 UTC       justIN version: 01.06.00