justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297921.94@dunegpschedd02.fnal.gov

Jobsub ID297921.94@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:58:26
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce504
Last heartbeat2026-02-04 01:05:59
From worker nodeHostnameb9p06p4022.cern.ch
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:54:28
Input filesvd-protodune:np02vd_raw_run042433_0316_df-s03-d0_dw_0_20260203T204214.hdf5
JobscriptExit code0
Real time1h (4149s)
CPU time1h (3992s = 96%)
Max RSS bytes3381604352 (3224 MiB)
Outputting started2026-02-04 01:03:39
Output files
Finished2026-02-04 01:05:59
Saved logsjustin-logs:297921.94-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

nt ptr... 0x87817d40, 0x87817ae0
Caught exception in looking up space point ptr... 0x87817ae0, 0x878179b0
Caught exception in looking up space point ptr... 0x627f49a0, 0x9e06280
Caught exception in looking up space point ptr... 0x12f07c40, 0x3add1490
Caught exception in looking up space point ptr... 0x3add1490, 0x627f49a0
Caught exception in looking up space point ptr... 0x5c2a2cf0, 0x5c2a27f0
Caught exception in looking up space point ptr... 0x5c2a27f0, 0x5c2a2cf0
Caught exception in looking up space point ptr... 0xf6a2a60, 0x1224da80
Caught exception in looking up space point ptr... 0x1224da80, 0xf6af440
Caught exception in looking up space point ptr... 0x805f30a0, 0x805f50b0
Caught exception in looking up space point ptr... 0x805f50b0, 0x805f30a0
Caught exception in looking up space point ptr... 0x130b9750, 0x12ee3950
Caught exception in looking up space point ptr... 0x12ee3950, 0x130b9750
Caught exception in looking up space point ptr... 0x130b9750, 0x19a4afe0
Caught exception in looking up space point ptr... 0x19a4afe0, 0x130b9750
Caught exception in looking up space point ptr... 0x130b9750, 0xf69c390
Caught exception in looking up space point ptr... 0xf69c390, 0x130b9750
Caught exception in looking up space point ptr... 0x130b9750, 0x610bc8f0
Caught exception in looking up space point ptr... 0x610bc8f0, 0x130b9750
Caught exception in looking up space point ptr... 0x5bb9c430, 0x5bb9e850
Caught exception in looking up space point ptr... 0x5bb9e850, 0x5bb9e5f0
Caught exception in looking up space point ptr... 0x314bd990, 0x314bda90
Caught exception in looking up space point ptr... 0x314bda90, 0x314c6e00
Caught exception in looking up space point ptr... 0x87832f90, 0x878330c0
Caught exception in looking up space point ptr... 0x878330c0, 0x87833320
Caught exception in looking up space point ptr... 0x5c29bb00, 0x5c29ba00
Caught exception in looking up space point ptr... 0x5c29ba00, 0x5c29bca0
Caught exception in looking up space point ptr... 0x6870b8c0, 0x6870a3c0
Caught exception in looking up space point ptr... 0x6870a3c0, 0x6870b8c0
Caught exception in looking up space point ptr... 0x1225db40, 0x1225da60
Caught exception in looking up space point ptr... 0x1225da60, 0x1225db40
Caught exception in looking up space point ptr... 0x1225db40, 0x1225d980
Caught exception in looking up space point ptr... 0x1225d980, 0x1225db40
Caught exception in looking up space point ptr... 0x8001d9e0, 0x8001db70
Caught exception in looking up space point ptr... 0x8001db70, 0x8001de90
Caught exception in looking up space point ptr... 0x37086b30, 0x37086a00
Caught exception in looking up space point ptr... 0x37086a00, 0x37086b30
Caught exception in looking up space point ptr... 0x12261bf0, 0x12260d50
Caught exception in looking up space point ptr... 0x12260d50, 0x12261bf0
Caught exception in looking up space point ptr... 0x12077b60, 0x12088380
Caught exception in looking up space point ptr... 0x12088380, 0x12076aa0
Caught exception in looking up space point ptr... 0x100bf510, 0x100bea10
Caught exception in looking up space point ptr... 0x100bea10, 0x100ba560
Caught exception in looking up space point ptr... 0x36700a80, 0x36709230
Caught exception in looking up space point ptr... 0x36709230, 0x36700a80
Caught exception in looking up space point ptr... 0x13b4d050, 0x31e2da70
Caught exception in looking up space point ptr... 0x31e2da70, 0x13b4d050
Caught exception in looking up space point ptr... 0x800168b0, 0x80014380
Caught exception in looking up space point ptr... 0x80014380, 0x800168b0
Caught exception in looking up space point ptr... 0x2f541a50, 0x2f541ba0
Caught exception in looking up space point ptr... 0x2f541ba0, 0x2f541a50
Caught exception in looking up space point ptr... 0x4e9b4e30, 0x4e9acb10
Caught exception in looking up space point ptr... 0x4e9acb10, 0x4e9b4e30
Caught exception in looking up space point ptr... 0x68835400, 0x68834de0
Caught exception in looking up space point ptr... 0x68834de0, 0x68835cb0
Caught exception in looking up space point ptr... 0x36ef1c30, 0x36ef1910
Caught exception in looking up space point ptr... 0x36ef1910, 0x36ef1c30
Caught exception in looking up space point ptr... 0x36ef1c30, 0x36ef1aa0
Caught exception in looking up space point ptr... 0x36ef1aa0, 0x36ef1c30
Caught exception in looking up space point ptr... 0x36ef1c30, 0x36ef1780
Caught exception in looking up space point ptr... 0x36ef1780, 0x36ef1c30
Caught exception in looking up space point ptr... 0x77b7f60, 0x85670d10
Caught exception in looking up space point ptr... 0x6e3c62d0, 0x77b7f60
Caught exception in looking up space point ptr... 0x36ee0ee0, 0x36ee11c0
Caught exception in looking up space point ptr... 0x36ee11c0, 0x36ee1030
Caught exception in looking up space point ptr... 0x7328db40, 0x7328e010
Caught exception in looking up space point ptr... 0x7328e010, 0x7328db40
++++>>>> total num hits: 88411, num free: 84675
04-Feb-2026 02:03:32 CET  Closed output file "np02vd_raw_run042433_0316_df-s03-d0_dw_0_20260203T204214_reco_stage0_20260204T010332_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         124.033       212.631       248.361       223.841       35.5674        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       6.5141e-05     0.0002459    0.000927034   0.000208913   0.000186069      19     
produce:tpcrawdecoder:PDVDTPCReader                6.35173       9.14421       12.2123       9.05695       1.26686        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.0123476     0.0477146     0.369528      0.0272753     0.0770244       19     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000586399   0.00324682     0.0123592    0.00140169    0.00392299       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.000470197   0.00122258    0.00423075    0.000952255   0.000904869      19     
produce:beamevent:BeamEvent                      9.2311e-05    0.000422398   0.00456953    0.000160112   0.000982152      19     
produce:pdvddaphne:DAPHNEReaderPDVD                2.12668       3.72002       4.8739        3.75998      0.759768        19     
produce:ophit:OpHitFinder                         0.0304089     0.0498372     0.067972      0.0496555    0.00990901       19     
produce:wclsdatavd:WireCellToolkit                 87.9615       163.716       202.794       172.436       32.2065        19     
produce:gaushit:GausHitFinder                     0.884846       2.20848       3.28807       2.12121      0.683014        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000263104   0.00150257    0.00332964    0.00126442    0.00108883       19     
produce:reco3d:SpacePointSolver                    12.947        22.3108       34.7367       19.7686       7.07374        19     
produce:hitpdune:DisambigFromSpacePoints          0.126382      0.325655      0.673743      0.328275      0.141771        19     
produce:cluster3d:Cluster3D                        1.61958       4.48807       9.54984       4.09978       2.19478        19     
[art]:TriggerResults:TriggerResultInserter       2.5431e-05    5.96498e-05   0.000147722   5.3451e-05    3.05448e-05      19     
end_path:out1:RootOutput                          8.72e-06     2.83675e-05   7.6632e-05    2.6131e-05    1.71617e-05      19     
end_path:out1:RootOutput(write)                    4.2908        6.53224       7.68874       6.7593       0.964184        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5350.89 MB
  Peak resident set size usage (VmHWM): 3381.6 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0316_df-s03-d0_dw_0_20260203T204214_reco_stage0_20260204T010332_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0316_df-s03-d0_dw_0_20260203T204214_reco_stage0_20260204T010332_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0316_df-s03-d0_dw_0_20260203T204214_reco_stage0_20260204T010332_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770162888.0,
    "core.end_time": 1770167013.0,
    "core.events": [
      36032,
      36038,
      36044,
      36050,
      36056,
      36062,
      36068,
      36074,
      36080,
      36086,
      36092,
      36098,
      36104,
      36110,
      36116,
      36122,
      36128,
      36134,
      36140
    ],
    "core.event_count": 19,
    "core.first_event_number": 36032,
    "core.last_event_number": 36140,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0316_df-s03-d0_dw_0_20260203T204214.hdf5"
    }
  ]
}
justIN time: 2026-02-04 07:36:10 UTC       justIN version: 01.06.00