justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303654.76@dunegpschedd01.fnal.gov

Jobsub ID303654.76@dunegpschedd01.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:52:26
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce504
Last heartbeat2026-02-04 00:58:54
From worker nodeHostnameb9p07p8370.cern.ch
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-04 00:02:07
Input filesvd-protodune:np02vd_raw_run042433_0069_df-s02-d1_dw_0_20260203T190820.hdf5
JobscriptExit code0
Real time52m (3133s)
CPU time50m (3021s = 96%)
Max RSS bytes3604754432 (3437 MiB)
Outputting started2026-02-04 00:54:21
Output files
Finished2026-02-04 00:58:54
Saved logsjustin-logs:303654.76-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ption in looking up space point ptr... 0x41ca9c00, 0x3e55e030
Caught exception in looking up space point ptr... 0x3e55e030, 0x407c3a10
Caught exception in looking up space point ptr... 0x407c3f40, 0x2e7bd1e0
Caught exception in looking up space point ptr... 0x2e7bd1e0, 0x41ca9c00
Caught exception in looking up space point ptr... 0x352fada0, 0x352fd2a0
Caught exception in looking up space point ptr... 0x352fd2a0, 0x177106c0
Caught exception in looking up space point ptr... 0x50d0fcf0, 0x50d27d30
Caught exception in looking up space point ptr... 0x159efc10, 0x50d0fcf0
Caught exception in looking up space point ptr... 0x794a5cd0, 0x794a59e0
Caught exception in looking up space point ptr... 0x794a59e0, 0x794a5b80
Caught exception in looking up space point ptr... 0x352c0410, 0x352c0180
Caught exception in looking up space point ptr... 0x352c0180, 0x352c0e50
Caught exception in looking up space point ptr... 0xf404450, 0xf404200
Caught exception in looking up space point ptr... 0xf404200, 0xf40a090
Caught exception in looking up space point ptr... 0x11097870, 0x11097620
Caught exception in looking up space point ptr... 0x11097620, 0x1109a210
Caught exception in looking up space point ptr... 0x5ef3d150, 0x5ef3cf00
Caught exception in looking up space point ptr... 0x5ef3cf00, 0x5ef48360
Caught exception in looking up space point ptr... 0x4f7c43c0, 0x4f7b7580
Caught exception in looking up space point ptr... 0x4f7b7580, 0x4f7c4230
Caught exception in looking up space point ptr... 0x6e066b70, 0x6e066d00
Caught exception in looking up space point ptr... 0x6e066d00, 0x6e067070
Caught exception in looking up space point ptr... 0x34134f30, 0x15aac1c0
Caught exception in looking up space point ptr... 0x5e347da0, 0x34134f30
Caught exception in looking up space point ptr... 0x36696ac0, 0x606250a0
Caught exception in looking up space point ptr... 0x606250a0, 0x60624f10
Caught exception in looking up space point ptr... 0x155c91e0, 0xe32dcf0
Caught exception in looking up space point ptr... 0xe32dcf0, 0xe32db60
Caught exception in looking up space point ptr... 0x5d8fc830, 0x2c820ff0
Caught exception in looking up space point ptr... 0x2c820ff0, 0x18d5cfa0
Caught exception in looking up space point ptr... 0x18d5cfa0, 0x2c820ff0
Caught exception in looking up space point ptr... 0x2c820ff0, 0x3dc6ab40
Caught exception in looking up space point ptr... 0x6e4fc730, 0x3f798ad0
Caught exception in looking up space point ptr... 0x5e412f10, 0x6e4fc730
Caught exception in looking up space point ptr... 0x4a517c60, 0x3cb8f4c0
Caught exception in looking up space point ptr... 0x3cb8f4c0, 0x5ebc3ec0
Caught exception in looking up space point ptr... 0x469347c0, 0x80b6d0a0
Caught exception in looking up space point ptr... 0x80b6d0a0, 0x4694d560
Caught exception in looking up space point ptr... 0x4694d560, 0x38272bd0
Caught exception in looking up space point ptr... 0xd518fc0, 0xd4dab00
Caught exception in looking up space point ptr... 0x4fd39fc0, 0xd518fc0
Caught exception in looking up space point ptr... 0x3dd2ea90, 0x3dd1f860
Caught exception in looking up space point ptr... 0x3dd406f0, 0x3dd2ea90
Caught exception in looking up space point ptr... 0x4072fef0, 0x40722f20
Caught exception in looking up space point ptr... 0x4073d1e0, 0x4072fef0
Caught exception in looking up space point ptr... 0x39474400, 0x59b68ce0
Caught exception in looking up space point ptr... 0x59b68ce0, 0x1505aa80
Caught exception in looking up space point ptr... 0x2de9e810, 0x2de9e680
Caught exception in looking up space point ptr... 0x2de9e680, 0x40ec1af0
Caught exception in looking up space point ptr... 0x17742950, 0x3905d180
Caught exception in looking up space point ptr... 0x352e6b30, 0x17742950
Caught exception in looking up space point ptr... 0x79ffe50, 0x2de93d80
Caught exception in looking up space point ptr... 0x79fffe0, 0x79ffe50
Caught exception in looking up space point ptr... 0x11159220, 0x11159090
Caught exception in looking up space point ptr... 0x11159090, 0x616f6870
Caught exception in looking up space point ptr... 0x17583310, 0x44604b90
Caught exception in looking up space point ptr... 0x44604b90, 0x18a7e110
Caught exception in looking up space point ptr... 0x41647570, 0x416477d0
Caught exception in looking up space point ptr... 0x416477d0, 0x41647570
Caught exception in looking up space point ptr... 0x34131740, 0x34131ae0
Caught exception in looking up space point ptr... 0x34131ae0, 0x34131ed0
Caught exception in looking up space point ptr... 0x52865430, 0x52864bc0
Caught exception in looking up space point ptr... 0x52864bc0, 0x2e7bd7c0
Caught exception in looking up space point ptr... 0x607ef380, 0x5b008a40
Caught exception in looking up space point ptr... 0x5b008a40, 0x607ef380
Caught exception in looking up space point ptr... 0xe4b2970, 0xe4b2b00
Caught exception in looking up space point ptr... 0xe4b2b00, 0xe4b2970
++++>>>> total num hits: 530784, num free: 521601
04-Feb-2026 01:54:13 CET  Closed output file "np02vd_raw_run042433_0069_df-s02-d1_dw_0_20260203T190820_reco_stage0_20260204T005413_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         132.924       161.851       196.341       156.668       21.5341        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       5.5391e-05    0.000109078   0.000256494   8.5852e-05    5.8262e-05       19     
produce:tpcrawdecoder:PDVDTPCReader                5.89145       6.96621       8.00422       7.00091       0.48694        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.0103577     0.0325772     0.175477      0.0130821     0.0423375       19     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000427438   0.000559968   0.00118014    0.000497189   0.000183109      19     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.000406618   0.00050506    0.00101486    0.000460789   0.000138437      19     
produce:beamevent:BeamEvent                      5.5341e-05    0.000164912   0.00113633    0.000107602   0.00023107       19     
produce:pdvddaphne:DAPHNEReaderPDVD                2.36886       2.88094       3.43488       2.86395      0.330744        19     
produce:ophit:OpHitFinder                         0.029482      0.0393124     0.0508121     0.0386017     0.0056659       19     
produce:wclsdatavd:WireCellToolkit                 89.5798       105.427       122.332       106.882       7.61016        19     
produce:gaushit:GausHitFinder                     0.751547       1.58238       2.47797       1.70391      0.450056        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000270385   0.000392176   0.000632251   0.000357806   0.000104532      19     
produce:reco3d:SpacePointSolver                    11.6209       20.105        26.0533       21.5751       4.18919        19     
produce:hitpdune:DisambigFromSpacePoints          0.112795      0.219341      0.323433      0.231523      0.0551915       19     
produce:cluster3d:Cluster3D                        1.52445       19.1928       51.101        3.22594       21.7956        19     
[art]:TriggerResults:TriggerResultInserter        2.402e-05    4.13551e-05   0.000120232   3.3021e-05    2.41394e-05      19     
end_path:out1:RootOutput                          5.39e-06     1.78914e-05   0.000177823    8.42e-06     3.79494e-05      19     
end_path:out1:RootOutput(write)                    4.4545        5.37407       6.89191       5.29252      0.715085        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5570.5 MB
  Peak resident set size usage (VmHWM): 3604.75 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0069_df-s02-d1_dw_0_20260203T190820_reco_stage0_20260204T005413_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0069_df-s02-d1_dw_0_20260203T190820_reco_stage0_20260204T005413_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0069_df-s02-d1_dw_0_20260203T190820_reco_stage0_20260204T005413_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770163337.0,
    "core.end_time": 1770166454.0,
    "core.events": [
      7873,
      7879,
      7885,
      7891,
      7897,
      7903,
      7909,
      7915,
      7921,
      7927,
      7933,
      7939,
      7945,
      7951,
      7957,
      7963,
      7969,
      7975,
      7981
    ],
    "core.event_count": 19,
    "core.first_event_number": 7873,
    "core.last_event_number": 7981,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0069_df-s02-d1_dw_0_20260203T190820.hdf5"
    }
  ]
}
justIN time: 2026-02-04 13:18:06 UTC       justIN version: 01.06.00