justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303653.160@dunegpschedd01.fnal.gov

Jobsub ID303653.160@dunegpschedd01.fnal.gov
Workflow ID12717
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:52:24
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce510
Last heartbeat2026-02-04 00:30:39
From worker nodeHostnameb9p09p3861.cern.ch
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:46:16
Input filesvd-protodune:np02vd_raw_run042421_0052_df-s03-d0_dw_0_20260203T135910.hdf5
JobscriptExit code0
Real time41m (2511s)
CPU time40m (2418s = 96%)
Max RSS bytes2594861056 (2474 MiB)
Outputting started2026-02-04 00:28:09
Output files
Finished2026-02-04 00:30:39
Saved logsjustin-logs:303653.160-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ptr... 0x4274f4a0, 0x4274fe10
Caught exception in looking up space point ptr... 0x4274fe10, 0x4274f970
Caught exception in looking up space point ptr... 0x3d658130, 0x427516c0
Caught exception in looking up space point ptr... 0x427516c0, 0x4274f4a0
Caught exception in looking up space point ptr... 0x3622b650, 0x3622a8e0
Caught exception in looking up space point ptr... 0x3622a8e0, 0x3622a750
Caught exception in looking up space point ptr... 0x3622a750, 0x3622a5c0
Caught exception in looking up space point ptr... 0x3622a5c0, 0x3622b4c0
Caught exception in looking up space point ptr... 0x15653340, 0x35ce1300
Caught exception in looking up space point ptr... 0x35ce1300, 0x2ea95d80
Caught exception in looking up space point ptr... 0x2ea95d80, 0x4f68cb80
Caught exception in looking up space point ptr... 0x60c90360, 0xc0498d0
Caught exception in looking up space point ptr... 0xc0498d0, 0x33b90470
Caught exception in looking up space point ptr... 0x33b90470, 0x5c6bce50
Caught exception in looking up space point ptr... 0x4bd5ae10, 0x1558aca0
Caught exception in looking up space point ptr... 0x1558aca0, 0x1558ab50
Caught exception in looking up space point ptr... 0x1558ab50, 0x1558aa00
Caught exception in looking up space point ptr... 0x1558aa00, 0x4bd5abc0
Caught exception in looking up space point ptr... 0x115586c0, 0x11558620
Caught exception in looking up space point ptr... 0x11558620, 0x4bd6aa80
Caught exception in looking up space point ptr... 0x4bd6aa80, 0x380cb920
Caught exception in looking up space point ptr... 0x380cba20, 0x244796a0
Caught exception in looking up space point ptr... 0x244796a0, 0x4bd69610
Caught exception in looking up space point ptr... 0x4bd69610, 0x115586c0
Caught exception in looking up space point ptr... 0x2dedaeb0, 0x2e62f750
Caught exception in looking up space point ptr... 0x2e62f750, 0x2dedaeb0
Caught exception in looking up space point ptr... 0x2db7ac50, 0x5097b940
Caught exception in looking up space point ptr... 0x5097b940, 0x35d64f10
Caught exception in looking up space point ptr... 0xa656c70, 0x57083610
Caught exception in looking up space point ptr... 0x57083610, 0xa656c70
Caught exception in looking up space point ptr... 0xa656c70, 0x37d81920
Caught exception in looking up space point ptr... 0x37d81920, 0xa656c70
Caught exception in looking up space point ptr... 0x5a919e90, 0x32d55f50
Caught exception in looking up space point ptr... 0x37f0fa90, 0x5a919e90
Caught exception in looking up space point ptr... 0x13f7be30, 0x14d8b4f0
Caught exception in looking up space point ptr... 0x113caae0, 0x13f7be30
Caught exception in looking up space point ptr... 0x2ec87ed0, 0x41edd0b0
Caught exception in looking up space point ptr... 0x41edd0b0, 0x2ec87ed0
Caught exception in looking up space point ptr... 0x4bf7b2e0, 0x42679ea0
Caught exception in looking up space point ptr... 0x238916a0, 0x4bf7b2e0
Caught exception in looking up space point ptr... 0x102b4870, 0x57ce3370
Caught exception in looking up space point ptr... 0x4f6a67d0, 0x102b4870
Caught exception in looking up space point ptr... 0x2c1846b0, 0x36b711f0
Caught exception in looking up space point ptr... 0x36b711f0, 0x36b710f0
Caught exception in looking up space point ptr... 0x240e4940, 0x240e4840
Caught exception in looking up space point ptr... 0x240e4840, 0x329e2ed0
Caught exception in looking up space point ptr... 0x4f699520, 0x616c1930
Caught exception in looking up space point ptr... 0x4bf247f0, 0x4f699520
Caught exception in looking up space point ptr... 0x3e96a200, 0x616c0c40
Caught exception in looking up space point ptr... 0x616c0c40, 0x4f698d30
Caught exception in looking up space point ptr... 0x329ff700, 0x329febe0
Caught exception in looking up space point ptr... 0x329febe0, 0x329ff700
Caught exception in looking up space point ptr... 0x329f2450, 0x329f2270
Caught exception in looking up space point ptr... 0x329f2270, 0x329f2090
Caught exception in looking up space point ptr... 0x41edb2f0, 0x2ec7a7d0
Caught exception in looking up space point ptr... 0x2ec7a7d0, 0x41edbf40
Caught exception in looking up space point ptr... 0x174b29a0, 0x174b27c0
Caught exception in looking up space point ptr... 0x133e0490, 0x174b29a0
Caught exception in looking up space point ptr... 0x5f7624e0, 0x2c264900
Caught exception in looking up space point ptr... 0x2c264900, 0x30f1cb20
Caught exception in looking up space point ptr... 0x2f27b300, 0x2f27b170
Caught exception in looking up space point ptr... 0x2f27b170, 0x2f27afe0
Caught exception in looking up space point ptr... 0x1434cef0, 0x1434cc90
Caught exception in looking up space point ptr... 0x1434cc90, 0x1434cb60
Caught exception in looking up space point ptr... 0x3ee29bb0, 0x3ee29a80
Caught exception in looking up space point ptr... 0x3ee29a80, 0x3ee29bb0
++++>>>> total num hits: 40960, num free: 39715
04-Feb-2026 01:28:00 CET  Closed output file "np02vd_raw_run042421_0052_df-s03-d0_dw_0_20260203T135910_reco_stage0_20260204T002800_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         77.234        90.6883       114.611       87.2788       8.85921        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       4.8931e-05    8.48636e-05   0.000160162   7.2161e-05    3.23427e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                3.71661       4.21548       4.60624       4.20497      0.212322        27     
produce:triggerrawdecoder:PDVDTriggerReader4      0.0109505     0.0208072     0.0964159     0.0135436     0.0174824       27     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000448488   0.000550269   0.000757092   0.000526559   8.38665e-05      27     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0016882    0.00202769     0.0027621    0.00201522    0.000231795      27     
produce:beamevent:BeamEvent                      7.0181e-05    0.000128152   0.00105471    8.1861e-05    0.00018282       27     
produce:pdvddaphne:DAPHNEReaderPDVD                2.34322       3.06971       3.44257       3.07602      0.251492        27     
produce:ophit:OpHitFinder                         0.0310337     0.0402891     0.0474844     0.0423908    0.00533077       27     
produce:wclsdatavd:WireCellToolkit                 54.565        66.0467       87.051        65.5265       7.27188        27     
produce:gaushit:GausHitFinder                     0.605341      0.975639       1.82856       0.89552      0.290751        27     
produce:nhitsfilter:NumberOfHitsFilter           0.000185283   0.000287309   0.000764811   0.000239763   0.00012874       27     
produce:reco3d:SpacePointSolver                    6.29792       10.6221       15.5264       10.5556       2.79286        27     
produce:hitpdune:DisambigFromSpacePoints          0.0577102     0.102578      0.194633      0.0894737     0.0366355       27     
produce:cluster3d:Cluster3D                       0.822407       1.77551       3.41162       1.8215       0.644041        27     
[art]:TriggerResults:TriggerResultInserter       1.9421e-05    2.83491e-05   9.3852e-05     2.501e-05    1.35322e-05      27     
end_path:out1:RootOutput                          4.16e-06     9.62678e-06   3.0531e-05     9.34e-06     4.79851e-06      27     
end_path:out1:RootOutput(write)                    2.90194       3.79168       4.91001       3.69042      0.551153        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4565.6 MB
  Peak resident set size usage (VmHWM): 2594.86 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042421_0052_df-s03-d0_dw_0_20260203T135910_reco_stage0_20260204T002800_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042421_0052_df-s03-d0_dw_0_20260203T135910_reco_stage0_20260204T002800_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042421_0052_df-s03-d0_dw_0_20260203T135910_reco_stage0_20260204T002800_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770162390.0,
    "core.end_time": 1770164881.0,
    "core.events": [
      8486,
      8492,
      8498,
      8504,
      8510,
      8516,
      8522,
      8528,
      8534,
      8540,
      8546,
      8552,
      8558,
      8564,
      8570,
      8576,
      8582,
      8588,
      8594,
      8600,
      8606,
      8612,
      8618,
      8624,
      8630,
      8636,
      8642
    ],
    "core.event_count": 27,
    "core.first_event_number": 8486,
    "core.last_event_number": 8642,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42421
    ],
    "core.runs_subruns": [
      4242100001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042421_0052_df-s03-d0_dw_0_20260203T135910.hdf5"
    }
  ]
}
justIN time: 2026-02-04 05:58:13 UTC       justIN version: 01.06.00