justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303653.30@dunegpschedd01.fnal.gov

Jobsub ID303653.30@dunegpschedd01.fnal.gov
Workflow ID12717
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:52:24
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce515
Last heartbeat2026-02-04 00:27:17
From worker nodeHostnameb9p35p0043.cern.ch
cpuinfoAMD EPYC 9655P 96-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:38:05
Input filesvd-protodune:np02vd_raw_run042421_0130_df-s03-d1_dw_0_20260203T144118.hdf5
JobscriptExit code0
Real time46m (2799s)
CPU time45m (2718s = 97%)
Max RSS bytes2739937280 (2613 MiB)
Outputting started2026-02-04 00:24:46
Output files
Finished2026-02-04 00:27:17
Saved logsjustin-logs:303653.30-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

tr... 0x1118d340, 0x28a94f60
Caught exception in looking up space point ptr... 0x28a94f60, 0x1118d340
Caught exception in looking up space point ptr... 0xdfd7760, 0x26990220
Caught exception in looking up space point ptr... 0x26990220, 0x526a2330
Caught exception in looking up space point ptr... 0x90f6c20, 0x54153100
Caught exception in looking up space point ptr... 0x54153100, 0x556e3870
Caught exception in looking up space point ptr... 0x9103160, 0x28a95f90
Caught exception in looking up space point ptr... 0x28a95f90, 0x9103160
Caught exception in looking up space point ptr... 0x9103160, 0xc4faa40
Caught exception in looking up space point ptr... 0xc4faa40, 0x9103160
Caught exception in looking up space point ptr... 0x9103160, 0x58649ed0
Caught exception in looking up space point ptr... 0x58649ed0, 0x9103160
Caught exception in looking up space point ptr... 0x9103160, 0x251434e0
Caught exception in looking up space point ptr... 0x251434e0, 0x9103160
Caught exception in looking up space point ptr... 0x9103160, 0x28a94c90
Caught exception in looking up space point ptr... 0x28a94c90, 0x9103160
Caught exception in looking up space point ptr... 0x9647ef0, 0x9647d60
Caught exception in looking up space point ptr... 0x9647d60, 0x105416c0
Caught exception in looking up space point ptr... 0x12cb08e0, 0x2646c4d0
Caught exception in looking up space point ptr... 0x2646c4d0, 0x2626a750
Caught exception in looking up space point ptr... 0x9ec03f0, 0xc4f30e0
Caught exception in looking up space point ptr... 0xc4f30e0, 0x9ec03f0
Caught exception in looking up space point ptr... 0x9ec03f0, 0x26121a50
Caught exception in looking up space point ptr... 0x26121a50, 0x9ec03f0
Caught exception in looking up space point ptr... 0x9ec03f0, 0x28a8f7b0
Caught exception in looking up space point ptr... 0x28a8f7b0, 0x9ec03f0
Caught exception in looking up space point ptr... 0x58656cd0, 0x28a90db0
Caught exception in looking up space point ptr... 0x28a90db0, 0x58656cd0
Caught exception in looking up space point ptr... 0x58656cd0, 0xc4f0cd0
Caught exception in looking up space point ptr... 0xc4f0cd0, 0x58656cd0
Caught exception in looking up space point ptr... 0xd8a2fd0, 0xd899f50
Caught exception in looking up space point ptr... 0x38df87c0, 0xd8a2fd0
Caught exception in looking up space point ptr... 0x24e2a220, 0x4a2d3200
Caught exception in looking up space point ptr... 0x4a2d3200, 0x518f5b60
Caught exception in looking up space point ptr... 0x518f5b60, 0x20135b20
Caught exception in looking up space point ptr... 0x113e4940, 0x24e2a220
Caught exception in looking up space point ptr... 0x1044ddd0, 0x573ca0b0
Caught exception in looking up space point ptr... 0x573ca0b0, 0x56933160
Caught exception in looking up space point ptr... 0x603617b0, 0x60361290
Caught exception in looking up space point ptr... 0x603641c0, 0x603617b0
Caught exception in looking up space point ptr... 0x501c8a30, 0x41ff2630
Caught exception in looking up space point ptr... 0x501c8c80, 0x501c8a30
Caught exception in looking up space point ptr... 0x3611ad50, 0x2c19f600
Caught exception in looking up space point ptr... 0x2c19f600, 0x2c1a3e00
Caught exception in looking up space point ptr... 0xc517de0, 0xc518070
Caught exception in looking up space point ptr... 0xc518070, 0xc51d5e0
Caught exception in looking up space point ptr... 0x5b0c54c0, 0x45a35990
Caught exception in looking up space point ptr... 0x5b0c5650, 0x5b0c54c0
Caught exception in looking up space point ptr... 0x51271650, 0x512711b0
Caught exception in looking up space point ptr... 0x24bb4710, 0x51271650
Caught exception in looking up space point ptr... 0x2a6b2e30, 0x4696b780
Caught exception in looking up space point ptr... 0x3f2ef680, 0x2a6b2e30
Caught exception in looking up space point ptr... 0x363353a0, 0x36326ad0
Caught exception in looking up space point ptr... 0x36326ad0, 0x36335210
Caught exception in looking up space point ptr... 0xe45e750, 0xe44caf0
Caught exception in looking up space point ptr... 0xe44caf0, 0x463c2dc0
Caught exception in looking up space point ptr... 0x1fd43860, 0x25c23a40
Caught exception in looking up space point ptr... 0x2b4c59c0, 0x1fd43860
Caught exception in looking up space point ptr... 0x28a9ea80, 0x1033a3d0
Caught exception in looking up space point ptr... 0x1033a3d0, 0x28a9e7b0
Caught exception in looking up space point ptr... 0x251b8fd0, 0x5b78c1b0
Caught exception in looking up space point ptr... 0x5b78c1b0, 0x251b8fd0
Caught exception in looking up space point ptr... 0x5107d5e0, 0x3f1ddef0
Caught exception in looking up space point ptr... 0x3f1ddef0, 0x5ee105b0
Caught exception in looking up space point ptr... 0x4d34dee0, 0x5111de40
Caught exception in looking up space point ptr... 0x5111de40, 0x5f02dbb0
++++>>>> total num hits: 291367, num free: 285648
04-Feb-2026 01:24:39 CET  Closed output file "np02vd_raw_run042421_0130_df-s03-d1_dw_0_20260203T144118_reco_stage0_20260204T002439_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         61.6333       101.533       124.969       108.789       19.2289        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                        5.219e-05    7.51789e-05   0.00017599     7.16e-05     2.14488e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                3.52461       3.76121       4.00248       3.75949      0.127877        27     
produce:triggerrawdecoder:PDVDTriggerReader4     0.00797256     0.0136798     0.0527519     0.0104889    0.00908626       27     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0003689    0.000524893   0.00095811    0.00050509    0.000119613      27     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.00116431    0.00150692    0.00196616    0.00142654    0.000216256      27     
produce:beamevent:BeamEvent                       6.284e-05    0.000109594   0.00064928     8.476e-05    0.00010731       27     
produce:pdvddaphne:DAPHNEReaderPDVD                1.82163       2.10583       2.31442       2.12658      0.136837        27     
produce:ophit:OpHitFinder                         0.0252572     0.035371      0.0401726     0.0360917    0.00330563       27     
produce:wclsdatavd:WireCellToolkit                 46.4166       54.2434       61.5292       55.0215       3.60253        27     
produce:gaushit:GausHitFinder                     0.449569       1.03231       1.44024       1.03392      0.284291        27     
produce:nhitsfilter:NumberOfHitsFilter           0.00011248    0.000213768   0.00034894     0.0002001    5.73411e-05      27     
produce:reco3d:SpacePointSolver                    3.76572       7.25047       13.3036       6.73472       2.33828        27     
produce:hitpdune:DisambigFromSpacePoints          0.0426944     0.158959      0.330582      0.164209      0.0687448       27     
produce:cluster3d:Cluster3D                       0.589258       28.6155        44.08        36.5001       16.459         27     
[art]:TriggerResults:TriggerResultInserter        2.229e-05    3.37604e-05    7.07e-05      3.142e-05    9.12905e-06      27     
end_path:out1:RootOutput                          4.51e-06     6.11296e-06    1.766e-05     5.54e-06     2.38103e-06      27     
end_path:out1:RootOutput(write)                    3.23873       4.28613       5.22236       4.33964      0.525533        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4697.55 MB
  Peak resident set size usage (VmHWM): 2739.94 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042421_0130_df-s03-d1_dw_0_20260203T144118_reco_stage0_20260204T002439_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042421_0130_df-s03-d1_dw_0_20260203T144118_reco_stage0_20260204T002439_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042421_0130_df-s03-d1_dw_0_20260203T144118_reco_stage0_20260204T002439_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770161894.0,
    "core.end_time": 1770164680.0,
    "core.events": [
      21129,
      21135,
      21141,
      21147,
      21153,
      21159,
      21165,
      21171,
      21177,
      21183,
      21189,
      21195,
      21201,
      21207,
      21213,
      21219,
      21225,
      21231,
      21237,
      21243,
      21249,
      21255,
      21261,
      21267,
      21273,
      21279,
      21285
    ],
    "core.event_count": 27,
    "core.first_event_number": 21129,
    "core.last_event_number": 21285,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42421
    ],
    "core.runs_subruns": [
      4242100001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042421_0130_df-s03-d1_dw_0_20260203T144118.hdf5"
    }
  ]
}
justIN time: 2026-02-04 09:13:25 UTC       justIN version: 01.06.00