justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297923.45@dunegpschedd02.fnal.gov

Jobsub ID297923.45@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 23:00:26
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce512
Last heartbeat2026-02-04 00:46:23
From worker nodeHostnameb9p10p5730.cern.ch
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:59:36
Input filesvd-protodune:np02vd_raw_run042433_0331_df-s03-d0_dw_0_20260203T204755.hdf5
JobscriptExit code0
Real time44m (2665s)
CPU time43m (2585s = 96%)
Max RSS bytes3496546304 (3334 MiB)
Outputting started2026-02-04 00:44:02
Output files
Finished2026-02-04 00:46:23
Saved logsjustin-logs:297923.45-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

.. 0x455ca190, 0x49aeb050
Caught exception in looking up space point ptr... 0x3b38a0a0, 0x5536c870
Caught exception in looking up space point ptr... 0x5536c870, 0x3b38a0a0
Caught exception in looking up space point ptr... 0x51cfb780, 0x51cfb870
Caught exception in looking up space point ptr... 0x51cfb870, 0x51cfb0d0
Caught exception in looking up space point ptr... 0x7ec7bb40, 0x51cfb780
Caught exception in looking up space point ptr... 0x3bd9d800, 0x3c33bb20
Caught exception in looking up space point ptr... 0x6f79ea70, 0x3bd9d800
Caught exception in looking up space point ptr... 0x14f2b370, 0x2a0a4fb0
Caught exception in looking up space point ptr... 0x2a0a4fb0, 0x14f2b370
Caught exception in looking up space point ptr... 0x1a863f60, 0x1a865850
Caught exception in looking up space point ptr... 0x1a865850, 0x1a8654c0
Caught exception in looking up space point ptr... 0x53ff4010, 0x11a86df0
Caught exception in looking up space point ptr... 0x3ee1bc40, 0x53ff4010
Caught exception in looking up space point ptr... 0x3bfbbee0, 0x3bfbc200
Caught exception in looking up space point ptr... 0x3bfbc200, 0x3bfbc070
Caught exception in looking up space point ptr... 0x45920910, 0x45920030
Caught exception in looking up space point ptr... 0x45920030, 0x45920910
Caught exception in looking up space point ptr... 0xd476f70, 0x44134ac0
Caught exception in looking up space point ptr... 0x44134ac0, 0x36867120
Caught exception in looking up space point ptr... 0x43fc4cc0, 0x2bbac6b0
Caught exception in looking up space point ptr... 0x2bbac6b0, 0x5408eb20
Caught exception in looking up space point ptr... 0x1a857750, 0x1a868880
Caught exception in looking up space point ptr... 0x1a868880, 0x1a857750
Caught exception in looking up space point ptr... 0x30c84a20, 0x30c83f30
Caught exception in looking up space point ptr... 0x30c83f30, 0x30c83490
Caught exception in looking up space point ptr... 0x637e6300, 0x18499410
Caught exception in looking up space point ptr... 0x18499410, 0x637e6300
Caught exception in looking up space point ptr... 0x637e6300, 0x3c1c0c60
Caught exception in looking up space point ptr... 0x3c1c0c60, 0x637e6300
Caught exception in looking up space point ptr... 0x411541a0, 0x3c087180
Caught exception in looking up space point ptr... 0x3c087180, 0x39a69f20
Caught exception in looking up space point ptr... 0x19274d50, 0x1927b120
Caught exception in looking up space point ptr... 0x1927b120, 0x19274d50
Caught exception in looking up space point ptr... 0x4c0fb380, 0x34154770
Caught exception in looking up space point ptr... 0x341548a0, 0x4c0fb380
Caught exception in looking up space point ptr... 0x639f79a0, 0x42f449c0
Caught exception in looking up space point ptr... 0x42f449c0, 0x639f79a0
Caught exception in looking up space point ptr... 0x639f79a0, 0x44112660
Caught exception in looking up space point ptr... 0x44112660, 0x639f79a0
Caught exception in looking up space point ptr... 0x1a85c030, 0x1a857400
Caught exception in looking up space point ptr... 0x1a857400, 0x1a85cd80
Caught exception in looking up space point ptr... 0x401df430, 0x1a85c030
Caught exception in looking up space point ptr... 0x31502570, 0x6f9b55e0
Caught exception in looking up space point ptr... 0x6f9b55e0, 0x31502570
Caught exception in looking up space point ptr... 0x7f9b2870, 0x59da94e0
Caught exception in looking up space point ptr... 0x59da7820, 0x7f9b2870
Caught exception in looking up space point ptr... 0x364ec510, 0x407b4550
Caught exception in looking up space point ptr... 0x407b4550, 0x364ec510
Caught exception in looking up space point ptr... 0x3b09fe80, 0x3b0a0060
Caught exception in looking up space point ptr... 0x3b0a0060, 0x3b0a01f0
Caught exception in looking up space point ptr... 0x15a45f80, 0x396265b0
Caught exception in looking up space point ptr... 0x396265b0, 0x15a45f80
Caught exception in looking up space point ptr... 0x15a45f80, 0x60f71880
Caught exception in looking up space point ptr... 0x60f71880, 0x15a45f80
Caught exception in looking up space point ptr... 0x336d1e60, 0x336dac00
Caught exception in looking up space point ptr... 0x336dac00, 0x336d1e60
Caught exception in looking up space point ptr... 0x336d1e60, 0x336dd1d0
Caught exception in looking up space point ptr... 0x336dd1d0, 0x336d1e60
Caught exception in looking up space point ptr... 0x2e6a5030, 0x884e0c00
Caught exception in looking up space point ptr... 0x884e0c00, 0x47a43440
Caught exception in looking up space point ptr... 0x2aaaee60, 0x4d793f90
Caught exception in looking up space point ptr... 0x4d793f90, 0x2aaaee60
Caught exception in looking up space point ptr... 0x3bd31d30, 0x369f02b0
Caught exception in looking up space point ptr... 0x369f02b0, 0x3b6f5bd0
Caught exception in looking up space point ptr... 0x5f041f20, 0x5f042560
Caught exception in looking up space point ptr... 0x5f042560, 0x5f042240
++++>>>> total num hits: 63117, num free: 60799
04-Feb-2026 01:43:53 CET  Closed output file "np02vd_raw_run042433_0331_df-s03-d0_dw_0_20260203T204755_reco_stage0_20260204T004353_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         118.988       137.001       167.597       134.15        11.7488        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                        5.135e-05    9.15616e-05   0.000196942   7.6561e-05    4.35777e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                4.65288       5.5008        6.25411       5.46502      0.403754        19     
produce:triggerrawdecoder:PDVDTriggerReader4     0.00996454     0.0130232     0.0208214     0.0115976    0.00291236       19     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000387465   0.000520471   0.000971831   0.000475816   0.000123126      19     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.000379454   0.000509619   0.00113637    0.000466045   0.000162362      19     
produce:beamevent:BeamEvent                       4.819e-05    0.000117842   0.000656038   8.3241e-05    0.000129325      19     
produce:pdvddaphne:DAPHNEReaderPDVD                2.20407       2.56827       2.98748       2.61679      0.259854        19     
produce:ophit:OpHitFinder                         0.0199604     0.0334589     0.041037      0.0338985     0.0049061       19     
produce:wclsdatavd:WireCellToolkit                 86.2551       95.1027       104.202       94.883        5.13755        19     
produce:gaushit:GausHitFinder                     0.797809       1.30383       1.82673       1.22627       0.36575        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000286704   0.000383465   0.000589947   0.000380684   7.56097e-05      19     
produce:reco3d:SpacePointSolver                    11.9868       24.3566       41.1446       23.0134       6.94367        19     
produce:hitpdune:DisambigFromSpacePoints          0.0794807     0.214257      0.366456       0.21386      0.0757402       19     
produce:cluster3d:Cluster3D                        1.30193       2.98946       6.88359       2.36066       1.41314        19     
[art]:TriggerResults:TriggerResultInserter       2.0241e-05    3.18973e-05   0.000138192    2.43e-05     2.54895e-05      19     
end_path:out1:RootOutput                          3.49e-06     8.47063e-06   3.2131e-05      6.8e-06     5.9829e-06       19     
end_path:out1:RootOutput(write)                    4.18961       4.88948       7.04142        4.667        0.72973        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5448.64 MB
  Peak resident set size usage (VmHWM): 3496.55 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0331_df-s03-d0_dw_0_20260203T204755_reco_stage0_20260204T004353_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0331_df-s03-d0_dw_0_20260203T204755_reco_stage0_20260204T004353_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0331_df-s03-d0_dw_0_20260203T204755_reco_stage0_20260204T004353_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770163187.0,
    "core.end_time": 1770165834.0,
    "core.events": [
      37742,
      37748,
      37754,
      37760,
      37766,
      37772,
      37778,
      37784,
      37790,
      37796,
      37802,
      37808,
      37814,
      37820,
      37826,
      37832,
      37838,
      37844,
      37850
    ],
    "core.event_count": 19,
    "core.first_event_number": 37742,
    "core.last_event_number": 37850,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0331_df-s03-d0_dw_0_20260203T204755.hdf5"
    }
  ]
}
justIN time: 2026-02-04 06:06:11 UTC       justIN version: 01.06.00