justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303652.163@dunegpschedd01.fnal.gov

Jobsub ID303652.163@dunegpschedd01.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:50:26
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce514
Last heartbeat2026-02-04 00:21:47
From worker nodeHostnameb9p06p2461.cern.ch
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:33:15
Input filesvd-protodune:np02vd_raw_run042433_0295_df-s03-d1_dw_0_20260203T203414.hdf5
JobscriptExit code0
Real time45m (2759s)
CPU time44m (2654s = 96%)
Max RSS bytes3435790336 (3276 MiB)
Outputting started2026-02-04 00:19:15
Output files
Finished2026-02-04 00:21:47
Saved logsjustin-logs:303652.163-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

... 0x126811d0, 0x126814b0
Caught exception in looking up space point ptr... 0x44229320, 0x1890d370
Caught exception in looking up space point ptr... 0x1890d370, 0x2ef45b20
Caught exception in looking up space point ptr... 0x4ef43210, 0x4ef2b950
Caught exception in looking up space point ptr... 0x4ef2b950, 0x4ef2b770
Caught exception in looking up space point ptr... 0x68cd0a10, 0x68cd05e0
Caught exception in looking up space point ptr... 0x68cd05e0, 0x68cd0340
Caught exception in looking up space point ptr... 0x68cd0340, 0x68cd0880
Caught exception in looking up space point ptr... 0x68cd0880, 0x68cd0a10
Caught exception in looking up space point ptr... 0x581b2b60, 0x581b2e80
Caught exception in looking up space point ptr... 0x581b2e80, 0x581b56e0
Caught exception in looking up space point ptr... 0x4405af70, 0x4405ade0
Caught exception in looking up space point ptr... 0x4405ade0, 0x4405a7a0
Caught exception in looking up space point ptr... 0x49096cf0, 0x490902d0
Caught exception in looking up space point ptr... 0x490902d0, 0x4909a100
Caught exception in looking up space point ptr... 0x69ab7b10, 0x69ab8cf0
Caught exception in looking up space point ptr... 0x69ab8cf0, 0x69ab8e40
Caught exception in looking up space point ptr... 0x458d4680, 0x458d4480
Caught exception in looking up space point ptr... 0x458d4480, 0x458d4680
Caught exception in looking up space point ptr... 0x8832cd0, 0x51f49a70
Caught exception in looking up space point ptr... 0x51f49a70, 0x8832cd0
Caught exception in looking up space point ptr... 0x6a7d1280, 0x6a7d10a0
Caught exception in looking up space point ptr... 0x6a7d10a0, 0x6a7d15a0
Caught exception in looking up space point ptr... 0x7d97f880, 0x45a02540
Caught exception in looking up space point ptr... 0x39c1e7e0, 0x7d97f880
Caught exception in looking up space point ptr... 0x4ef4f8b0, 0x4ef2bea0
Caught exception in looking up space point ptr... 0x4ef2bea0, 0x4ef2bcc0
Caught exception in looking up space point ptr... 0x4ef3dae0, 0x4ef3dbe0
Caught exception in looking up space point ptr... 0x4ef3dbe0, 0x4ef4f8b0
Caught exception in looking up space point ptr... 0x10af6190, 0x11df33b0
Caught exception in looking up space point ptr... 0x7fbd29a0, 0x10af6190
Caught exception in looking up space point ptr... 0x5b6ef070, 0x5b6edb00
Caught exception in looking up space point ptr... 0x5b6eec20, 0x5b6ef070
Caught exception in looking up space point ptr... 0x14b7bad0, 0x14b7be40
Caught exception in looking up space point ptr... 0x14b7b940, 0x14b7bad0
Caught exception in looking up space point ptr... 0x73276830, 0x732766e0
Caught exception in looking up space point ptr... 0x732766e0, 0x73276830
Caught exception in looking up space point ptr... 0x6a7f6360, 0x6a7f6040
Caught exception in looking up space point ptr... 0x6a7f6040, 0x6a7f6360
Caught exception in looking up space point ptr... 0x582a71e0, 0x582a6f90
Caught exception in looking up space point ptr... 0x582a6f90, 0x582a71e0
Caught exception in looking up space point ptr... 0x44100a40, 0x3af3d280
Caught exception in looking up space point ptr... 0x72aed610, 0x44100a40
Caught exception in looking up space point ptr... 0x446b1ff0, 0x446b1e60
Caught exception in looking up space point ptr... 0x446b1e60, 0x446b1ff0
Caught exception in looking up space point ptr... 0x446b1ff0, 0x446b1cd0
Caught exception in looking up space point ptr... 0x446b1cd0, 0x446b1ff0
Caught exception in looking up space point ptr... 0x66498b00, 0x664928d0
Caught exception in looking up space point ptr... 0x664928d0, 0x66498b00
Caught exception in looking up space point ptr... 0x695ac960, 0x6b0d5af0
Caught exception in looking up space point ptr... 0x6b0d5af0, 0x695a5fc0
Caught exception in looking up space point ptr... 0x6f865a70, 0x6f865c00
Caught exception in looking up space point ptr... 0x6f865c00, 0x5e7478b0
Caught exception in looking up space point ptr... 0x63c2fe60, 0x63c2f960
Caught exception in looking up space point ptr... 0x63c2f960, 0x63c2f7d0
Caught exception in looking up space point ptr... 0x468e7de0, 0x468e7c90
Caught exception in looking up space point ptr... 0x468e7c90, 0x468e7de0
Caught exception in looking up space point ptr... 0x461f4ff0, 0x43c38880
Caught exception in looking up space point ptr... 0x183375a0, 0x461f4ff0
Caught exception in looking up space point ptr... 0x44071180, 0x44070e60
Caught exception in looking up space point ptr... 0x44070e60, 0x44071180
Caught exception in looking up space point ptr... 0x6986c950, 0x6986cd10
Caught exception in looking up space point ptr... 0x6986cd10, 0x6986c950
Caught exception in looking up space point ptr... 0x18c594b0, 0x18c59320
Caught exception in looking up space point ptr... 0x18c59320, 0x18c594b0
Caught exception in looking up space point ptr... 0x68cff370, 0x68d03b60
Caught exception in looking up space point ptr... 0x68d03b60, 0x68d03c90
++++>>>> total num hits: 78938, num free: 75858
04-Feb-2026 01:19:04 CET  Closed output file "np02vd_raw_run042433_0295_df-s03-d1_dw_0_20260203T203414_reco_stage0_20260204T001904_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         107.146       142.02        175.446       138.679       17.4981        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                        4.351e-05    9.25941e-05   0.000170302    8.703e-05    3.10185e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                5.30365       6.78645       7.8027        6.94219      0.745202        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.0113154     0.023583      0.104072      0.0160178     0.0214084       19     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000411885   0.000660873   0.00133468    0.000622567   0.000203314      19     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.000427855   0.000615973   0.00088563    0.000611066   0.000137738      19     
produce:beamevent:BeamEvent                      6.5261e-05    0.000148162   0.00089538    0.000105531   0.000177757      19     
produce:pdvddaphne:DAPHNEReaderPDVD                2.03349       2.84682       3.44952       2.91187      0.389981        19     
produce:ophit:OpHitFinder                         0.024813      0.036324      0.0456839     0.0372994    0.00602372       19     
produce:wclsdatavd:WireCellToolkit                 86.825        106.431       124.415       108.594       11.5137        19     
produce:gaushit:GausHitFinder                     0.676664       1.41805       2.09256       1.34296      0.429468        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000217432   0.000352132   0.000644967   0.000363364   0.000102874      19     
produce:reco3d:SpacePointSolver                    6.41769       16.2187       25.7868       16.1589       4.70413        19     
produce:hitpdune:DisambigFromSpacePoints          0.0478503     0.199905      0.321193      0.208988      0.0710407       19     
produce:cluster3d:Cluster3D                       0.714742       2.80734       6.47096       2.67858       1.29592        19     
[art]:TriggerResults:TriggerResultInserter        1.779e-05    2.99935e-05   9.4131e-05     2.753e-05    1.58012e-05      19     
end_path:out1:RootOutput                          4.02e-06     8.29742e-06    2.877e-05     7.17e-06     4.93538e-06      19     
end_path:out1:RootOutput(write)                    3.66838       5.21701       6.80519       5.10145      0.841508        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5403.39 MB
  Peak resident set size usage (VmHWM): 3435.79 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0295_df-s03-d1_dw_0_20260203T203414_reco_stage0_20260204T001904_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0295_df-s03-d1_dw_0_20260203T203414_reco_stage0_20260204T001904_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0295_df-s03-d1_dw_0_20260203T203414_reco_stage0_20260204T001904_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770161606.0,
    "core.end_time": 1770164346.0,
    "core.events": [
      33639,
      33645,
      33651,
      33657,
      33663,
      33669,
      33675,
      33681,
      33687,
      33693,
      33699,
      33705,
      33711,
      33717,
      33723,
      33729,
      33735,
      33741,
      33747
    ],
    "core.event_count": 19,
    "core.first_event_number": 33639,
    "core.last_event_number": 33747,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0295_df-s03-d1_dw_0_20260203T203414.hdf5"
    }
  ]
}
justIN time: 2026-02-04 07:45:18 UTC       justIN version: 01.06.00