justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303653.96@dunegpschedd01.fnal.gov

Jobsub ID303653.96@dunegpschedd01.fnal.gov
Workflow ID12717
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:52:24
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce508
Last heartbeat2026-02-04 00:17:04
From worker nodeHostnameb9p34p0003.cern.ch
cpuinfoAMD EPYC 9655P 96-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:41:21
Input filesvd-protodune:np02vd_raw_run042421_0096_df-s03-d1_dw_0_20260203T142257.hdf5
JobscriptExit code0
Real time33m (2011s)
CPU time32m (1921s = 95%)
Max RSS bytes2641059840 (2518 MiB)
Outputting started2026-02-04 00:14:53
Output files
Finished2026-02-04 00:17:04
Saved logsjustin-logs:303653.96-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

0x3b298640
Caught exception in looking up space point ptr... 0x3b298640, 0x3b2983f0
Caught exception in looking up space point ptr... 0x4b62c150, 0x5d006160
Caught exception in looking up space point ptr... 0x5d006160, 0x4b62c150
Caught exception in looking up space point ptr... 0xa7da360, 0xa7db020
Caught exception in looking up space point ptr... 0xa7db020, 0xa7da360
Caught exception in looking up space point ptr... 0x8311980, 0x83117f0
Caught exception in looking up space point ptr... 0x83117f0, 0x8311980
Caught exception in looking up space point ptr... 0x15f807f0, 0x14b066f0
Caught exception in looking up space point ptr... 0x345411e0, 0x15f807f0
Caught exception in looking up space point ptr... 0x1019bd30, 0x101941e0
Caught exception in looking up space point ptr... 0x101941e0, 0x1019bd30
Caught exception in looking up space point ptr... 0x5cb13730, 0x5cb0f470
Caught exception in looking up space point ptr... 0x5cb0f470, 0x5cb13a50
Caught exception in looking up space point ptr... 0x2ee92220, 0x2ee91f80
Caught exception in looking up space point ptr... 0x2ee91f80, 0x2ee900c0
Caught exception in looking up space point ptr... 0x4b84b8d0, 0xa0c69b0
Caught exception in looking up space point ptr... 0xa0c69b0, 0x4b84b8d0
Caught exception in looking up space point ptr... 0x59c36b70, 0x59c37030
Caught exception in looking up space point ptr... 0x59c32680, 0x59c36b70
Caught exception in looking up space point ptr... 0x573f9f10, 0x51c2a880
Caught exception in looking up space point ptr... 0x51c2a880, 0x2fa9fe10
Caught exception in looking up space point ptr... 0x41ee4f20, 0x41ee4c90
Caught exception in looking up space point ptr... 0x41ee4c90, 0x41ee4d90
Caught exception in looking up space point ptr... 0x4fad2da0, 0x82b1af0
Caught exception in looking up space point ptr... 0x82b6840, 0x4fad2da0
Caught exception in looking up space point ptr... 0xa917710, 0xa9173e0
Caught exception in looking up space point ptr... 0xa9173e0, 0xa917710
Caught exception in looking up space point ptr... 0x5d00aba0, 0x5d00ae90
Caught exception in looking up space point ptr... 0x4b620040, 0x5d00aba0
Caught exception in looking up space point ptr... 0x5caeddd0, 0xedc6e10
Caught exception in looking up space point ptr... 0xedc6e10, 0x5caeddd0
Caught exception in looking up space point ptr... 0x3fabd8a0, 0x3fab7e10
Caught exception in looking up space point ptr... 0x3faac360, 0x3fabd8a0
Caught exception in looking up space point ptr... 0x18773ef0, 0x2ceec080
Caught exception in looking up space point ptr... 0x2ceec080, 0x59c199c0
Caught exception in looking up space point ptr... 0x29043310, 0x36920e80
Caught exception in looking up space point ptr... 0x36920e80, 0x4a113460
Caught exception in looking up space point ptr... 0x32220940, 0x5cb06d00
Caught exception in looking up space point ptr... 0x5cb06d00, 0x4e617960
Caught exception in looking up space point ptr... 0x4c2406f0, 0x4c2468d0
Caught exception in looking up space point ptr... 0x4c2468d0, 0x4c2406f0
Caught exception in looking up space point ptr... 0x2f295f70, 0x2f2955c0
Caught exception in looking up space point ptr... 0x2f2955c0, 0x2f295f70
Caught exception in looking up space point ptr... 0x2f295f70, 0x2f295d20
Caught exception in looking up space point ptr... 0x2f295d20, 0x2f295f70
Caught exception in looking up space point ptr... 0x2f295f70, 0x2f295710
Caught exception in looking up space point ptr... 0x2f295710, 0x2f295f70
Caught exception in looking up space point ptr... 0x18843610, 0x5753edf0
Caught exception in looking up space point ptr... 0x5753edf0, 0x5753ecf0
Caught exception in looking up space point ptr... 0x18778470, 0x18818740
Caught exception in looking up space point ptr... 0x1880f170, 0x18778470
Caught exception in looking up space point ptr... 0x1708bec0, 0x1708a460
Caught exception in looking up space point ptr... 0x1708a460, 0x1708bec0
Caught exception in looking up space point ptr... 0x3174cb10, 0x37df3d30
Caught exception in looking up space point ptr... 0x37df3d30, 0x4473e580
Caught exception in looking up space point ptr... 0x5cb13dc0, 0x5caf38f0
Caught exception in looking up space point ptr... 0x5caf38f0, 0x5cb13dc0
Caught exception in looking up space point ptr... 0x4e621a90, 0x4e6257d0
Caught exception in looking up space point ptr... 0x4e6257d0, 0x2fcf1ff0
Caught exception in looking up space point ptr... 0x4f789ac0, 0x4f77ede0
Caught exception in looking up space point ptr... 0x4f77ede0, 0x4f789ac0
Caught exception in looking up space point ptr... 0x5cb11730, 0x5cb0c620
Caught exception in looking up space point ptr... 0x5cb0c620, 0x5cb11730
Caught exception in looking up space point ptr... 0x2ade7b60, 0x2ade6df0
Caught exception in looking up space point ptr... 0x2ade6df0, 0x2ade7b60
++++>>>> total num hits: 62535, num free: 60129
04-Feb-2026 01:14:47 CET  Closed output file "np02vd_raw_run042421_0096_df-s03-d1_dw_0_20260203T142257_reco_stage0_20260204T001447_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         62.1645       72.477        85.2558       72.1573       5.73452        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       5.3911e-05    7.68679e-05   0.000178431    7.312e-05    2.08503e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                3.95012       4.30226       4.75103       4.2305       0.211957        27     
produce:triggerrawdecoder:PDVDTriggerReader4     0.00753182     0.0135982     0.0326954    0.00997536    0.00675888       27     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000327712   0.000501103   0.000781494   0.000489203   0.000106581      27     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.00112001    0.00143648    0.00191455    0.00138762    0.000241592      27     
produce:beamevent:BeamEvent                       5.997e-05    0.000113753   0.000830934   8.1491e-05    0.000141752      27     
produce:pdvddaphne:DAPHNEReaderPDVD                1.86666       2.13346       2.41282       2.14685      0.131317        27     
produce:ophit:OpHitFinder                         0.0281484     0.0356855     0.0410246     0.0354817    0.00327885       27     
produce:wclsdatavd:WireCellToolkit                 42.3854       51.108        59.1477       51.2295       4.25129        27     
produce:gaushit:GausHitFinder                      0.44392      0.858892       1.48189      0.796755      0.260326        27     
produce:nhitsfilter:NumberOfHitsFilter           0.000114421   0.000234393   0.000441212   0.000204431   8.96393e-05      27     
produce:reco3d:SpacePointSolver                    5.69228       8.49659       12.6822       8.15164       1.69885        27     
produce:hitpdune:DisambigFromSpacePoints          0.0501011     0.0884387     0.211753       0.07174      0.0387053       27     
produce:cluster3d:Cluster3D                       0.656101       1.42689       2.7871        1.38824      0.516945        27     
[art]:TriggerResults:TriggerResultInserter        2.178e-05    3.54664e-05    8.612e-05     3.366e-05    1.22541e-05      27     
end_path:out1:RootOutput                          4.11e-06     6.18037e-06    1.758e-05      5.7e-06     2.43375e-06      27     
end_path:out1:RootOutput(write)                    3.26628       3.9821        4.67411       3.97201      0.359039        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4692.25 MB
  Peak resident set size usage (VmHWM): 2641.06 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042421_0096_df-s03-d1_dw_0_20260203T142257_reco_stage0_20260204T001447_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042421_0096_df-s03-d1_dw_0_20260203T142257_reco_stage0_20260204T001447_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042421_0096_df-s03-d1_dw_0_20260203T142257_reco_stage0_20260204T001447_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770162089.0,
    "core.end_time": 1770164088.0,
    "core.events": [
      15621,
      15627,
      15633,
      15639,
      15645,
      15651,
      15657,
      15663,
      15669,
      15675,
      15681,
      15687,
      15693,
      15699,
      15705,
      15711,
      15717,
      15723,
      15729,
      15735,
      15741,
      15747,
      15753,
      15759,
      15765,
      15771,
      15777
    ],
    "core.event_count": 27,
    "core.first_event_number": 15621,
    "core.last_event_number": 15777,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42421
    ],
    "core.runs_subruns": [
      4242100001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042421_0096_df-s03-d1_dw_0_20260203T142257.hdf5"
    }
  ]
}
justIN time: 2026-02-04 10:36:52 UTC       justIN version: 01.06.00