justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297923.31@dunegpschedd02.fnal.gov

Jobsub ID297923.31@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 23:00:26
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce513
Last heartbeat2026-02-04 00:49:22
From worker nodeHostnameb9p35p0008.cern.ch
cpuinfoAMD EPYC 9655P 96-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:57:59
Input filesvd-protodune:np02vd_raw_run042433_0525_df-s03-d0_dw_0_20260203T220138.hdf5
JobscriptExit code0
Real time48m (2936s)
CPU time47m (2845s = 96%)
Max RSS bytes3610222592 (3442 MiB)
Outputting started2026-02-04 00:46:57
Output files
Finished2026-02-04 00:49:22
Saved logsjustin-logs:297923.31-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

t ptr... 0x167ccd10, 0x167ccb80
Caught exception in looking up space point ptr... 0x54ea7b60, 0x54ea79d0
Caught exception in looking up space point ptr... 0x54ea79d0, 0x54ed8a00
Caught exception in looking up space point ptr... 0x4f023cc0, 0x4f765f60
Caught exception in looking up space point ptr... 0x4f765f60, 0x4f6ee8e0
Caught exception in looking up space point ptr... 0x124678b0, 0x12467270
Caught exception in looking up space point ptr... 0x12467270, 0xc949b70
Caught exception in looking up space point ptr... 0x55c870f0, 0x55c86c40
Caught exception in looking up space point ptr... 0x55c87410, 0x55c870f0
Caught exception in looking up space point ptr... 0x50791e20, 0x108685b0
Caught exception in looking up space point ptr... 0x50791fb0, 0x50791e20
Caught exception in looking up space point ptr... 0x3c914de0, 0x3c914660
Caught exception in looking up space point ptr... 0x3c915420, 0x3c914de0
Caught exception in looking up space point ptr... 0x62a666f0, 0x62a661b0
Caught exception in looking up space point ptr... 0x62a661b0, 0x62a65b20
Caught exception in looking up space point ptr... 0x51728640, 0x51727e60
Caught exception in looking up space point ptr... 0x51728cd0, 0x51728640
Caught exception in looking up space point ptr... 0x516df230, 0x516dea50
Caught exception in looking up space point ptr... 0x516dea50, 0x516de120
Caught exception in looking up space point ptr... 0x54652590, 0x396ad2c0
Caught exception in looking up space point ptr... 0x54658b50, 0x54652590
Caught exception in looking up space point ptr... 0x3e386860, 0x827ad830
Caught exception in looking up space point ptr... 0x3e3869f0, 0x3e386860
Caught exception in looking up space point ptr... 0x185ff2c0, 0x185cab10
Caught exception in looking up space point ptr... 0x185ff450, 0x185ff2c0
Caught exception in looking up space point ptr... 0x3e9eb850, 0x3e9c3790
Caught exception in looking up space point ptr... 0x3e9eb9e0, 0x3e9eb850
Caught exception in looking up space point ptr... 0x776f4110, 0x776dd430
Caught exception in looking up space point ptr... 0x776f42a0, 0x776f4110
Caught exception in looking up space point ptr... 0x327cb7b0, 0x327bafa0
Caught exception in looking up space point ptr... 0x327dcdd0, 0x327cb7b0
Caught exception in looking up space point ptr... 0x4b98c990, 0x3199a150
Caught exception in looking up space point ptr... 0x3199a150, 0x3241eeb0
Caught exception in looking up space point ptr... 0x4d1977e0, 0x4d184f00
Caught exception in looking up space point ptr... 0x3c29b820, 0x4d1977e0
Caught exception in looking up space point ptr... 0x32f07a50, 0x18a84e90
Caught exception in looking up space point ptr... 0x18a84e90, 0x32f07a50
Caught exception in looking up space point ptr... 0xbeb4620, 0xbea7330
Caught exception in looking up space point ptr... 0xbec1780, 0xbeb4620
Caught exception in looking up space point ptr... 0x2bcd00f0, 0x12d57ce0
Caught exception in looking up space point ptr... 0x12d57ce0, 0x68cc1550
Caught exception in looking up space point ptr... 0x3c3220a0, 0x3c30f770
Caught exception in looking up space point ptr... 0x3c30f770, 0x3c2fd020
Caught exception in looking up space point ptr... 0x5f5749d0, 0x129ab490
Caught exception in looking up space point ptr... 0x8609ab30, 0x5f5749d0
Caught exception in looking up space point ptr... 0x790dcd00, 0x31bfea90
Caught exception in looking up space point ptr... 0x31bfea90, 0x5261a870
Caught exception in looking up space point ptr... 0x32cf1f80, 0x151aa900
Caught exception in looking up space point ptr... 0x151aa900, 0x32cf1f80
Caught exception in looking up space point ptr... 0x166f7530, 0x166f76c0
Caught exception in looking up space point ptr... 0x166f76c0, 0x166f7530
Caught exception in looking up space point ptr... 0x61657020, 0x2bb47230
Caught exception in looking up space point ptr... 0x2bb47230, 0x61649e50
Caught exception in looking up space point ptr... 0x49608d70, 0x164fea30
Caught exception in looking up space point ptr... 0x164fea30, 0x49608d70
Caught exception in looking up space point ptr... 0x4839c320, 0xffcd380
Caught exception in looking up space point ptr... 0xffcd380, 0x5006a410
Caught exception in looking up space point ptr... 0x60e78120, 0x424e95a0
Caught exception in looking up space point ptr... 0x424e95a0, 0x60e78120
Caught exception in looking up space point ptr... 0x52de24d0, 0x52de11b0
Caught exception in looking up space point ptr... 0x52de11b0, 0x52de24d0
Caught exception in looking up space point ptr... 0xcd83570, 0x61666680
Caught exception in looking up space point ptr... 0x61666680, 0xcd83570
Caught exception in looking up space point ptr... 0x52e13700, 0x54459560
Caught exception in looking up space point ptr... 0x544577b0, 0x52e13700
Caught exception in looking up space point ptr... 0x32ab49a0, 0x152431b0
Caught exception in looking up space point ptr... 0x152431b0, 0x32ab49a0
++++>>>> total num hits: 577995, num free: 567696
04-Feb-2026 01:46:50 CET  Closed output file "np02vd_raw_run042433_0525_df-s03-d0_dw_0_20260203T220138_reco_stage0_20260204T004650_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         97.9704       152.05        186.62        157.396       20.0953        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       5.3111e-05    7.69621e-05   0.00016012    7.2061e-05    2.35735e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                6.07081       6.76616       8.14805       6.70608      0.455727        19     
produce:triggerrawdecoder:PDVDTriggerReader4     0.00722702     0.0238481     0.0845231     0.0128464     0.022139        19     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000319461   0.000504971   0.00100931    0.000476383   0.000159428      19     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.000261881   0.000405309   0.000729543   0.000365932   0.000122105      19     
produce:beamevent:BeamEvent                       5.654e-05    0.000112189   0.000811204    7.513e-05    0.000165064      19     
produce:pdvddaphne:DAPHNEReaderPDVD                1.76259       2.11849       2.52288       2.16594      0.184216        19     
produce:ophit:OpHitFinder                         0.025682      0.0316333     0.037487      0.0312549    0.00307833       19     
produce:wclsdatavd:WireCellToolkit                 73.6978       82.0487       94.3077       81.805        5.62476        19     
produce:gaushit:GausHitFinder                     0.705611       1.70963       2.47496       1.70842      0.433516        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000175361   0.000234972   0.000443013   0.000222011   5.84608e-05      19     
produce:reco3d:SpacePointSolver                    7.66407       10.8353       21.4969       10.3811       2.94188        19     
produce:hitpdune:DisambigFromSpacePoints          0.101852      0.243783      0.417746      0.215822      0.090769        19     
produce:cluster3d:Cluster3D                        1.12191       42.4995       59.6323       48.6763       17.8601        19     
[art]:TriggerResults:TriggerResultInserter        2.46e-05     3.62107e-05   9.0981e-05     3.23e-05     1.40671e-05      19     
end_path:out1:RootOutput                          4.87e-06     6.52263e-06    2.306e-05     5.63e-06     3.92367e-06      19     
end_path:out1:RootOutput(write)                    4.01854       5.73782       7.47012       5.65403      0.737649        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5677.45 MB
  Peak resident set size usage (VmHWM): 3610.22 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0525_df-s03-d0_dw_0_20260203T220138_reco_stage0_20260204T004650_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0525_df-s03-d0_dw_0_20260203T220138_reco_stage0_20260204T004650_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0525_df-s03-d0_dw_0_20260203T220138_reco_stage0_20260204T004650_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770163085.0,
    "core.end_time": 1770166011.0,
    "core.events": [
      59858,
      59864,
      59870,
      59876,
      59882,
      59888,
      59894,
      59900,
      59906,
      59912,
      59918,
      59924,
      59930,
      59936,
      59942,
      59948,
      59954,
      59960,
      59966
    ],
    "core.event_count": 19,
    "core.first_event_number": 59858,
    "core.last_event_number": 59966,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0525_df-s03-d0_dw_0_20260203T220138.hdf5"
    }
  ]
}
justIN time: 2026-02-04 07:43:29 UTC       justIN version: 01.06.00