justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297921.156@dunegpschedd02.fnal.gov

Jobsub ID297921.156@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:58:26
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce505
Last heartbeat2026-02-04 00:33:29
From worker nodeHostnameb9p35p0031.cern.ch
cpuinfoAMD EPYC 9655P 96-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:55:56
Input filesvd-protodune:np02vd_raw_run042433_0450_df-s03-d1_dw_0_20260203T213308.hdf5
JobscriptExit code0
Real time35m (2128s)
CPU time34m (2051s = 96%)
Max RSS bytes3399929856 (3242 MiB)
Outputting started2026-02-04 00:31:26
Output files
Finished2026-02-04 00:33:29
Saved logsjustin-logs:297921.156-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ce point ptr... 0x4fbfc8b0, 0x8fb2c20
Caught exception in looking up space point ptr... 0x2a7ac0d0, 0x114dd600
Caught exception in looking up space point ptr... 0x114dd600, 0x2a7ac3f0
Caught exception in looking up space point ptr... 0x114dd500, 0x2c2ec290
Caught exception in looking up space point ptr... 0x2c2ec290, 0x2c2ec0b0
Caught exception in looking up space point ptr... 0x37f41ca0, 0x37f41b10
Caught exception in looking up space point ptr... 0x37f41b10, 0x85e50120
Caught exception in looking up space point ptr... 0x85e453f0, 0x6fd0ec70
Caught exception in looking up space point ptr... 0x6fd0ec70, 0x1368e710
Caught exception in looking up space point ptr... 0xece82b0, 0x5c54e920
Caught exception in looking up space point ptr... 0x5c54e920, 0x66bc6b00
Caught exception in looking up space point ptr... 0x6ad19000, 0x6ad17e10
Caught exception in looking up space point ptr... 0x6ad17e10, 0x6ad19000
Caught exception in looking up space point ptr... 0x505b2be0, 0x505b1df0
Caught exception in looking up space point ptr... 0x505b1df0, 0x505b2be0
Caught exception in looking up space point ptr... 0x96b5290, 0x4c468b00
Caught exception in looking up space point ptr... 0x4c468b00, 0x4c468ce0
Caught exception in looking up space point ptr... 0x4c468e70, 0x96b5470
Caught exception in looking up space point ptr... 0x96b5470, 0x96b5290
Caught exception in looking up space point ptr... 0x6656f690, 0x6656ebe0
Caught exception in looking up space point ptr... 0x6656ebe0, 0x60745e30
Caught exception in looking up space point ptr... 0x38d4cc90, 0x11cd9110
Caught exception in looking up space point ptr... 0x11cd9110, 0x38d4cc90
Caught exception in looking up space point ptr... 0x3df341b0, 0x586a6bc0
Caught exception in looking up space point ptr... 0x586a6bc0, 0x3df341b0
Caught exception in looking up space point ptr... 0x358043d0, 0x35804240
Caught exception in looking up space point ptr... 0x35804240, 0x358043d0
Caught exception in looking up space point ptr... 0x38052970, 0x67e5dd40
Caught exception in looking up space point ptr... 0x67e5ded0, 0x67e5e290
Caught exception in looking up space point ptr... 0x67e5e290, 0x67e5e0b0
Caught exception in looking up space point ptr... 0x67e5e0b0, 0x38052970
Caught exception in looking up space point ptr... 0x56d7dff0, 0x56d7e120
Caught exception in looking up space point ptr... 0x56d7e120, 0x56d7dff0
Caught exception in looking up space point ptr... 0x7677f620, 0x7677f500
Caught exception in looking up space point ptr... 0x7677f500, 0x7677f620
Caught exception in looking up space point ptr... 0x76530340, 0x7652f1a0
Caught exception in looking up space point ptr... 0x7652f1a0, 0x41894c50
Caught exception in looking up space point ptr... 0x568d0730, 0x568d0530
Caught exception in looking up space point ptr... 0x568d0530, 0x568d0630
Caught exception in looking up space point ptr... 0x3afdd590, 0x698e56a0
Caught exception in looking up space point ptr... 0x698e56a0, 0x698e1960
Caught exception in looking up space point ptr... 0x38055130, 0x38054c80
Caught exception in looking up space point ptr... 0x38054c80, 0x38054af0
Caught exception in looking up space point ptr... 0x56ebe5f0, 0x4fb82ac0
Caught exception in looking up space point ptr... 0x4fb82ac0, 0x56ebe5f0
Caught exception in looking up space point ptr... 0xe72ea70, 0xe72e900
Caught exception in looking up space point ptr... 0xe72e900, 0xe72ea70
Caught exception in looking up space point ptr... 0x38054070, 0x380541c0
Caught exception in looking up space point ptr... 0x380541c0, 0x380544a0
Caught exception in looking up space point ptr... 0x67e5c7e0, 0x38053b40
Caught exception in looking up space point ptr... 0x38053b40, 0x67e5c7e0
Caught exception in looking up space point ptr... 0x2c2ee320, 0x2c2ee4b0
Caught exception in looking up space point ptr... 0x2c2ee4b0, 0x114ddd40
Caught exception in looking up space point ptr... 0x3fa15970, 0x3fa15b10
Caught exception in looking up space point ptr... 0x3fa15b10, 0x3fa15970
Caught exception in looking up space point ptr... 0x395eca10, 0x395ed030
Caught exception in looking up space point ptr... 0x395ed030, 0x395ed1c0
Caught exception in looking up space point ptr... 0xe72cd70, 0xe72cf50
Caught exception in looking up space point ptr... 0xe72cf50, 0xe72d130
Caught exception in looking up space point ptr... 0x395ea0f0, 0x395ea5a0
Caught exception in looking up space point ptr... 0x395ea5a0, 0x395ea730
Caught exception in looking up space point ptr... 0x37c12cf0, 0x37c13240
Caught exception in looking up space point ptr... 0x37c13240, 0x37c12cf0
Caught exception in looking up space point ptr... 0x65e41fe0, 0x65e42170
Caught exception in looking up space point ptr... 0x65e42170, 0x65e41e50
Caught exception in looking up space point ptr... 0x5e3933a0, 0x7621fb70
Caught exception in looking up space point ptr... 0x7621fb70, 0x5e3933a0
++++>>>> total num hits: 53726, num free: 51188
04-Feb-2026 01:31:19 CET  Closed output file "np02vd_raw_run042433_0450_df-s03-d1_dw_0_20260203T213308_reco_stage0_20260204T003119_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         98.0052       109.381       123.089       109.791       7.49004        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                        5.714e-05    7.98543e-05   0.00014021    7.7491e-05    1.72316e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                5.83614       6.25243       6.83757       6.29309      0.300672        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.0103053     0.0540627     0.147788      0.0537287     0.0325263       19     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.00045433    0.00498463     0.0228485    0.00099216    0.00613024       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.00046105    0.000648865   0.00103428    0.00064022    0.000161588      19     
produce:beamevent:BeamEvent                       6.917e-05    0.000131718   0.00081987     8.981e-05    0.00016285       19     
produce:pdvddaphne:DAPHNEReaderPDVD                1.69176       2.03793       2.37236       2.0506       0.178028        19     
produce:ophit:OpHitFinder                         0.0232893     0.0332008     0.0393725     0.0326724    0.00514718       19     
produce:wclsdatavd:WireCellToolkit                 70.4041       81.0581       92.6906       80.9764       5.05007        19     
produce:gaushit:GausHitFinder                     0.715131       1.07344       1.92798       1.0095       0.319923        19     
produce:nhitsfilter:NumberOfHitsFilter           0.00015804    0.000219491   0.00036366    0.00019455    5.52034e-05      19     
produce:reco3d:SpacePointSolver                    7.45687       12.0615       19.6386       12.2926       2.7799         19     
produce:hitpdune:DisambigFromSpacePoints          0.096095      0.152235      0.244465      0.134449      0.0444884       19     
produce:cluster3d:Cluster3D                       0.998479       1.77396       2.48383       1.93979      0.468865        19     
[art]:TriggerResults:TriggerResultInserter        2.561e-05    3.61416e-05   0.00010783     3.124e-05    1.76887e-05      19     
end_path:out1:RootOutput                          4.58e-06     6.53579e-06    2.33e-05      5.65e-06     3.99114e-06      19     
end_path:out1:RootOutput(write)                    4.08195       4.82961       6.37316       4.80672      0.555929        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5420.38 MB
  Peak resident set size usage (VmHWM): 3399.93 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0450_df-s03-d1_dw_0_20260203T213308_reco_stage0_20260204T003119_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0450_df-s03-d1_dw_0_20260203T213308_reco_stage0_20260204T003119_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0450_df-s03-d1_dw_0_20260203T213308_reco_stage0_20260204T003119_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770162964.0,
    "core.end_time": 1770165081.0,
    "core.events": [
      51309,
      51315,
      51321,
      51327,
      51333,
      51339,
      51345,
      51351,
      51357,
      51363,
      51369,
      51375,
      51381,
      51387,
      51393,
      51399,
      51405,
      51411,
      51417
    ],
    "core.event_count": 19,
    "core.first_event_number": 51309,
    "core.last_event_number": 51417,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0450_df-s03-d1_dw_0_20260203T213308.hdf5"
    }
  ]
}
justIN time: 2026-02-04 06:07:54 UTC       justIN version: 01.06.00