justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303652.148@dunegpschedd01.fnal.gov

Jobsub ID303652.148@dunegpschedd01.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:50:26
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce512
Last heartbeat2026-02-04 00:34:21
From worker nodeHostnameb9g24p4713.cern.ch
cpuinfoAMD EPYC 7302 16-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:32:14
Input filesvd-protodune:np02vd_raw_run042433_0431_df-s03-d1_dw_0_20260203T212557.hdf5
JobscriptExit code0
Real time59m (3569s)
CPU time57m (3461s = 96%)
Max RSS bytes3606986752 (3439 MiB)
Outputting started2026-02-04 00:31:45
Output files
Finished2026-02-04 00:34:21
Saved logsjustin-logs:303652.148-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

int ptr... 0x5da36490, 0xeedcd70
Caught exception in looking up space point ptr... 0xeedcd70, 0xeec7460
Caught exception in looking up space point ptr... 0x2fd9f4b0, 0x6b56b270
Caught exception in looking up space point ptr... 0x6b56b270, 0x67b56530
Caught exception in looking up space point ptr... 0x38f74800, 0x48cef680
Caught exception in looking up space point ptr... 0x48cef680, 0x38f74800
Caught exception in looking up space point ptr... 0x4cb00f90, 0x467c5bf0
Caught exception in looking up space point ptr... 0x467c5bf0, 0x11d5a6b0
Caught exception in looking up space point ptr... 0x7c3ab350, 0x51ce88a0
Caught exception in looking up space point ptr... 0x51cea3f0, 0x7c3ab350
Caught exception in looking up space point ptr... 0x18566ad0, 0x18555000
Caught exception in looking up space point ptr... 0x18566c60, 0x18566ad0
Caught exception in looking up space point ptr... 0x137e80c0, 0x137e7df0
Caught exception in looking up space point ptr... 0x137e7df0, 0x784a8f20
Caught exception in looking up space point ptr... 0x784a8f20, 0x784a9620
Caught exception in looking up space point ptr... 0x784a9620, 0x137e80c0
Caught exception in looking up space point ptr... 0x8d7e9470, 0x7849a840
Caught exception in looking up space point ptr... 0x7849a840, 0x7849ab60
Caught exception in looking up space point ptr... 0x7849ab60, 0x7849b480
Caught exception in looking up space point ptr... 0x4185a530, 0xa8d5ee0
Caught exception in looking up space point ptr... 0xa8d5ee0, 0xa873fc0
Caught exception in looking up space point ptr... 0x784b1e60, 0x137eff00
Caught exception in looking up space point ptr... 0x137eff00, 0x137efc70
Caught exception in looking up space point ptr... 0x54597eb0, 0x2c510d30
Caught exception in looking up space point ptr... 0x49a9c730, 0x54597eb0
Caught exception in looking up space point ptr... 0x3a706dc0, 0x4adfecc0
Caught exception in looking up space point ptr... 0xf3b4530, 0x3a706dc0
Caught exception in looking up space point ptr... 0x4aec7410, 0x50624870
Caught exception in looking up space point ptr... 0x50624870, 0x4aec7410
Caught exception in looking up space point ptr... 0x3a11f880, 0x3a11f6f0
Caught exception in looking up space point ptr... 0x3a11f6f0, 0x3a11f560
Caught exception in looking up space point ptr... 0x3b98ee30, 0xaee30f0
Caught exception in looking up space point ptr... 0x49ce8740, 0x3b98ee30
Caught exception in looking up space point ptr... 0x63131c40, 0x723c51c0
Caught exception in looking up space point ptr... 0x723c8600, 0x63131c40
Caught exception in looking up space point ptr... 0x739a7a50, 0x739a78c0
Caught exception in looking up space point ptr... 0x739a78c0, 0x7398b220
Caught exception in looking up space point ptr... 0x1727d2b0, 0xd674420
Caught exception in looking up space point ptr... 0xd674420, 0xd6745b0
Caught exception in looking up space point ptr... 0x4963a4b0, 0x61849540
Caught exception in looking up space point ptr... 0x49643b00, 0x4963a4b0
Caught exception in looking up space point ptr... 0x301b2950, 0x392bd850
Caught exception in looking up space point ptr... 0x392bd850, 0x62658330
Caught exception in looking up space point ptr... 0x81d04830, 0x18b03df0
Caught exception in looking up space point ptr... 0x18b03df0, 0x47a597a0
Caught exception in looking up space point ptr... 0x3843ed30, 0x1a075020
Caught exception in looking up space point ptr... 0x1a075020, 0x3843ed30
Caught exception in looking up space point ptr... 0x3843ed30, 0x2fb7a3c0
Caught exception in looking up space point ptr... 0x2fb7a3c0, 0x3843ed30
Caught exception in looking up space point ptr... 0x2bd64930, 0x4b48e3c0
Caught exception in looking up space point ptr... 0x4b48e3c0, 0x2bd64930
Caught exception in looking up space point ptr... 0x6a0a9ff0, 0x5db2d4b0
Caught exception in looking up space point ptr... 0x5db2d4b0, 0x6a0a9ff0
Caught exception in looking up space point ptr... 0x2d945ef0, 0x6b51e280
Caught exception in looking up space point ptr... 0x6b51e280, 0x2d945ef0
Caught exception in looking up space point ptr... 0x3f4946f0, 0x4c0502a0
Caught exception in looking up space point ptr... 0x4c0502a0, 0x3f4946f0
Caught exception in looking up space point ptr... 0x3add23d0, 0x50649630
Caught exception in looking up space point ptr... 0x50649630, 0x2d17b150
Caught exception in looking up space point ptr... 0x2d5291e0, 0x161b2180
Caught exception in looking up space point ptr... 0x161b2180, 0x2f436df0
Caught exception in looking up space point ptr... 0x6b40f930, 0x80491840
Caught exception in looking up space point ptr... 0x80491840, 0x6b40f930
Caught exception in looking up space point ptr... 0x50302550, 0x503026d0
Caught exception in looking up space point ptr... 0x503026d0, 0x50302550
Caught exception in looking up space point ptr... 0x4db129b0, 0x4db11830
Caught exception in looking up space point ptr... 0x4db11830, 0x300bf730
++++>>>> total num hits: 513685, num free: 504743
04-Feb-2026 01:31:37 CET  Closed output file "np02vd_raw_run042433_0431_df-s03-d1_dw_0_20260203T212557_reco_stage0_20260204T003137_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         126.941       184.569       210.876       191.711       24.6531        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       5.2421e-05    7.89278e-05   0.000131852   6.9102e-05    2.46278e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                6.03385       7.06589       8.04031       7.19022      0.577147        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.0102025     0.0233846     0.0675158     0.0138456     0.0177975       19     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000406037   0.00063065    0.00108123    0.00054602    0.000188922      19     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.000362237   0.000468172   0.000834997   0.000451769   0.000104645      19     
produce:beamevent:BeamEvent                      6.1691e-05    0.000140936   0.000867727   0.000107472   0.00017355       19     
produce:pdvddaphne:DAPHNEReaderPDVD                2.15815       2.76159       5.25969       2.6791       0.654499        19     
produce:ophit:OpHitFinder                         0.0261259     0.0342533     0.0487305     0.0318832    0.00611394       19     
produce:wclsdatavd:WireCellToolkit                 98.3906       113.267       141.752       113.277       9.02236        19     
produce:gaushit:GausHitFinder                     0.919532       1.78739       2.34922       1.92788      0.450653        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000281756    0.0003942    0.000518271   0.000400057   6.28985e-05      19     
produce:reco3d:SpacePointSolver                    7.79062       14.1968       22.5242       13.5218       3.59673        19     
produce:hitpdune:DisambigFromSpacePoints          0.154686      0.272584      0.445213      0.270576      0.0730788       19     
produce:cluster3d:Cluster3D                        1.79604       39.2857       59.7865       51.494        22.3457        19     
[art]:TriggerResults:TriggerResultInserter        2.462e-05    3.26891e-05   8.5891e-05     3.011e-05    1.31056e-05      19     
end_path:out1:RootOutput                          3.44e-06     7.29484e-06   2.5081e-05     7.54e-06     4.71765e-06      19     
end_path:out1:RootOutput(write)                    4.60402       5.84538       6.59668       6.10365      0.660741        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5580.12 MB
  Peak resident set size usage (VmHWM): 3606.99 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0431_df-s03-d1_dw_0_20260203T212557_reco_stage0_20260204T003137_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0431_df-s03-d1_dw_0_20260203T212557_reco_stage0_20260204T003137_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0431_df-s03-d1_dw_0_20260203T212557_reco_stage0_20260204T003137_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770161546.0,
    "core.end_time": 1770165098.0,
    "core.events": [
      49143,
      49149,
      49155,
      49161,
      49167,
      49173,
      49179,
      49185,
      49191,
      49197,
      49203,
      49209,
      49215,
      49221,
      49227,
      49233,
      49239,
      49245,
      49251
    ],
    "core.event_count": 19,
    "core.first_event_number": 49143,
    "core.last_event_number": 49251,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0431_df-s03-d1_dw_0_20260203T212557.hdf5"
    }
  ]
}
justIN time: 2026-02-04 07:37:10 UTC       justIN version: 01.06.00