justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303647.87@dunegpschedd01.fnal.gov

Jobsub ID303647.87@dunegpschedd01.fnal.gov
Workflow ID12717
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:46:24
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce512
Last heartbeat2026-02-03 23:24:03
From worker nodeHostnameb9p10p5730.cern.ch
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 22:47:22
Input filesvd-protodune:np02vd_raw_run042421_0099_df-s02-d0_dw_0_20260203T142435.hdf5
JobscriptExit code0
Real time34m (2068s)
CPU time33m (1983s = 95%)
Max RSS bytes2627833856 (2506 MiB)
Outputting started2026-02-03 23:21:52
Output files
Finished2026-02-03 23:24:03
Saved logsjustin-logs:303647.87-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

exception in looking up space point ptr... 0x3984d420, 0x3984d600
Caught exception in looking up space point ptr... 0x3984d600, 0x3984cf70
Caught exception in looking up space point ptr... 0x3984cf70, 0x3984cc50
Caught exception in looking up space point ptr... 0x3984cc50, 0x3984cde0
Caught exception in looking up space point ptr... 0x3984cde0, 0x3984c7a0
Caught exception in looking up space point ptr... 0x3984c7a0, 0x3984cac0
Caught exception in looking up space point ptr... 0x3984cac0, 0x3984d420
Caught exception in looking up space point ptr... 0x22bbf6a0, 0x26275490
Caught exception in looking up space point ptr... 0x26275490, 0x454e2b10
Caught exception in looking up space point ptr... 0x350e2780, 0x350e28d0
Caught exception in looking up space point ptr... 0x350e28d0, 0x350e2780
Caught exception in looking up space point ptr... 0x27e3e0d0, 0x27e3ddb0
Caught exception in looking up space point ptr... 0x27e3ddb0, 0x27e3e0d0
Caught exception in looking up space point ptr... 0x60043f90, 0x60043e00
Caught exception in looking up space point ptr... 0x60043e00, 0x60043f90
Caught exception in looking up space point ptr... 0x3d19a1e0, 0x3d199f20
Caught exception in looking up space point ptr... 0x3d199f20, 0x3d193d80
Caught exception in looking up space point ptr... 0x2ee16400, 0x2ee16200
Caught exception in looking up space point ptr... 0x2ee16200, 0x2ee16400
Caught exception in looking up space point ptr... 0x12c53020, 0x398e1f50
Caught exception in looking up space point ptr... 0x398e1f50, 0x12c53020
Caught exception in looking up space point ptr... 0x12c53020, 0x27c442c0
Caught exception in looking up space point ptr... 0x27c442c0, 0x12c53020
Caught exception in looking up space point ptr... 0x12c53020, 0x27c440a0
Caught exception in looking up space point ptr... 0x27c440a0, 0x12c53020
Caught exception in looking up space point ptr... 0x55610310, 0x556106e0
Caught exception in looking up space point ptr... 0x556106e0, 0x55610310
Caught exception in looking up space point ptr... 0x39dd07a0, 0x42a9c9a0
Caught exception in looking up space point ptr... 0x42a9c9a0, 0x42a9cb30
Caught exception in looking up space point ptr... 0x55df2610, 0x3be07500
Caught exception in looking up space point ptr... 0x3be07500, 0x3be162f0
Caught exception in looking up space point ptr... 0x2f302370, 0x622cc670
Caught exception in looking up space point ptr... 0x29fd24d0, 0x2f302370
Caught exception in looking up space point ptr... 0x264ee920, 0x225505c0
Caught exception in looking up space point ptr... 0x225505c0, 0x39f9e220
Caught exception in looking up space point ptr... 0x3d19b360, 0x3d19c850
Caught exception in looking up space point ptr... 0x3d19c390, 0x3d19b360
Caught exception in looking up space point ptr... 0x40467ed0, 0x4045d630
Caught exception in looking up space point ptr... 0x4045d630, 0x40467ed0
Caught exception in looking up space point ptr... 0x3bd769c0, 0x22585e10
Caught exception in looking up space point ptr... 0x22585e10, 0x3bd769c0
Caught exception in looking up space point ptr... 0x664c3780, 0x664c3430
Caught exception in looking up space point ptr... 0x664c3430, 0x664c3780
Caught exception in looking up space point ptr... 0x416e6a50, 0x416e0a60
Caught exception in looking up space point ptr... 0x416e0a60, 0x416e9c00
Caught exception in looking up space point ptr... 0x3be22fb0, 0x5c71d630
Caught exception in looking up space point ptr... 0x5c71d630, 0x5c71d950
Caught exception in looking up space point ptr... 0x4b4ab850, 0x4b4abbf0
Caught exception in looking up space point ptr... 0x4b4abbf0, 0x4b4ab850
Caught exception in looking up space point ptr... 0x5497e9e0, 0x55ab80f0
Caught exception in looking up space point ptr... 0x55ab80f0, 0x54983380
Caught exception in looking up space point ptr... 0x2622d8c0, 0x2622d780
Caught exception in looking up space point ptr... 0x2622d780, 0x2622d8c0
Caught exception in looking up space point ptr... 0x2a0dcc20, 0x2a0dc8b0
Caught exception in looking up space point ptr... 0x2a0dc8b0, 0x2a0dcc20
Caught exception in looking up space point ptr... 0x486f97c0, 0x29d49390
Caught exception in looking up space point ptr... 0x29d49390, 0x486f97c0
Caught exception in looking up space point ptr... 0x398574b0, 0x39857320
Caught exception in looking up space point ptr... 0x39857320, 0x398574b0
Caught exception in looking up space point ptr... 0x4fb542f0, 0x4fb543f0
Caught exception in looking up space point ptr... 0x4fb543f0, 0x4fb542f0
Caught exception in looking up space point ptr... 0x63222400, 0x2f373390
Caught exception in looking up space point ptr... 0x2f373390, 0x63222400
Caught exception in looking up space point ptr... 0x32624060, 0x32623f10
Caught exception in looking up space point ptr... 0x32623f10, 0x32624060
++++>>>> total num hits: 79972, num free: 76087
04-Feb-2026 00:21:46 CET  Closed output file "np02vd_raw_run042421_0099_df-s02-d0_dw_0_20260203T142435_reco_stage0_20260203T232146_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         66.1354       74.6333       95.2185       70.3804       8.26174        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       4.4841e-05    8.10529e-05   0.000166582   7.2141e-05    3.61847e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                3.66402       4.28687       4.84393       4.33989       0.33422        27     
produce:triggerrawdecoder:PDVDTriggerReader4     0.00960081     0.0165435     0.0458104     0.0132795    0.00911089       27     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000391084   0.000561937   0.00119607    0.000525025   0.000155362      27     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.00161469    0.00217091    0.00508862    0.00205767    0.000645064      27     
produce:beamevent:BeamEvent                      5.2211e-05    0.000116793   0.000947682   7.8801e-05    0.00016431       27     
produce:pdvddaphne:DAPHNEReaderPDVD                2.08673       2.60309       3.18094       2.50681      0.332766        27     
produce:ophit:OpHitFinder                         0.0253439     0.0349433     0.0431682     0.0346987    0.00488738       27     
produce:wclsdatavd:WireCellToolkit                 47.2167       53.6189       63.7644       50.9263       4.88416        27     
produce:gaushit:GausHitFinder                     0.477097      0.743943       1.41838      0.713201      0.201894        27     
produce:nhitsfilter:NumberOfHitsFilter           0.000146152   0.000219266   0.000434275   0.000197283   6.62089e-05      27     
produce:reco3d:SpacePointSolver                    4.82751       8.66711       16.8384       8.37443       2.8716         27     
produce:hitpdune:DisambigFromSpacePoints          0.0436988     0.0759493     0.185223      0.0681348     0.0313482       27     
produce:cluster3d:Cluster3D                       0.645266       1.42632       2.69512       1.28552      0.525285        27     
[art]:TriggerResults:TriggerResultInserter        1.847e-05    2.65589e-05   9.2001e-05    2.3571e-05    1.33984e-05      27     
end_path:out1:RootOutput                           3.7e-06     7.62348e-06   2.7571e-05     7.18e-06     4.30235e-06      27     
end_path:out1:RootOutput(write)                    2.71387       3.13828       4.51757       3.04139      0.414961        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4584.15 MB
  Peak resident set size usage (VmHWM): 2627.83 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042421_0099_df-s02-d0_dw_0_20260203T142435_reco_stage0_20260203T232146_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042421_0099_df-s02-d0_dw_0_20260203T142435_reco_stage0_20260203T232146_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042421_0099_df-s02-d0_dw_0_20260203T142435_reco_stage0_20260203T232146_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770158854.0,
    "core.end_time": 1770160907.0,
    "core.events": [
      16104,
      16110,
      16116,
      16122,
      16128,
      16134,
      16140,
      16146,
      16152,
      16158,
      16164,
      16170,
      16176,
      16182,
      16188,
      16194,
      16200,
      16206,
      16212,
      16218,
      16224,
      16230,
      16236,
      16242,
      16248,
      16254,
      16260
    ],
    "core.event_count": 27,
    "core.first_event_number": 16104,
    "core.last_event_number": 16260,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42421
    ],
    "core.runs_subruns": [
      4242100001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042421_0099_df-s02-d0_dw_0_20260203T142435.hdf5"
    }
  ]
}
justIN time: 2026-02-04 07:45:38 UTC       justIN version: 01.06.00