justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303647.108@dunegpschedd01.fnal.gov

Jobsub ID303647.108@dunegpschedd01.fnal.gov
Workflow ID12717
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:46:24
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce515
Last heartbeat2026-02-03 23:22:42
From worker nodeHostnameb9p12p0201.cern.ch
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 22:47:13
Input filesvd-protodune:np02vd_raw_run042421_0097_df-s02-d0_dw_0_20260203T142329.hdf5
JobscriptExit code0
Real time33m (1985s)
CPU time31m (1888s = 95%)
Max RSS bytes2640732160 (2518 MiB)
Outputting started2026-02-03 23:20:19
Output files
Finished2026-02-03 23:22:42
Saved logsjustin-logs:303647.108-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

r... 0xd5f7600, 0xd5f7920
Caught exception in looking up space point ptr... 0x86f40b0, 0x514cd8e0
Caught exception in looking up space point ptr... 0x514cd8e0, 0x86f40b0
Caught exception in looking up space point ptr... 0x57600310, 0x575fe910
Caught exception in looking up space point ptr... 0x575fe910, 0x575f9a30
Caught exception in looking up space point ptr... 0x575f9a30, 0x575fe580
Caught exception in looking up space point ptr... 0x575fe580, 0x57600310
Caught exception in looking up space point ptr... 0x57bd51c0, 0x57bd6390
Caught exception in looking up space point ptr... 0x57bd6390, 0x57bd52f0
Caught exception in looking up space point ptr... 0x44a6d930, 0x38f8b360
Caught exception in looking up space point ptr... 0x38f8b360, 0x38f938d0
Caught exception in looking up space point ptr... 0x413179d0, 0x41317bb0
Caught exception in looking up space point ptr... 0x41317bb0, 0x413179d0
Caught exception in looking up space point ptr... 0xe631c20, 0x35d9ad30
Caught exception in looking up space point ptr... 0x35d9ad30, 0x129a4510
Caught exception in looking up space point ptr... 0x40983510, 0x40985d30
Caught exception in looking up space point ptr... 0x40985d30, 0x40983510
Caught exception in looking up space point ptr... 0x40983510, 0x40985ff0
Caught exception in looking up space point ptr... 0x40985ff0, 0x40983510
Caught exception in looking up space point ptr... 0x848a120, 0x84854d0
Caught exception in looking up space point ptr... 0x84854d0, 0x848f420
Caught exception in looking up space point ptr... 0x13369750, 0x13368ed0
Caught exception in looking up space point ptr... 0x13368ed0, 0x13369a00
Caught exception in looking up space point ptr... 0x2845f670, 0x2845f7e0
Caught exception in looking up space point ptr... 0x2845f7e0, 0x5061b5d0
Caught exception in looking up space point ptr... 0x4ebe6bd0, 0x4ebe6960
Caught exception in looking up space point ptr... 0x4ebe6960, 0x4ebe6bd0
Caught exception in looking up space point ptr... 0x3bd1cec0, 0x3bd13bd0
Caught exception in looking up space point ptr... 0x3bd13bd0, 0x3bd1cec0
Caught exception in looking up space point ptr... 0xd6006a0, 0xd6009c0
Caught exception in looking up space point ptr... 0xd6009c0, 0xd600b50
Caught exception in looking up space point ptr... 0x5d35efc0, 0x8f04780
Caught exception in looking up space point ptr... 0x8f04780, 0x8f04370
Caught exception in looking up space point ptr... 0xcddd7e0, 0xcddd970
Caught exception in looking up space point ptr... 0xcddd970, 0xcddd1a0
Caught exception in looking up space point ptr... 0xcddd1a0, 0xcddd4c0
Caught exception in looking up space point ptr... 0xcddd4c0, 0xcddd330
Caught exception in looking up space point ptr... 0xcddd330, 0xcddd7e0
Caught exception in looking up space point ptr... 0xe632c20, 0x27597a80
Caught exception in looking up space point ptr... 0x27597a80, 0xe632c20
Caught exception in looking up space point ptr... 0xe632c20, 0x275974e0
Caught exception in looking up space point ptr... 0x275974e0, 0xe632c20
Caught exception in looking up space point ptr... 0xe632c20, 0xe632150
Caught exception in looking up space point ptr... 0xe632150, 0xe632c20
Caught exception in looking up space point ptr... 0x20fa8c40, 0x20fa8960
Caught exception in looking up space point ptr... 0x20fa8960, 0x20fa8c40
Caught exception in looking up space point ptr... 0x2554be40, 0x4f6f8c10
Caught exception in looking up space point ptr... 0x4f6f8c10, 0x2554be40
Caught exception in looking up space point ptr... 0x2aed0750, 0x2aed07f0
Caught exception in looking up space point ptr... 0x2aed07f0, 0x2aecdc50
Caught exception in looking up space point ptr... 0x4598d210, 0x4598d6c0
Caught exception in looking up space point ptr... 0x4598d6c0, 0x4598d210
Caught exception in looking up space point ptr... 0x4598d210, 0x4598d3a0
Caught exception in looking up space point ptr... 0x4598d3a0, 0x4598d210
Caught exception in looking up space point ptr... 0x2e859600, 0x2e8594b0
Caught exception in looking up space point ptr... 0x2e8594b0, 0x2e85cd00
Caught exception in looking up space point ptr... 0x50fd0240, 0x50fcfb70
Caught exception in looking up space point ptr... 0x50fcfb70, 0x50fd0240
Caught exception in looking up space point ptr... 0x255c88b0, 0x255c8a40
Caught exception in looking up space point ptr... 0x255c8a40, 0x255c8590
Caught exception in looking up space point ptr... 0x27598f00, 0x27598d60
Caught exception in looking up space point ptr... 0x27598d60, 0x275991a0
Caught exception in looking up space point ptr... 0x255c8400, 0x255c8270
Caught exception in looking up space point ptr... 0x255c8270, 0x255c8400
Caught exception in looking up space point ptr... 0x1036f990, 0x3bd2afe0
Caught exception in looking up space point ptr... 0x3bd2afe0, 0x1036f990
++++>>>> total num hits: 93780, num free: 89093
04-Feb-2026 00:20:09 CET  Closed output file "np02vd_raw_run042421_0097_df-s02-d0_dw_0_20260203T142329_reco_stage0_20260203T232009_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                          61.83        71.1948       93.5738       68.2053       9.03077        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       4.2921e-05    7.31218e-05   0.000200751    5.846e-05     3.956e-05       27     
produce:tpcrawdecoder:PDVDTPCReader                 2.922        3.59304       4.25495       3.52002      0.337674        27     
produce:triggerrawdecoder:PDVDTriggerReader4      0.0093996     0.0180448     0.0508404     0.0128351     0.0106324       27     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000373311   0.000459391   0.000669993   0.000457661   5.66507e-05      27     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0013898    0.00172635    0.00239291    0.00167001    0.000211011      27     
produce:beamevent:BeamEvent                       5.216e-05    0.000112507   0.000704182   8.3561e-05    0.000125087      27     
produce:pdvddaphne:DAPHNEReaderPDVD                1.84923       2.24959       3.02601       2.16832       0.26901        27     
produce:ophit:OpHitFinder                         0.0182889     0.0297014     0.0377006     0.0311338    0.00431508       27     
produce:wclsdatavd:WireCellToolkit                 46.7398       50.5019       62.4119        48.29        4.91624        27     
produce:gaushit:GausHitFinder                     0.387009      0.759461       1.33384      0.714656      0.211833        27     
produce:nhitsfilter:NumberOfHitsFilter           0.000136891   0.000238145   0.000504681   0.000225431   7.69444e-05      27     
produce:reco3d:SpacePointSolver                    4.00787       8.61326       13.4232       7.94698       2.72902        27     
produce:hitpdune:DisambigFromSpacePoints          0.0429413     0.0809401     0.131225      0.0709062     0.0253047       27     
produce:cluster3d:Cluster3D                       0.613073       2.29002       9.73988       1.34557       2.27464        27     
[art]:TriggerResults:TriggerResultInserter        1.752e-05    2.64352e-05   0.00010734     2.253e-05    1.66221e-05      27     
end_path:out1:RootOutput                          3.39e-06     8.38078e-06    3.281e-05     7.28e-06     5.40975e-06      27     
end_path:out1:RootOutput(write)                    2.65467       3.04076       4.16612       2.93869      0.387464        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4607.22 MB
  Peak resident set size usage (VmHWM): 2640.73 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042421_0097_df-s02-d0_dw_0_20260203T142329_reco_stage0_20260203T232009_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042421_0097_df-s02-d0_dw_0_20260203T142329_reco_stage0_20260203T232009_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042421_0097_df-s02-d0_dw_0_20260203T142329_reco_stage0_20260203T232009_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770158847.0,
    "core.end_time": 1770160810.0,
    "core.events": [
      15780,
      15786,
      15792,
      15798,
      15804,
      15810,
      15816,
      15822,
      15828,
      15834,
      15840,
      15846,
      15852,
      15858,
      15864,
      15870,
      15876,
      15882,
      15888,
      15894,
      15900,
      15906,
      15912,
      15918,
      15924,
      15930,
      15936
    ],
    "core.event_count": 27,
    "core.first_event_number": 15780,
    "core.last_event_number": 15936,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42421
    ],
    "core.runs_subruns": [
      4242100001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042421_0097_df-s02-d0_dw_0_20260203T142329.hdf5"
    }
  ]
}
justIN time: 2026-02-04 10:46:51 UTC       justIN version: 01.06.00