justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303652.137@dunegpschedd01.fnal.gov

Jobsub ID303652.137@dunegpschedd01.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:50:26
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce506
Last heartbeat2026-02-04 00:26:20
From worker nodeHostnameb9p06p2822.cern.ch
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:29:43
Input filesvd-protodune:np02vd_raw_run042433_0623_df-s02-d1_dw_0_20260203T223854.hdf5
JobscriptExit code0
Real time54m (3246s)
CPU time52m (3136s = 96%)
Max RSS bytes3625529344 (3457 MiB)
Outputting started2026-02-04 00:23:51
Output files
Finished2026-02-04 00:26:20
Saved logsjustin-logs:303652.137-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

oint ptr... 0x376cc2c0, 0x616f8f50
Caught exception in looking up space point ptr... 0x7c905a40, 0x76d8b230
Caught exception in looking up space point ptr... 0x76d8b230, 0x7c905a40
Caught exception in looking up space point ptr... 0x7c905a40, 0x285f9b70
Caught exception in looking up space point ptr... 0x285f9b70, 0x7c905a40
Caught exception in looking up space point ptr... 0x3910c1c0, 0x391091a0
Caught exception in looking up space point ptr... 0x391091a0, 0x3910c1c0
Caught exception in looking up space point ptr... 0xcf57460, 0xcf57f10
Caught exception in looking up space point ptr... 0xcf57f10, 0xcf57cb0
Caught exception in looking up space point ptr... 0x12ab4230, 0x12ab43b0
Caught exception in looking up space point ptr... 0x12ab43b0, 0x12ab4020
Caught exception in looking up space point ptr... 0x7838f1e0, 0x52bd26b0
Caught exception in looking up space point ptr... 0x52bd26b0, 0x7838f1e0
Caught exception in looking up space point ptr... 0x52ba0120, 0x52bb4e30
Caught exception in looking up space point ptr... 0x52bb4e30, 0x52ba0120
Caught exception in looking up space point ptr... 0x19494e80, 0x19496e50
Caught exception in looking up space point ptr... 0x19496e50, 0x19496b60
Caught exception in looking up space point ptr... 0x49504010, 0x49503300
Caught exception in looking up space point ptr... 0x49503300, 0x495030a0
Caught exception in looking up space point ptr... 0x433f0e00, 0x433f0c70
Caught exception in looking up space point ptr... 0x433f0c70, 0x433f0950
Caught exception in looking up space point ptr... 0x598b72e0, 0x51dea0c0
Caught exception in looking up space point ptr... 0x51dea0c0, 0x3fc71c90
Caught exception in looking up space point ptr... 0x599aa980, 0x599aa610
Caught exception in looking up space point ptr... 0x599aa610, 0x599aa980
Caught exception in looking up space point ptr... 0x1071d480, 0x1071d330
Caught exception in looking up space point ptr... 0x1071d330, 0x1071d7a0
Caught exception in looking up space point ptr... 0x2f26ee60, 0x2f26ed10
Caught exception in looking up space point ptr... 0x2f26ed10, 0x2f26ebc0
Caught exception in looking up space point ptr... 0x783a5720, 0x1a02c610
Caught exception in looking up space point ptr... 0x494bd550, 0x783a5720
Caught exception in looking up space point ptr... 0xcf6c720, 0xcf6c590
Caught exception in looking up space point ptr... 0xcf6c590, 0xcf6c400
Caught exception in looking up space point ptr... 0x512961e0, 0x522dd080
Caught exception in looking up space point ptr... 0x522dd080, 0x512961e0
Caught exception in looking up space point ptr... 0x599acc30, 0x599ac6e0
Caught exception in looking up space point ptr... 0x599ac6e0, 0x599acc30
Caught exception in looking up space point ptr... 0xefff210, 0x1e548c50
Caught exception in looking up space point ptr... 0x1e548c50, 0xefff210
Caught exception in looking up space point ptr... 0x164bca30, 0x586c3b70
Caught exception in looking up space point ptr... 0x586c3b70, 0x164bca30
Caught exception in looking up space point ptr... 0x76bf4580, 0x6a5825a0
Caught exception in looking up space point ptr... 0x6a5825a0, 0x76bf4580
Caught exception in looking up space point ptr... 0x6a923c50, 0x60d66710
Caught exception in looking up space point ptr... 0x60d66710, 0x6a923ef0
Caught exception in looking up space point ptr... 0x5c1fd940, 0x5c1fd7b0
Caught exception in looking up space point ptr... 0x5c1fd7b0, 0x5c1fd490
Caught exception in looking up space point ptr... 0x4b999f70, 0x4b999e00
Caught exception in looking up space point ptr... 0x4b999e00, 0x4b999cb0
Caught exception in looking up space point ptr... 0x60d43f10, 0x525d9170
Caught exception in looking up space point ptr... 0x525d9170, 0x628c3e80
Caught exception in looking up space point ptr... 0x599ac1e0, 0x599abec0
Caught exception in looking up space point ptr... 0x599abec0, 0x599abb50
Caught exception in looking up space point ptr... 0x2c66fd20, 0x2c6702d0
Caught exception in looking up space point ptr... 0x2c6702d0, 0x2c66fc30
Caught exception in looking up space point ptr... 0x62819db0, 0x62819390
Caught exception in looking up space point ptr... 0x62819390, 0x62819db0
Caught exception in looking up space point ptr... 0x5a145020, 0x5a144e90
Caught exception in looking up space point ptr... 0x5a144e90, 0x5a145020
Caught exception in looking up space point ptr... 0x236204a0, 0x23620310
Caught exception in looking up space point ptr... 0x23620310, 0x23620180
Caught exception in looking up space point ptr... 0xdaf1f20, 0x47e56210
Caught exception in looking up space point ptr... 0x47e56210, 0xdaf1d90
Caught exception in looking up space point ptr... 0x2361cae0, 0x2361cc30
Caught exception in looking up space point ptr... 0x2361cc30, 0x2361cae0
Caught exception in looking up space point ptr... 0x49cf2cb0, 0x49cf6250
Caught exception in looking up space point ptr... 0x49cf6250, 0x49cf4470
++++>>>> total num hits: 143265, num free: 138263
04-Feb-2026 01:23:43 CET  Closed output file "np02vd_raw_run042433_0623_df-s02-d1_dw_0_20260203T223854_reco_stage0_20260204T002343_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         121.348       167.75        207.888       174.716       26.6555        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                        4.98e-05     0.000103003   0.000227311   0.00010258    4.70924e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                5.71061       7.09877       8.16868       7.09161      0.584409        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.0109349     0.0146248     0.0273936     0.013158     0.00413221       19     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000428232   0.000569356   0.000895984   0.000519802   0.000120739      19     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.000407381   0.000526466   0.000790504   0.000503572   9.91602e-05      19     
produce:beamevent:BeamEvent                       6.834e-05    0.000146774   0.000872484    9.934e-05    0.000173415      19     
produce:pdvddaphne:DAPHNEReaderPDVD                2.0788        2.67162       3.22165       2.6364       0.362722        19     
produce:ophit:OpHitFinder                         0.0256417     0.0345062     0.0430103     0.0336105    0.00553918       19     
produce:wclsdatavd:WireCellToolkit                 92.4775       105.083       122.925       104.978       8.20001        19     
produce:gaushit:GausHitFinder                     0.870556       1.51021       2.32738       1.52409      0.455628        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000258081   0.000357148   0.000527853   0.000339772   7.74776e-05      19     
produce:reco3d:SpacePointSolver                    7.76653       16.6605       22.7536       17.4638       3.97105        19     
produce:hitpdune:DisambigFromSpacePoints          0.107776      0.226129      0.511843      0.213545      0.0870207       19     
produce:cluster3d:Cluster3D                        1.64677       29.0865       59.8206       42.832        23.0468        19     
[art]:TriggerResults:TriggerResultInserter        2.452e-05    3.67265e-05   9.1161e-05     3.147e-05    1.48116e-05      19     
end_path:out1:RootOutput                          3.77e-06     8.13158e-06    3.634e-05      5.5e-06     7.7485e-06       19     
end_path:out1:RootOutput(write)                    4.11431       5.32804       7.57639       5.29029      0.912978        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5657.03 MB
  Peak resident set size usage (VmHWM): 3625.53 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0623_df-s02-d1_dw_0_20260203T223854_reco_stage0_20260204T002343_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0623_df-s02-d1_dw_0_20260203T223854_reco_stage0_20260204T002343_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0623_df-s02-d1_dw_0_20260203T223854_reco_stage0_20260204T002343_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770161396.0,
    "core.end_time": 1770164624.0,
    "core.events": [
      71029,
      71035,
      71041,
      71047,
      71053,
      71059,
      71065,
      71071,
      71077,
      71083,
      71089,
      71095,
      71101,
      71107,
      71113,
      71119,
      71125,
      71131,
      71137
    ],
    "core.event_count": 19,
    "core.first_event_number": 71029,
    "core.last_event_number": 71137,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0623_df-s02-d1_dw_0_20260203T223854.hdf5"
    }
  ]
}
justIN time: 2026-02-04 05:59:15 UTC       justIN version: 01.06.00