justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297923.87@dunegpschedd02.fnal.gov

Jobsub ID297923.87@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 23:00:26
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce504
Last heartbeat2026-02-04 00:44:41
From worker nodeHostnameb9p07p8370.cern.ch
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-04 00:00:18
Input filesvd-protodune:np02vd_raw_run042433_0291_df-s02-d1_dw_0_20260203T203242.hdf5
JobscriptExit code0
Real time41m (2517s)
CPU time40m (2403s = 95%)
Max RSS bytes3397853184 (3240 MiB)
Outputting started2026-02-04 00:42:17
Output files
Finished2026-02-04 00:44:41
Saved logsjustin-logs:297923.87-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

up space point ptr... 0x63fb1630, 0x3c95a040
Caught exception in looking up space point ptr... 0x496d7cb0, 0x496d7b20
Caught exception in looking up space point ptr... 0x496d7b20, 0x496d7e40
Caught exception in looking up space point ptr... 0x3eddb7e0, 0x4f406f40
Caught exception in looking up space point ptr... 0x4f406f40, 0x3eddb7e0
Caught exception in looking up space point ptr... 0xe4fbe00, 0xe4fbf90
Caught exception in looking up space point ptr... 0xe4fbf90, 0x156e88c0
Caught exception in looking up space point ptr... 0x3ec39280, 0x2a94b960
Caught exception in looking up space point ptr... 0x2a94b960, 0x3ec39280
Caught exception in looking up space point ptr... 0x3cd9ac10, 0x1eaaac30
Caught exception in looking up space point ptr... 0x1eaaac30, 0x3cd9bb00
Caught exception in looking up space point ptr... 0x584e9430, 0x584d17c0
Caught exception in looking up space point ptr... 0x584d17c0, 0x584e9430
Caught exception in looking up space point ptr... 0x584e9430, 0x1ccbd8e0
Caught exception in looking up space point ptr... 0x1ccbd8e0, 0x584e9430
Caught exception in looking up space point ptr... 0x13ac5280, 0x13ac1da0
Caught exception in looking up space point ptr... 0x13ac1da0, 0x13ac5280
Caught exception in looking up space point ptr... 0x5a8ff0d0, 0x5a8fed80
Caught exception in looking up space point ptr... 0x5a8fed80, 0x5a8ff0d0
Caught exception in looking up space point ptr... 0x39a2a930, 0x51b52260
Caught exception in looking up space point ptr... 0x51b52260, 0x39a2a930
Caught exception in looking up space point ptr... 0x564d20f0, 0x564d1f90
Caught exception in looking up space point ptr... 0xe4fc0d0, 0x564d20f0
Caught exception in looking up space point ptr... 0x58c25860, 0x58c23650
Caught exception in looking up space point ptr... 0x58c23650, 0x58c25860
Caught exception in looking up space point ptr... 0x15b06880, 0x15afd0b0
Caught exception in looking up space point ptr... 0x15afd0b0, 0x15b06880
Caught exception in looking up space point ptr... 0xdf14960, 0xdf15b70
Caught exception in looking up space point ptr... 0xdf12600, 0xdf14960
Caught exception in looking up space point ptr... 0xa6d1c30, 0xa6c4cb0
Caught exception in looking up space point ptr... 0xa6c4cb0, 0xa6d1c30
Caught exception in looking up space point ptr... 0x16602100, 0x16601de0
Caught exception in looking up space point ptr... 0x16601de0, 0x16601ac0
Caught exception in looking up space point ptr... 0x16601ac0, 0x16601f70
Caught exception in looking up space point ptr... 0x16601f70, 0x16602100
Caught exception in looking up space point ptr... 0x2e512300, 0x2e512400
Caught exception in looking up space point ptr... 0x2e512400, 0x2e512300
Caught exception in looking up space point ptr... 0x2e512300, 0x4253cd90
Caught exception in looking up space point ptr... 0x4253cd90, 0x2e512300
Caught exception in looking up space point ptr... 0x42d61950, 0x199f5830
Caught exception in looking up space point ptr... 0x199f5830, 0x42d61950
Caught exception in looking up space point ptr... 0x15bc2550, 0x9234340
Caught exception in looking up space point ptr... 0x9234340, 0x15bc2550
Caught exception in looking up space point ptr... 0x33f6e730, 0x33f6ec40
Caught exception in looking up space point ptr... 0x33f6e9f0, 0x33f6e730
Caught exception in looking up space point ptr... 0x496c7cd0, 0x496c04e0
Caught exception in looking up space point ptr... 0x496c04e0, 0x496c7cd0
Caught exception in looking up space point ptr... 0x13aed530, 0x13aed680
Caught exception in looking up space point ptr... 0x13aed680, 0x13aed530
Caught exception in looking up space point ptr... 0x212fd8a0, 0x40075750
Caught exception in looking up space point ptr... 0x40075750, 0x212fd8a0
Caught exception in looking up space point ptr... 0x37cc0bf0, 0xed4d290
Caught exception in looking up space point ptr... 0xed52490, 0x37cc0bf0
Caught exception in looking up space point ptr... 0x2bbf6610, 0x2bbf67a0
Caught exception in looking up space point ptr... 0x2bbf67a0, 0x2bbf6ac0
Caught exception in looking up space point ptr... 0x57efde20, 0x57f02390
Caught exception in looking up space point ptr... 0x57f02390, 0x57efde20
Caught exception in looking up space point ptr... 0x581d3a40, 0x581d4590
Caught exception in looking up space point ptr... 0x581d4590, 0x581d2c30
Caught exception in looking up space point ptr... 0x58c283e0, 0x398fa5d0
Caught exception in looking up space point ptr... 0x398fa5d0, 0x58c283e0
Caught exception in looking up space point ptr... 0x37cccac0, 0xed55bb0
Caught exception in looking up space point ptr... 0xed55bb0, 0x37ccc930
Caught exception in looking up space point ptr... 0x13be2810, 0x13bf6ea0
Caught exception in looking up space point ptr... 0x13bf6ea0, 0x13bf8560
Caught exception in looking up space point ptr... 0x1e1ca3a0, 0xaf16da0
Caught exception in looking up space point ptr... 0xaf16da0, 0x1e1ca3a0
++++>>>> total num hits: 93824, num free: 89913
04-Feb-2026 01:42:08 CET  Closed output file "np02vd_raw_run042433_0291_df-s02-d1_dw_0_20260203T203242_reco_stage0_20260204T004208_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         107.704       128.974       146.185       133.049       11.1233        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       5.6551e-05    9.58649e-05   0.000160583   8.6392e-05    3.26811e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                5.7253        7.46907       26.2671       6.48272       4.44799        19     
produce:triggerrawdecoder:PDVDTriggerReader4     0.00985785     0.0202628     0.0739776     0.0137076     0.0150738       19     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000400818   0.000486802   0.000629061   0.000477259   5.39934e-05      19     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.000378287   0.000491867   0.000720674   0.000466389   9.56009e-05      19     
produce:beamevent:BeamEvent                      5.8911e-05    0.000140174   0.000773534   0.000116782   0.000151214      19     
produce:pdvddaphne:DAPHNEReaderPDVD                2.14397       2.62034       3.31796       2.5807       0.360353        19     
produce:ophit:OpHitFinder                         0.0200502     0.0343778     0.0437307     0.0344359    0.00531857       19     
produce:wclsdatavd:WireCellToolkit                 82.7291       94.7283       102.451       95.781        5.04966        19     
produce:gaushit:GausHitFinder                      0.80223       1.16702       1.73684       1.09875      0.287444        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000197074   0.000313657   0.000595651   0.000284795   9.54867e-05      19     
produce:reco3d:SpacePointSolver                    8.08813       15.912        31.0236       13.9133       6.26024        19     
produce:hitpdune:DisambigFromSpacePoints          0.0815878     0.157307      0.348792      0.137976      0.0579108       19     
produce:cluster3d:Cluster3D                        1.25448       2.13858       3.47598       1.74915      0.720866        19     
[art]:TriggerResults:TriggerResultInserter       2.3371e-05    3.63975e-05   0.000100682     3.1e-05     1.62275e-05      19     
end_path:out1:RootOutput                          6.33e-06     8.50611e-06   2.7371e-05     7.47e-06     4.50876e-06      19     
end_path:out1:RootOutput(write)                    3.75429       4.69664       6.55207       4.40471      0.764891        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5402.99 MB
  Peak resident set size usage (VmHWM): 3397.85 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0291_df-s02-d1_dw_0_20260203T203242_reco_stage0_20260204T004208_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0291_df-s02-d1_dw_0_20260203T203242_reco_stage0_20260204T004208_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0291_df-s02-d1_dw_0_20260203T203242_reco_stage0_20260204T004208_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770163229.0,
    "core.end_time": 1770165730.0,
    "core.events": [
      33181,
      33187,
      33193,
      33199,
      33205,
      33211,
      33217,
      33223,
      33229,
      33235,
      33241,
      33247,
      33253,
      33259,
      33265,
      33271,
      33277,
      33283,
      33289
    ],
    "core.event_count": 19,
    "core.first_event_number": 33181,
    "core.last_event_number": 33289,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0291_df-s02-d1_dw_0_20260203T203242.hdf5"
    }
  ]
}
justIN time: 2026-02-04 06:06:06 UTC       justIN version: 01.06.00