justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303653.133@dunegpschedd01.fnal.gov

Jobsub ID303653.133@dunegpschedd01.fnal.gov
Workflow ID12717
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:52:24
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce510
Last heartbeat2026-02-04 00:28:59
From worker nodeHostnameb9p09p3861.cern.ch
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:46:13
Input filesvd-protodune:np02vd_raw_run042421_0031_df-s04-d1_dw_0_20260203T134748.hdf5
JobscriptExit code0
Real time40m (2417s)
CPU time38m (2311s = 95%)
Max RSS bytes2550464512 (2432 MiB)
Outputting started2026-02-04 00:26:31
Output files
Finished2026-02-04 00:28:59
Saved logsjustin-logs:303653.133-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

p space point ptr... 0x18ed2250, 0x18ed27a0
Caught exception in looking up space point ptr... 0x18ece850, 0x18ec6460
Caught exception in looking up space point ptr... 0x18ec6460, 0x18ed3310
Caught exception in looking up space point ptr... 0x18ed3310, 0x18ed2250
Caught exception in looking up space point ptr... 0x2e2e4df0, 0x48c2e5e0
Caught exception in looking up space point ptr... 0x48c2e5e0, 0x48c2e7c0
Caught exception in looking up space point ptr... 0xfe33b40, 0x7f96090
Caught exception in looking up space point ptr... 0x7f96090, 0xfe33b40
Caught exception in looking up space point ptr... 0x11532410, 0x11532280
Caught exception in looking up space point ptr... 0x1151c4b0, 0x11532410
Caught exception in looking up space point ptr... 0x171e1c70, 0x171e1a70
Caught exception in looking up space point ptr... 0x171dd8b0, 0x171e1c70
Caught exception in looking up space point ptr... 0x1166f960, 0x1166f830
Caught exception in looking up space point ptr... 0x1166f830, 0x1166f960
Caught exception in looking up space point ptr... 0x511e4240, 0x511e15c0
Caught exception in looking up space point ptr... 0x511e15c0, 0x511e4240
Caught exception in looking up space point ptr... 0x37cf3b20, 0x109f3180
Caught exception in looking up space point ptr... 0x109f3180, 0x37cf3b20
Caught exception in looking up space point ptr... 0x4da24b40, 0x3c443eb0
Caught exception in looking up space point ptr... 0x3c443eb0, 0x4da24c60
Caught exception in looking up space point ptr... 0x5304e590, 0x50f96aa0
Caught exception in looking up space point ptr... 0x5304e400, 0x5304e590
Caught exception in looking up space point ptr... 0x3dbe8c90, 0x16c389a0
Caught exception in looking up space point ptr... 0x16c389a0, 0x3dbe8c90
Caught exception in looking up space point ptr... 0x48c338d0, 0xc08baa0
Caught exception in looking up space point ptr... 0x48c33740, 0x48c338d0
Caught exception in looking up space point ptr... 0x50fbaa50, 0x2e5e7b40
Caught exception in looking up space point ptr... 0x50fa2740, 0x50fbaa50
Caught exception in looking up space point ptr... 0x44e4d860, 0x461b89a0
Caught exception in looking up space point ptr... 0x461b89a0, 0x44e4d860
Caught exception in looking up space point ptr... 0x151d0fc0, 0x151bff30
Caught exception in looking up space point ptr... 0x151bff30, 0x151d0fc0
Caught exception in looking up space point ptr... 0x151d0fc0, 0x151d0da0
Caught exception in looking up space point ptr... 0x151d0da0, 0x151d0fc0
Caught exception in looking up space point ptr... 0x386df8e0, 0x386e9800
Caught exception in looking up space point ptr... 0x386e9800, 0x1a6c0bc0
Caught exception in looking up space point ptr... 0x28f8dfa0, 0x28f8de10
Caught exception in looking up space point ptr... 0x28f8de10, 0x28f8daf0
Caught exception in looking up space point ptr... 0x3b0e6530, 0x3b0e63a0
Caught exception in looking up space point ptr... 0x3b0e63a0, 0x3b0e66c0
Caught exception in looking up space point ptr... 0x385bf1e0, 0x13eeb800
Caught exception in looking up space point ptr... 0x13eeb800, 0x385bf1e0
Caught exception in looking up space point ptr... 0x511d5410, 0x511d4f60
Caught exception in looking up space point ptr... 0x511d4f60, 0x511d5410
Caught exception in looking up space point ptr... 0x4e70e30, 0x4e709c0
Caught exception in looking up space point ptr... 0x4e709c0, 0x4e70e30
Caught exception in looking up space point ptr... 0x4e70e30, 0x4e7d0a0
Caught exception in looking up space point ptr... 0x4e7d0a0, 0x4e70e30
Caught exception in looking up space point ptr... 0x15217bd0, 0x29cdf930
Caught exception in looking up space point ptr... 0x29cdf930, 0xe690b50
Caught exception in looking up space point ptr... 0x38be22d0, 0x3c9e7550
Caught exception in looking up space point ptr... 0x3c9e7550, 0x38be22d0
Caught exception in looking up space point ptr... 0xfe2bf80, 0xfe2bdf0
Caught exception in looking up space point ptr... 0xfe2bdf0, 0xfe2bf80
Caught exception in looking up space point ptr... 0x12e74b50, 0x10af7740
Caught exception in looking up space point ptr... 0x10af7740, 0x3bd66100
Caught exception in looking up space point ptr... 0x151ac1f0, 0x151ac3d0
Caught exception in looking up space point ptr... 0x151ac3d0, 0x151ac1f0
Caught exception in looking up space point ptr... 0x2a363490, 0x1a47d950
Caught exception in looking up space point ptr... 0x1a47d950, 0x586939b0
Caught exception in looking up space point ptr... 0x481a26a0, 0x47363580
Caught exception in looking up space point ptr... 0x47363580, 0xd534810
Caught exception in looking up space point ptr... 0x7ac57e0, 0x42680270
Caught exception in looking up space point ptr... 0x42680270, 0x7ac57e0
Caught exception in looking up space point ptr... 0x34eda960, 0x34ed9260
Caught exception in looking up space point ptr... 0x34ed9260, 0x34ed04d0
++++>>>> total num hits: 63884, num free: 61039
04-Feb-2026 01:26:21 CET  Closed output file "np02vd_raw_run042421_0031_df-s04-d1_dw_0_20260203T134748_reco_stage0_20260204T002621_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         69.7959       86.9716       110.719       86.5627       9.44649        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       4.8061e-05    0.000105269   0.000262254   7.9981e-05    5.64657e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                3.6236        4.65073       11.9958       4.41861       1.47842        27     
produce:triggerrawdecoder:PDVDTriggerReader4      0.0104558     0.0245854     0.0788602     0.0180523     0.0166181       27     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000389556   0.00334195     0.0219962    0.000627291   0.00591028       27     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.00156243    0.00203105    0.00328371    0.00201077    0.000324163      27     
produce:beamevent:BeamEvent                      5.2741e-05    0.000139717   0.00112522    8.6542e-05    0.000196581      27     
produce:pdvddaphne:DAPHNEReaderPDVD                2.05026       3.01878       3.5483        3.14073      0.450202        27     
produce:ophit:OpHitFinder                         0.0309158     0.0402067     0.0495366     0.0415725     0.0049453       27     
produce:wclsdatavd:WireCellToolkit                 54.4287       65.0068       80.4879       64.1222       6.75086        27     
produce:gaushit:GausHitFinder                     0.565379      0.978621       1.87095      0.922323      0.327239        27     
produce:nhitsfilter:NumberOfHitsFilter           0.000152952   0.000235404   0.000419246   0.000225924   6.70729e-05      27     
produce:reco3d:SpacePointSolver                    4.88252       7.83493       12.8558        7.551        1.83328        27     
produce:hitpdune:DisambigFromSpacePoints          0.0508516     0.0888416     0.226868      0.0807022     0.0360898       27     
produce:cluster3d:Cluster3D                       0.874071       1.66938       3.26981       1.70835      0.585885        27     
[art]:TriggerResults:TriggerResultInserter        1.704e-05    3.17964e-05   8.8681e-05     2.778e-05    1.44098e-05      27     
end_path:out1:RootOutput                          3.18e-06      7.11e-06      2.518e-05     4.51e-06     5.58192e-06      27     
end_path:out1:RootOutput(write)                    2.88947       3.63232       4.75173       3.41897      0.554443        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4512.15 MB
  Peak resident set size usage (VmHWM): 2550.46 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042421_0031_df-s04-d1_dw_0_20260203T134748_reco_stage0_20260204T002621_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042421_0031_df-s04-d1_dw_0_20260203T134748_reco_stage0_20260204T002621_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042421_0031_df-s04-d1_dw_0_20260203T134748_reco_stage0_20260204T002621_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770162385.0,
    "core.end_time": 1770164783.0,
    "core.events": [
      5075,
      5081,
      5087,
      5093,
      5099,
      5105,
      5111,
      5117,
      5123,
      5129,
      5135,
      5141,
      5147,
      5153,
      5159,
      5165,
      5171,
      5177,
      5183,
      5189,
      5195,
      5201,
      5207,
      5213,
      5219,
      5225,
      5231
    ],
    "core.event_count": 27,
    "core.first_event_number": 5075,
    "core.last_event_number": 5231,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42421
    ],
    "core.runs_subruns": [
      4242100001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042421_0031_df-s04-d1_dw_0_20260203T134748.hdf5"
    }
  ]
}
justIN time: 2026-02-04 05:58:16 UTC       justIN version: 01.06.00