justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303654.146@dunegpschedd01.fnal.gov

Jobsub ID303654.146@dunegpschedd01.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:52:26
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce503
Last heartbeat2026-02-04 00:54:55
From worker nodeHostnameb9p02p6824.cern.ch
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-04 00:09:46
Input filesvd-protodune:np02vd_raw_run042433_0558_df-s03-d1_dw_0_20260203T221411.hdf5
JobscriptExit code0
Real time42m (2564s)
CPU time41m (2468s = 96%)
Max RSS bytes3376984064 (3220 MiB)
Outputting started2026-02-04 00:52:31
Output files
Finished2026-02-04 00:54:55
Saved logsjustin-logs:303654.146-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ace point ptr... 0x31b178d0, 0x334b1fa0
Caught exception in looking up space point ptr... 0xce5f8a0, 0xce5bd80
Caught exception in looking up space point ptr... 0xce5f770, 0xce5f8a0
Caught exception in looking up space point ptr... 0x2d9ea330, 0x2d9eee40
Caught exception in looking up space point ptr... 0x2d9eee40, 0x2d9ea330
Caught exception in looking up space point ptr... 0x24c46790, 0x24c46890
Caught exception in looking up space point ptr... 0x24c46890, 0x24c46790
Caught exception in looking up space point ptr... 0x18f4d5b0, 0x43041180
Caught exception in looking up space point ptr... 0x43041180, 0x6e400990
Caught exception in looking up space point ptr... 0x22c36f40, 0x22c370e0
Caught exception in looking up space point ptr... 0x22c370e0, 0x22c36f40
Caught exception in looking up space point ptr... 0x3920fc80, 0x39229770
Caught exception in looking up space point ptr... 0x39229770, 0x13f83290
Caught exception in looking up space point ptr... 0x14c5c280, 0x42f259b0
Caught exception in looking up space point ptr... 0x42f259b0, 0x68daa050
Caught exception in looking up space point ptr... 0x65a83510, 0xcf3c8d0
Caught exception in looking up space point ptr... 0xcf3c8d0, 0xcf3c7d0
Caught exception in looking up space point ptr... 0x446eb4b0, 0x6771e6a0
Caught exception in looking up space point ptr... 0x6e250560, 0x446eb4b0
Caught exception in looking up space point ptr... 0x4a8fccc0, 0x44841340
Caught exception in looking up space point ptr... 0x44841340, 0x44841240
Caught exception in looking up space point ptr... 0x4344aa10, 0x3e5b69a0
Caught exception in looking up space point ptr... 0x3e5b69a0, 0xa61e280
Caught exception in looking up space point ptr... 0x131058f0, 0x13101950
Caught exception in looking up space point ptr... 0x13101950, 0x131058f0
Caught exception in looking up space point ptr... 0x273baa30, 0x273b74f0
Caught exception in looking up space point ptr... 0x273b74f0, 0x273baa30
Caught exception in looking up space point ptr... 0x366124a0, 0x366121e0
Caught exception in looking up space point ptr... 0x366121e0, 0x366124a0
Caught exception in looking up space point ptr... 0x366124a0, 0x36612040
Caught exception in looking up space point ptr... 0x36612040, 0x366124a0
Caught exception in looking up space point ptr... 0x552770c0, 0x552733f0
Caught exception in looking up space point ptr... 0x552733f0, 0x552770c0
Caught exception in looking up space point ptr... 0x58b57380, 0x58b57030
Caught exception in looking up space point ptr... 0x58b57030, 0x2db90f20
Caught exception in looking up space point ptr... 0x1489a5a0, 0x208946d0
Caught exception in looking up space point ptr... 0x208946d0, 0x1489a5a0
Caught exception in looking up space point ptr... 0x54dea500, 0x27b9bdb0
Caught exception in looking up space point ptr... 0x27b9bdb0, 0x54dea500
Caught exception in looking up space point ptr... 0x95eb2f0, 0x95eec80
Caught exception in looking up space point ptr... 0x95eec80, 0x95eb2f0
Caught exception in looking up space point ptr... 0x42848250, 0x42847f30
Caught exception in looking up space point ptr... 0x42847f30, 0x42848250
Caught exception in looking up space point ptr... 0x420d9490, 0x10f23030
Caught exception in looking up space point ptr... 0x10f23030, 0x420d9490
Caught exception in looking up space point ptr... 0x35faf250, 0x35faf570
Caught exception in looking up space point ptr... 0x35faf570, 0x35faf250
Caught exception in looking up space point ptr... 0x659bbfd0, 0x659c03a0
Caught exception in looking up space point ptr... 0x659c03a0, 0x659bbfd0
Caught exception in looking up space point ptr... 0x270c8020, 0x6e7cb5f0
Caught exception in looking up space point ptr... 0x6e7cb5f0, 0x694e0620
Caught exception in looking up space point ptr... 0x3edb6960, 0x3edbcf30
Caught exception in looking up space point ptr... 0x3edbcf30, 0x3edbd0a0
Caught exception in looking up space point ptr... 0x13efcdc0, 0x6e79e080
Caught exception in looking up space point ptr... 0x6e7be1b0, 0x13efcdc0
Caught exception in looking up space point ptr... 0x3edd15e0, 0x173706b0
Caught exception in looking up space point ptr... 0x173706b0, 0x3edd15e0
Caught exception in looking up space point ptr... 0x3614d800, 0x36149f70
Caught exception in looking up space point ptr... 0x36149f70, 0x3614d800
Caught exception in looking up space point ptr... 0x273ce5f0, 0x22c34380
Caught exception in looking up space point ptr... 0x22c34380, 0x273ce5f0
Caught exception in looking up space point ptr... 0x3609e740, 0x3609e4a0
Caught exception in looking up space point ptr... 0x3609e4a0, 0x36095af0
Caught exception in looking up space point ptr... 0x13f889a0, 0x13f88260
Caught exception in looking up space point ptr... 0x13f88260, 0xd0a41a0
Caught exception in looking up space point ptr... 0x9ecb0b0, 0x13ef2eb0
Caught exception in looking up space point ptr... 0x13ef2eb0, 0x9ecb0b0
++++>>>> total num hits: 83919, num free: 80445
04-Feb-2026 01:52:21 CET  Closed output file "np02vd_raw_run042433_0558_df-s03-d1_dw_0_20260203T221411_reco_stage0_20260204T005221_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         116.02        132.06        150.746       129.872       9.5223         19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                        4.877e-05    0.000120407   0.000400533   8.6541e-05    8.98091e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                5.82724       6.82183       7.48111       6.80695      0.377735        19     
produce:triggerrawdecoder:PDVDTriggerReader4     0.00966186     0.0147158     0.0259849     0.0123678     0.0046973       19     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000375123   0.000508195   0.000641795   0.000496634   7.2294e-05       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.000371273   0.000461284   0.000582525   0.000453744   5.4653e-05       19     
produce:beamevent:BeamEvent                       6.194e-05    0.000133089   0.000676836   0.000112131   0.000131822      19     
produce:pdvddaphne:DAPHNEReaderPDVD                2.14748       2.81397       3.42886       2.86325      0.397294        19     
produce:ophit:OpHitFinder                         0.0269182     0.0359425     0.0418796     0.035854     0.00431782       19     
produce:wclsdatavd:WireCellToolkit                 85.1162       100.197       115.214       99.9003       8.21626        19     
produce:gaushit:GausHitFinder                     0.762661       1.18957       1.88687       1.1311       0.327605        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000208412   0.000297591   0.000493704   0.000276182   7.27576e-05      19     
produce:reco3d:SpacePointSolver                    8.53247       13.8999       19.9957       13.517        2.88844        19     
produce:hitpdune:DisambigFromSpacePoints          0.0860243     0.151261      0.269469      0.148372      0.0400548       19     
produce:cluster3d:Cluster3D                        1.27337       2.13309       3.55312       1.95019      0.578394        19     
[art]:TriggerResults:TriggerResultInserter        2.568e-05    3.72149e-05   0.000126331    3.156e-05    2.14897e-05      19     
end_path:out1:RootOutput                          3.89e-06     8.49063e-06   3.1181e-05     6.97e-06     5.9231e-06       19     
end_path:out1:RootOutput(write)                     3.835        4.77169       5.96517       4.55626      0.703642        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5422.68 MB
  Peak resident set size usage (VmHWM): 3376.98 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0558_df-s03-d1_dw_0_20260203T221411_reco_stage0_20260204T005221_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0558_df-s03-d1_dw_0_20260203T221411_reco_stage0_20260204T005221_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0558_df-s03-d1_dw_0_20260203T221411_reco_stage0_20260204T005221_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770163795.0,
    "core.end_time": 1770166342.0,
    "core.events": [
      63621,
      63627,
      63633,
      63639,
      63645,
      63651,
      63657,
      63663,
      63669,
      63675,
      63681,
      63687,
      63693,
      63699,
      63705,
      63711,
      63717,
      63723,
      63729
    ],
    "core.event_count": 19,
    "core.first_event_number": 63621,
    "core.last_event_number": 63729,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0558_df-s03-d1_dw_0_20260203T221411.hdf5"
    }
  ]
}
justIN time: 2026-02-04 07:37:43 UTC       justIN version: 01.06.00