justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297916.37@dunegpschedd02.fnal.gov

Jobsub ID297916.37@dunegpschedd02.fnal.gov
Workflow ID12718
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:46:26
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce505
Last heartbeat2026-02-03 23:20:45
From worker nodeHostnameb9p35p0031.cern.ch
cpuinfoAMD EPYC 9655P 96-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 22:47:14
Input filesvd-protodune:np02vd_raw_run042422_0126_df-s03-d0_dw_0_20260203T162255.hdf5
JobscriptExit code0
Real time31m (1884s)
CPU time30m (1805s = 95%)
Max RSS bytes2563735552 (2444 MiB)
Outputting started2026-02-03 23:18:40
Output files
Finished2026-02-03 23:20:45
Saved logsjustin-logs:297916.37-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

803be90
Caught exception in looking up space point ptr... 0x3ec49050, 0x25bdec90
Caught exception in looking up space point ptr... 0x25bdec90, 0x219b43b0
Caught exception in looking up space point ptr... 0x5cd91530, 0x5cd916f0
Caught exception in looking up space point ptr... 0x5cd89b00, 0x5cd91530
Caught exception in looking up space point ptr... 0x15bf4830, 0xe157e20
Caught exception in looking up space point ptr... 0xe157e20, 0x3ec4b110
Caught exception in looking up space point ptr... 0x4ba875c0, 0x4ba87850
Caught exception in looking up space point ptr... 0x54353980, 0x4ba875c0
Caught exception in looking up space point ptr... 0x4f7adbf0, 0x53e54380
Caught exception in looking up space point ptr... 0x53e54380, 0x53e09470
Caught exception in looking up space point ptr... 0x54456ec0, 0x549ce130
Caught exception in looking up space point ptr... 0x549ce130, 0x549cdea0
Caught exception in looking up space point ptr... 0x26faf620, 0x26facff0
Caught exception in looking up space point ptr... 0x26fae7f0, 0x26faf620
Caught exception in looking up space point ptr... 0x1533d780, 0x1533dc30
Caught exception in looking up space point ptr... 0x1533daa0, 0x1533d780
Caught exception in looking up space point ptr... 0x4b8c0820, 0x24ca7b00
Caught exception in looking up space point ptr... 0x24ca7b00, 0x4b8c0ab0
Caught exception in looking up space point ptr... 0x40fed910, 0x40fedaa0
Caught exception in looking up space point ptr... 0x40fedaa0, 0x40fed910
Caught exception in looking up space point ptr... 0x2cbd0360, 0x2cbd01d0
Caught exception in looking up space point ptr... 0x2cbd01d0, 0x2cbd0360
Caught exception in looking up space point ptr... 0x27f80500, 0x33a80850
Caught exception in looking up space point ptr... 0x33a80850, 0x5a384c70
Caught exception in looking up space point ptr... 0xa59b200, 0x27f80500
Caught exception in looking up space point ptr... 0xba7bf60, 0xba7c090
Caught exception in looking up space point ptr... 0xba7c090, 0xba7c420
Caught exception in looking up space point ptr... 0x8d17cf0, 0x8d22de0
Caught exception in looking up space point ptr... 0x8d22de0, 0x8d17cf0
Caught exception in looking up space point ptr... 0x45a50590, 0x24e9d650
Caught exception in looking up space point ptr... 0x24e9d650, 0x24e9d550
Caught exception in looking up space point ptr... 0x3624a350, 0x1694a800
Caught exception in looking up space point ptr... 0x1694a800, 0x3624a350
Caught exception in looking up space point ptr... 0xe797450, 0xe7971f0
Caught exception in looking up space point ptr... 0xe7971f0, 0x4b9dc1a0
Caught exception in looking up space point ptr... 0x508cf200, 0x508cf4f0
Caught exception in looking up space point ptr... 0x508cf4f0, 0x508cf060
Caught exception in looking up space point ptr... 0x141e2000, 0x4c176880
Caught exception in looking up space point ptr... 0x4c176880, 0x141e2000
Caught exception in looking up space point ptr... 0x3b55dce0, 0x27e8d470
Caught exception in looking up space point ptr... 0x27e8d470, 0x3b55dce0
Caught exception in looking up space point ptr... 0x36ddc020, 0x36ddf430
Caught exception in looking up space point ptr... 0x36ddf430, 0x36ddf320
Caught exception in looking up space point ptr... 0x13106860, 0x13106b80
Caught exception in looking up space point ptr... 0x13106b80, 0x13106860
Caught exception in looking up space point ptr... 0x28f494f0, 0x28f493c0
Caught exception in looking up space point ptr... 0x28f493c0, 0x28f494f0
Caught exception in looking up space point ptr... 0x101c62f0, 0x101c60d0
Caught exception in looking up space point ptr... 0x101c60d0, 0x101d2d60
Caught exception in looking up space point ptr... 0x4bb98ff0, 0x4349d340
Caught exception in looking up space point ptr... 0x4349d340, 0x4bb98ff0
Caught exception in looking up space point ptr... 0x42cb02f0, 0x42caf2c0
Caught exception in looking up space point ptr... 0x42caf2c0, 0x42cb02f0
Caught exception in looking up space point ptr... 0x141d2310, 0x141cf070
Caught exception in looking up space point ptr... 0x141cf070, 0x141d2310
Caught exception in looking up space point ptr... 0x2f56a350, 0x250dc250
Caught exception in looking up space point ptr... 0x250dc250, 0x3c7aa530
Caught exception in looking up space point ptr... 0x550590b0, 0x2a7f53e0
Caught exception in looking up space point ptr... 0x2a7f53e0, 0x550590b0
Caught exception in looking up space point ptr... 0x4f793eb0, 0x3a9f3f00
Caught exception in looking up space point ptr... 0x3a9f3f00, 0x4f793eb0
Caught exception in looking up space point ptr... 0x42caf980, 0x42caf8a0
Caught exception in looking up space point ptr... 0x42caf8a0, 0x42caf980
Caught exception in looking up space point ptr... 0x32881090, 0x32881e00
Caught exception in looking up space point ptr... 0x32881e00, 0x32881090
++++>>>> total num hits: 82985, num free: 80181
04-Feb-2026 00:18:34 CET  Closed output file "np02vd_raw_run042422_0126_df-s03-d0_dw_0_20260203T162255_reco_stage0_20260203T231834_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         60.1704       67.7798       76.536        67.1891       4.66975        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                        5.26e-05     7.0296e-05    0.00014961     6.94e-05     1.77586e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                3.88368       4.24757       4.60949       4.2411       0.186278        27     
produce:triggerrawdecoder:PDVDTriggerReader4     0.00764975     0.0134232     0.0563997    0.00985012    0.00959182       27     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0003436    0.00049164    0.00072471    0.00047702    0.000110093      27     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.00109745    0.00139798    0.00190102    0.00138805    0.000211715      27     
produce:beamevent:BeamEvent                       6.068e-05    0.00010945    0.00072513     8.075e-05    0.000121995      27     
produce:pdvddaphne:DAPHNEReaderPDVD                1.65547       2.11733       2.39007       2.15201      0.149955        27     
produce:ophit:OpHitFinder                         0.0230646     0.0343578     0.0405812     0.0339555    0.00436475       27     
produce:wclsdatavd:WireCellToolkit                 43.9039       48.6084       57.0253       48.5891       3.12373        27     
produce:gaushit:GausHitFinder                     0.383189      0.710158       1.18465       0.65447      0.187417        27     
produce:nhitsfilter:NumberOfHitsFilter           0.00011723    0.000184457   0.00069446    0.00014838    0.000107855      27     
produce:reco3d:SpacePointSolver                    4.61726       7.23509       11.1327       6.93262       1.88167        27     
produce:hitpdune:DisambigFromSpacePoints          0.042062      0.0718882     0.131122      0.0682389     0.0195307       27     
produce:cluster3d:Cluster3D                       0.506927       1.28844       2.35312       1.24617       0.4233         27     
[art]:TriggerResults:TriggerResultInserter        2.313e-05    3.17119e-05   8.7601e-05     2.836e-05    1.24186e-05      27     
end_path:out1:RootOutput                          4.51e-06     5.93889e-06    2.083e-05     5.22e-06     3.01654e-06      27     
end_path:out1:RootOutput(write)                    2.76898       3.42594       4.40668       3.38658      0.377726        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4599.19 MB
  Peak resident set size usage (VmHWM): 2563.74 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042422_0126_df-s03-d0_dw_0_20260203T162255_reco_stage0_20260203T231834_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042422_0126_df-s03-d0_dw_0_20260203T162255_reco_stage0_20260203T231834_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042422_0126_df-s03-d0_dw_0_20260203T162255_reco_stage0_20260203T231834_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770158842.0,
    "core.end_time": 1770160715.0,
    "core.events": [
      20432,
      20438,
      20444,
      20450,
      20456,
      20462,
      20468,
      20474,
      20480,
      20486,
      20492,
      20498,
      20504,
      20510,
      20516,
      20522,
      20528,
      20534,
      20540,
      20546,
      20552,
      20558,
      20564,
      20570,
      20576,
      20582,
      20588
    ],
    "core.event_count": 27,
    "core.first_event_number": 20432,
    "core.last_event_number": 20588,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42422
    ],
    "core.runs_subruns": [
      4242200001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042422_0126_df-s03-d0_dw_0_20260203T162255.hdf5"
    }
  ]
}
justIN time: 2026-02-04 07:37:50 UTC       justIN version: 01.06.00