justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297922.28@dunegpschedd02.fnal.gov

Jobsub ID297922.28@dunegpschedd02.fnal.gov
Workflow ID12717
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 23:00:24
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce507
Last heartbeat2026-02-04 01:00:01
From worker nodeHostnameb9p17p9437.cern.ch
cpuinfoIntel(R) Xeon(R) Silver 4216 CPU @ 2.10GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:57:34
Input filesvd-protodune:np02vd_raw_run042421_0029_df-s04-d0_dw_0_20260203T134642.hdf5
JobscriptExit code0
Real time1h (3612s)
CPU time58m (3517s = 97%)
Max RSS bytes2612383744 (2491 MiB)
Outputting started2026-02-04 00:57:48
Output files
Finished2026-02-04 01:00:01
Saved logsjustin-logs:297922.28-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

space point ptr... 0x2dda9ac0, 0x500bb3d0
Caught exception in looking up space point ptr... 0x5a48ef80, 0x2dda9ac0
Caught exception in looking up space point ptr... 0x2a816ac0, 0x2fdfa7b0
Caught exception in looking up space point ptr... 0x2fdfa7b0, 0x49673540
Caught exception in looking up space point ptr... 0x2d211780, 0x931f4e0
Caught exception in looking up space point ptr... 0x931f4e0, 0x130bb6d0
Caught exception in looking up space point ptr... 0x10353c60, 0x10352d90
Caught exception in looking up space point ptr... 0x10354cb0, 0x10353c60
Caught exception in looking up space point ptr... 0x4cb63790, 0x4cbb0910
Caught exception in looking up space point ptr... 0x4cbb0910, 0xd9fcce0
Caught exception in looking up space point ptr... 0x462b8ce0, 0x256fd290
Caught exception in looking up space point ptr... 0x256fd290, 0x462b8ce0
Caught exception in looking up space point ptr... 0x3d71c040, 0x3d71bda0
Caught exception in looking up space point ptr... 0x3d71bda0, 0x3d71c040
Caught exception in looking up space point ptr... 0x2c014880, 0x20b82330
Caught exception in looking up space point ptr... 0x20b82330, 0x20b82430
Caught exception in looking up space point ptr... 0x4cc024c0, 0x679b0bd0
Caught exception in looking up space point ptr... 0x679b0bd0, 0x4cc024c0
Caught exception in looking up space point ptr... 0x29a4fcb0, 0x29a50960
Caught exception in looking up space point ptr... 0x29a50960, 0x29a4fcb0
Caught exception in looking up space point ptr... 0x4cb730e0, 0x4cb75470
Caught exception in looking up space point ptr... 0x4cb75220, 0x4cb730e0
Caught exception in looking up space point ptr... 0x40064f70, 0x400630b0
Caught exception in looking up space point ptr... 0x400630b0, 0x48823df0
Caught exception in looking up space point ptr... 0x462bdbb0, 0x462bd200
Caught exception in looking up space point ptr... 0x462bd200, 0x462bdbb0
Caught exception in looking up space point ptr... 0x2be9cf40, 0x34baa6d0
Caught exception in looking up space point ptr... 0x34baa6d0, 0x2be9cf40
Caught exception in looking up space point ptr... 0xaf6b830, 0xaf6b730
Caught exception in looking up space point ptr... 0xaf6b730, 0x496bc280
Caught exception in looking up space point ptr... 0x30c35f10, 0x4956d840
Caught exception in looking up space point ptr... 0x4956d840, 0x245fd790
Caught exception in looking up space point ptr... 0x59da2c90, 0x45d0ca30
Caught exception in looking up space point ptr... 0x45d0ca30, 0x45d0cad0
Caught exception in looking up space point ptr... 0x5a3fdad0, 0x2d2282c0
Caught exception in looking up space point ptr... 0x2d2282c0, 0xe6332b0
Caught exception in looking up space point ptr... 0x2e347680, 0x2e347820
Caught exception in looking up space point ptr... 0x8711cb0, 0x2e347680
Caught exception in looking up space point ptr... 0x455ff1f0, 0x4560c990
Caught exception in looking up space point ptr... 0x455ff8e0, 0x455ff1f0
Caught exception in looking up space point ptr... 0x935a5a0, 0x935a6d0
Caught exception in looking up space point ptr... 0x935a6d0, 0x935a5a0
Caught exception in looking up space point ptr... 0x10351190, 0x1034c590
Caught exception in looking up space point ptr... 0x1034c590, 0x1034c630
Caught exception in looking up space point ptr... 0x24601700, 0x24601830
Caught exception in looking up space point ptr... 0x24601830, 0x24601700
Caught exception in looking up space point ptr... 0xaa39120, 0x209cd070
Caught exception in looking up space point ptr... 0x209cd070, 0x4e3b6960
Caught exception in looking up space point ptr... 0xe63a7f0, 0xe63a430
Caught exception in looking up space point ptr... 0xe630960, 0xe63a7f0
Caught exception in looking up space point ptr... 0x4e1ef720, 0x28bc39d0
Caught exception in looking up space point ptr... 0x28bc39d0, 0x4e1ef720
Caught exception in looking up space point ptr... 0xe6d8190, 0x578d36d0
Caught exception in looking up space point ptr... 0x578d36d0, 0x3e59d9c0
Caught exception in looking up space point ptr... 0x4208f660, 0x4208aba0
Caught exception in looking up space point ptr... 0x4208aba0, 0x4208f660
Caught exception in looking up space point ptr... 0x3006ad60, 0x3006ab10
Caught exception in looking up space point ptr... 0x3006ab10, 0x3006ad60
Caught exception in looking up space point ptr... 0x23a41510, 0x23a40b10
Caught exception in looking up space point ptr... 0x23a40b10, 0x23a41510
Caught exception in looking up space point ptr... 0x15c139a0, 0x15c0f5a0
Caught exception in looking up space point ptr... 0x15c0f5a0, 0x15c139a0
Caught exception in looking up space point ptr... 0x445b5c90, 0x367e01a0
Caught exception in looking up space point ptr... 0x367e01a0, 0x446abb50
Caught exception in looking up space point ptr... 0x30106290, 0x4e8d5a00
Caught exception in looking up space point ptr... 0x4e8d5a00, 0x30106290
++++>>>> total num hits: 53914, num free: 51709
04-Feb-2026 01:57:39 CET  Closed output file "np02vd_raw_run042421_0029_df-s04-d0_dw_0_20260203T134642_reco_stage0_20260204T005739_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         118.719       129.905       168.123       128.535       9.48656        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       6.6932e-05    0.000105704   0.00028404    9.4446e-05    4.55996e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                5.17137       6.43084       7.65056       6.56463      0.608278        27     
produce:triggerrawdecoder:PDVDTriggerReader4      0.0225276     0.0831432     0.201785      0.0706552     0.0446134       27     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.00052548    0.00583026     0.040076     0.000868518   0.00985244       27     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.00150146    0.00222319    0.00477143    0.00201528    0.000679243      27     
produce:beamevent:BeamEvent                      7.5047e-05    0.000173147   0.00144057    0.000117778   0.000250151      27     
produce:pdvddaphne:DAPHNEReaderPDVD                5.38939       6.2126        7.99145       5.88932      0.709315        27     
produce:ophit:OpHitFinder                         0.0359028     0.0540898     0.0813674     0.0478148     0.0147741       27     
produce:wclsdatavd:WireCellToolkit                 90.4189       96.2613       108.826       95.6579       4.52602        27     
produce:gaushit:GausHitFinder                     0.804031       1.38736       1.99586       1.38155      0.274602        27     
produce:nhitsfilter:NumberOfHitsFilter           0.000306809   0.000434225   0.000791422   0.000394443   0.000115467      27     
produce:reco3d:SpacePointSolver                    5.93449       11.1425       15.1755       11.3199       2.44546        27     
produce:hitpdune:DisambigFromSpacePoints          0.0982529     0.149643      0.304943      0.140918      0.0451124       27     
produce:cluster3d:Cluster3D                        1.01266       3.30968       36.2634       1.84234       6.50299        27     
[art]:TriggerResults:TriggerResultInserter       2.8043e-05    3.79816e-05   0.000104861   3.3886e-05    1.43011e-05      27     
end_path:out1:RootOutput                          4.701e-06    8.74215e-06   2.9267e-05     6.946e-06    5.46036e-06      27     
end_path:out1:RootOutput(write)                    4.21871       4.83855       7.53773       4.4943       0.832045        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4572.2 MB
  Peak resident set size usage (VmHWM): 2612.38 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042421_0029_df-s04-d0_dw_0_20260203T134642_reco_stage0_20260204T005739_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042421_0029_df-s04-d0_dw_0_20260203T134642_reco_stage0_20260204T005739_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042421_0029_df-s04-d0_dw_0_20260203T134642_reco_stage0_20260204T005739_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770163069.0,
    "core.end_time": 1770166661.0,
    "core.events": [
      4744,
      4750,
      4756,
      4762,
      4768,
      4774,
      4780,
      4786,
      4792,
      4798,
      4804,
      4810,
      4816,
      4822,
      4828,
      4834,
      4840,
      4846,
      4852,
      4858,
      4864,
      4870,
      4876,
      4882,
      4888,
      4894,
      4900
    ],
    "core.event_count": 27,
    "core.first_event_number": 4744,
    "core.last_event_number": 4900,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42421
    ],
    "core.runs_subruns": [
      4242100001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042421_0029_df-s04-d0_dw_0_20260203T134642.hdf5"
    }
  ]
}
justIN time: 2026-02-04 08:55:56 UTC       justIN version: 01.06.00