justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303654.55@dunegpschedd01.fnal.gov

Jobsub ID303654.55@dunegpschedd01.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:52:26
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce510
Last heartbeat2026-02-04 00:52:40
From worker nodeHostnameb9p34p1010.cern.ch
cpuinfoAMD EPYC 9655 96-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:57:16
Input filesvd-protodune:np02vd_raw_run042433_0587_df-s02-d1_dw_0_20260203T222513.hdf5
JobscriptExit code0
Real time52m (3179s)
CPU time51m (3069s = 96%)
Max RSS bytes7445528576 (7100 MiB)
Outputting started2026-02-04 00:50:17
Output files
Finished2026-02-04 00:52:40
Saved logsjustin-logs:303654.55-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

nt ptr... 0xca9225b0, 0xca926450
Caught exception in looking up space point ptr... 0xc26f9df0, 0xc26fa160
Caught exception in looking up space point ptr... 0xc26fa160, 0xc26f9df0
Caught exception in looking up space point ptr... 0xa41fc430, 0x10c048c0
Caught exception in looking up space point ptr... 0x10c048c0, 0x37a38ec0
Caught exception in looking up space point ptr... 0x39fed5c0, 0x39fedc50
Caught exception in looking up space point ptr... 0x39fedc50, 0x39fed5c0
Caught exception in looking up space point ptr... 0x7f3f5dc0, 0x7f3f7e60
Caught exception in looking up space point ptr... 0x7f3f7e60, 0x7f3f5dc0
Caught exception in looking up space point ptr... 0xa459d900, 0x8f6f430
Caught exception in looking up space point ptr... 0x8f6f430, 0x8f6f560
Caught exception in looking up space point ptr... 0xb6634350, 0x837045e0
Caught exception in looking up space point ptr... 0x837045e0, 0xc5a9aed0
Caught exception in looking up space point ptr... 0x14abc4f0, 0x7dcc5dc0
Caught exception in looking up space point ptr... 0x7dcc5dc0, 0x43c920f0
Caught exception in looking up space point ptr... 0xc5abfb00, 0xc5abf7f0
Caught exception in looking up space point ptr... 0xc5ac04e0, 0xc5abfb00
Caught exception in looking up space point ptr... 0x37acf510, 0x39ec0660
Caught exception in looking up space point ptr... 0x96e59d60, 0x37acf510
Caught exception in looking up space point ptr... 0x39ebeed0, 0x39ebf1d0
Caught exception in looking up space point ptr... 0x39ebf1d0, 0x39ebeed0
Caught exception in looking up space point ptr... 0x3b365400, 0x94dc8750
Caught exception in looking up space point ptr... 0x94dc8750, 0x3b362b20
Caught exception in looking up space point ptr... 0x376dc7d0, 0x3bb8c580
Caught exception in looking up space point ptr... 0x3bb8c580, 0x376dc7d0
Caught exception in looking up space point ptr... 0x53b288d0, 0xa7f15960
Caught exception in looking up space point ptr... 0xa7f15960, 0x10d43030
Caught exception in looking up space point ptr... 0x11453a80, 0x114538f0
Caught exception in looking up space point ptr... 0x1144e840, 0x11453a80
Caught exception in looking up space point ptr... 0x85383060, 0x85382d40
Caught exception in looking up space point ptr... 0x85382d40, 0x19b5b0d0
Caught exception in looking up space point ptr... 0x7f3f7630, 0x7f3f7460
Caught exception in looking up space point ptr... 0x7f3f7460, 0x7f3f7630
Caught exception in looking up space point ptr... 0x83392380, 0xa49d4b60
Caught exception in looking up space point ptr... 0xa49d4b60, 0x83392380
Caught exception in looking up space point ptr... 0x83392380, 0x37bed570
Caught exception in looking up space point ptr... 0x37bed570, 0x83392380
Caught exception in looking up space point ptr... 0x84a551b0, 0x8f43e40
Caught exception in looking up space point ptr... 0x8f43e40, 0x84a556f0
Caught exception in looking up space point ptr... 0x11446d70, 0x11446c20
Caught exception in looking up space point ptr... 0x11446c20, 0x11446290
Caught exception in looking up space point ptr... 0x1a490860, 0x9444dbb0
Caught exception in looking up space point ptr... 0x9444dbb0, 0x1a493cf0
Caught exception in looking up space point ptr... 0x11453760, 0x11453220
Caught exception in looking up space point ptr... 0x11453220, 0x11453760
Caught exception in looking up space point ptr... 0x11453760, 0x11453610
Caught exception in looking up space point ptr... 0x11453610, 0x11453760
Caught exception in looking up space point ptr... 0x8c12c9a0, 0x8f460b0
Caught exception in looking up space point ptr... 0x8f460b0, 0x7e952840
Caught exception in looking up space point ptr... 0x4c7fa8c0, 0x4c7fab60
Caught exception in looking up space point ptr... 0x4c7fab60, 0x4c7faa10
Caught exception in looking up space point ptr... 0x2247aff0, 0x2247b460
Caught exception in looking up space point ptr... 0x2247b460, 0x2247aff0
Caught exception in looking up space point ptr... 0x12a8ecd0, 0x12a8eba0
Caught exception in looking up space point ptr... 0x12a8eba0, 0x12a8ecd0
Caught exception in looking up space point ptr... 0x12a8ecd0, 0x12a8ea70
Caught exception in looking up space point ptr... 0x12a8ea70, 0x12a8ecd0
Caught exception in looking up space point ptr... 0x8f8e610, 0x872c91d0
Caught exception in looking up space point ptr... 0x872c91d0, 0x7d0d9a00
Caught exception in looking up space point ptr... 0xad193f70, 0xad1943b0
Caught exception in looking up space point ptr... 0xad1943b0, 0xad193f70
Caught exception in looking up space point ptr... 0x38b0ab40, 0x38b0ae60
Caught exception in looking up space point ptr... 0x38b0ae60, 0x38b0ab40
Caught exception in looking up space point ptr... 0x1a866d70, 0x4101c310
Caught exception in looking up space point ptr... 0x1a754990, 0x1a866d70
Caught exception in looking up space point ptr... 0x9df00db0, 0x7dcc2920
Caught exception in looking up space point ptr... 0x7dcc2920, 0x9df00db0
++++>>>> total num hits: 75137, num free: 72091
04-Feb-2026 01:50:06 CET  Closed output file "np02vd_raw_run042433_0587_df-s02-d1_dw_0_20260203T222513_reco_stage0_20260204T005006_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         96.384        163.884       580.108       118.578       136.504        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       6.9344e-05    9.76445e-05   0.000194801   8.5538e-05    2.6595e-05       19     
produce:tpcrawdecoder:PDVDTPCReader                5.40647       6.87864       11.089        6.5412        1.19716        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.0102965     0.0263484     0.0606294     0.0205205     0.0149477       19     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000377612   0.00360773     0.0543232    0.000680143    0.0119655       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.000354698   0.000573109   0.000810608   0.000539825   0.000128336      19     
produce:beamevent:BeamEvent                      8.1301e-05    0.000153758   0.000891047   0.000114801   0.000174537      19     
produce:pdvddaphne:DAPHNEReaderPDVD                1.88311       2.1968        2.53624       2.19359       0.16686        19     
produce:ophit:OpHitFinder                         0.0263281     0.0350323     0.0425075     0.0347517    0.00486416       19     
produce:wclsdatavd:WireCellToolkit                 74.8879       84.5157       94.6761       83.3744       6.45445        19     
produce:gaushit:GausHitFinder                     0.519352       1.56195       5.66797       1.18116       1.18599        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000196493   0.000342169   0.00102495    0.00029585    0.000178575      19     
produce:reco3d:SpacePointSolver                    7.99277       19.9835       54.4168       17.1672       11.4624        19     
produce:hitpdune:DisambigFromSpacePoints          0.0686618     0.490794       3.96701      0.182357      0.965476        19     
produce:cluster3d:Cluster3D                       0.710683       42.4686       405.168       2.04771       117.957        19     
[art]:TriggerResults:TriggerResultInserter       2.7351e-05    4.69602e-05   0.000134481   4.0821e-05    2.39571e-05      19     
end_path:out1:RootOutput                          8.843e-06    1.14535e-05   2.5929e-05    1.0305e-05    4.14728e-06      19     
end_path:out1:RootOutput(write)                    3.49732       5.66723       14.4015       4.97057       2.74719        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 9544.29 MB
  Peak resident set size usage (VmHWM): 7445.53 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0587_df-s02-d1_dw_0_20260203T222513_reco_stage0_20260204T005006_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0587_df-s02-d1_dw_0_20260203T222513_reco_stage0_20260204T005006_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0587_df-s02-d1_dw_0_20260203T222513_reco_stage0_20260204T005006_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770163045.0,
    "core.end_time": 1770166210.0,
    "core.events": [
      66925,
      66931,
      66937,
      66943,
      66949,
      66955,
      66961,
      66967,
      66973,
      66979,
      66985,
      66991,
      66997,
      67003,
      67009,
      67015,
      67021,
      67027,
      67033
    ],
    "core.event_count": 19,
    "core.first_event_number": 66925,
    "core.last_event_number": 67033,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0587_df-s02-d1_dw_0_20260203T222513.hdf5"
    }
  ]
}
justIN time: 2026-02-04 05:56:27 UTC       justIN version: 01.06.00