justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303654.33@dunegpschedd01.fnal.gov

Jobsub ID303654.33@dunegpschedd01.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:52:26
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce508
Last heartbeat2026-02-04 00:25:40
From worker nodeHostnameb9p35p0022.cern.ch
cpuinfoAMD EPYC 9655P 96-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:51:01
Input filesvd-protodune:np02vd_raw_run042433_0154_df-s04-d0_dw_0_20260203T194039.hdf5
JobscriptExit code0
Real time32m (1958s)
CPU time31m (1884s = 96%)
Max RSS bytes3385532416 (3228 MiB)
Outputting started2026-02-04 00:23:40
Output files
Finished2026-02-04 00:25:40
Saved logsjustin-logs:303654.33-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ptr... 0x38eabf20, 0x4c591bd0
Caught exception in looking up space point ptr... 0x4c591bd0, 0x15a7dc10
Caught exception in looking up space point ptr... 0x58c94200, 0x58c94520
Caught exception in looking up space point ptr... 0x58c94520, 0x58c94200
Caught exception in looking up space point ptr... 0x3edd99c0, 0x3eddac30
Caught exception in looking up space point ptr... 0x4c52a3f0, 0x3edd99c0
Caught exception in looking up space point ptr... 0x3509a630, 0x3509e4b0
Caught exception in looking up space point ptr... 0x3509e4b0, 0x3509e260
Caught exception in looking up space point ptr... 0x195bac30, 0x6327adc0
Caught exception in looking up space point ptr... 0x6327adc0, 0x54ff1ca0
Caught exception in looking up space point ptr... 0x36791c60, 0x2bad1a00
Caught exception in looking up space point ptr... 0x2bad1a00, 0x36385580
Caught exception in looking up space point ptr... 0x17baeba0, 0x17baeaa0
Caught exception in looking up space point ptr... 0x17baeaa0, 0x3880a870
Caught exception in looking up space point ptr... 0x2b2dd3e0, 0x2b2dd070
Caught exception in looking up space point ptr... 0x2b2dd070, 0x2b2dd3e0
Caught exception in looking up space point ptr... 0x3ea985f0, 0x42a4eff0
Caught exception in looking up space point ptr... 0x42a4eff0, 0x379fd940
Caught exception in looking up space point ptr... 0x3f7c8310, 0x3b006180
Caught exception in looking up space point ptr... 0x3b006180, 0x38885e10
Caught exception in looking up space point ptr... 0x3a0cdc30, 0x43959b30
Caught exception in looking up space point ptr... 0x43959b30, 0x3a0cdc30
Caught exception in looking up space point ptr... 0x6778ca30, 0x29806690
Caught exception in looking up space point ptr... 0x29806690, 0x6778ca30
Caught exception in looking up space point ptr... 0x3ac014f0, 0x13f6efd0
Caught exception in looking up space point ptr... 0x13f6efd0, 0x3ac014f0
Caught exception in looking up space point ptr... 0x35131940, 0x58c96500
Caught exception in looking up space point ptr... 0x58c96500, 0x35131940
Caught exception in looking up space point ptr... 0x78b0b4d0, 0x78b0b1e0
Caught exception in looking up space point ptr... 0x78b0b1e0, 0x78b0b040
Caught exception in looking up space point ptr... 0x19378cc0, 0x13ac6ca0
Caught exception in looking up space point ptr... 0x13ac6ca0, 0x19378cc0
Caught exception in looking up space point ptr... 0x32af0190, 0x32af05d0
Caught exception in looking up space point ptr... 0x32af05d0, 0x32af0190
Caught exception in looking up space point ptr... 0x3bc12440, 0x36ff6e40
Caught exception in looking up space point ptr... 0x36ff6e40, 0x3bc12440
Caught exception in looking up space point ptr... 0x4219f3f0, 0x4219fad0
Caught exception in looking up space point ptr... 0x4219fad0, 0x4219f3f0
Caught exception in looking up space point ptr... 0x3f0aebd0, 0x17cb05a0
Caught exception in looking up space point ptr... 0xa076190, 0x3f0aebd0
Caught exception in looking up space point ptr... 0x639d9530, 0x36be9350
Caught exception in looking up space point ptr... 0x36be9350, 0x36be66c0
Caught exception in looking up space point ptr... 0x303cf610, 0x303d4120
Caught exception in looking up space point ptr... 0x303d4120, 0x303d3a60
Caught exception in looking up space point ptr... 0x303d3a60, 0x303d3fc0
Caught exception in looking up space point ptr... 0x3d28a7a0, 0xd013c20
Caught exception in looking up space point ptr... 0xd013c20, 0x3d20dca0
Caught exception in looking up space point ptr... 0x38b86ac0, 0x685b12e0
Caught exception in looking up space point ptr... 0x685b12e0, 0x38b86ac0
Caught exception in looking up space point ptr... 0x6ed95950, 0x6ed95630
Caught exception in looking up space point ptr... 0x6ed95630, 0x6ed957c0
Caught exception in looking up space point ptr... 0x68e36ce0, 0x68e36920
Caught exception in looking up space point ptr... 0x68e36bb0, 0x68e36ce0
Caught exception in looking up space point ptr... 0x17dbfb80, 0x439047c0
Caught exception in looking up space point ptr... 0x439047c0, 0x17dbfb80
Caught exception in looking up space point ptr... 0x63035bf0, 0x63035e50
Caught exception in looking up space point ptr... 0x63035e50, 0x63035bf0
Caught exception in looking up space point ptr... 0x17dc4170, 0x17dc4bf0
Caught exception in looking up space point ptr... 0x17dc4410, 0x17dc4170
Caught exception in looking up space point ptr... 0x3b2c5af0, 0x78b4c240
Caught exception in looking up space point ptr... 0x78b4c240, 0x84ee5940
Caught exception in looking up space point ptr... 0x9bde650, 0x9bddab0
Caught exception in looking up space point ptr... 0x9bddab0, 0x299837f0
Caught exception in looking up space point ptr... 0x433216f0, 0x4c55b780
Caught exception in looking up space point ptr... 0x4c55b780, 0x301a0490
Caught exception in looking up space point ptr... 0x389208c0, 0x585481f0
Caught exception in looking up space point ptr... 0x585481f0, 0x389208c0
++++>>>> total num hits: 58798, num free: 56109
04-Feb-2026 01:23:34 CET  Closed output file "np02vd_raw_run042433_0154_df-s04-d0_dw_0_20260203T194039_reco_stage0_20260204T002334_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         92.2558       100.787       118.244       98.7789       6.30038        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                        5.989e-05    7.76651e-05   0.000195381    7.235e-05    2.96145e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                5.31886       5.65156       6.14018       5.69463      0.229505        19     
produce:triggerrawdecoder:PDVDTriggerReader4     0.00945575     0.0177375     0.0718296     0.0142442     0.0134559       19     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000367482   0.000611239   0.00117581    0.000525263   0.000225664      19     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.000306052   0.000490042   0.000900134   0.000414702   0.00015752       19     
produce:beamevent:BeamEvent                       5.364e-05    0.000112174   0.000570502    8.108e-05    0.000110676      19     
produce:pdvddaphne:DAPHNEReaderPDVD                1.53764       1.91425       2.22032       1.96104       0.1853         19     
produce:ophit:OpHitFinder                         0.0205884     0.0311569     0.0392991     0.0325544    0.00543607       19     
produce:wclsdatavd:WireCellToolkit                 66.2783       74.2964       89.634        75.0568       5.56164        19     
produce:gaushit:GausHitFinder                     0.720825      0.919464       1.1935       0.912517       0.13062        19     
produce:nhitsfilter:NumberOfHitsFilter            0.0001595    0.000232682   0.000435422   0.000191541   8.32011e-05      19     
produce:reco3d:SpacePointSolver                    8.23754       11.1145       15.2924       11.2677       1.68233        19     
produce:hitpdune:DisambigFromSpacePoints          0.0664343     0.125373      0.257326      0.113445      0.044618        19     
produce:cluster3d:Cluster3D                       0.899092       1.80005       2.64398       1.70165      0.481636        19     
[art]:TriggerResults:TriggerResultInserter       2.3131e-05    3.24397e-05   8.5121e-05    2.6831e-05    1.36351e-05      19     
end_path:out1:RootOutput                          4.38e-06     6.93526e-06    1.996e-05     5.46e-06     3.86036e-06      19     
end_path:out1:RootOutput(write)                    3.76171       4.87558       6.31199       4.91335       0.62084        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5519.22 MB
  Peak resident set size usage (VmHWM): 3385.53 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0154_df-s04-d0_dw_0_20260203T194039_reco_stage0_20260204T002334_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0154_df-s04-d0_dw_0_20260203T194039_reco_stage0_20260204T002334_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0154_df-s04-d0_dw_0_20260203T194039_reco_stage0_20260204T002334_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770162668.0,
    "core.end_time": 1770164615.0,
    "core.events": [
      17566,
      17572,
      17578,
      17584,
      17590,
      17596,
      17602,
      17608,
      17614,
      17620,
      17626,
      17632,
      17638,
      17644,
      17650,
      17656,
      17662,
      17668,
      17674
    ],
    "core.event_count": 19,
    "core.first_event_number": 17566,
    "core.last_event_number": 17674,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0154_df-s04-d0_dw_0_20260203T194039.hdf5"
    }
  ]
}
justIN time: 2026-02-04 07:45:24 UTC       justIN version: 01.06.00