justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297921.42@dunegpschedd02.fnal.gov

Jobsub ID297921.42@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:58:26
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce507
Last heartbeat2026-02-04 00:23:01
From worker nodeHostnameb9p35p0020.cern.ch
cpuinfoAMD EPYC 9655P 96-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:49:04
Input filesvd-protodune:np02vd_raw_run042433_0502_df-s04-d1_dw_0_20260203T215256.hdf5
JobscriptExit code0
Real time31m (1914s)
CPU time30m (1823s = 95%)
Max RSS bytes3399520256 (3242 MiB)
Outputting started2026-02-04 00:20:59
Output files
Finished2026-02-04 00:23:01
Saved logsjustin-logs:297921.42-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ptr... 0x43bee060, 0x43bee360
Caught exception in looking up space point ptr... 0x43bee360, 0x43bee160
Caught exception in looking up space point ptr... 0x6377dd60, 0x18f920a0
Caught exception in looking up space point ptr... 0x18f920a0, 0x6377dd60
Caught exception in looking up space point ptr... 0x174bfd90, 0x174bff20
Caught exception in looking up space point ptr... 0x174bff20, 0x174bfd90
Caught exception in looking up space point ptr... 0x7e2063d0, 0x5c814570
Caught exception in looking up space point ptr... 0x5c814570, 0x3a5838d0
Caught exception in looking up space point ptr... 0x62e4a420, 0x62e49f20
Caught exception in looking up space point ptr... 0x62e49f20, 0x62e4a420
Caught exception in looking up space point ptr... 0xb836990, 0xb836b20
Caught exception in looking up space point ptr... 0xb836b20, 0xb836990
Caught exception in looking up space point ptr... 0x7ad7c190, 0x7ad6ad20
Caught exception in looking up space point ptr... 0x7ad6ad20, 0x7ad7c190
Caught exception in looking up space point ptr... 0x3d221a60, 0x3d2252a0
Caught exception in looking up space point ptr... 0x3d2252a0, 0x3d221e20
Caught exception in looking up space point ptr... 0x3d221e20, 0x3d223e70
Caught exception in looking up space point ptr... 0x3d228570, 0x3d228a10
Caught exception in looking up space point ptr... 0x3d228a10, 0x3d228570
Caught exception in looking up space point ptr... 0x3d228570, 0x3d21d650
Caught exception in looking up space point ptr... 0x3d21d650, 0x3d228570
Caught exception in looking up space point ptr... 0x3d228570, 0x3d228200
Caught exception in looking up space point ptr... 0x3d228200, 0x3d228570
Caught exception in looking up space point ptr... 0x4408a4f0, 0x44074560
Caught exception in looking up space point ptr... 0x440793c0, 0x4408a4f0
Caught exception in looking up space point ptr... 0x1991f510, 0x5f38be60
Caught exception in looking up space point ptr... 0x5f38be60, 0x1991f5b0
Caught exception in looking up space point ptr... 0x12ffa3a0, 0x62b22c20
Caught exception in looking up space point ptr... 0x62b22c20, 0x12ffa3a0
Caught exception in looking up space point ptr... 0x2dddcd80, 0x2dddca60
Caught exception in looking up space point ptr... 0x2dddca60, 0x2dddcd80
Caught exception in looking up space point ptr... 0x4a2de680, 0x4a328620
Caught exception in looking up space point ptr... 0x4a328620, 0x4a332de0
Caught exception in looking up space point ptr... 0x3d6052b0, 0x3d5e7e50
Caught exception in looking up space point ptr... 0x3d5e7e50, 0x3d5e7d50
Caught exception in looking up space point ptr... 0x62e147b0, 0x62e14680
Caught exception in looking up space point ptr... 0x62e14680, 0x62e147b0
Caught exception in looking up space point ptr... 0x7daed470, 0x7daf08e0
Caught exception in looking up space point ptr... 0x7daf08e0, 0x7daf06f0
Caught exception in looking up space point ptr... 0x6f0ba5d0, 0x6f0ba2f0
Caught exception in looking up space point ptr... 0x6f0ba2f0, 0x6f0ba5d0
Caught exception in looking up space point ptr... 0x3f03fb30, 0x3f03f9a0
Caught exception in looking up space point ptr... 0x3f03f9a0, 0x3f03fb30
Caught exception in looking up space point ptr... 0x7a98df70, 0x7a989a30
Caught exception in looking up space point ptr... 0x7a989a30, 0x7a98bae0
Caught exception in looking up space point ptr... 0x6030b920, 0x39460950
Caught exception in looking up space point ptr... 0x39460950, 0x6030b920
Caught exception in looking up space point ptr... 0x48aa2a70, 0x48a9e250
Caught exception in looking up space point ptr... 0x48a9e250, 0x48aa2720
Caught exception in looking up space point ptr... 0x2ddca550, 0x2ddca1c0
Caught exception in looking up space point ptr... 0x2ddca1c0, 0x2ddca550
Caught exception in looking up space point ptr... 0x11e07ba0, 0x11e13c60
Caught exception in looking up space point ptr... 0x11e13c60, 0x11e07d80
Caught exception in looking up space point ptr... 0x532a5920, 0x532a4b90
Caught exception in looking up space point ptr... 0x532a4b90, 0x532a5920
Caught exception in looking up space point ptr... 0x532a5920, 0x532a2a10
Caught exception in looking up space point ptr... 0x532a2a10, 0x532a5920
Caught exception in looking up space point ptr... 0x30f6ee10, 0x30f6eaa0
Caught exception in looking up space point ptr... 0x30f6eaa0, 0x30f6ee10
Caught exception in looking up space point ptr... 0x66b240c0, 0x5695cd50
Caught exception in looking up space point ptr... 0x5695cd50, 0x39be74b0
Caught exception in looking up space point ptr... 0x5329f970, 0x5329f7e0
Caught exception in looking up space point ptr... 0x5329f7e0, 0x5329f970
Caught exception in looking up space point ptr... 0x6345abf0, 0xb60e480
Caught exception in looking up space point ptr... 0xb60e480, 0x6345abf0
Caught exception in looking up space point ptr... 0x7abb9380, 0x7147bb00
Caught exception in looking up space point ptr... 0x7147bb00, 0x4a32d5e0
++++>>>> total num hits: 82166, num free: 78549
04-Feb-2026 01:20:54 CET  Closed output file "np02vd_raw_run042433_0502_df-s04-d1_dw_0_20260203T215256_reco_stage0_20260204T002054_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         89.6731       98.2545       106.737       98.965        4.31484        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                        5.551e-05    7.10074e-05   0.000161731    6.921e-05    2.30391e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                6.18758       6.71305       7.60965       6.61682      0.404214        19     
produce:triggerrawdecoder:PDVDTriggerReader4     0.00771306     0.0213916     0.0597522     0.015376      0.0142263       19     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000288692   0.000466639   0.000797835   0.000457593   0.000108167      19     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.000257572   0.000425647   0.000635075   0.000404543   0.000104556      19     
produce:beamevent:BeamEvent                      5.3151e-05    0.000102072   0.000673245    6.997e-05    0.000134932      19     
produce:pdvddaphne:DAPHNEReaderPDVD                1.57852       1.89779       2.69791       1.86155      0.236033        19     
produce:ophit:OpHitFinder                         0.0166413     0.0258878     0.0317558     0.0270786    0.00423427       19     
produce:wclsdatavd:WireCellToolkit                 62.3721       69.5193       74.3733       69.978        3.43143        19     
produce:gaushit:GausHitFinder                     0.651587      0.899955       1.27947      0.896211      0.151984        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000157101   0.000225555   0.000370692   0.000213621   5.2493e-05       19     
produce:reco3d:SpacePointSolver                    7.99166       12.7064       15.9235       12.6676       1.99038        19     
produce:hitpdune:DisambigFromSpacePoints          0.0941805     0.133902       0.19741      0.132477      0.027753        19     
produce:cluster3d:Cluster3D                        1.33741       1.72427       2.28024       1.62798      0.303454        19     
[art]:TriggerResults:TriggerResultInserter        2.123e-05    2.9643e-05     7.387e-05    2.6571e-05    1.11605e-05      19     
end_path:out1:RootOutput                          4.49e-06     5.89695e-06   1.8091e-05     5.04e-06     2.92072e-06      19     
end_path:out1:RootOutput(write)                    3.97189       4.57944       5.61511       4.58064      0.438194        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5456.99 MB
  Peak resident set size usage (VmHWM): 3399.52 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0502_df-s04-d1_dw_0_20260203T215256_reco_stage0_20260204T002054_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0502_df-s04-d1_dw_0_20260203T215256_reco_stage0_20260204T002054_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0502_df-s04-d1_dw_0_20260203T215256_reco_stage0_20260204T002054_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770162552.0,
    "core.end_time": 1770164455.0,
    "core.events": [
      57239,
      57245,
      57251,
      57257,
      57263,
      57269,
      57275,
      57281,
      57287,
      57293,
      57299,
      57305,
      57311,
      57317,
      57323,
      57329,
      57335,
      57341,
      57347
    ],
    "core.event_count": 19,
    "core.first_event_number": 57239,
    "core.last_event_number": 57347,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0502_df-s04-d1_dw_0_20260203T215256.hdf5"
    }
  ]
}
justIN time: 2026-02-04 10:33:58 UTC       justIN version: 01.06.00