justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297916.59@dunegpschedd02.fnal.gov

Jobsub ID297916.59@dunegpschedd02.fnal.gov
Workflow ID12718
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:46:26
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce505
Last heartbeat2026-02-03 23:21:11
From worker nodeHostnameb9p35p0031.cern.ch
cpuinfoAMD EPYC 9655P 96-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 22:47:11
Input filesvd-protodune:np02vd_raw_run042422_0048_df-s04-d0_dw_0_20260203T154045.hdf5
JobscriptExit code0
Real time31m (1915s)
CPU time30m (1810s = 94%)
Max RSS bytes2675580928 (2551 MiB)
Outputting started2026-02-03 23:19:07
Output files
Finished2026-02-03 23:21:11
Saved logsjustin-logs:297916.59-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

r... 0x54da5540, 0x54d98da0
Caught exception in looking up space point ptr... 0x54d9d900, 0x54da57d0
Caught exception in looking up space point ptr... 0x54da57d0, 0x54da5540
Caught exception in looking up space point ptr... 0x683aa4a0, 0x3be71b10
Caught exception in looking up space point ptr... 0x3be71b10, 0x683aa4a0
Caught exception in looking up space point ptr... 0x423903e0, 0x42390570
Caught exception in looking up space point ptr... 0x42390570, 0x423903e0
Caught exception in looking up space point ptr... 0x3356a9d0, 0x3356ad40
Caught exception in looking up space point ptr... 0x3356ad40, 0x8222740
Caught exception in looking up space point ptr... 0x8222740, 0x3356aed0
Caught exception in looking up space point ptr... 0x3356aed0, 0x3356ab60
Caught exception in looking up space point ptr... 0x4c950c10, 0xef22140
Caught exception in looking up space point ptr... 0xef22140, 0x549b2140
Caught exception in looking up space point ptr... 0x549b2140, 0x14f3f060
Caught exception in looking up space point ptr... 0x2288aa60, 0x4ae19110
Caught exception in looking up space point ptr... 0x4ae19110, 0x2288b660
Caught exception in looking up space point ptr... 0x4f475220, 0x305609e0
Caught exception in looking up space point ptr... 0x305609e0, 0x30546280
Caught exception in looking up space point ptr... 0x46e64db0, 0x4d543c50
Caught exception in looking up space point ptr... 0x4d543c50, 0x46e6c180
Caught exception in looking up space point ptr... 0x510bc260, 0x510bc3f0
Caught exception in looking up space point ptr... 0x510bc3f0, 0x510bc260
Caught exception in looking up space point ptr... 0xc8c9720, 0x627d2850
Caught exception in looking up space point ptr... 0x627d2850, 0x28c14110
Caught exception in looking up space point ptr... 0x30a1f240, 0x30a1f0f0
Caught exception in looking up space point ptr... 0x30a1f0f0, 0x30a1d180
Caught exception in looking up space point ptr... 0x5414bf80, 0x5414ebf0
Caught exception in looking up space point ptr... 0x5414ebf0, 0x5414bf80
Caught exception in looking up space point ptr... 0x302a6100, 0x302a5cb0
Caught exception in looking up space point ptr... 0x302a5cb0, 0x302a6100
Caught exception in looking up space point ptr... 0x4f7f6510, 0x4f7f63e0
Caught exception in looking up space point ptr... 0x4f7f63e0, 0x4f7f6510
Caught exception in looking up space point ptr... 0x55a56880, 0x55a5b9a0
Caught exception in looking up space point ptr... 0x55a5b9a0, 0x55a56880
Caught exception in looking up space point ptr... 0x30a237b0, 0x30a22d70
Caught exception in looking up space point ptr... 0x30a2d720, 0x30a237b0
Caught exception in looking up space point ptr... 0x4ae172b0, 0x4ae16fb0
Caught exception in looking up space point ptr... 0x4ae16fb0, 0x4ae172b0
Caught exception in looking up space point ptr... 0x513dd7d0, 0x513df620
Caught exception in looking up space point ptr... 0x513df620, 0x513dd7d0
Caught exception in looking up space point ptr... 0x4aaeae50, 0x4aade5f0
Caught exception in looking up space point ptr... 0x4aade5f0, 0x4aaeae50
Caught exception in looking up space point ptr... 0x4aaeae50, 0x4aadc1b0
Caught exception in looking up space point ptr... 0x4aadc1b0, 0x4aaeae50
Caught exception in looking up space point ptr... 0x147f03d0, 0x147f0560
Caught exception in looking up space point ptr... 0x147f0560, 0x147f03d0
Caught exception in looking up space point ptr... 0x513d92b0, 0x513cb600
Caught exception in looking up space point ptr... 0x513cb600, 0x513e10b0
Caught exception in looking up space point ptr... 0x4adf51b0, 0x4adf4e40
Caught exception in looking up space point ptr... 0x4adf4e40, 0x4adf51b0
Caught exception in looking up space point ptr... 0x28fd8c90, 0x29bf9a10
Caught exception in looking up space point ptr... 0x29bf9a10, 0x28fd8c90
Caught exception in looking up space point ptr... 0x32dc6290, 0x32dc6420
Caught exception in looking up space point ptr... 0x32dc6420, 0x32dc6740
Caught exception in looking up space point ptr... 0x57015a10, 0x5701edd0
Caught exception in looking up space point ptr... 0x5701edd0, 0x57015a10
Caught exception in looking up space point ptr... 0x12d6d0b0, 0x4621afb0
Caught exception in looking up space point ptr... 0x4621afb0, 0x1291f4a0
Caught exception in looking up space point ptr... 0x14e44cd0, 0x30a833f0
Caught exception in looking up space point ptr... 0x30a833f0, 0x14e44cd0
Caught exception in looking up space point ptr... 0x67fab9b0, 0x2b822640
Caught exception in looking up space point ptr... 0x2b822640, 0x61b71920
Caught exception in looking up space point ptr... 0x61ffbbe0, 0x462c0330
Caught exception in looking up space point ptr... 0x462c0330, 0x61ffbbe0
Caught exception in looking up space point ptr... 0x3dd71400, 0x14c1af70
Caught exception in looking up space point ptr... 0x14c1af70, 0x3dd71400
++++>>>> total num hits: 57272, num free: 54813
04-Feb-2026 00:19:01 CET  Closed output file "np02vd_raw_run042422_0048_df-s04-d0_dw_0_20260203T154045_reco_stage0_20260203T231901_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         61.7295       68.7381       81.8037       67.5954       4.86665        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                        5.668e-05    7.41944e-05   0.00016812     7.169e-05    2.06225e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                4.38957       5.07362       8.02677       4.92216      0.696797        27     
produce:triggerrawdecoder:PDVDTriggerReader4     0.00896097     0.0211803     0.0641739     0.0144451     0.0157212       27     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.00041188    0.00101577     0.0127083    0.00056302    0.00229511       27     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.00126424    0.00156542    0.00223613    0.00144954    0.000273393      27     
produce:beamevent:BeamEvent                       6.496e-05    0.000117997   0.00084681     8.479e-05    0.000143967      27     
produce:pdvddaphne:DAPHNEReaderPDVD                1.77796       2.09445       2.36765       2.0974       0.160774        27     
produce:ophit:OpHitFinder                         0.0175847     0.0326506     0.0396839     0.0328744    0.00509508       27     
produce:wclsdatavd:WireCellToolkit                 42.6142       47.8041       51.9972       48.1001       2.2312         27     
produce:gaushit:GausHitFinder                     0.468483      0.776588       1.35938      0.783072      0.228709        27     
produce:nhitsfilter:NumberOfHitsFilter           0.00012154    0.000182656   0.000421841   0.00016764    5.62967e-05      27     
produce:reco3d:SpacePointSolver                    3.8378        8.01041       15.9293       7.33558       2.66535        27     
produce:hitpdune:DisambigFromSpacePoints          0.0322126     0.0787798     0.176668      0.0682916     0.0311052       27     
produce:cluster3d:Cluster3D                       0.404549       1.31048       2.44133       1.07512      0.555559        27     
[art]:TriggerResults:TriggerResultInserter        2.204e-05    3.30804e-05    8.171e-05     3.116e-05    1.11476e-05      27     
end_path:out1:RootOutput                          4.19e-06     5.82889e-06    1.865e-05     5.03e-06     2.63803e-06      27     
end_path:out1:RootOutput(write)                    2.73947       3.50603       4.60099       3.44323      0.465874        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4617.54 MB
  Peak resident set size usage (VmHWM): 2675.58 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042422_0048_df-s04-d0_dw_0_20260203T154045_reco_stage0_20260203T231901_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042422_0048_df-s04-d0_dw_0_20260203T154045_reco_stage0_20260203T231901_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042422_0048_df-s04-d0_dw_0_20260203T154045_reco_stage0_20260203T231901_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770158838.0,
    "core.end_time": 1770160742.0,
    "core.events": [
      7786,
      7792,
      7798,
      7804,
      7810,
      7816,
      7822,
      7828,
      7834,
      7840,
      7846,
      7852,
      7858,
      7864,
      7870,
      7876,
      7882,
      7888,
      7894,
      7900,
      7906,
      7912,
      7918,
      7924,
      7930,
      7936,
      7942
    ],
    "core.event_count": 27,
    "core.first_event_number": 7786,
    "core.last_event_number": 7942,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42422
    ],
    "core.runs_subruns": [
      4242200001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042422_0048_df-s04-d0_dw_0_20260203T154045.hdf5"
    }
  ]
}
justIN time: 2026-02-04 07:38:28 UTC       justIN version: 01.06.00