justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297923.56@dunegpschedd02.fnal.gov

Jobsub ID297923.56@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 23:00:26
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce515
Last heartbeat2026-02-04 00:39:31
From worker nodeHostnameb9p35p0043.cern.ch
cpuinfoAMD EPYC 9655P 96-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:59:11
Input filesvd-protodune:np02vd_raw_run042433_0112_df-s04-d0_dw_0_20260203T192441.hdf5
JobscriptExit code0
Real time38m (2283s)
CPU time36m (2187s = 95%)
Max RSS bytes3410898944 (3252 MiB)
Outputting started2026-02-04 00:37:15
Output files
Finished2026-02-04 00:39:31
Saved logsjustin-logs:297923.56-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ce point ptr... 0x18a9fab0, 0x18ab0460
Caught exception in looking up space point ptr... 0x18ab0460, 0x18ab0310
Caught exception in looking up space point ptr... 0x7903f230, 0x7903ef90
Caught exception in looking up space point ptr... 0x7903ef90, 0x7903f230
Caught exception in looking up space point ptr... 0x491ff2a0, 0xf454d40
Caught exception in looking up space point ptr... 0xf454d40, 0x792f1420
Caught exception in looking up space point ptr... 0x7909f7e0, 0x7909f9e0
Caught exception in looking up space point ptr... 0x7909f9e0, 0x7909f7e0
Caught exception in looking up space point ptr... 0x2a22f7d0, 0x2a2285e0
Caught exception in looking up space point ptr... 0x2a2285e0, 0x2a22f7d0
Caught exception in looking up space point ptr... 0x1528e550, 0x1528e3c0
Caught exception in looking up space point ptr... 0x1528e3c0, 0x1528e550
Caught exception in looking up space point ptr... 0x849ec80, 0x849ee10
Caught exception in looking up space point ptr... 0x849ee10, 0x849efa0
Caught exception in looking up space point ptr... 0x17e44110, 0x17e44370
Caught exception in looking up space point ptr... 0x17e44370, 0x17e431a0
Caught exception in looking up space point ptr... 0x3380f0b0, 0x33814480
Caught exception in looking up space point ptr... 0x33814480, 0x33813fc0
Caught exception in looking up space point ptr... 0x19238070, 0x1920e980
Caught exception in looking up space point ptr... 0x1920e980, 0x1923a320
Caught exception in looking up space point ptr... 0x1702fb70, 0x17031730
Caught exception in looking up space point ptr... 0x17031730, 0x1702fb70
Caught exception in looking up space point ptr... 0x13bd3ce0, 0x13bce6f0
Caught exception in looking up space point ptr... 0x13bd6d70, 0x13bd3ce0
Caught exception in looking up space point ptr... 0x4a55cd20, 0x37194e00
Caught exception in looking up space point ptr... 0x37194e00, 0x122e2bd0
Caught exception in looking up space point ptr... 0x2ca0dc90, 0x2ca0d8c0
Caught exception in looking up space point ptr... 0x2ca0d8c0, 0x2ca0dc90
Caught exception in looking up space point ptr... 0x2d396a30, 0x2d3b89f0
Caught exception in looking up space point ptr... 0x15b42100, 0x2d396a30
Caught exception in looking up space point ptr... 0x4ae8af20, 0x4ae88d10
Caught exception in looking up space point ptr... 0x4ae88d10, 0x4ae8af20
Caught exception in looking up space point ptr... 0x45e37bc0, 0x192324a0
Caught exception in looking up space point ptr... 0x192324a0, 0x192323b0
Caught exception in looking up space point ptr... 0x1a3dad20, 0x1a3c6f40
Caught exception in looking up space point ptr... 0x4d5942a0, 0x1a3dad20
Caught exception in looking up space point ptr... 0x45e2fd20, 0x45e2feb0
Caught exception in looking up space point ptr... 0x45e2feb0, 0x45e2fa00
Caught exception in looking up space point ptr... 0x3e7956f0, 0x47453d50
Caught exception in looking up space point ptr... 0x47453d50, 0x3e7956f0
Caught exception in looking up space point ptr... 0x493ef3c0, 0x493ef770
Caught exception in looking up space point ptr... 0x493ef770, 0x493ef3c0
Caught exception in looking up space point ptr... 0xf5df970, 0x7c6c6190
Caught exception in looking up space point ptr... 0x7c6c6190, 0xf5df970
Caught exception in looking up space point ptr... 0xf97c950, 0xf97c7c0
Caught exception in looking up space point ptr... 0xf97c7c0, 0xf97cc70
Caught exception in looking up space point ptr... 0x3b7bf510, 0x41452fc0
Caught exception in looking up space point ptr... 0x41452fc0, 0x3b7bf510
Caught exception in looking up space point ptr... 0x13bd9ff0, 0x13bda310
Caught exception in looking up space point ptr... 0x13bda310, 0x13bda110
Caught exception in looking up space point ptr... 0x13bc3600, 0x13bc3210
Caught exception in looking up space point ptr... 0x13bc3210, 0x13bc3470
Caught exception in looking up space point ptr... 0x5a176d90, 0x36107530
Caught exception in looking up space point ptr... 0x36107530, 0x5a173340
Caught exception in looking up space point ptr... 0xcf7b3a0, 0x378b5e60
Caught exception in looking up space point ptr... 0x378b5e60, 0xcf7b3a0
Caught exception in looking up space point ptr... 0x3b4eba00, 0x3b4eb6e0
Caught exception in looking up space point ptr... 0x3b4eb6e0, 0x3b4eb550
Caught exception in looking up space point ptr... 0x187c0e70, 0x187bef60
Caught exception in looking up space point ptr... 0x187bef60, 0x187c0e70
Caught exception in looking up space point ptr... 0x187b3ab0, 0x187b3920
Caught exception in looking up space point ptr... 0x187b3920, 0x3b5f0d10
Caught exception in looking up space point ptr... 0xab8bd90, 0x187b3ab0
Caught exception in looking up space point ptr... 0x10437eb0, 0x31e83030
Caught exception in looking up space point ptr... 0x31e83030, 0x31e82ee0
Caught exception in looking up space point ptr... 0x1527cb90, 0x48fb6450
Caught exception in looking up space point ptr... 0x48fb6450, 0x48fb7690
++++>>>> total num hits: 191482, num free: 185041
04-Feb-2026 01:37:09 CET  Closed output file "np02vd_raw_run042433_0112_df-s04-d0_dw_0_20260203T192441_reco_stage0_20260204T003709_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         103.792       117.412       129.552       116.679       7.57905        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                        6.089e-05    7.62605e-05   0.00013567     7.299e-05    1.5683e-05       19     
produce:tpcrawdecoder:PDVDTPCReader                5.73852       6.96263       15.131        6.6282         1.973         19     
produce:triggerrawdecoder:PDVDTriggerReader4     0.00977479     0.0247555     0.0522947     0.0237249     0.0143369       19     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000441771   0.000921354   0.00395222    0.00061328    0.000849937      19     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.00032875    0.000531716   0.00087403    0.00050022    0.000140454      19     
produce:beamevent:BeamEvent                       5.996e-05    0.000119897   0.00078601     8.288e-05    0.000157351      19     
produce:pdvddaphne:DAPHNEReaderPDVD                1.67376       2.09666       2.35915       2.09058      0.143627        19     
produce:ophit:OpHitFinder                         0.0245247     0.0334839     0.0399447     0.0342004    0.00430691       19     
produce:wclsdatavd:WireCellToolkit                 71.8402       80.7445       91.7787       79.0898       5.52126        19     
produce:gaushit:GausHitFinder                     0.729566       1.10506       1.35523       1.11138      0.165935        19     
produce:nhitsfilter:NumberOfHitsFilter           0.00017728    0.000295616    0.0009598    0.00023855    0.000168916      19     
produce:reco3d:SpacePointSolver                    13.079        18.748        24.5839       17.6589       3.48224        19     
produce:hitpdune:DisambigFromSpacePoints          0.0978967     0.170362      0.270011      0.157495      0.0471511       19     
produce:cluster3d:Cluster3D                        1.44261       2.25075       3.40048       2.07182      0.522573        19     
[art]:TriggerResults:TriggerResultInserter        2.444e-05    3.48337e-05   0.00010212     2.843e-05    1.74523e-05      19     
end_path:out1:RootOutput                          4.51e-06     6.43263e-06    2.32e-05      5.19e-06     4.03111e-06      19     
end_path:out1:RootOutput(write)                    3.97608       5.23149       6.20932       5.26644      0.593672        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5415.61 MB
  Peak resident set size usage (VmHWM): 3410.9 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0112_df-s04-d0_dw_0_20260203T192441_reco_stage0_20260204T003709_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0112_df-s04-d0_dw_0_20260203T192441_reco_stage0_20260204T003709_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0112_df-s04-d0_dw_0_20260203T192441_reco_stage0_20260204T003709_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770163160.0,
    "core.end_time": 1770165430.0,
    "core.events": [
      12778,
      12784,
      12790,
      12796,
      12802,
      12808,
      12814,
      12820,
      12826,
      12832,
      12838,
      12844,
      12850,
      12856,
      12862,
      12868,
      12874,
      12880,
      12886
    ],
    "core.event_count": 19,
    "core.first_event_number": 12778,
    "core.last_event_number": 12886,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0112_df-s04-d0_dw_0_20260203T192441.hdf5"
    }
  ]
}
justIN time: 2026-02-04 07:38:28 UTC       justIN version: 01.06.00