justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303649.144@dunegpschedd01.fnal.gov

Jobsub ID303649.144@dunegpschedd01.fnal.gov
Workflow ID12718
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:48:26
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce515
Last heartbeat2026-02-03 23:57:15
From worker nodeHostnameb9p35p0043.cern.ch
cpuinfoAMD EPYC 9655P 96-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:07:12
Input filesvd-protodune:np02vd_raw_run042422_0103_df-s03-d1_dw_0_20260203T161028.hdf5
JobscriptExit code0
Real time47m (2851s)
CPU time46m (2768s = 97%)
Max RSS bytes2756841472 (2629 MiB)
Outputting started2026-02-03 23:54:44
Output files
Finished2026-02-03 23:57:11
Saved logsjustin-logs:303649.144-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

0x3ea11e90
Caught exception in looking up space point ptr... 0x6da0c340, 0x6d98ffd0
Caught exception in looking up space point ptr... 0x6d98ffd0, 0x6d98fe50
Caught exception in looking up space point ptr... 0x28df5e00, 0x83c8f20
Caught exception in looking up space point ptr... 0x83c8f20, 0x6da0c340
Caught exception in looking up space point ptr... 0x24a37a30, 0x6d990530
Caught exception in looking up space point ptr... 0x6d990530, 0x6d990280
Caught exception in looking up space point ptr... 0x6d990280, 0xe784610
Caught exception in looking up space point ptr... 0x28df3440, 0x7b6f330
Caught exception in looking up space point ptr... 0x7b6f330, 0x11d06940
Caught exception in looking up space point ptr... 0x2715f060, 0x26312f10
Caught exception in looking up space point ptr... 0x26312f10, 0x632437b0
Caught exception in looking up space point ptr... 0x632437b0, 0x6d991970
Caught exception in looking up space point ptr... 0x6d991970, 0x6d9915e0
Caught exception in looking up space point ptr... 0x2a479a10, 0x14b83c00
Caught exception in looking up space point ptr... 0x14b83c00, 0x11d086c0
Caught exception in looking up space point ptr... 0x11d086c0, 0x2c5431b0
Caught exception in looking up space point ptr... 0x2b424a50, 0x2a479a10
Caught exception in looking up space point ptr... 0x2970fbb0, 0x2970fd40
Caught exception in looking up space point ptr... 0x45f69a80, 0x2970fbb0
Caught exception in looking up space point ptr... 0x27189260, 0x4b0d8da0
Caught exception in looking up space point ptr... 0x4b0be580, 0x27189260
Caught exception in looking up space point ptr... 0x2a48d8e0, 0x42b50b20
Caught exception in looking up space point ptr... 0x42b50b20, 0x42b52140
Caught exception in looking up space point ptr... 0x44a352e0, 0x264ef300
Caught exception in looking up space point ptr... 0x44a35470, 0x44a352e0
Caught exception in looking up space point ptr... 0x44b436b0, 0x44b31410
Caught exception in looking up space point ptr... 0x44b43840, 0x44b436b0
Caught exception in looking up space point ptr... 0x4ee6d690, 0x17ac1160
Caught exception in looking up space point ptr... 0x4ee6d500, 0x4ee6d690
Caught exception in looking up space point ptr... 0x4b0e0e70, 0x3ff1acd0
Caught exception in looking up space point ptr... 0x3ff1acd0, 0x739bb00
Caught exception in looking up space point ptr... 0x26da3450, 0x2e87dee0
Caught exception in looking up space point ptr... 0x2e87dee0, 0x393d44a0
Caught exception in looking up space point ptr... 0x30509110, 0x30508f80
Caught exception in looking up space point ptr... 0x30508f80, 0x30508df0
Caught exception in looking up space point ptr... 0x511ce860, 0x511c9d50
Caught exception in looking up space point ptr... 0x511ce590, 0x511ce860
Caught exception in looking up space point ptr... 0x3fe62b10, 0x3fe55050
Caught exception in looking up space point ptr... 0x3fe55050, 0x3fe62ca0
Caught exception in looking up space point ptr... 0x13b8d420, 0x65596b70
Caught exception in looking up space point ptr... 0x65596b70, 0x13b8d290
Caught exception in looking up space point ptr... 0x13b8f9a0, 0x65599280
Caught exception in looking up space point ptr... 0x65599280, 0x13b8fb30
Caught exception in looking up space point ptr... 0x3d9acd70, 0x3d9ac8c0
Caught exception in looking up space point ptr... 0x3d9ac8c0, 0x2da5bfc0
Caught exception in looking up space point ptr... 0x2c5e0310, 0x2c5e00b0
Caught exception in looking up space point ptr... 0x2c5e00b0, 0x1255fa20
Caught exception in looking up space point ptr... 0xbdfd790, 0x6974fb30
Caught exception in looking up space point ptr... 0x3e08d850, 0xbdfd790
Caught exception in looking up space point ptr... 0x4b3350d0, 0x4b322980
Caught exception in looking up space point ptr... 0x4b322980, 0x4b334f40
Caught exception in looking up space point ptr... 0x69741d10, 0x304ed3e0
Caught exception in looking up space point ptr... 0x4b585d90, 0x69741d10
Caught exception in looking up space point ptr... 0x248126f0, 0x24801a30
Caught exception in looking up space point ptr... 0x24801a30, 0x2b7e96c0
Caught exception in looking up space point ptr... 0x4a646170, 0x4a633ed0
Caught exception in looking up space point ptr... 0x4a633ed0, 0x4a645fe0
Caught exception in looking up space point ptr... 0x27ba5690, 0x2a9de0b0
Caught exception in looking up space point ptr... 0x2a9de0b0, 0x13fad620
Caught exception in looking up space point ptr... 0xe573160, 0xe572c60
Caught exception in looking up space point ptr... 0xe572c60, 0xe573160
Caught exception in looking up space point ptr... 0xe573160, 0xe572e40
Caught exception in looking up space point ptr... 0xe572e40, 0xe573160
Caught exception in looking up space point ptr... 0x7b709d0, 0x7b70770
Caught exception in looking up space point ptr... 0x7b70770, 0x7b709d0
++++>>>> total num hits: 284423, num free: 278828
04-Feb-2026 00:54:38 CET  Closed output file "np02vd_raw_run042422_0103_df-s03-d1_dw_0_20260203T161028_reco_stage0_20260203T235438_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         65.9036       103.708       130.045       110.34        18.9949        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                        6.476e-05    7.82367e-05   0.00018331     7.339e-05    2.1932e-05       27     
produce:tpcrawdecoder:PDVDTPCReader                3.45034       3.74357       4.03571       3.75686      0.145087        27     
produce:triggerrawdecoder:PDVDTriggerReader4     0.00970803     0.0154782     0.0749371     0.0115954     0.0126437       27     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.00036759    0.000531607   0.00068679    0.00054745    9.05835e-05      27     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.00119866    0.00154975    0.00192906    0.00159784    0.000190113      27     
produce:beamevent:BeamEvent                       6.467e-05     0.0001134    0.000700031    8.818e-05    0.000116698      27     
produce:pdvddaphne:DAPHNEReaderPDVD                1.86094       2.16362       2.43144       2.1525       0.142067        27     
produce:ophit:OpHitFinder                         0.0290067     0.0366037     0.0442833     0.0374362    0.00344892       27     
produce:wclsdatavd:WireCellToolkit                 46.5306       54.1866       62.6635       52.4155       4.37001        27     
produce:gaushit:GausHitFinder                     0.558636       1.10024       1.72772       1.07086       0.28558        27     
produce:nhitsfilter:NumberOfHitsFilter           0.00012567    0.00021959    0.00040869    0.00020717    5.90997e-05      27     
produce:reco3d:SpacePointSolver                    4.12357       7.55156        13.11        6.66113       2.66634        27     
produce:hitpdune:DisambigFromSpacePoints          0.0478408     0.155888      0.269544       0.13857      0.0612421       27     
produce:cluster3d:Cluster3D                       0.577329       30.2248       44.2597       37.0276       15.7775        27     
[art]:TriggerResults:TriggerResultInserter        2.474e-05    3.49881e-05    6.973e-05     3.478e-05    8.6421e-06       27     
end_path:out1:RootOutput                          4.63e-06     6.56704e-06    1.869e-05     5.61e-06     3.26196e-06      27     
end_path:out1:RootOutput(write)                    3.01301       4.49601       5.61137       4.45971      0.558014        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4733.93 MB
  Peak resident set size usage (VmHWM): 2756.84 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042422_0103_df-s03-d1_dw_0_20260203T161028_reco_stage0_20260203T235438_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042422_0103_df-s03-d1_dw_0_20260203T161028_reco_stage0_20260203T235438_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042422_0103_df-s03-d1_dw_0_20260203T161028_reco_stage0_20260203T235438_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770160040.0,
    "core.end_time": 1770162879.0,
    "core.events": [
      16701,
      16707,
      16713,
      16719,
      16725,
      16731,
      16737,
      16743,
      16749,
      16755,
      16761,
      16767,
      16773,
      16779,
      16785,
      16791,
      16797,
      16803,
      16809,
      16815,
      16821,
      16827,
      16833,
      16839,
      16845,
      16851,
      16857
    ],
    "core.event_count": 27,
    "core.first_event_number": 16701,
    "core.last_event_number": 16857,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42422
    ],
    "core.runs_subruns": [
      4242200001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042422_0103_df-s03-d1_dw_0_20260203T161028.hdf5"
    }
  ]
}
justIN time: 2026-02-04 10:46:22 UTC       justIN version: 01.06.00