justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303656.27@dunegpschedd01.fnal.gov

Jobsub ID303656.27@dunegpschedd01.fnal.gov
Workflow ID12717
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:56:25
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce512
Last heartbeat2026-02-04 00:48:47
From worker nodeHostnameb9p10p5730.cern.ch
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-04 00:11:15
Input filesvd-protodune:np02vd_raw_run042421_0066_df-s03-d0_dw_0_20260203T140646.hdf5
JobscriptExit code0
Real time35m (2115s)
CPU time33m (2038s = 96%)
Max RSS bytes2606870528 (2486 MiB)
Outputting started2026-02-04 00:46:30
Output files
Finished2026-02-04 00:48:47
Saved logsjustin-logs:303656.27-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

0x7770240
Caught exception in looking up space point ptr... 0x7770940, 0x776acd0
Caught exception in looking up space point ptr... 0x3bae1770, 0x10d04880
Caught exception in looking up space point ptr... 0x10d04880, 0x3baf4ea0
Caught exception in looking up space point ptr... 0xf80e2d0, 0x354672e0
Caught exception in looking up space point ptr... 0x354672e0, 0x641f35e0
Caught exception in looking up space point ptr... 0x6878d2b0, 0x520d9f10
Caught exception in looking up space point ptr... 0x518723b0, 0x6878d400
Caught exception in looking up space point ptr... 0x6878d400, 0x6878d2b0
Caught exception in looking up space point ptr... 0x44b4eff0, 0x44b4eea0
Caught exception in looking up space point ptr... 0x44b4eca0, 0x44b521b0
Caught exception in looking up space point ptr... 0x44b521b0, 0x44b522e0
Caught exception in looking up space point ptr... 0x44b522e0, 0x44b509f0
Caught exception in looking up space point ptr... 0x44b509f0, 0x44b4eff0
Caught exception in looking up space point ptr... 0x44b63da0, 0x44b63580
Caught exception in looking up space point ptr... 0x44b63580, 0x44b62e00
Caught exception in looking up space point ptr... 0x32c28540, 0x32c26150
Caught exception in looking up space point ptr... 0x32c297c0, 0x32c28540
Caught exception in looking up space point ptr... 0x315b1100, 0x2b741530
Caught exception in looking up space point ptr... 0x2b741530, 0x2b6c58c0
Caught exception in looking up space point ptr... 0x56a68c60, 0x56a68d90
Caught exception in looking up space point ptr... 0x56a68d90, 0x56a68b30
Caught exception in looking up space point ptr... 0xb2d5eb0, 0xb062de0
Caught exception in looking up space point ptr... 0xb062de0, 0x5e770340
Caught exception in looking up space point ptr... 0xaf63990, 0x51eaa6b0
Caught exception in looking up space point ptr... 0x51eaa6b0, 0xb2d5eb0
Caught exception in looking up space point ptr... 0x3d44bfb0, 0x10c36c30
Caught exception in looking up space point ptr... 0x10c36c30, 0x3d44c320
Caught exception in looking up space point ptr... 0x375075a0, 0x375076a0
Caught exception in looking up space point ptr... 0x375076a0, 0x375075a0
Caught exception in looking up space point ptr... 0x5f3a7a10, 0x9358b70
Caught exception in looking up space point ptr... 0x9358b70, 0x4996ff60
Caught exception in looking up space point ptr... 0x56575f60, 0x49dda640
Caught exception in looking up space point ptr... 0x49dda640, 0x56575f60
Caught exception in looking up space point ptr... 0x50e53ba0, 0x2a1719a0
Caught exception in looking up space point ptr... 0x1ca40e10, 0x50e53ba0
Caught exception in looking up space point ptr... 0x1cb6b3c0, 0x5c589cb0
Caught exception in looking up space point ptr... 0x5e76f8a0, 0x1cb6b3c0
Caught exception in looking up space point ptr... 0x41286810, 0x41286770
Caught exception in looking up space point ptr... 0x41286770, 0x412868b0
Caught exception in looking up space point ptr... 0x55064200, 0x193706b0
Caught exception in looking up space point ptr... 0x193706b0, 0x55064200
Caught exception in looking up space point ptr... 0x55064200, 0x2b6bb220
Caught exception in looking up space point ptr... 0x2b6bb220, 0x55064200
Caught exception in looking up space point ptr... 0x2c695620, 0xd194410
Caught exception in looking up space point ptr... 0xd194410, 0xd1945a0
Caught exception in looking up space point ptr... 0x3547b610, 0x521aec70
Caught exception in looking up space point ptr... 0x521aec70, 0x3547b610
Caught exception in looking up space point ptr... 0x32c19b60, 0x64bb0bc0
Caught exception in looking up space point ptr... 0x64bb0bc0, 0x32c19b60
Caught exception in looking up space point ptr... 0x33175850, 0x331756c0
Caught exception in looking up space point ptr... 0x331756c0, 0x1257cda0
Caught exception in looking up space point ptr... 0x1ca79130, 0x37320b20
Caught exception in looking up space point ptr... 0x37320b20, 0x17b31d90
Caught exception in looking up space point ptr... 0xf7e8fb0, 0x13103c00
Caught exception in looking up space point ptr... 0x13103c00, 0x131041a0
Caught exception in looking up space point ptr... 0x3d08c700, 0x3d08c3e0
Caught exception in looking up space point ptr... 0x3d08c3e0, 0x3d08c570
Caught exception in looking up space point ptr... 0x777e0d0, 0x1693ecf0
Caught exception in looking up space point ptr... 0x1693ecf0, 0x777e0d0
Caught exception in looking up space point ptr... 0x427e9100, 0x3d087a70
Caught exception in looking up space point ptr... 0x3d087a70, 0x427e9100
Caught exception in looking up space point ptr... 0x30878130, 0x27462790
Caught exception in looking up space point ptr... 0x27462790, 0x30878130
Caught exception in looking up space point ptr... 0x4e6cad60, 0x53018160
Caught exception in looking up space point ptr... 0x53018160, 0x4e6cc120
++++>>>> total num hits: 47083, num free: 45536
04-Feb-2026 01:46:23 CET  Closed output file "np02vd_raw_run042421_0066_df-s03-d0_dw_0_20260203T140646_reco_stage0_20260204T004623_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         67.9967       76.2833       98.9822       73.9154       7.51884        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       3.9751e-05    7.62969e-05   0.000189873   6.4301e-05    3.5767e-05       27     
produce:tpcrawdecoder:PDVDTPCReader                3.03427       3.71803       4.68584       3.68624      0.360536        27     
produce:triggerrawdecoder:PDVDTriggerReader4      0.0101815     0.0222967     0.0748559     0.0152858     0.0158222       27     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000367535   0.000599996   0.00241431    0.000496296   0.000370994      27     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.00139812    0.00304055     0.0349361    0.00180221    0.00625852       27     
produce:beamevent:BeamEvent                      4.7091e-05    0.000112088   0.000893141   8.1561e-05    0.000153976      27     
produce:pdvddaphne:DAPHNEReaderPDVD                1.90778       2.48078       3.3431        2.41246      0.361465        27     
produce:ophit:OpHitFinder                         0.0234763     0.0329302     0.0435838     0.0316777    0.00524961       27     
produce:wclsdatavd:WireCellToolkit                 49.1646       55.5084       70.5747       54.7165       4.58868        27     
produce:gaushit:GausHitFinder                     0.456434      0.867322       1.67485      0.811514      0.227333        27     
produce:nhitsfilter:NumberOfHitsFilter           0.000148711   0.000234595   0.000358685   0.000215803   6.17472e-05      27     
produce:reco3d:SpacePointSolver                    4.46366       8.93283       14.5929       8.77301       2.65486        27     
produce:hitpdune:DisambigFromSpacePoints          0.0368156     0.077477      0.151686      0.0666039     0.0257826       27     
produce:cluster3d:Cluster3D                       0.578395       1.42583       2.44817       1.37786      0.442795        27     
[art]:TriggerResults:TriggerResultInserter        1.825e-05    2.70861e-05   8.1861e-05     2.328e-05    1.19603e-05      27     
end_path:out1:RootOutput                          3.16e-06     7.06937e-06    2.621e-05     6.38e-06     4.41768e-06      27     
end_path:out1:RootOutput(write)                    2.8256         3.196        4.10008       3.10872      0.297397        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4566.53 MB
  Peak resident set size usage (VmHWM): 2606.87 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042421_0066_df-s03-d0_dw_0_20260203T140646_reco_stage0_20260204T004623_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042421_0066_df-s03-d0_dw_0_20260203T140646_reco_stage0_20260204T004623_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042421_0066_df-s03-d0_dw_0_20260203T140646_reco_stage0_20260204T004623_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770163886.0,
    "core.end_time": 1770165984.0,
    "core.events": [
      10760,
      10766,
      10772,
      10778,
      10784,
      10790,
      10796,
      10802,
      10808,
      10814,
      10820,
      10826,
      10832,
      10838,
      10844,
      10850,
      10856,
      10862,
      10868,
      10874,
      10880,
      10886,
      10892,
      10898,
      10904,
      10910,
      10916
    ],
    "core.event_count": 27,
    "core.first_event_number": 10760,
    "core.last_event_number": 10916,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42421
    ],
    "core.runs_subruns": [
      4242100001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042421_0066_df-s03-d0_dw_0_20260203T140646.hdf5"
    }
  ]
}
justIN time: 2026-02-04 10:34:23 UTC       justIN version: 01.06.00