justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303654.90@dunegpschedd01.fnal.gov

Jobsub ID303654.90@dunegpschedd01.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:52:26
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce510
Last heartbeat2026-02-04 00:41:31
From worker nodeHostnameb9p35p0023.cern.ch
cpuinfoAMD EPYC 9655P 96-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-04 00:05:24
Input filesvd-protodune:np02vd_raw_run042433_0134_df-s04-d0_dw_0_20260203T193303.hdf5
JobscriptExit code0
Real time34m (2046s)
CPU time32m (1966s = 96%)
Max RSS bytes3484475392 (3323 MiB)
Outputting started2026-02-04 00:39:31
Output files
Finished2026-02-04 00:41:31
Saved logsjustin-logs:303654.90-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

t ptr... 0x2ffdd880, 0x2ffdd5e0
Caught exception in looking up space point ptr... 0xa5a1f00, 0xa5b9cf0
Caught exception in looking up space point ptr... 0xa5b9cf0, 0xa5cc530
Caught exception in looking up space point ptr... 0x42a98030, 0x42a98350
Caught exception in looking up space point ptr... 0x42a98350, 0x42a981c0
Caught exception in looking up space point ptr... 0x4758acf0, 0x47589930
Caught exception in looking up space point ptr... 0x18d51460, 0x4758acf0
Caught exception in looking up space point ptr... 0x1473d520, 0x1473d390
Caught exception in looking up space point ptr... 0x1473d390, 0x1473d520
Caught exception in looking up space point ptr... 0x3f782e50, 0x3f782d00
Caught exception in looking up space point ptr... 0x3f782d00, 0x3f782e50
Caught exception in looking up space point ptr... 0x330b2090, 0x330b2270
Caught exception in looking up space point ptr... 0x330b2270, 0x330b2090
Caught exception in looking up space point ptr... 0x3aea2a60, 0x51f7dbf0
Caught exception in looking up space point ptr... 0x51f7dbf0, 0x3aea2a60
Caught exception in looking up space point ptr... 0x4fcadba0, 0x4fcb08d0
Caught exception in looking up space point ptr... 0x4fcb08d0, 0x4fcadba0
Caught exception in looking up space point ptr... 0x4b72f630, 0x4b72f530
Caught exception in looking up space point ptr... 0x4b72f530, 0x4b72f920
Caught exception in looking up space point ptr... 0x1a770ef0, 0x1a770140
Caught exception in looking up space point ptr... 0x1a770140, 0x1a770ef0
Caught exception in looking up space point ptr... 0x13b80b10, 0x13b80db0
Caught exception in looking up space point ptr... 0x13b80db0, 0x13b80b10
Caught exception in looking up space point ptr... 0x606e3110, 0x606e9600
Caught exception in looking up space point ptr... 0x606e9600, 0x606e3110
Caught exception in looking up space point ptr... 0x14762c10, 0x14765dd0
Caught exception in looking up space point ptr... 0x14765dd0, 0x14762c10
Caught exception in looking up space point ptr... 0xa8b1890, 0x395c8e70
Caught exception in looking up space point ptr... 0x395c8e70, 0x395c8b50
Caught exception in looking up space point ptr... 0x321aebf0, 0x3ea34cb0
Caught exception in looking up space point ptr... 0x3ea34cb0, 0x321aebf0
Caught exception in looking up space point ptr... 0x3ca2d3c0, 0x3ca2d4f0
Caught exception in looking up space point ptr... 0x3ca2d4f0, 0x3ca2d3c0
Caught exception in looking up space point ptr... 0x4abc9d70, 0x4abca2a0
Caught exception in looking up space point ptr... 0x4abca2a0, 0x4abc8ca0
Caught exception in looking up space point ptr... 0x33012190, 0x3f779bf0
Caught exception in looking up space point ptr... 0x3f779bf0, 0x33012190
Caught exception in looking up space point ptr... 0x4e7a4fd0, 0x4e7a40a0
Caught exception in looking up space point ptr... 0x4e7a40a0, 0x4e7a4fd0
Caught exception in looking up space point ptr... 0x46dc4830, 0x46dc4b50
Caught exception in looking up space point ptr... 0x46dc4b50, 0x46dc4830
Caught exception in looking up space point ptr... 0x40e97c60, 0x3bdcf420
Caught exception in looking up space point ptr... 0x3bdcf420, 0x40e97c60
Caught exception in looking up space point ptr... 0x40159db0, 0x40b017b0
Caught exception in looking up space point ptr... 0x40b017b0, 0x3840b600
Caught exception in looking up space point ptr... 0x46062960, 0x46064a80
Caught exception in looking up space point ptr... 0x46064a80, 0x46062c00
Caught exception in looking up space point ptr... 0x46062c00, 0x46064a80
Caught exception in looking up space point ptr... 0x46064a80, 0x46064bd0
Caught exception in looking up space point ptr... 0x7396c320, 0x7396c220
Caught exception in looking up space point ptr... 0x7396c220, 0x7396d790
Caught exception in looking up space point ptr... 0x11d1e980, 0x11d1e5c0
Caught exception in looking up space point ptr... 0x11d1e5c0, 0x11d1e980
Caught exception in looking up space point ptr... 0x867dfe10, 0x867dffa0
Caught exception in looking up space point ptr... 0x867dffa0, 0x867dfc80
Caught exception in looking up space point ptr... 0xaac19f0, 0xaac1450
Caught exception in looking up space point ptr... 0xaac1450, 0xaac19f0
Caught exception in looking up space point ptr... 0x33034ce0, 0x33034e70
Caught exception in looking up space point ptr... 0x33034e70, 0x33034ce0
Caught exception in looking up space point ptr... 0x539c8220, 0x539c8540
Caught exception in looking up space point ptr... 0x539c8540, 0x539c8220
Caught exception in looking up space point ptr... 0x3c8d00b0, 0x5397a5c0
Caught exception in looking up space point ptr... 0x5397a5c0, 0x11452ed0
Caught exception in looking up space point ptr... 0x33ae7630, 0x33ae6990
Caught exception in looking up space point ptr... 0x33ae6990, 0x33ae7630
Caught exception in looking up space point ptr... 0x3c21f800, 0x3c21fb20
Caught exception in looking up space point ptr... 0x3c21fb20, 0x3c21f800
++++>>>> total num hits: 139861, num free: 134298
04-Feb-2026 01:39:26 CET  Closed output file "np02vd_raw_run042433_0134_df-s04-d0_dw_0_20260203T193303_reco_stage0_20260204T003926_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         93.9828       105.065       130.265       105.452       9.12638        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                        5.618e-05    7.68737e-05   0.000205771    7.157e-05    3.17246e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                5.63402       6.0589        6.96215       6.02956       0.31928        19     
produce:triggerrawdecoder:PDVDTriggerReader4     0.00830964     0.0121248     0.0342756    0.00995355    0.00564847       19     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000397671   0.000540264   0.000772142   0.000513902   0.000103726      19     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.000356411   0.000533987   0.000816062   0.000479032   0.000143637      19     
produce:beamevent:BeamEvent                       6.129e-05     0.0001137    0.000784183   7.4311e-05    0.000158238      19     
produce:pdvddaphne:DAPHNEReaderPDVD                1.58024       1.93372       2.21847       1.97432      0.174705        19     
produce:ophit:OpHitFinder                         0.017025      0.0302384     0.0376949     0.0313736    0.00545351       19     
produce:wclsdatavd:WireCellToolkit                 65.5433       72.997        79.8429       72.0069       3.90477        19     
produce:gaushit:GausHitFinder                      0.66755       1.00948       1.76748      0.965425      0.251617        19     
produce:nhitsfilter:NumberOfHitsFilter           0.00016945    0.000234486   0.000548992   0.000213661   7.92754e-05      19     
produce:reco3d:SpacePointSolver                    8.82527       15.593         25.67        14.7041       4.62967        19     
produce:hitpdune:DisambigFromSpacePoints          0.0672848      0.17083      0.397749      0.149249      0.0798779       19     
produce:cluster3d:Cluster3D                       0.855708       2.29522       9.31469       1.85057       1.83624        19     
[art]:TriggerResults:TriggerResultInserter        2.396e-05    3.15353e-05    7.895e-05     2.809e-05     1.221e-05       19     
end_path:out1:RootOutput                           4.5e-06     5.82947e-06    1.958e-05     5.01e-06     3.2594e-06       19     
end_path:out1:RootOutput(write)                    3.66689       4.92333       6.40588       5.01515      0.708849        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5510.91 MB
  Peak resident set size usage (VmHWM): 3484.48 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0134_df-s04-d0_dw_0_20260203T193303_reco_stage0_20260204T003926_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0134_df-s04-d0_dw_0_20260203T193303_reco_stage0_20260204T003926_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0134_df-s04-d0_dw_0_20260203T193303_reco_stage0_20260204T003926_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770163532.0,
    "core.end_time": 1770165568.0,
    "core.events": [
      15286,
      15292,
      15298,
      15304,
      15310,
      15316,
      15322,
      15328,
      15334,
      15340,
      15346,
      15352,
      15358,
      15364,
      15370,
      15376,
      15382,
      15388,
      15394
    ],
    "core.event_count": 19,
    "core.first_event_number": 15286,
    "core.last_event_number": 15394,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0134_df-s04-d0_dw_0_20260203T193303.hdf5"
    }
  ]
}
justIN time: 2026-02-04 06:00:29 UTC       justIN version: 01.06.00