justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303652.136@dunegpschedd01.fnal.gov

Jobsub ID303652.136@dunegpschedd01.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:50:26
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce511
Last heartbeat2026-02-04 00:17:22
From worker nodeHostnameb9p02p3983.cern.ch
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:30:04
Input filesvd-protodune:np02vd_raw_run042433_0174_df-s04-d1_dw_0_20260203T194815.hdf5
JobscriptExit code0
Real time44m (2694s)
CPU time43m (2602s = 96%)
Max RSS bytes3392032768 (3234 MiB)
Outputting started2026-02-04 00:15:00
Output files
Finished2026-02-04 00:17:22
Saved logsjustin-logs:303652.136-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

r... 0x12023280, 0x58295750
Caught exception in looking up space point ptr... 0x6e2ee430, 0x11e17b40
Caught exception in looking up space point ptr... 0x6e2edb20, 0x6e2ee430
Caught exception in looking up space point ptr... 0x1320b180, 0x13205160
Caught exception in looking up space point ptr... 0x13205160, 0x13207900
Caught exception in looking up space point ptr... 0x13207900, 0x1320b2b0
Caught exception in looking up space point ptr... 0x12105900, 0x3c0ac190
Caught exception in looking up space point ptr... 0x3c0ac190, 0x6d11cd10
Caught exception in looking up space point ptr... 0x2ab517f0, 0x11d879c0
Caught exception in looking up space point ptr... 0x11d879c0, 0x34e96f80
Caught exception in looking up space point ptr... 0x3ff9ff90, 0x72e6e510
Caught exception in looking up space point ptr... 0x72e6e510, 0x39320970
Caught exception in looking up space point ptr... 0x378de6f0, 0x5427b4c0
Caught exception in looking up space point ptr... 0x5427b4c0, 0x3f02f340
Caught exception in looking up space point ptr... 0x6e2d9d10, 0x33dadc30
Caught exception in looking up space point ptr... 0x33dadc30, 0x34f37840
Caught exception in looking up space point ptr... 0x34f37840, 0x34f34ab0
Caught exception in looking up space point ptr... 0x34f34ab0, 0x5965b6d0
Caught exception in looking up space point ptr... 0x5965b6d0, 0x3c7e0d80
Caught exception in looking up space point ptr... 0x3668cbf0, 0x3668ccf0
Caught exception in looking up space point ptr... 0x3668ccf0, 0x3668cbf0
Caught exception in looking up space point ptr... 0x63d11af0, 0x63d12270
Caught exception in looking up space point ptr... 0x63d12270, 0x402283c0
Caught exception in looking up space point ptr... 0x33870880, 0x40224190
Caught exception in looking up space point ptr... 0x40224190, 0x40223e70
Caught exception in looking up space point ptr... 0x40223e70, 0x338703d0
Caught exception in looking up space point ptr... 0x37f35710, 0x37f355c0
Caught exception in looking up space point ptr... 0x37f355c0, 0x37f35710
Caught exception in looking up space point ptr... 0x5a1ee150, 0x5a1ede70
Caught exception in looking up space point ptr... 0x5a1ede70, 0x5a1ee150
Caught exception in looking up space point ptr... 0x17b50070, 0x66bdf560
Caught exception in looking up space point ptr... 0x66bdf560, 0x4b9a21c0
Caught exception in looking up space point ptr... 0x6186e0c0, 0x6186bf00
Caught exception in looking up space point ptr... 0x6186bf00, 0x6186e0c0
Caught exception in looking up space point ptr... 0x322cdee0, 0x322d7030
Caught exception in looking up space point ptr... 0x322d7030, 0x322cdee0
Caught exception in looking up space point ptr... 0x153fdd80, 0x153fdfe0
Caught exception in looking up space point ptr... 0x153fdfe0, 0xb2998e0
Caught exception in looking up space point ptr... 0x582ab030, 0x582ab4e0
Caught exception in looking up space point ptr... 0x582ab4e0, 0x582ab030
Caught exception in looking up space point ptr... 0x582a0a80, 0x33e87cc0
Caught exception in looking up space point ptr... 0x33e87cc0, 0x582a0a80
Caught exception in looking up space point ptr... 0x37b3fb40, 0x37b3dc70
Caught exception in looking up space point ptr... 0x37b3dc70, 0x5f438e50
Caught exception in looking up space point ptr... 0x5f438e50, 0x37b429b0
Caught exception in looking up space point ptr... 0x37b429b0, 0x37b3fb40
Caught exception in looking up space point ptr... 0x5f4745a0, 0x5f474450
Caught exception in looking up space point ptr... 0x5f474450, 0x5f4745a0
Caught exception in looking up space point ptr... 0x5f43b840, 0x5f43b740
Caught exception in looking up space point ptr... 0x5f43b740, 0x5f43b840
Caught exception in looking up space point ptr... 0x879b44e0, 0x3adcea30
Caught exception in looking up space point ptr... 0x3adcea30, 0x3c2cdcd0
Caught exception in looking up space point ptr... 0xa34b1d0, 0x879b44e0
Caught exception in looking up space point ptr... 0x59687c60, 0x59681900
Caught exception in looking up space point ptr... 0x59681900, 0x596866f0
Caught exception in looking up space point ptr... 0x1a804f60, 0x6350cdb0
Caught exception in looking up space point ptr... 0x6350cdb0, 0x5824c660
Caught exception in looking up space point ptr... 0x48719830, 0x88028fa0
Caught exception in looking up space point ptr... 0x673b9ea0, 0x48719830
Caught exception in looking up space point ptr... 0x63bf6770, 0x63bf9ab0
Caught exception in looking up space point ptr... 0x63bf9ab0, 0x63bf6770
Caught exception in looking up space point ptr... 0x63bf6770, 0x5548c7e0
Caught exception in looking up space point ptr... 0x5548c7e0, 0x63bf6770
Caught exception in looking up space point ptr... 0x331083e0, 0x16300d70
Caught exception in looking up space point ptr... 0x16300d70, 0x16300f40
Caught exception in looking up space point ptr... 0x14e61900, 0x14e616a0
Caught exception in looking up space point ptr... 0x14e616a0, 0x14e61900
++++>>>> total num hits: 79185, num free: 76336
04-Feb-2026 01:14:51 CET  Closed output file "np02vd_raw_run042433_0174_df-s04-d1_dw_0_20260203T194815_reco_stage0_20260204T001451_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         119.831       138.706       156.998       136.731       11.5967        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       5.7671e-05    9.85199e-05   0.000172991   8.9331e-05    2.86284e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                5.60479       6.24675       6.74285        6.262       0.309374        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.0110999     0.0327543     0.0893473     0.0258032     0.0214515       19     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000403751    0.0027392     0.0117561    0.000751872   0.00325666       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.000373931   0.000590914   0.000916421   0.000569402   0.000145679      19     
produce:beamevent:BeamEvent                       6.702e-05    0.000152814   0.000918833   0.000110611   0.000181744      19     
produce:pdvddaphne:DAPHNEReaderPDVD                2.09317       2.70567       3.33074       2.76194      0.337231        19     
produce:ophit:OpHitFinder                         0.021196      0.0346798     0.0418851     0.0351702    0.00482435       19     
produce:wclsdatavd:WireCellToolkit                 91.365        102.024       119.101       98.4122       8.44397        19     
produce:gaushit:GausHitFinder                     0.722102       1.25703       2.14544       1.24366      0.367803        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000245651   0.000334362   0.000545921   0.000320551   7.42845e-05      19     
produce:reco3d:SpacePointSolver                    10.1589       18.808        25.7973       20.1831       5.09515        19     
produce:hitpdune:DisambigFromSpacePoints          0.0757104     0.155755      0.233182      0.170965      0.047082        19     
produce:cluster3d:Cluster3D                       0.981179       2.24331       3.64745       2.45634      0.766156        19     
[art]:TriggerResults:TriggerResultInserter        2.438e-05    5.01612e-05   0.00021246     3.279e-05    4.59294e-05      19     
end_path:out1:RootOutput                          3.44e-06     7.28158e-06    2.965e-05     6.92e-06     5.55605e-06      19     
end_path:out1:RootOutput(write)                    4.30905       5.16052       6.10653       5.1083       0.546187        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5362.9 MB
  Peak resident set size usage (VmHWM): 3392.03 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0174_df-s04-d1_dw_0_20260203T194815_reco_stage0_20260204T001451_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0174_df-s04-d1_dw_0_20260203T194815_reco_stage0_20260204T001451_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0174_df-s04-d1_dw_0_20260203T194815_reco_stage0_20260204T001451_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770161416.0,
    "core.end_time": 1770164093.0,
    "core.events": [
      19847,
      19853,
      19859,
      19865,
      19871,
      19877,
      19883,
      19889,
      19895,
      19901,
      19907,
      19913,
      19919,
      19925,
      19931,
      19937,
      19943,
      19949,
      19955
    ],
    "core.event_count": 19,
    "core.first_event_number": 19847,
    "core.last_event_number": 19955,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0174_df-s04-d1_dw_0_20260203T194815.hdf5"
    }
  ]
}
justIN time: 2026-02-04 09:14:46 UTC       justIN version: 01.06.00