justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297919.60@dunegpschedd02.fnal.gov

Jobsub ID297919.60@dunegpschedd02.fnal.gov
Workflow ID12717
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:54:25
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce515
Last heartbeat2026-02-04 00:14:47
From worker nodeHostnameb9p12p0201.cern.ch
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:30:45
Input filesvd-protodune:np02vd_raw_run042421_0112_df-s03-d0_dw_0_20260203T143138.hdf5
JobscriptExit code0
Real time41m (2491s)
CPU time39m (2377s = 95%)
Max RSS bytes2731515904 (2604 MiB)
Outputting started2026-02-04 00:12:18
Output files
Finished2026-02-04 00:14:47
Saved logsjustin-logs:297919.60-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

190
Caught exception in looking up space point ptr... 0x2f183190, 0x42c6b3b0
Caught exception in looking up space point ptr... 0x52d96e00, 0x2cd56c40
Caught exception in looking up space point ptr... 0x53ab58b0, 0x4ae5b510
Caught exception in looking up space point ptr... 0x4ae5b510, 0x4fbd4fc0
Caught exception in looking up space point ptr... 0x2ebaa510, 0x2ebaa380
Caught exception in looking up space point ptr... 0x2ebaa380, 0x13aeb5b0
Caught exception in looking up space point ptr... 0x529975c0, 0x427e60f0
Caught exception in looking up space point ptr... 0x427e60f0, 0x13ab7710
Caught exception in looking up space point ptr... 0xeda8fa0, 0xedad200
Caught exception in looking up space point ptr... 0xeda5480, 0xeda8fa0
Caught exception in looking up space point ptr... 0x17472270, 0x26648170
Caught exception in looking up space point ptr... 0x17478730, 0x17472270
Caught exception in looking up space point ptr... 0x2ab544d0, 0x41d816e0
Caught exception in looking up space point ptr... 0x331f9b50, 0x2ab544d0
Caught exception in looking up space point ptr... 0x905eb20, 0x26582000
Caught exception in looking up space point ptr... 0x26582000, 0x103684f0
Caught exception in looking up space point ptr... 0x103684f0, 0x49afdd90
Caught exception in looking up space point ptr... 0x28a98d30, 0x4f14dfb0
Caught exception in looking up space point ptr... 0x4f14dfb0, 0x4f14e140
Caught exception in looking up space point ptr... 0x2f2aca20, 0x2f29fbe0
Caught exception in looking up space point ptr... 0x2f2acbb0, 0x2f2aca20
Caught exception in looking up space point ptr... 0x3c36b010, 0x4405cb10
Caught exception in looking up space point ptr... 0x4405cb10, 0x3c3826e0
Caught exception in looking up space point ptr... 0x4b576950, 0x4b564b60
Caught exception in looking up space point ptr... 0x4b564b60, 0x4e4a8ee0
Caught exception in looking up space point ptr... 0x4b575050, 0x4b563260
Caught exception in looking up space point ptr... 0x4b563260, 0x4e4a75e0
Caught exception in looking up space point ptr... 0x4b322230, 0x4b311a20
Caught exception in looking up space point ptr... 0x4b311890, 0x4b322230
Caught exception in looking up space point ptr... 0x5807dfd0, 0x3542cad0
Caught exception in looking up space point ptr... 0x3542cad0, 0x40bfab40
Caught exception in looking up space point ptr... 0x4f00c300, 0xa9a9f30
Caught exception in looking up space point ptr... 0xa9a9f30, 0x550cdc60
Caught exception in looking up space point ptr... 0xfd12e70, 0x52875670
Caught exception in looking up space point ptr... 0x2e6239c0, 0xfd12e70
Caught exception in looking up space point ptr... 0x44b5dd80, 0x44b4e330
Caught exception in looking up space point ptr... 0x53a13a90, 0x44b5dd80
Caught exception in looking up space point ptr... 0x420ea210, 0x175b5920
Caught exception in looking up space point ptr... 0x175b5920, 0x4a711270
Caught exception in looking up space point ptr... 0x30554890, 0x30542780
Caught exception in looking up space point ptr... 0x60f1c890, 0x30554890
Caught exception in looking up space point ptr... 0x3f7082b0, 0x3f6f6330
Caught exception in looking up space point ptr... 0x3f6f6330, 0x2f305200
Caught exception in looking up space point ptr... 0x3f70a830, 0x3f6f88b0
Caught exception in looking up space point ptr... 0x3f6f88b0, 0x2f307780
Caught exception in looking up space point ptr... 0x602cd2e0, 0x4c5d7220
Caught exception in looking up space point ptr... 0x4c5d7220, 0x602cd2e0
Caught exception in looking up space point ptr... 0x5ef961e0, 0x5ef95f80
Caught exception in looking up space point ptr... 0x5ef95f80, 0x5ef961e0
Caught exception in looking up space point ptr... 0x5ef96e60, 0x5ef96c00
Caught exception in looking up space point ptr... 0x5ef96c00, 0x5ef96900
Caught exception in looking up space point ptr... 0x4a754a90, 0x4fbccaa0
Caught exception in looking up space point ptr... 0x4fbccaa0, 0xed6fd00
Caught exception in looking up space point ptr... 0x133deef0, 0x54eb6250
Caught exception in looking up space point ptr... 0x54eb6250, 0x133deef0
Caught exception in looking up space point ptr... 0x311b1490, 0x311b12c0
Caught exception in looking up space point ptr... 0x311b12c0, 0x3eaf1190
Caught exception in looking up space point ptr... 0x3cf054d0, 0x303c72d0
Caught exception in looking up space point ptr... 0x303c72d0, 0x17a3e900
Caught exception in looking up space point ptr... 0x3b01e5e0, 0x2aadcd70
Caught exception in looking up space point ptr... 0x2aadcd70, 0x3b01e5e0
Caught exception in looking up space point ptr... 0x325434e0, 0x45c09aa0
Caught exception in looking up space point ptr... 0x45c09aa0, 0x325434e0
Caught exception in looking up space point ptr... 0x3156f3b0, 0xa2c3aa0
Caught exception in looking up space point ptr... 0xa2c3aa0, 0x3156f3b0
++++>>>> total num hits: 303642, num free: 296883
04-Feb-2026 01:12:11 CET  Closed output file "np02vd_raw_run042421_0112_df-s03-d0_dw_0_20260203T143138_reco_stage0_20260204T001211_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         67.1955       89.866        114.963       94.5062       12.3038        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                        4.685e-05    8.23634e-05   0.000336631    6.046e-05    5.95035e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                3.54386       4.25168       10.2419       3.92907       1.23121        27     
produce:triggerrawdecoder:PDVDTriggerReader4     0.00960431     0.0163189     0.0485982     0.0118395     0.0103194       27     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000346981   0.000457885   0.000877793   0.000448021   0.000100743      27     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.00131028    0.00168028    0.00256977    0.00154849    0.000317657      27     
produce:beamevent:BeamEvent                       4.295e-05    0.000113537   0.000888222    6.871e-05    0.000157305      27     
produce:pdvddaphne:DAPHNEReaderPDVD                1.95755       2.19403       2.64253       2.21319      0.159452        27     
produce:ophit:OpHitFinder                         0.0229153     0.0314543     0.0379489     0.0316139    0.00320168       27     
produce:wclsdatavd:WireCellToolkit                 51.2729       53.9006       67.0794       53.2612       3.06667        27     
produce:gaushit:GausHitFinder                     0.515886      0.945662       1.29443      0.947555      0.183007        27     
produce:nhitsfilter:NumberOfHitsFilter           0.00015418    0.000221824   0.000409801   0.000217891   4.79171e-05      27     
produce:reco3d:SpacePointSolver                    3.73712       5.72169       9.15858       5.30909       1.33305        27     
produce:hitpdune:DisambigFromSpacePoints          0.0525931     0.131372      0.225583      0.131899      0.0456277       27     
produce:cluster3d:Cluster3D                       0.753189       19.1725       30.5582       25.8924       11.6563        27     
[art]:TriggerResults:TriggerResultInserter       2.0731e-05    3.09708e-05   0.000116851    2.714e-05    1.71123e-05      27     
end_path:out1:RootOutput                          3.69e-06      7.48e-06      3.371e-05     6.99e-06     5.3976e-06       27     
end_path:out1:RootOutput(write)                    2.69685       3.47973       4.09374       3.58031      0.343492        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4673.91 MB
  Peak resident set size usage (VmHWM): 2731.52 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042421_0112_df-s03-d0_dw_0_20260203T143138_reco_stage0_20260204T001211_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042421_0112_df-s03-d0_dw_0_20260203T143138_reco_stage0_20260204T001211_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042421_0112_df-s03-d0_dw_0_20260203T143138_reco_stage0_20260204T001211_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770161459.0,
    "core.end_time": 1770163932.0,
    "core.events": [
      18218,
      18224,
      18230,
      18236,
      18242,
      18248,
      18254,
      18260,
      18266,
      18272,
      18278,
      18284,
      18290,
      18296,
      18302,
      18308,
      18314,
      18320,
      18326,
      18332,
      18338,
      18344,
      18350,
      18356,
      18362,
      18368,
      18374
    ],
    "core.event_count": 27,
    "core.first_event_number": 18218,
    "core.last_event_number": 18374,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42421
    ],
    "core.runs_subruns": [
      4242100001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042421_0112_df-s03-d0_dw_0_20260203T143138.hdf5"
    }
  ]
}
justIN time: 2026-02-04 06:06:08 UTC       justIN version: 01.06.00