justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297923.139@dunegpschedd02.fnal.gov

Jobsub ID297923.139@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 23:00:26
SiteIT_CNAF
EntryCMSHTPC_T1_IT_CNAF_condor_ce02
Last heartbeat2026-02-04 01:48:36
From worker nodeHostnamewn-f01-15-05-05.cr.cnaf.infn.it
cpuinfoIntel(R) Xeon(R) CPU E5-2697 v4 @ 2.30GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-04 00:05:07
Input filesvd-protodune:np02vd_raw_run042433_0145_df-s04-d1_dw_0_20260203T193714.hdf5
JobscriptExit code0
Real time1h (6073s)
CPU time1h (5747s = 94%)
Max RSS bytes3532296192 (3368 MiB)
Outputting started2026-02-04 01:46:22
Output files
Finished2026-02-04 01:48:36
Saved logsjustin-logs:297923.139-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

t ptr... 0x6b508c90, 0x6b508b00
Caught exception in looking up space point ptr... 0x6b508b00, 0x6b508c90
Caught exception in looking up space point ptr... 0x6b5628a0, 0x6b562750
Caught exception in looking up space point ptr... 0x6b562750, 0x6b5628a0
Caught exception in looking up space point ptr... 0x6b56f9c0, 0x6b56f870
Caught exception in looking up space point ptr... 0x6b56f870, 0x6b56f9c0
Caught exception in looking up space point ptr... 0x6b6056b0, 0x6b605840
Caught exception in looking up space point ptr... 0x6b605840, 0x6b6056b0
Caught exception in looking up space point ptr... 0x11ed3170, 0x6b55df00
Caught exception in looking up space point ptr... 0x6b55df00, 0x6b506e50
Caught exception in looking up space point ptr... 0x32b906e0, 0x32b5c390
Caught exception in looking up space point ptr... 0x32b5c390, 0x32b906e0
Caught exception in looking up space point ptr... 0x547c49d0, 0x64c2b3a0
Caught exception in looking up space point ptr... 0x64c2b3a0, 0x343d7420
Caught exception in looking up space point ptr... 0x343d7420, 0x404ea160
Caught exception in looking up space point ptr... 0x404ea160, 0x404ea2f0
Caught exception in looking up space point ptr... 0x11d737a0, 0x54e83cc0
Caught exception in looking up space point ptr... 0x54e83cc0, 0x54e83dc0
Caught exception in looking up space point ptr... 0x11ed1de0, 0x11ed2b60
Caught exception in looking up space point ptr... 0x11ed2b60, 0x11ed1de0
Caught exception in looking up space point ptr... 0x11ed5c20, 0x11ed5a90
Caught exception in looking up space point ptr... 0x11ed5a90, 0x11ed5c20
Caught exception in looking up space point ptr... 0x6b601830, 0x11e34690
Caught exception in looking up space point ptr... 0x11e34690, 0x6b601830
Caught exception in looking up space point ptr... 0x3ddd48a0, 0x37aa4a00
Caught exception in looking up space point ptr... 0x37aa4a00, 0x37a2a770
Caught exception in looking up space point ptr... 0x1425eb00, 0x170f14b0
Caught exception in looking up space point ptr... 0x170f14b0, 0x12b46a70
Caught exception in looking up space point ptr... 0x3ef3d290, 0x5e5ed2f0
Caught exception in looking up space point ptr... 0x5e5ed2f0, 0x3ef3d290
Caught exception in looking up space point ptr... 0x11e29a10, 0x11e1b0d0
Caught exception in looking up space point ptr... 0x6b5de5a0, 0x11e29a10
Caught exception in looking up space point ptr... 0x6b604bc0, 0x11e1eb00
Caught exception in looking up space point ptr... 0x11e1eb00, 0x6b604bc0
Caught exception in looking up space point ptr... 0x7ff36630, 0x7ff364f0
Caught exception in looking up space point ptr... 0x7ff364f0, 0x7ff385c0
Caught exception in looking up space point ptr... 0x7ff385c0, 0x7ff384c0
Caught exception in looking up space point ptr... 0x11e18630, 0x11df9c60
Caught exception in looking up space point ptr... 0x11df9c60, 0x6b603db0
Caught exception in looking up space point ptr... 0x11e1bd80, 0x11df1820
Caught exception in looking up space point ptr... 0x6b5dbe70, 0x11e1bd80
Caught exception in looking up space point ptr... 0x6b607460, 0x11dfac50
Caught exception in looking up space point ptr... 0x11dfac50, 0x6b607460
Caught exception in looking up space point ptr... 0x2bb019c0, 0x2bb01830
Caught exception in looking up space point ptr... 0x2bb01830, 0x2bb019c0
Caught exception in looking up space point ptr... 0x11dfbff0, 0x11e1dc00
Caught exception in looking up space point ptr... 0x11e1dc00, 0x6b56df00
Caught exception in looking up space point ptr... 0x6b605070, 0x11e16c70
Caught exception in looking up space point ptr... 0x11e16c70, 0x6b605070
Caught exception in looking up space point ptr... 0x6b603900, 0x6b603770
Caught exception in looking up space point ptr... 0x6b603770, 0x6b603900
Caught exception in looking up space point ptr... 0x6b602af0, 0x11e18e40
Caught exception in looking up space point ptr... 0x11e18e40, 0x6b602af0
Caught exception in looking up space point ptr... 0x8b3fef0, 0x8b40080
Caught exception in looking up space point ptr... 0x8b40080, 0x8b3fef0
Caught exception in looking up space point ptr... 0x8b3fef0, 0x8b3fd60
Caught exception in looking up space point ptr... 0x8b3fd60, 0x8b3fef0
Caught exception in looking up space point ptr... 0x1270d100, 0x1270d000
Caught exception in looking up space point ptr... 0x439fb5e0, 0x1270d100
Caught exception in looking up space point ptr... 0x30a65140, 0x7ec77160
Caught exception in looking up space point ptr... 0x7ec77160, 0x30a65140
Caught exception in looking up space point ptr... 0x4a586ca0, 0x318fcfe0
Caught exception in looking up space point ptr... 0x318fcfe0, 0x1599fe50
Caught exception in looking up space point ptr... 0x5efcfd50, 0x5af57110
Caught exception in looking up space point ptr... 0x5af57110, 0x5af5b660
Caught exception in looking up space point ptr... 0x59c51ed0, 0x59c51d40
Caught exception in looking up space point ptr... 0x59c51d40, 0x59c51bb0
++++>>>> total num hits: 831917, num free: 812142
04-Feb-2026 02:46:07 CET  Closed output file "np02vd_raw_run042433_0145_df-s04-d1_dw_0_20260203T193714_reco_stage0_20260204T014607_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         245.494       313.564       500.594       286.84        83.6472        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       7.4509e-05    0.000149758   0.00029596    0.000118888   5.90131e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                16.4781       18.828        27.1562       18.4614       2.23743        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.317536      0.394295      0.636182      0.368376      0.0769646       19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0103472     0.0142981     0.0289283     0.0114958     0.004777        19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.010286      0.0104918     0.0109864     0.0104465    0.000179063      19     
produce:beamevent:BeamEvent                      9.0363e-05    0.00022188    0.00126271    0.000182503   0.000248544      19     
produce:pdvddaphne:DAPHNEReaderPDVD                4.98276       6.72747       8.5237        6.84808      0.758641        19     
produce:ophit:OpHitFinder                         0.0356028     0.0528583     0.0651855     0.0558014    0.00881524       19     
produce:wclsdatavd:WireCellToolkit                 174.679       211.245       246.852       205.821       21.0818        19     
produce:gaushit:GausHitFinder                      1.12174       2.66044       4.67399       2.52316      0.998139        19     
produce:nhitsfilter:NumberOfHitsFilter           0.00037576    0.000706692   0.00147946    0.000653481   0.000242278      19     
produce:reco3d:SpacePointSolver                    12.5381       22.7797       33.4212       22.5678       5.7577         19     
produce:hitpdune:DisambigFromSpacePoints          0.119519      0.485946       1.0812       0.427898       0.25073        19     
produce:cluster3d:Cluster3D                        1.39002       40.4125       188.672       4.50509       70.6829        19     
[art]:TriggerResults:TriggerResultInserter       2.5685e-05    4.49521e-05   8.5111e-05    3.9424e-05    1.62803e-05      19     
end_path:out1:RootOutput                          8.202e-06    1.21796e-05   2.5078e-05    1.0384e-05    3.93304e-06      19     
end_path:out1:RootOutput(write)                    5.31674       9.91198       15.4494       9.66835       2.69971        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5738.68 MB
  Peak resident set size usage (VmHWM): 3532.3 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0145_df-s04-d1_dw_0_20260203T193714_reco_stage0_20260204T014607_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0145_df-s04-d1_dw_0_20260203T193714_reco_stage0_20260204T014607_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0145_df-s04-d1_dw_0_20260203T193714_reco_stage0_20260204T014607_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770163523.0,
    "core.end_time": 1770169570.0,
    "core.events": [
      16541,
      16547,
      16553,
      16559,
      16565,
      16571,
      16577,
      16583,
      16589,
      16595,
      16601,
      16607,
      16613,
      16619,
      16625,
      16631,
      16637,
      16643,
      16649
    ],
    "core.event_count": 19,
    "core.first_event_number": 16541,
    "core.last_event_number": 16649,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0145_df-s04-d1_dw_0_20260203T193714.hdf5"
    }
  ]
}
justIN time: 2026-02-04 04:49:07 UTC       justIN version: 01.06.00