justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303647.52@dunegpschedd01.fnal.gov

Jobsub ID303647.52@dunegpschedd01.fnal.gov
Workflow ID12717
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:46:24
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce514
Last heartbeat2026-02-03 23:35:12
From worker nodeHostnameb9p08p6650.cern.ch
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 22:47:04
Input filesvd-protodune:np02vd_raw_run042421_0011_df-s04-d1_dw_0_20260203T133653.hdf5
JobscriptExit code0
Real time45m (2736s)
CPU time43m (2611s = 95%)
Max RSS bytes2729820160 (2603 MiB)
Outputting started2026-02-03 23:32:41
Output files
Finished2026-02-03 23:35:12
Saved logsjustin-logs:303647.52-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

, 0x57e750e0
Caught exception in looking up space point ptr... 0x57e750e0, 0x18482250
Caught exception in looking up space point ptr... 0x39b42880, 0x39b426f0
Caught exception in looking up space point ptr... 0x39b426f0, 0x5b0bc400
Caught exception in looking up space point ptr... 0x67fa1f70, 0x67fa47b0
Caught exception in looking up space point ptr... 0x67fa47b0, 0x67fa1f70
Caught exception in looking up space point ptr... 0x42726f40, 0x42726db0
Caught exception in looking up space point ptr... 0x42726db0, 0x42726a40
Caught exception in looking up space point ptr... 0x2c88d600, 0x2c88d2e0
Caught exception in looking up space point ptr... 0x2c88d2e0, 0x2c88d600
Caught exception in looking up space point ptr... 0x3c3466b0, 0x3c343e20
Caught exception in looking up space point ptr... 0x3c343e20, 0x3c3466b0
Caught exception in looking up space point ptr... 0x5e581100, 0x47ead660
Caught exception in looking up space point ptr... 0x47ead660, 0x5e581100
Caught exception in looking up space point ptr... 0x384bce30, 0x110ab5c0
Caught exception in looking up space point ptr... 0x110ab5c0, 0x384bce30
Caught exception in looking up space point ptr... 0x61fc33a0, 0x61fc3210
Caught exception in looking up space point ptr... 0x61fc3210, 0x61fc33a0
Caught exception in looking up space point ptr... 0x61f602b0, 0x61f600b0
Caught exception in looking up space point ptr... 0x61f600b0, 0x61f602b0
Caught exception in looking up space point ptr... 0x42726680, 0x42726540
Caught exception in looking up space point ptr... 0x42726540, 0x42725560
Caught exception in looking up space point ptr... 0x2db3b7b0, 0x2db3b620
Caught exception in looking up space point ptr... 0x2db3b620, 0x2db3b940
Caught exception in looking up space point ptr... 0x42728730, 0x42727cd0
Caught exception in looking up space point ptr... 0x42727cd0, 0x42727560
Caught exception in looking up space point ptr... 0x67fb09d0, 0x48927b20
Caught exception in looking up space point ptr... 0x48927b20, 0x4a1a65c0
Caught exception in looking up space point ptr... 0x4a1a65c0, 0x2d7bf8e0
Caught exception in looking up space point ptr... 0x2d7bf8e0, 0x67fb09d0
Caught exception in looking up space point ptr... 0x53466950, 0x53454340
Caught exception in looking up space point ptr... 0x53454340, 0x53466950
Caught exception in looking up space point ptr... 0x53466950, 0x534682b0
Caught exception in looking up space point ptr... 0x534682b0, 0x53466950
Caught exception in looking up space point ptr... 0x53466950, 0x417efc50
Caught exception in looking up space point ptr... 0x417efc50, 0x53466950
Caught exception in looking up space point ptr... 0x3c341d70, 0x3c341c20
Caught exception in looking up space point ptr... 0x3c341c20, 0x3c342010
Caught exception in looking up space point ptr... 0x480bf8d0, 0x480bf420
Caught exception in looking up space point ptr... 0x480bf420, 0x480bf8d0
Caught exception in looking up space point ptr... 0x5e679b20, 0x48917770
Caught exception in looking up space point ptr... 0x48917770, 0x4890b880
Caught exception in looking up space point ptr... 0x394ba5c0, 0x394b9180
Caught exception in looking up space point ptr... 0x394afe10, 0x394ba5c0
Caught exception in looking up space point ptr... 0x3c35dbf0, 0x129bc700
Caught exception in looking up space point ptr... 0x129bc700, 0x3c35dbf0
Caught exception in looking up space point ptr... 0x4483d910, 0x4483d780
Caught exception in looking up space point ptr... 0x4483d780, 0x4483d910
Caught exception in looking up space point ptr... 0x3682d5a0, 0x3682d7a0
Caught exception in looking up space point ptr... 0x3682d7a0, 0x3682d8f0
Caught exception in looking up space point ptr... 0x4483a8a0, 0x4483a710
Caught exception in looking up space point ptr... 0x4483a710, 0x4483a8a0
Caught exception in looking up space point ptr... 0x2d889190, 0xbd8bd10
Caught exception in looking up space point ptr... 0xbd8bd10, 0x2d888d10
Caught exception in looking up space point ptr... 0x5e58a490, 0x5e58a140
Caught exception in looking up space point ptr... 0x5e58a140, 0x5e589d10
Caught exception in looking up space point ptr... 0x58747ba0, 0x58747cf0
Caught exception in looking up space point ptr... 0x58747cf0, 0x58747ba0
Caught exception in looking up space point ptr... 0x2d29fc00, 0x2d29f980
Caught exception in looking up space point ptr... 0x2d29f980, 0x2d29fc00
Caught exception in looking up space point ptr... 0x2cbd9bd0, 0x2cbd7fb0
Caught exception in looking up space point ptr... 0x2cbd7fb0, 0x2cbd9bd0
Caught exception in looking up space point ptr... 0x5e67c5d0, 0x127c6300
Caught exception in looking up space point ptr... 0x127c6300, 0x127c2550
Caught exception in looking up space point ptr... 0x392b2930, 0x392b4e10
Caught exception in looking up space point ptr... 0x392b4e10, 0x392b2930
++++>>>> total num hits: 92809, num free: 88858
04-Feb-2026 00:32:34 CET  Closed output file "np02vd_raw_run042421_0011_df-s04-d1_dw_0_20260203T133653_reco_stage0_20260203T233234_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         74.4636       95.4753       117.199       94.7189       9.0344         28     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                        6.345e-05    9.7134e-05    0.000273423   8.4666e-05    4.14297e-05      28     
produce:tpcrawdecoder:PDVDTPCReader                4.12751       5.02707       11.8499       4.59801       1.52425        28     
produce:triggerrawdecoder:PDVDTriggerReader4      0.0101564     0.0185526     0.0423101     0.0140328    0.00947678       28     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000451655   0.000592091    0.0010055    0.000566867   0.000127821      28     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.00148523    0.00202998    0.00423316    0.00187543    0.000606956      28     
produce:beamevent:BeamEvent                      6.5031e-05    0.000142461   0.000954612   0.000115056   0.000159552      28     
produce:pdvddaphne:DAPHNEReaderPDVD                2.45952       3.08443       3.49098       3.07632      0.231421        28     
produce:ophit:OpHitFinder                         0.0267604     0.0381163      0.04521      0.0383633    0.00458399       28     
produce:wclsdatavd:WireCellToolkit                 54.6137       69.7762       82.8881       68.3186       6.29669        28     
produce:gaushit:GausHitFinder                     0.588867       2.26469       8.89803       1.1754        2.51238        28     
produce:nhitsfilter:NumberOfHitsFilter           0.000175952   0.000499569   0.00200621    0.000298388   0.000470462      28     
produce:reco3d:SpacePointSolver                    4.83597       10.965        21.4323       11.1126       4.52868        23     
produce:hitpdune:DisambigFromSpacePoints          0.0510618      0.11834      0.235984      0.109546      0.0470977       23     
produce:cluster3d:Cluster3D                       0.596867       2.05883       3.66863       1.92109      0.717042        23     
[art]:TriggerResults:TriggerResultInserter        1.757e-05    3.82419e-05   0.000147342   3.4386e-05    2.23405e-05      28     
end_path:out1:RootOutput                          3.34e-06     6.23607e-06    3.234e-05     4.745e-06    5.32094e-06      28     
end_path:out1:RootOutput(write)                    3.09631       4.44301       6.40451       4.4106       0.856217        28     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4722.12 MB
  Peak resident set size usage (VmHWM): 2729.82 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042421_0011_df-s04-d1_dw_0_20260203T133653_reco_stage0_20260203T233234_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042421_0011_df-s04-d1_dw_0_20260203T133653_reco_stage0_20260203T233234_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042421_0011_df-s04-d1_dw_0_20260203T133653_reco_stage0_20260203T233234_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770158838.0,
    "core.end_time": 1770161555.0,
    "core.events": [
      1805,
      1811,
      1817,
      1823,
      1829,
      1835,
      1841,
      1847,
      1853,
      1859,
      1865,
      1871,
      1877,
      1883,
      1889,
      1895,
      1901,
      1907,
      1913,
      1919,
      1925,
      1931,
      1937,
      1943,
      1949,
      1955,
      1961,
      1967
    ],
    "core.event_count": 28,
    "core.first_event_number": 1805,
    "core.last_event_number": 1967,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42421
    ],
    "core.runs_subruns": [
      4242100001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042421_0011_df-s04-d1_dw_0_20260203T133653.hdf5"
    }
  ]
}
justIN time: 2026-02-04 07:45:33 UTC       justIN version: 01.06.00