justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303647.174@dunegpschedd01.fnal.gov

Jobsub ID303647.174@dunegpschedd01.fnal.gov
Workflow ID12717
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:46:24
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce512
Last heartbeat2026-02-03 23:31:43
From worker nodeHostnameb9g24p4713.cern.ch
cpuinfoAMD EPYC 7302 16-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 22:46:57
Input filesvd-protodune:np02vd_raw_run042421_0088_df-s04-d1_dw_0_20260203T141838.hdf5
JobscriptExit code0
Real time42m (2545s)
CPU time40m (2435s = 95%)
Max RSS bytes2695221248 (2570 MiB)
Outputting started2026-02-03 23:29:23
Output files
Finished2026-02-03 23:31:43
Saved logsjustin-logs:303647.174-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

28f330
Caught exception in looking up space point ptr... 0xe501a00, 0xe529890
Caught exception in looking up space point ptr... 0x2893a2e0, 0x4a729570
Caught exception in looking up space point ptr... 0x4a729570, 0x64f169c0
Caught exception in looking up space point ptr... 0x28b09510, 0x1475f2f0
Caught exception in looking up space point ptr... 0x1475f2f0, 0x28b09510
Caught exception in looking up space point ptr... 0x35f4d0b0, 0x2b94d720
Caught exception in looking up space point ptr... 0x2b94d720, 0x35f4d0b0
Caught exception in looking up space point ptr... 0x57acc4a0, 0x57acc350
Caught exception in looking up space point ptr... 0x57acc350, 0x57acc4a0
Caught exception in looking up space point ptr... 0x34ece120, 0x29586bb0
Caught exception in looking up space point ptr... 0x29586bb0, 0x64eecc30
Caught exception in looking up space point ptr... 0x3cc511e0, 0x46f28010
Caught exception in looking up space point ptr... 0x46f28010, 0x46f281a0
Caught exception in looking up space point ptr... 0x1006cc60, 0x1006cdf0
Caught exception in looking up space point ptr... 0x48f33430, 0x1006cc60
Caught exception in looking up space point ptr... 0x4d0f5200, 0x14d962e0
Caught exception in looking up space point ptr... 0x14d962e0, 0x1503ec80
Caught exception in looking up space point ptr... 0x28debd40, 0x5aebdf50
Caught exception in looking up space point ptr... 0x5aebdf50, 0xc5d0c20
Caught exception in looking up space point ptr... 0xc5d0c20, 0x4d555170
Caught exception in looking up space point ptr... 0x2ac768a0, 0x365c2d90
Caught exception in looking up space point ptr... 0x365c2d90, 0x36695f10
Caught exception in looking up space point ptr... 0xa3a05a0, 0xef309b0
Caught exception in looking up space point ptr... 0xef309b0, 0xef31d40
Caught exception in looking up space point ptr... 0x2bfaf950, 0x59b1e500
Caught exception in looking up space point ptr... 0x59b1e500, 0x4a392800
Caught exception in looking up space point ptr... 0x59655ba0, 0x306a1880
Caught exception in looking up space point ptr... 0x306a1880, 0x306a1ad0
Caught exception in looking up space point ptr... 0x10081540, 0x10081220
Caught exception in looking up space point ptr... 0x10081220, 0x10081540
Caught exception in looking up space point ptr... 0x128b6400, 0x3c24c680
Caught exception in looking up space point ptr... 0x3c24c680, 0x66238610
Caught exception in looking up space point ptr... 0x2bb38030, 0x2bb37d10
Caught exception in looking up space point ptr... 0x2bb37d10, 0x3bfd8cf0
Caught exception in looking up space point ptr... 0x49beb300, 0x291302a0
Caught exception in looking up space point ptr... 0x472291f0, 0x49beb300
Caught exception in looking up space point ptr... 0x4d64c170, 0x15883aa0
Caught exception in looking up space point ptr... 0x15883aa0, 0x52a2a320
Caught exception in looking up space point ptr... 0x52a2a320, 0x14762920
Caught exception in looking up space point ptr... 0x5a069420, 0x5a068f70
Caught exception in looking up space point ptr... 0x5a068f70, 0x5a069420
Caught exception in looking up space point ptr... 0x47e76c50, 0x32497a50
Caught exception in looking up space point ptr... 0x32497a50, 0x47e76c50
Caught exception in looking up space point ptr... 0x335bbac0, 0x3093e300
Caught exception in looking up space point ptr... 0x3093e300, 0xc5c15a0
Caught exception in looking up space point ptr... 0x5e8f5bc0, 0x28914750
Caught exception in looking up space point ptr... 0x28914750, 0x28912440
Caught exception in looking up space point ptr... 0x3663f940, 0x14728f80
Caught exception in looking up space point ptr... 0x14728f80, 0x266231c0
Caught exception in looking up space point ptr... 0x3c18f270, 0x2d1bdd70
Caught exception in looking up space point ptr... 0x2d1bdd70, 0x24d348a0
Caught exception in looking up space point ptr... 0x3cbe3c10, 0x3cbe3a10
Caught exception in looking up space point ptr... 0x3cbe3a10, 0x3cbe3c10
Caught exception in looking up space point ptr... 0x5be041e0, 0x2195e1e0
Caught exception in looking up space point ptr... 0x2195e1e0, 0x5d4fc310
Caught exception in looking up space point ptr... 0x5d4fc310, 0x292016d0
Caught exception in looking up space point ptr... 0x292016d0, 0x46f5b240
Caught exception in looking up space point ptr... 0x46f5b240, 0x15891b40
Caught exception in looking up space point ptr... 0x15891b40, 0x5be041e0
Caught exception in looking up space point ptr... 0x87e15a0, 0x87e1450
Caught exception in looking up space point ptr... 0x87e1450, 0x87e15a0
Caught exception in looking up space point ptr... 0x4ee6e9c0, 0x4ee6eb50
Caught exception in looking up space point ptr... 0x4ee6eb50, 0x4ee6e6a0
Caught exception in looking up space point ptr... 0x4ee6e380, 0x4ee6e510
Caught exception in looking up space point ptr... 0x4ee6e510, 0x4ee6e380
++++>>>> total num hits: 43601, num free: 41775
04-Feb-2026 00:29:17 CET  Closed output file "np02vd_raw_run042421_0088_df-s04-d1_dw_0_20260203T141838_reco_stage0_20260203T232917_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         74.3224       91.7564       108.476       93.0359       9.45517        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       5.3441e-05    0.000104681   0.000166883   9.5352e-05    3.73908e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                4.01161       4.97284       10.854        4.48645       1.64684        27     
produce:triggerrawdecoder:PDVDTriggerReader4     0.00871626     0.0192974     0.0609076     0.0124029     0.0144767       27     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000341847   0.000495957   0.000818326   0.000445649   0.000123927      27     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.00125565    0.00158953    0.00225242    0.00155458    0.00022111       27     
produce:beamevent:BeamEvent                      6.0952e-05    0.000138792   0.00110303    9.1822e-05    0.000191952      27     
produce:pdvddaphne:DAPHNEReaderPDVD                2.21013       2.81051       3.3135        2.85665      0.284502        27     
produce:ophit:OpHitFinder                         0.0234005     0.0350619     0.0455351     0.0347208    0.00655007       27     
produce:wclsdatavd:WireCellToolkit                 56.3506       65.9912       73.444        67.0902       4.68066        27     
produce:gaushit:GausHitFinder                     0.473988      0.883444       1.49095      0.815842      0.224865        27     
produce:nhitsfilter:NumberOfHitsFilter           0.000180403   0.000317815   0.000733104   0.000294806   0.000119193      27     
produce:reco3d:SpacePointSolver                    5.41768       11.0377       20.726        9.11549       4.15986        27     
produce:hitpdune:DisambigFromSpacePoints          0.063459      0.108729      0.242392      0.0992223     0.0395489       27     
produce:cluster3d:Cluster3D                       0.851108       2.00973       4.28142       1.96044      0.898898        27     
[art]:TriggerResults:TriggerResultInserter        2.379e-05    3.3492e-05    0.000143153    2.831e-05    2.21239e-05      27     
end_path:out1:RootOutput                          3.74e-06     8.52085e-06    3.79e-05      8.081e-06    6.17652e-06      27     
end_path:out1:RootOutput(write)                    3.19379       3.86505       4.8663        3.80281      0.506796        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4569.01 MB
  Peak resident set size usage (VmHWM): 2695.22 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042421_0088_df-s04-d1_dw_0_20260203T141838_reco_stage0_20260203T232917_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042421_0088_df-s04-d1_dw_0_20260203T141838_reco_stage0_20260203T232917_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042421_0088_df-s04-d1_dw_0_20260203T141838_reco_stage0_20260203T232917_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770158828.0,
    "core.end_time": 1770161358.0,
    "core.events": [
      14321,
      14327,
      14333,
      14339,
      14345,
      14351,
      14357,
      14363,
      14369,
      14375,
      14381,
      14387,
      14393,
      14399,
      14405,
      14411,
      14417,
      14423,
      14429,
      14435,
      14441,
      14447,
      14453,
      14459,
      14465,
      14471,
      14477
    ],
    "core.event_count": 27,
    "core.first_event_number": 14321,
    "core.last_event_number": 14477,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42421
    ],
    "core.runs_subruns": [
      4242100001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042421_0088_df-s04-d1_dw_0_20260203T141838.hdf5"
    }
  ]
}
justIN time: 2026-02-04 10:46:54 UTC       justIN version: 01.06.00