justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303647.22@dunegpschedd01.fnal.gov

Jobsub ID303647.22@dunegpschedd01.fnal.gov
Workflow ID12717
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:46:24
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce514
Last heartbeat2026-02-03 23:33:23
From worker nodeHostnameb9p08p6650.cern.ch
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 22:47:36
Input filesvd-protodune:np02vd_raw_run042421_0044_df-s04-d0_dw_0_20260203T135447.hdf5
JobscriptExit code0
Real time43m (2599s)
CPU time41m (2503s = 96%)
Max RSS bytes2585640960 (2465 MiB)
Outputting started2026-02-03 23:30:57
Output files
Finished2026-02-03 23:33:23
Saved logsjustin-logs:303647.22-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

space point ptr... 0x4a44b4c0, 0x4a44b760
Caught exception in looking up space point ptr... 0x2bca9a50, 0x621f7300
Caught exception in looking up space point ptr... 0x621f7300, 0x30d48b40
Caught exception in looking up space point ptr... 0x40925560, 0x3e5b8990
Caught exception in looking up space point ptr... 0x4092a090, 0x40925560
Caught exception in looking up space point ptr... 0x4a3344d0, 0x4b8f4d20
Caught exception in looking up space point ptr... 0x4b8f4d20, 0x32042640
Caught exception in looking up space point ptr... 0x5aa16b50, 0x5871bf70
Caught exception in looking up space point ptr... 0x5871bf70, 0x5aa0a250
Caught exception in looking up space point ptr... 0xa4cbdc0, 0x2df170d0
Caught exception in looking up space point ptr... 0x2df170d0, 0xa4cbdc0
Caught exception in looking up space point ptr... 0x30d5df30, 0x66717f50
Caught exception in looking up space point ptr... 0x66717f50, 0x30d5df30
Caught exception in looking up space point ptr... 0xa4b46a0, 0xa4b5210
Caught exception in looking up space point ptr... 0xa4b5210, 0xa4b46a0
Caught exception in looking up space point ptr... 0x2631a590, 0x26318f70
Caught exception in looking up space point ptr... 0x26318f70, 0x2631a590
Caught exception in looking up space point ptr... 0x2631a590, 0x26319090
Caught exception in looking up space point ptr... 0x26319090, 0x2631a590
Caught exception in looking up space point ptr... 0x39181a00, 0xe7acfb0
Caught exception in looking up space point ptr... 0xe7acfb0, 0x39181a00
Caught exception in looking up space point ptr... 0x33da3c90, 0x33da3d30
Caught exception in looking up space point ptr... 0x33da3d30, 0x33da3c90
Caught exception in looking up space point ptr... 0x10c6dc00, 0x51176a20
Caught exception in looking up space point ptr... 0x51176a20, 0x10c6dc00
Caught exception in looking up space point ptr... 0x4b8f0ac0, 0x4b8da010
Caught exception in looking up space point ptr... 0x4b8da010, 0x4b8f0ac0
Caught exception in looking up space point ptr... 0xad677f0, 0xad68b00
Caught exception in looking up space point ptr... 0xad68b00, 0xad677f0
Caught exception in looking up space point ptr... 0x3121d650, 0x42e195a0
Caught exception in looking up space point ptr... 0x42e195a0, 0xe7ddb20
Caught exception in looking up space point ptr... 0x4b8dd7a0, 0x4b8d6690
Caught exception in looking up space point ptr... 0x4b8d6690, 0x4b8dd610
Caught exception in looking up space point ptr... 0xf9ee270, 0x433e0b70
Caught exception in looking up space point ptr... 0x433e0b70, 0xf9ee270
Caught exception in looking up space point ptr... 0xf9ee270, 0x688bfa90
Caught exception in looking up space point ptr... 0x688bfa90, 0xf9ee270
Caught exception in looking up space point ptr... 0xf9ee270, 0x30908da0
Caught exception in looking up space point ptr... 0x30908da0, 0xf9ee270
Caught exception in looking up space point ptr... 0x182982d0, 0x18298980
Caught exception in looking up space point ptr... 0x18298980, 0x182982d0
Caught exception in looking up space point ptr... 0x2df7b380, 0x45fa8a00
Caught exception in looking up space point ptr... 0x45fa8a00, 0x2df7b380
Caught exception in looking up space point ptr... 0x13c68a50, 0x13c5a070
Caught exception in looking up space point ptr... 0x13c5a070, 0x49e48960
Caught exception in looking up space point ptr... 0x33361090, 0x33360df0
Caught exception in looking up space point ptr... 0x33360df0, 0x33361090
Caught exception in looking up space point ptr... 0x33346830, 0x33346510
Caught exception in looking up space point ptr... 0x33346510, 0x33346830
Caught exception in looking up space point ptr... 0xaea1b10, 0x58dd6bc0
Caught exception in looking up space point ptr... 0x58dd6bc0, 0x5aa1e6e0
Caught exception in looking up space point ptr... 0x33b175b0, 0xaea1b10
Caught exception in looking up space point ptr... 0x13c5db30, 0x13c5d9a0
Caught exception in looking up space point ptr... 0x13c5d9a0, 0x13c5db30
Caught exception in looking up space point ptr... 0x2cc7c2c0, 0x2cc7c680
Caught exception in looking up space point ptr... 0x2cc7c680, 0x2cc7c2c0
Caught exception in looking up space point ptr... 0x3496ee00, 0x349701f0
Caught exception in looking up space point ptr... 0x349701f0, 0x3496ee00
Caught exception in looking up space point ptr... 0x2e29d530, 0x2e29e010
Caught exception in looking up space point ptr... 0x2e29e010, 0x2e29d530
Caught exception in looking up space point ptr... 0x39180460, 0x4cc3c670
Caught exception in looking up space point ptr... 0x4cc3c670, 0x39180460
Caught exception in looking up space point ptr... 0x13c73650, 0x13c73570
Caught exception in looking up space point ptr... 0x13c73570, 0x13c72c60
Caught exception in looking up space point ptr... 0x4f525080, 0x4f525180
Caught exception in looking up space point ptr... 0x4f525180, 0x4f525080
++++>>>> total num hits: 122608, num free: 118250
04-Feb-2026 00:30:45 CET  Closed output file "np02vd_raw_run042421_0044_df-s04-d0_dw_0_20260203T135447_reco_stage0_20260203T233045_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         79.8938       93.5662       112.038       93.4733       8.75518        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       5.0451e-05    9.45374e-05   0.000205632   8.6731e-05    3.51871e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                4.12567       4.38443       5.3015        4.33176      0.214305        27     
produce:triggerrawdecoder:PDVDTriggerReader4      0.0121334     0.0242418     0.0726151     0.014986      0.0161196       27     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000492916   0.000723261   0.00194731    0.000569387   0.000353841      27     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.001761     0.00213819    0.00291938    0.00200966    0.000300221      27     
produce:beamevent:BeamEvent                      6.9021e-05    0.000159604   0.00102028    0.000121921   0.000176255      27     
produce:pdvddaphne:DAPHNEReaderPDVD                2.44268       3.00087       3.48779       3.08485      0.262603        27     
produce:ophit:OpHitFinder                         0.0302917     0.0406565     0.0477541     0.0408149    0.00344008       27     
produce:wclsdatavd:WireCellToolkit                 58.6277       67.0623       79.5811       66.779        5.08027        27     
produce:gaushit:GausHitFinder                     0.542638       1.00606       1.56266      0.987093      0.278505        27     
produce:nhitsfilter:NumberOfHitsFilter           0.000164902   0.000297379   0.000486886   0.000287033   6.64803e-05      27     
produce:reco3d:SpacePointSolver                    5.57412       12.0218       18.7155       11.7448       3.50347        27     
produce:hitpdune:DisambigFromSpacePoints          0.0469864     0.101608      0.197326      0.0854683     0.0397124       27     
produce:cluster3d:Cluster3D                       0.628098       1.99886       3.76338       1.92969      0.888407        27     
[art]:TriggerResults:TriggerResultInserter        2.427e-05    3.61981e-05   0.000138722    3.196e-05    2.11307e-05      27     
end_path:out1:RootOutput                           5.8e-06     9.50522e-06    3.379e-05     8.61e-06     5.05886e-06      27     
end_path:out1:RootOutput(write)                    2.97813       3.89672       5.06646       3.68811      0.690462        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4533.66 MB
  Peak resident set size usage (VmHWM): 2585.64 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042421_0044_df-s04-d0_dw_0_20260203T135447_reco_stage0_20260203T233045_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042421_0044_df-s04-d0_dw_0_20260203T135447_reco_stage0_20260203T233045_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042421_0044_df-s04-d0_dw_0_20260203T135447_reco_stage0_20260203T233045_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770158869.0,
    "core.end_time": 1770161447.0,
    "core.events": [
      7174,
      7180,
      7186,
      7192,
      7198,
      7204,
      7210,
      7216,
      7222,
      7228,
      7234,
      7240,
      7246,
      7252,
      7258,
      7264,
      7270,
      7276,
      7282,
      7288,
      7294,
      7300,
      7306,
      7312,
      7318,
      7324,
      7330
    ],
    "core.event_count": 27,
    "core.first_event_number": 7174,
    "core.last_event_number": 7330,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42421
    ],
    "core.runs_subruns": [
      4242100001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042421_0044_df-s04-d0_dw_0_20260203T135447.hdf5"
    }
  ]
}
justIN time: 2026-02-04 06:07:23 UTC       justIN version: 01.06.00