justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303654.32@dunegpschedd01.fnal.gov

Jobsub ID303654.32@dunegpschedd01.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:52:26
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce515
Last heartbeat2026-02-04 00:42:33
From worker nodeHostnameb9p35p0043.cern.ch
cpuinfoAMD EPYC 9655P 96-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:51:19
Input filesvd-protodune:np02vd_raw_run042433_0258_df-s03-d0_dw_0_20260203T202010.hdf5
JobscriptExit code0
Real time48m (2930s)
CPU time47m (2832s = 96%)
Max RSS bytes3592224768 (3425 MiB)
Outputting started2026-02-04 00:40:11
Output files
Finished2026-02-04 00:42:33
Saved logsjustin-logs:303654.32-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

king up space point ptr... 0xe5d8a00, 0x3b1932f0
Caught exception in looking up space point ptr... 0x5a612f80, 0x856e2830
Caught exception in looking up space point ptr... 0x5a612df0, 0x5a612f80
Caught exception in looking up space point ptr... 0x711d32b0, 0x5d12f540
Caught exception in looking up space point ptr... 0x5d12f540, 0x5d12f3b0
Caught exception in looking up space point ptr... 0x737cc1d0, 0x3b876c40
Caught exception in looking up space point ptr... 0x1b3660f0, 0x737cc1d0
Caught exception in looking up space point ptr... 0x5475df80, 0x5474c7d0
Caught exception in looking up space point ptr... 0x5474c7d0, 0x5473a9e0
Caught exception in looking up space point ptr... 0x83635830, 0x836355e0
Caught exception in looking up space point ptr... 0x836355e0, 0x11c79160
Caught exception in looking up space point ptr... 0x1619b220, 0x161ad7e0
Caught exception in looking up space point ptr... 0x1618c310, 0x1619b220
Caught exception in looking up space point ptr... 0xbcbb330, 0xbcbb7e0
Caught exception in looking up space point ptr... 0x13302470, 0xbcbb330
Caught exception in looking up space point ptr... 0x7c07d8b0, 0x855cc710
Caught exception in looking up space point ptr... 0x7c07da40, 0x7c07d8b0
Caught exception in looking up space point ptr... 0x19656930, 0x19634280
Caught exception in looking up space point ptr... 0x19656ac0, 0x19656930
Caught exception in looking up space point ptr... 0x39bb0ef0, 0x39bb0bd0
Caught exception in looking up space point ptr... 0x39bb0bd0, 0x39bb0720
Caught exception in looking up space point ptr... 0x3c3c64c0, 0x3c3c6330
Caught exception in looking up space point ptr... 0x3c3c6330, 0x3c3ae460
Caught exception in looking up space point ptr... 0x1ab28930, 0x1ab287a0
Caught exception in looking up space point ptr... 0x1ab287a0, 0x5db1be10
Caught exception in looking up space point ptr... 0x47eeea40, 0x47eee8b0
Caught exception in looking up space point ptr... 0x47eee8b0, 0x47ed7b60
Caught exception in looking up space point ptr... 0xf73bf20, 0xf729640
Caught exception in looking up space point ptr... 0xf74e4e0, 0xf73bf20
Caught exception in looking up space point ptr... 0x4daa5480, 0x70816200
Caught exception in looking up space point ptr... 0x70816200, 0x4daa5480
Caught exception in looking up space point ptr... 0x3a219930, 0x3a21a0f0
Caught exception in looking up space point ptr... 0x3a21a0f0, 0x3a21aca0
Caught exception in looking up space point ptr... 0x4d9b2670, 0xf6d9420
Caught exception in looking up space point ptr... 0xf6d9420, 0x34121fa0
Caught exception in looking up space point ptr... 0x1055c420, 0x1055c8d0
Caught exception in looking up space point ptr... 0x1055c8d0, 0x1055c420
Caught exception in looking up space point ptr... 0x6bf163e0, 0x41707fb0
Caught exception in looking up space point ptr... 0x41707fb0, 0x4f5fc710
Caught exception in looking up space point ptr... 0x94c3400, 0x6bf15740
Caught exception in looking up space point ptr... 0x6bf15740, 0x94c38c0
Caught exception in looking up space point ptr... 0x853b0e10, 0xb8e7950
Caught exception in looking up space point ptr... 0xb8e7950, 0xb8e7be0
Caught exception in looking up space point ptr... 0x183976f0, 0xebb9620
Caught exception in looking up space point ptr... 0xebb9620, 0x183976f0
Caught exception in looking up space point ptr... 0xebbd270, 0x388f0940
Caught exception in looking up space point ptr... 0x388f0940, 0xebbd270
Caught exception in looking up space point ptr... 0xebbd270, 0x353b5220
Caught exception in looking up space point ptr... 0x353b5220, 0xebbd270
Caught exception in looking up space point ptr... 0xebbd270, 0xebbd120
Caught exception in looking up space point ptr... 0xebbd120, 0xebbd270
Caught exception in looking up space point ptr... 0x4831e410, 0x35fef000
Caught exception in looking up space point ptr... 0x35fef000, 0x4831e410
Caught exception in looking up space point ptr... 0x708a60f0, 0x3f96e3f0
Caught exception in looking up space point ptr... 0x3f96e3f0, 0x5464c4c0
Caught exception in looking up space point ptr... 0x2a68f4e0, 0x9b00240
Caught exception in looking up space point ptr... 0x9b00240, 0x2a68f4e0
Caught exception in looking up space point ptr... 0x117749c0, 0x2eed01e0
Caught exception in looking up space point ptr... 0x2eed01e0, 0x117749c0
Caught exception in looking up space point ptr... 0x837a8bf0, 0x837a80f0
Caught exception in looking up space point ptr... 0x837a80f0, 0x837a8a20
Caught exception in looking up space point ptr... 0x737bb5d0, 0x3a332d40
Caught exception in looking up space point ptr... 0x3a332d40, 0x737bb5d0
Caught exception in looking up space point ptr... 0x11b28620, 0x11b288f0
Caught exception in looking up space point ptr... 0x11b288f0, 0x11b28620
Caught exception in looking up space point ptr... 0x155f0ca0, 0x155f0ae0
Caught exception in looking up space point ptr... 0x155f0ae0, 0x155f0e30
++++>>>> total num hits: 541538, num free: 532924
04-Feb-2026 01:40:04 CET  Closed output file "np02vd_raw_run042433_0258_df-s03-d0_dw_0_20260203T202010_reco_stage0_20260204T004004_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         104.623       151.461       181.853       159.217       23.1167        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                        5.243e-05    7.88516e-05   0.00017912     7.364e-05    2.5744e-05       19     
produce:tpcrawdecoder:PDVDTPCReader                5.84426       6.74149       7.99551       6.66344      0.485653        19     
produce:triggerrawdecoder:PDVDTriggerReader4     0.00927642     0.0172866     0.0498056     0.0124326     0.011339        19     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.00046818    0.000703012   0.00136271    0.00065882    0.000204194      19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0003615    0.000554144   0.00073029    0.00057701    0.000116494      19     
produce:beamevent:BeamEvent                       6.802e-05    0.000126807   0.000817171    8.824e-05    0.000163502      19     
produce:pdvddaphne:DAPHNEReaderPDVD                1.71129       2.0732        2.28055       2.09143      0.147855        19     
produce:ophit:OpHitFinder                         0.0267469     0.0343564     0.041714      0.0341255    0.00404028       19     
produce:wclsdatavd:WireCellToolkit                 72.2355       82.5667       97.2659       81.2667       6.1321         19     
produce:gaushit:GausHitFinder                     0.718973       1.64342       2.29418       1.70783        0.457         19     
produce:nhitsfilter:NumberOfHitsFilter           0.00017374    0.000249631   0.00050745    0.00021158    7.66603e-05      19     
produce:reco3d:SpacePointSolver                    10.1358       12.1633       16.1996       11.442        1.63589        19     
produce:hitpdune:DisambigFromSpacePoints          0.103973      0.215255      0.363007      0.192009      0.070712        19     
produce:cluster3d:Cluster3D                        1.15464       40.102        56.4727       49.1331       20.2479        19     
[art]:TriggerResults:TriggerResultInserter        2.706e-05    3.81605e-05    9.168e-05     3.567e-05    1.35819e-05      19     
end_path:out1:RootOutput                          4.64e-06     6.45368e-06    2.021e-05      5.8e-06     3.29191e-06      19     
end_path:out1:RootOutput(write)                    4.32989       5.86036       6.97542       5.92071       0.84343        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5643.85 MB
  Peak resident set size usage (VmHWM): 3592.22 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0258_df-s03-d0_dw_0_20260203T202010_reco_stage0_20260204T004004_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0258_df-s03-d0_dw_0_20260203T202010_reco_stage0_20260204T004004_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0258_df-s03-d0_dw_0_20260203T202010_reco_stage0_20260204T004004_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770162685.0,
    "core.end_time": 1770165605.0,
    "core.events": [
      29420,
      29426,
      29432,
      29438,
      29444,
      29450,
      29456,
      29462,
      29468,
      29474,
      29480,
      29486,
      29492,
      29498,
      29504,
      29510,
      29516,
      29522,
      29528
    ],
    "core.event_count": 19,
    "core.first_event_number": 29420,
    "core.last_event_number": 29528,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0258_df-s03-d0_dw_0_20260203T202010.hdf5"
    }
  ]
}
justIN time: 2026-02-04 09:13:27 UTC       justIN version: 01.06.00