justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303647.143@dunegpschedd01.fnal.gov

Jobsub ID303647.143@dunegpschedd01.fnal.gov
Workflow ID12717
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:46:24
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce515
Last heartbeat2026-02-03 23:22:42
From worker nodeHostnameb9p12p0201.cern.ch
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 22:47:13
Input filesvd-protodune:np02vd_raw_run042421_0164_df-s03-d1_dw_0_20260203T145944.hdf5
JobscriptExit code0
Real time33m (1985s)
CPU time30m (1845s = 92%)
Max RSS bytes2657550336 (2534 MiB)
Outputting started2026-02-03 23:20:19
Output files
Finished2026-02-03 23:22:42
Saved logsjustin-logs:303647.143-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

f90
Caught exception in looking up space point ptr... 0x42a2a460, 0x33719da0
Caught exception in looking up space point ptr... 0x33719da0, 0x3f732ea0
Caught exception in looking up space point ptr... 0x3f732ea0, 0x3ee29a90
Caught exception in looking up space point ptr... 0x3ee29a90, 0x3eec3650
Caught exception in looking up space point ptr... 0x3eec3650, 0x42a2a460
Caught exception in looking up space point ptr... 0x2acd6a10, 0x2acd6e20
Caught exception in looking up space point ptr... 0x2acd6e20, 0x3335a0c0
Caught exception in looking up space point ptr... 0x25f387f0, 0x25f386a0
Caught exception in looking up space point ptr... 0x25f386a0, 0x25f38980
Caught exception in looking up space point ptr... 0x30b958d0, 0x30b95740
Caught exception in looking up space point ptr... 0x30b95740, 0x30b958d0
Caught exception in looking up space point ptr... 0x573955d0, 0x57390390
Caught exception in looking up space point ptr... 0x57390390, 0x573955d0
Caught exception in looking up space point ptr... 0x852cd20, 0x8537530
Caught exception in looking up space point ptr... 0x8537530, 0x852cd20
Caught exception in looking up space point ptr... 0x55446450, 0x55446770
Caught exception in looking up space point ptr... 0x55446770, 0x55446900
Caught exception in looking up space point ptr... 0x3fbf32e0, 0x3fbf3150
Caught exception in looking up space point ptr... 0x3fbf3150, 0x3fbf32e0
Caught exception in looking up space point ptr... 0x3fb33120, 0x3fb3b960
Caught exception in looking up space point ptr... 0x3fb3b960, 0x3fb2c150
Caught exception in looking up space point ptr... 0x25f04c50, 0x25f0a6d0
Caught exception in looking up space point ptr... 0x25f0a6d0, 0x25f0d4e0
Caught exception in looking up space point ptr... 0x44ad4fe0, 0x2ae1ac50
Caught exception in looking up space point ptr... 0x2ae1ac50, 0x44ad4fe0
Caught exception in looking up space point ptr... 0x3c705cb0, 0x3c705b80
Caught exception in looking up space point ptr... 0x3c705b80, 0x3c702a60
Caught exception in looking up space point ptr... 0x5aee3a70, 0x5aee38e0
Caught exception in looking up space point ptr... 0x5aee38e0, 0x5aee3a70
Caught exception in looking up space point ptr... 0x5aee3a70, 0x5aee3750
Caught exception in looking up space point ptr... 0x5aee3750, 0x5aee3a70
Caught exception in looking up space point ptr... 0x3c83c360, 0x3c83c040
Caught exception in looking up space point ptr... 0x3c83c040, 0x3c83c360
Caught exception in looking up space point ptr... 0xd5ad660, 0xd5ad7f0
Caught exception in looking up space point ptr... 0xd5ad7f0, 0xd5a69a0
Caught exception in looking up space point ptr... 0x4a8eb510, 0x4a8eb710
Caught exception in looking up space point ptr... 0x4a8eb710, 0x4a8eb510
Caught exception in looking up space point ptr... 0x12d526b0, 0x12d52840
Caught exception in looking up space point ptr... 0x12d52840, 0x12d52b60
Caught exception in looking up space point ptr... 0x5b093320, 0x5b092c70
Caught exception in looking up space point ptr... 0x5b092c70, 0x5b090620
Caught exception in looking up space point ptr... 0xd59f160, 0x5aef95f0
Caught exception in looking up space point ptr... 0x5aef95f0, 0xd59f060
Caught exception in looking up space point ptr... 0x2bcad1d0, 0x11000040
Caught exception in looking up space point ptr... 0x11000040, 0x2bcad330
Caught exception in looking up space point ptr... 0x142e4300, 0x14306660
Caught exception in looking up space point ptr... 0x142f6530, 0x142e4300
Caught exception in looking up space point ptr... 0x3fbeeaa0, 0x3fbe9610
Caught exception in looking up space point ptr... 0x3fbe9610, 0x3fbeeaa0
Caught exception in looking up space point ptr... 0x245f45f0, 0x40459950
Caught exception in looking up space point ptr... 0x40459950, 0x245f45f0
Caught exception in looking up space point ptr... 0x64f5e710, 0x141fefa0
Caught exception in looking up space point ptr... 0x141fefa0, 0x141ff190
Caught exception in looking up space point ptr... 0x5db47df0, 0x5db48110
Caught exception in looking up space point ptr... 0x5db48110, 0x5db47df0
Caught exception in looking up space point ptr... 0x5db47df0, 0x5db48020
Caught exception in looking up space point ptr... 0x5db48020, 0x5db47df0
Caught exception in looking up space point ptr... 0x46f0e3f0, 0x46f0e580
Caught exception in looking up space point ptr... 0x46f0e580, 0x46f0e8a0
Caught exception in looking up space point ptr... 0x2f320170, 0x5b0e0b00
Caught exception in looking up space point ptr... 0x5b0e0b00, 0x2f320170
Caught exception in looking up space point ptr... 0x32c22900, 0x5d2aa490
Caught exception in looking up space point ptr... 0x5d2aa490, 0x32c22900
Caught exception in looking up space point ptr... 0x5aefa070, 0x5aefa390
Caught exception in looking up space point ptr... 0x5aefa390, 0x5aefa520
++++>>>> total num hits: 59112, num free: 56086
04-Feb-2026 00:20:10 CET  Closed output file "np02vd_raw_run042421_0164_df-s03-d1_dw_0_20260203T145944_reco_stage0_20260203T232010_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         59.3349       71.2584       95.4816       69.0207       8.22805        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                        3.889e-05    6.87183e-05   0.000187321   5.4721e-05    3.28017e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                3.51338       5.33809       39.6444       4.00816       6.74689        27     
produce:triggerrawdecoder:PDVDTriggerReader4     0.00904487     0.0157975     0.0517163     0.0107104     0.0116236       27     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000345721   0.000806632   0.00991803    0.000458221   0.00178818       27     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.00131035    0.00160408     0.0021517    0.00157227    0.000216227      27     
produce:beamevent:BeamEvent                       5.07e-05     0.000103519   0.000673792   8.5951e-05    0.000114297      27     
produce:pdvddaphne:DAPHNEReaderPDVD                1.80886       2.21826       2.92494       2.15875      0.278692        27     
produce:ophit:OpHitFinder                         0.0227351     0.031577      0.0456255     0.0313073    0.00491123       27     
produce:wclsdatavd:WireCellToolkit                 45.683        49.9593       61.0574       47.9255       4.65175        27     
produce:gaushit:GausHitFinder                     0.391009      0.739136       1.31195      0.694523       0.19442        27     
produce:nhitsfilter:NumberOfHitsFilter           0.000133261   0.000225356   0.000715663   0.00021807    0.000103443      27     
produce:reco3d:SpacePointSolver                    3.20122       8.4493        12.9512       8.76593       2.61398        27     
produce:hitpdune:DisambigFromSpacePoints          0.025326      0.074698      0.177004      0.0673342     0.0304366       27     
produce:cluster3d:Cluster3D                       0.452857       1.40565       3.14749       1.37223       0.57353        27     
[art]:TriggerResults:TriggerResultInserter        1.534e-05    2.97694e-05    0.0001269     2.552e-05    2.02312e-05      27     
end_path:out1:RootOutput                           3.6e-06     7.31111e-06    3.973e-05     5.08e-06     7.46307e-06      27     
end_path:out1:RootOutput(write)                    2.53789       3.0056        3.98652       2.88637       0.38602        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4617.63 MB
  Peak resident set size usage (VmHWM): 2657.55 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042421_0164_df-s03-d1_dw_0_20260203T145944_reco_stage0_20260203T232010_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042421_0164_df-s03-d1_dw_0_20260203T145944_reco_stage0_20260203T232010_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042421_0164_df-s03-d1_dw_0_20260203T145944_reco_stage0_20260203T232010_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770158847.0,
    "core.end_time": 1770160811.0,
    "core.events": [
      26649,
      26655,
      26661,
      26667,
      26673,
      26679,
      26685,
      26691,
      26697,
      26703,
      26709,
      26715,
      26721,
      26727,
      26733,
      26739,
      26745,
      26751,
      26757,
      26763,
      26769,
      26775,
      26781,
      26787,
      26793,
      26799,
      26805
    ],
    "core.event_count": 27,
    "core.first_event_number": 26649,
    "core.last_event_number": 26805,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42421
    ],
    "core.runs_subruns": [
      4242100001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042421_0164_df-s03-d1_dw_0_20260203T145944.hdf5"
    }
  ]
}
justIN time: 2026-02-04 07:45:13 UTC       justIN version: 01.06.00