justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303652.168@dunegpschedd01.fnal.gov

Jobsub ID303652.168@dunegpschedd01.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:50:26
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce514
Last heartbeat2026-02-04 00:27:44
From worker nodeHostnameb9p08p6650.cern.ch
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:34:43
Input filesvd-protodune:np02vd_raw_run042433_0622_df-s02-d1_dw_0_20260203T223831.hdf5
JobscriptExit code0
Real time50m (3035s)
CPU time48m (2936s = 96%)
Max RSS bytes3386331136 (3229 MiB)
Outputting started2026-02-04 00:25:20
Output files
Finished2026-02-04 00:27:44
Saved logsjustin-logs:303652.168-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

point ptr... 0x4d949b60, 0x35a5c7d0
Caught exception in looking up space point ptr... 0x35a5c7d0, 0x35a5ade0
Caught exception in looking up space point ptr... 0x4d948c00, 0x4d948e20
Caught exception in looking up space point ptr... 0x4d948e20, 0x4d949b60
Caught exception in looking up space point ptr... 0x325e4000, 0x325dc400
Caught exception in looking up space point ptr... 0x325cde30, 0x325e4000
Caught exception in looking up space point ptr... 0x4d233330, 0x4d23f8b0
Caught exception in looking up space point ptr... 0x4d23f8b0, 0x4d23a1e0
Caught exception in looking up space point ptr... 0x4d23a1e0, 0x31a585f0
Caught exception in looking up space point ptr... 0x6247c690, 0x6247c820
Caught exception in looking up space point ptr... 0x6247c820, 0x624815f0
Caught exception in looking up space point ptr... 0x72347480, 0x72347610
Caught exception in looking up space point ptr... 0x72347610, 0x5103a000
Caught exception in looking up space point ptr... 0x5784b930, 0x5784bac0
Caught exception in looking up space point ptr... 0x5784bac0, 0x57851370
Caught exception in looking up space point ptr... 0x6778aa90, 0x54d8d1f0
Caught exception in looking up space point ptr... 0x54d8d1f0, 0x17fef690
Caught exception in looking up space point ptr... 0x8017e2f0, 0x8016e9e0
Caught exception in looking up space point ptr... 0x8016e9e0, 0xd88a970
Caught exception in looking up space point ptr... 0x43e4feb0, 0x611a0320
Caught exception in looking up space point ptr... 0x57722df0, 0x43e4feb0
Caught exception in looking up space point ptr... 0x4d23efb0, 0x4d23f0e0
Caught exception in looking up space point ptr... 0x4d23f0e0, 0x31a45030
Caught exception in looking up space point ptr... 0x43404ab0, 0x433f3c40
Caught exception in looking up space point ptr... 0x433f3c40, 0x43404ab0
Caught exception in looking up space point ptr... 0x193df810, 0x193cf6b0
Caught exception in looking up space point ptr... 0x193e8ae0, 0x193df810
Caught exception in looking up space point ptr... 0x62701970, 0x51835a30
Caught exception in looking up space point ptr... 0x51835a30, 0x62701620
Caught exception in looking up space point ptr... 0x632a7cb0, 0x1746b7f0
Caught exception in looking up space point ptr... 0x1746b7f0, 0x632a7cb0
Caught exception in looking up space point ptr... 0x4d267010, 0x4d242880
Caught exception in looking up space point ptr... 0x4d242880, 0x4d267010
Caught exception in looking up space point ptr... 0x2d2d8740, 0x2d2d8870
Caught exception in looking up space point ptr... 0x2d2d8610, 0x2d2d8740
Caught exception in looking up space point ptr... 0x35a2cdb0, 0x35a2cc60
Caught exception in looking up space point ptr... 0x35a2cc60, 0x35a2cf00
Caught exception in looking up space point ptr... 0x43a5bf90, 0x3394c9c0
Caught exception in looking up space point ptr... 0x3394c9c0, 0x6631d330
Caught exception in looking up space point ptr... 0x37dd5410, 0x15c1fa00
Caught exception in looking up space point ptr... 0x15c1fa00, 0x37dd5410
Caught exception in looking up space point ptr... 0x59dfb260, 0x59dfd590
Caught exception in looking up space point ptr... 0x59dfd590, 0x59df9920
Caught exception in looking up space point ptr... 0x57842d80, 0xd818870
Caught exception in looking up space point ptr... 0xd818870, 0xd828740
Caught exception in looking up space point ptr... 0xd99dac0, 0xd99d750
Caught exception in looking up space point ptr... 0xd99d750, 0xd99dac0
Caught exception in looking up space point ptr... 0x35ec3500, 0x5148b160
Caught exception in looking up space point ptr... 0x5148b160, 0x35ec3500
Caught exception in looking up space point ptr... 0x5a20c4c0, 0xd8a2a10
Caught exception in looking up space point ptr... 0xd8a2a10, 0x57726e00
Caught exception in looking up space point ptr... 0x375d3ab0, 0x375d3960
Caught exception in looking up space point ptr... 0x375d3960, 0x375d3ab0
Caught exception in looking up space point ptr... 0x3d1c1860, 0x51177800
Caught exception in looking up space point ptr... 0x3a5ce1e0, 0x3d1c1860
Caught exception in looking up space point ptr... 0x31a14e30, 0x31a11ab0
Caught exception in looking up space point ptr... 0x4d96cb70, 0x31a14e30
Caught exception in looking up space point ptr... 0x35a2c0f0, 0x35a2bf60
Caught exception in looking up space point ptr... 0x35a2bf60, 0x35a2c0f0
Caught exception in looking up space point ptr... 0x1268d460, 0x433eaea0
Caught exception in looking up space point ptr... 0x433eaea0, 0x10d43070
Caught exception in looking up space point ptr... 0xa4424e0, 0x5a5be430
Caught exception in looking up space point ptr... 0x5a5be430, 0x370f1030
Caught exception in looking up space point ptr... 0x192fe2c0, 0x1a142490
Caught exception in looking up space point ptr... 0x1a142490, 0x192fe3c0
Caught exception in looking up space point ptr... 0x54d05dd0, 0x29973ea0
Caught exception in looking up space point ptr... 0x29973ea0, 0x54d05dd0
++++>>>> total num hits: 77346, num free: 74410
04-Feb-2026 01:25:10 CET  Closed output file "np02vd_raw_run042433_0622_df-s02-d1_dw_0_20260203T223831_reco_stage0_20260204T002510_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         139.851       156.316       173.157       156.892       8.37822        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       6.3771e-05    9.82048e-05   0.000201933   8.1681e-05    3.97219e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                6.47842       6.7539        7.00501       6.76062      0.157957        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.0124388     0.0193043     0.0625141     0.0152718     0.0109487       19     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000456205   0.000674067   0.00113198    0.000611387   0.000196339      19     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.000448835   0.000658488   0.00110881    0.000592197   0.000197575      19     
produce:beamevent:BeamEvent                      7.9331e-05    0.000159492   0.00087994    0.000107521   0.000176669      19     
produce:pdvddaphne:DAPHNEReaderPDVD                2.35052       2.90287       3.41274       2.93751      0.253891        19     
produce:ophit:OpHitFinder                         0.0294315     0.0399725      0.04541      0.0414064     0.004299        19     
produce:wclsdatavd:WireCellToolkit                 106.314       118.127       132.491       116.839       7.01165        19     
produce:gaushit:GausHitFinder                      1.01301       1.60412       2.09672       1.64804      0.322552        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000274433   0.00039285    0.000793599   0.000374345   0.000108309      19     
produce:reco3d:SpacePointSolver                    12.2213       18.1553       27.8603       17.2104       3.59255        19     
produce:hitpdune:DisambigFromSpacePoints          0.124728      0.198857      0.291554      0.210405      0.0519852       19     
produce:cluster3d:Cluster3D                        1.84599       2.66641       4.53465       2.49814      0.675494        19     
[art]:TriggerResults:TriggerResultInserter       2.2501e-05    3.49721e-05   0.000104632    3.061e-05    1.7436e-05       19     
end_path:out1:RootOutput                          4.59e-06     9.37489e-06    2.593e-05     8.89e-06     4.37934e-06      19     
end_path:out1:RootOutput(write)                    4.23755       5.80566       6.90437       5.8407       0.758209        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5356.17 MB
  Peak resident set size usage (VmHWM): 3386.33 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0622_df-s02-d1_dw_0_20260203T223831_reco_stage0_20260204T002510_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0622_df-s02-d1_dw_0_20260203T223831_reco_stage0_20260204T002510_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0622_df-s02-d1_dw_0_20260203T223831_reco_stage0_20260204T002510_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770161697.0,
    "core.end_time": 1770164712.0,
    "core.events": [
      70915,
      70921,
      70927,
      70933,
      70939,
      70945,
      70951,
      70957,
      70963,
      70969,
      70975,
      70981,
      70987,
      70993,
      70999,
      71005,
      71011,
      71017,
      71023
    ],
    "core.event_count": 19,
    "core.first_event_number": 70915,
    "core.last_event_number": 71023,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0622_df-s02-d1_dw_0_20260203T223831.hdf5"
    }
  ]
}
justIN time: 2026-02-04 07:45:28 UTC       justIN version: 01.06.00