justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297921.84@dunegpschedd02.fnal.gov

Jobsub ID297921.84@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:58:26
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce508
Last heartbeat2026-02-04 00:28:18
From worker nodeHostnameb9p35p0022.cern.ch
cpuinfoAMD EPYC 9655P 96-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:52:59
Input filesvd-protodune:np02vd_raw_run042433_0291_df-s04-d1_dw_0_20260203T203244.hdf5
JobscriptExit code0
Real time33m (1997s)
CPU time32m (1926s = 96%)
Max RSS bytes3387572224 (3230 MiB)
Outputting started2026-02-04 00:26:17
Output files
Finished2026-02-04 00:28:18
Saved logsjustin-logs:297921.84-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

up space point ptr... 0x3701cff0, 0x3701d110
Caught exception in looking up space point ptr... 0x6b240720, 0x6b2402b0
Caught exception in looking up space point ptr... 0x6b2402b0, 0x6b2423c0
Caught exception in looking up space point ptr... 0x4c72e680, 0x4c72e820
Caught exception in looking up space point ptr... 0x4c72e820, 0x4c72fc70
Caught exception in looking up space point ptr... 0x12272d10, 0x12272eb0
Caught exception in looking up space point ptr... 0x12272eb0, 0x12272d10
Caught exception in looking up space point ptr... 0x787fce60, 0x787fcb50
Caught exception in looking up space point ptr... 0x787fcb50, 0x787fdf00
Caught exception in looking up space point ptr... 0x15f5a380, 0x15f5a060
Caught exception in looking up space point ptr... 0x15f5a060, 0x7b095320
Caught exception in looking up space point ptr... 0x14549c60, 0x14549b60
Caught exception in looking up space point ptr... 0x14549b60, 0x14549c60
Caught exception in looking up space point ptr... 0x7829ddb0, 0x7829da90
Caught exception in looking up space point ptr... 0x7829da90, 0x7829ddb0
Caught exception in looking up space point ptr... 0x820277e0, 0x82026750
Caught exception in looking up space point ptr... 0x82026750, 0x82026850
Caught exception in looking up space point ptr... 0x35edfe60, 0x35edfd60
Caught exception in looking up space point ptr... 0x35edfd60, 0x35edfe60
Caught exception in looking up space point ptr... 0x126ed6f0, 0x126ed5b0
Caught exception in looking up space point ptr... 0x126ed5b0, 0x126ebae0
Caught exception in looking up space point ptr... 0x1841b9b0, 0x8201f460
Caught exception in looking up space point ptr... 0x8201f460, 0x8201f560
Caught exception in looking up space point ptr... 0x7acb650, 0x7abc920
Caught exception in looking up space point ptr... 0x7acdfe0, 0x7acb650
Caught exception in looking up space point ptr... 0x3a28ac40, 0xa87d1f0
Caught exception in looking up space point ptr... 0xa87d1f0, 0x323422f0
Caught exception in looking up space point ptr... 0x184635b0, 0x18463420
Caught exception in looking up space point ptr... 0x86a48ba0, 0x184635b0
Caught exception in looking up space point ptr... 0x7ae1ee0, 0x7adace0
Caught exception in looking up space point ptr... 0x7adace0, 0x7ae1ee0
Caught exception in looking up space point ptr... 0x79b156b0, 0x79b159d0
Caught exception in looking up space point ptr... 0x79b159d0, 0x79b15b60
Caught exception in looking up space point ptr... 0x2db79c00, 0x2db78d20
Caught exception in looking up space point ptr... 0x2db79000, 0x2db79330
Caught exception in looking up space point ptr... 0x2db79330, 0x2db798e0
Caught exception in looking up space point ptr... 0x2db798e0, 0x2db79c00
Caught exception in looking up space point ptr... 0x6070cb80, 0x6070c860
Caught exception in looking up space point ptr... 0x6070c860, 0x6070c6d0
Caught exception in looking up space point ptr... 0x4c6b3330, 0x4c6b54a0
Caught exception in looking up space point ptr... 0x4c6b54a0, 0x4c6b3820
Caught exception in looking up space point ptr... 0xec7b720, 0xec7bf10
Caught exception in looking up space point ptr... 0xec7bf10, 0xec7b720
Caught exception in looking up space point ptr... 0x690f80d0, 0x690f7db0
Caught exception in looking up space point ptr... 0x690ff630, 0x690f80d0
Caught exception in looking up space point ptr... 0x77c58fc0, 0x77c59350
Caught exception in looking up space point ptr... 0x77c59350, 0x77c58fc0
Caught exception in looking up space point ptr... 0x5a1b5b60, 0x5a1b5f90
Caught exception in looking up space point ptr... 0x5a1b5f90, 0x5a1b5b60
Caught exception in looking up space point ptr... 0x77b6e080, 0x77b6dcc0
Caught exception in looking up space point ptr... 0x77b6dcc0, 0x77b6e080
Caught exception in looking up space point ptr... 0x6641bab0, 0x9963950
Caught exception in looking up space point ptr... 0x9963950, 0x67780d30
Caught exception in looking up space point ptr... 0x359f18a0, 0x369e70e0
Caught exception in looking up space point ptr... 0x369e70e0, 0x6641bab0
Caught exception in looking up space point ptr... 0x7aa46a0, 0x7aa49c0
Caught exception in looking up space point ptr... 0x7aa49c0, 0x7aa4b50
Caught exception in looking up space point ptr... 0x3661d290, 0x33438cc0
Caught exception in looking up space point ptr... 0x33438cc0, 0x33438ee0
Caught exception in looking up space point ptr... 0x69a66340, 0x12274640
Caught exception in looking up space point ptr... 0x12274640, 0x69a66340
Caught exception in looking up space point ptr... 0x3e0f7430, 0x77bec4e0
Caught exception in looking up space point ptr... 0x77bec4e0, 0x3e0f7430
Caught exception in looking up space point ptr... 0x86a652b0, 0x81ffba50
Caught exception in looking up space point ptr... 0x81ffba50, 0x86a654b0
Caught exception in looking up space point ptr... 0x19f2a6f0, 0x19f2a990
Caught exception in looking up space point ptr... 0x19f2a990, 0x19f2a6f0
++++>>>> total num hits: 90158, num free: 86404
04-Feb-2026 01:26:12 CET  Closed output file "np02vd_raw_run042433_0291_df-s04-d1_dw_0_20260203T203244_reco_stage0_20260204T002612_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         94.5432       102.86        112.732       103.815       5.14785        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                        5.711e-05    7.86297e-05   0.000183791    7.513e-05     2.66e-05        19     
produce:tpcrawdecoder:PDVDTPCReader                5.1444        5.66081       6.30953       5.63424      0.313891        19     
produce:triggerrawdecoder:PDVDTriggerReader4     0.00809362     0.0172783     0.0287512     0.0185497     0.0053583       19     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000340812   0.000567114    0.0012479    0.000503913   0.000209924      19     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.000288591   0.000563212   0.00151571    0.000477313   0.000300337      19     
produce:beamevent:BeamEvent                       5.865e-05    0.000122268   0.000860774    7.98e-05     0.000174834      19     
produce:pdvddaphne:DAPHNEReaderPDVD                1.46459       1.87311       2.27359       1.83685      0.181515        19     
produce:ophit:OpHitFinder                         0.0206705     0.0305529     0.0375118     0.0315893    0.00450067       19     
produce:wclsdatavd:WireCellToolkit                 66.2555       73.922        83.1371       73.1942       4.40908        19     
produce:gaushit:GausHitFinder                     0.740028       1.01951       1.72925       1.00575      0.222192        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000172171   0.000306824   0.000933915   0.000244902   0.000198306      19     
produce:reco3d:SpacePointSolver                    8.77552       13.4416       20.7512        13.1         3.22698        19     
produce:hitpdune:DisambigFromSpacePoints          0.102563      0.143602      0.260803      0.124727      0.045671        19     
produce:cluster3d:Cluster3D                        1.30085       1.96537       3.74167       1.75579      0.622438        19     
[art]:TriggerResults:TriggerResultInserter        2.224e-05    3.27039e-05   9.5961e-05     2.901e-05    1.54619e-05      19     
end_path:out1:RootOutput                          4.521e-06    6.41005e-06    2.243e-05     5.27e-06     3.8879e-06       19     
end_path:out1:RootOutput(write)                    4.15051       4.74402       5.46184       4.7224       0.365704        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5517.91 MB
  Peak resident set size usage (VmHWM): 3387.57 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0291_df-s04-d1_dw_0_20260203T203244_reco_stage0_20260204T002612_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0291_df-s04-d1_dw_0_20260203T203244_reco_stage0_20260204T002612_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0291_df-s04-d1_dw_0_20260203T203244_reco_stage0_20260204T002612_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770162786.0,
    "core.end_time": 1770164773.0,
    "core.events": [
      33185,
      33191,
      33197,
      33203,
      33209,
      33215,
      33221,
      33227,
      33233,
      33239,
      33245,
      33251,
      33257,
      33263,
      33269,
      33275,
      33281,
      33287,
      33293
    ],
    "core.event_count": 19,
    "core.first_event_number": 33185,
    "core.last_event_number": 33293,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0291_df-s04-d1_dw_0_20260203T203244.hdf5"
    }
  ]
}
justIN time: 2026-02-04 07:41:53 UTC       justIN version: 01.06.00