justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303652.131@dunegpschedd01.fnal.gov

Jobsub ID303652.131@dunegpschedd01.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:50:26
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce511
Last heartbeat2026-02-04 00:18:30
From worker nodeHostnameb9p02p3983.cern.ch
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:29:43
Input filesvd-protodune:np02vd_raw_run042433_0188_df-s04-d0_dw_0_20260203T195335.hdf5
JobscriptExit code0
Real time46m (2787s)
CPU time44m (2673s = 95%)
Max RSS bytes3387334656 (3230 MiB)
Outputting started2026-02-04 00:16:11
Output files
Finished2026-02-04 00:18:30
Saved logsjustin-logs:303652.131-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

int ptr... 0x8000c290, 0x8000c420
Caught exception in looking up space point ptr... 0x3c156080, 0x18b813b0
Caught exception in looking up space point ptr... 0x18b813b0, 0x3c156080
Caught exception in looking up space point ptr... 0x4cd27040, 0x4cd27190
Caught exception in looking up space point ptr... 0x4cd27190, 0x4e3e5510
Caught exception in looking up space point ptr... 0xdc06980, 0xdc06ae0
Caught exception in looking up space point ptr... 0x1a389ed0, 0xdc06980
Caught exception in looking up space point ptr... 0x18b0bc10, 0x18b0baa0
Caught exception in looking up space point ptr... 0x18b0baa0, 0x18b0bc10
Caught exception in looking up space point ptr... 0x48f0f1e0, 0x48f1e270
Caught exception in looking up space point ptr... 0x48f1e270, 0x48f0f1e0
Caught exception in looking up space point ptr... 0x57654ac0, 0x57649300
Caught exception in looking up space point ptr... 0x57649300, 0x57654ac0
Caught exception in looking up space point ptr... 0x9cc03f0, 0x5985d6a0
Caught exception in looking up space point ptr... 0x5985d6a0, 0x9cc03f0
Caught exception in looking up space point ptr... 0x3c158380, 0x3c16bcb0
Caught exception in looking up space point ptr... 0x3c16bcb0, 0x3c158380
Caught exception in looking up space point ptr... 0x45e0cd30, 0x45e0cec0
Caught exception in looking up space point ptr... 0x45e0cec0, 0x45e0d1e0
Caught exception in looking up space point ptr... 0x3234a670, 0x3234a520
Caught exception in looking up space point ptr... 0x3234a520, 0x3234a670
Caught exception in looking up space point ptr... 0x86f154c0, 0x60f6d080
Caught exception in looking up space point ptr... 0x60f6d080, 0x86f154c0
Caught exception in looking up space point ptr... 0xb948bb0, 0x18fc5660
Caught exception in looking up space point ptr... 0x18fc5660, 0xb948bb0
Caught exception in looking up space point ptr... 0x61d9aac0, 0x61d9a930
Caught exception in looking up space point ptr... 0x61d9a930, 0x61d9aac0
Caught exception in looking up space point ptr... 0x3c137800, 0x33d73550
Caught exception in looking up space point ptr... 0x33d73550, 0x3c137800
Caught exception in looking up space point ptr... 0x3fed1620, 0x3fed14d0
Caught exception in looking up space point ptr... 0x3fed14d0, 0x3fed1620
Caught exception in looking up space point ptr... 0x61d99cb0, 0x61d99990
Caught exception in looking up space point ptr... 0x61d99990, 0x61d99800
Caught exception in looking up space point ptr... 0x61d951b0, 0x61d94d00
Caught exception in looking up space point ptr... 0x61d94d00, 0x61d94b70
Caught exception in looking up space point ptr... 0x61d94b70, 0x61d94e90
Caught exception in looking up space point ptr... 0x61d94e90, 0x61d95020
Caught exception in looking up space point ptr... 0x48217ab0, 0x48217600
Caught exception in looking up space point ptr... 0x48217600, 0x48217ab0
Caught exception in looking up space point ptr... 0xe1a6910, 0x3be13860
Caught exception in looking up space point ptr... 0x3be13860, 0xe1a6910
Caught exception in looking up space point ptr... 0x51950d10, 0x739543a0
Caught exception in looking up space point ptr... 0x739543a0, 0x51950d10
Caught exception in looking up space point ptr... 0x48946e70, 0x48947000
Caught exception in looking up space point ptr... 0x48947000, 0x48947190
Caught exception in looking up space point ptr... 0x8000ae30, 0x4895f960
Caught exception in looking up space point ptr... 0x4895f960, 0x8000ae30
Caught exception in looking up space point ptr... 0x34276120, 0x2e5b2430
Caught exception in looking up space point ptr... 0x2e5b2430, 0x34276120
Caught exception in looking up space point ptr... 0x48ebfc50, 0x48ec7c20
Caught exception in looking up space point ptr... 0x48ec7c20, 0x48ec7aa0
Caught exception in looking up space point ptr... 0x2ffae980, 0x90cd970
Caught exception in looking up space point ptr... 0x90cd970, 0x2ffae980
Caught exception in looking up space point ptr... 0x598c3500, 0x598c3240
Caught exception in looking up space point ptr... 0x598c3240, 0x598c3500
Caught exception in looking up space point ptr... 0x410c7db0, 0x410c7a90
Caught exception in looking up space point ptr... 0x410c7a90, 0x410c7db0
Caught exception in looking up space point ptr... 0x65c594a0, 0x65c57450
Caught exception in looking up space point ptr... 0x65c57450, 0x65c594a0
Caught exception in looking up space point ptr... 0x47d46b90, 0x47d46de0
Caught exception in looking up space point ptr... 0x47d46de0, 0x47d46b90
Caught exception in looking up space point ptr... 0x5f830700, 0x5f82f480
Caught exception in looking up space point ptr... 0x5f82f480, 0x5f830700
Caught exception in looking up space point ptr... 0x410ab2b0, 0x410ab160
Caught exception in looking up space point ptr... 0x410ab160, 0x410ab550
Caught exception in looking up space point ptr... 0x86bf4190, 0x577ba8e0
Caught exception in looking up space point ptr... 0x577ba8e0, 0x3c4ea360
++++>>>> total num hits: 111575, num free: 106922
04-Feb-2026 01:16:02 CET  Closed output file "np02vd_raw_run042433_0188_df-s04-d0_dw_0_20260203T195335_reco_stage0_20260204T001602_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         124.917       143.373       171.497       139.44        13.1028        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       5.4321e-05    9.73603e-05   0.00019713    9.3141e-05    3.44944e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                6.33351       7.37827       8.73527       7.31582      0.552315        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.0098023     0.0198525     0.0611626     0.0138887     0.0125153       19     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000414571   0.000658277   0.00139629    0.000553222   0.000258991      19     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.000382441   0.00174687     0.0235519    0.000510941    0.0051406       19     
produce:beamevent:BeamEvent                       6.185e-05    0.000149693   0.000866783   0.00010984    0.000173251      19     
produce:pdvddaphne:DAPHNEReaderPDVD                2.18397       2.79393       3.52395       2.66776      0.386277        19     
produce:ophit:OpHitFinder                         0.0222769     0.0358626     0.0480672     0.0381866    0.00647216       19     
produce:wclsdatavd:WireCellToolkit                 94.5499       109.995       129.809       106.907       10.0913        19     
produce:gaushit:GausHitFinder                     0.856864       1.14298       1.9261        1.11677      0.227774        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000232531   0.000309937   0.000496401   0.000282272   6.78451e-05      19     
produce:reco3d:SpacePointSolver                    10.1817       14.2694       23.627        13.4073       3.54574        19     
produce:hitpdune:DisambigFromSpacePoints          0.0928115     0.164934      0.240675      0.159521       0.04057        19     
produce:cluster3d:Cluster3D                        1.22981       2.21194       3.33646       2.11587      0.514234        19     
[art]:TriggerResults:TriggerResultInserter        2.512e-05    3.44958e-05    9.119e-05     3.177e-05    1.3899e-05       19     
end_path:out1:RootOutput                          3.91e-06     9.49895e-06    3.79e-05      7.54e-06     7.74726e-06      19     
end_path:out1:RootOutput(write)                    4.35346       5.31969       6.40817       5.35665      0.530529        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5346.57 MB
  Peak resident set size usage (VmHWM): 3387.33 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0188_df-s04-d0_dw_0_20260203T195335_reco_stage0_20260204T001602_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0188_df-s04-d0_dw_0_20260203T195335_reco_stage0_20260204T001602_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0188_df-s04-d0_dw_0_20260203T195335_reco_stage0_20260204T001602_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770161394.0,
    "core.end_time": 1770164163.0,
    "core.events": [
      21442,
      21448,
      21454,
      21460,
      21466,
      21472,
      21478,
      21484,
      21490,
      21496,
      21502,
      21508,
      21514,
      21520,
      21526,
      21532,
      21538,
      21544,
      21550
    ],
    "core.event_count": 19,
    "core.first_event_number": 21442,
    "core.last_event_number": 21550,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0188_df-s04-d0_dw_0_20260203T195335.hdf5"
    }
  ]
}
justIN time: 2026-02-04 07:36:52 UTC       justIN version: 01.06.00