justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303652.127@dunegpschedd01.fnal.gov

Jobsub ID303652.127@dunegpschedd01.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:50:26
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce506
Last heartbeat2026-02-04 00:19:00
From worker nodeHostnameb9p06p2822.cern.ch
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:29:53
Input filesvd-protodune:np02vd_raw_run042433_0533_df-s03-d1_dw_0_20260203T220441.hdf5
JobscriptExit code0
Real time46m (2802s)
CPU time45m (2704s = 96%)
Max RSS bytes3427901440 (3269 MiB)
Outputting started2026-02-04 00:16:37
Output files
Finished2026-02-04 00:19:00
Saved logsjustin-logs:303652.127-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ptr... 0x6eefe5e0, 0x6eefcef0
Caught exception in looking up space point ptr... 0x3def9870, 0x1a1346f0
Caught exception in looking up space point ptr... 0x1a1346f0, 0x3def9870
Caught exception in looking up space point ptr... 0x380f33b0, 0x57d2c550
Caught exception in looking up space point ptr... 0x57d2c550, 0x57d2c680
Caught exception in looking up space point ptr... 0x6ee895c0, 0x6ee894a0
Caught exception in looking up space point ptr... 0x6ee894a0, 0x6ee895c0
Caught exception in looking up space point ptr... 0x653c8780, 0x31cf2b80
Caught exception in looking up space point ptr... 0x31cf2b80, 0x5e0c83c0
Caught exception in looking up space point ptr... 0x417e43f0, 0x417e06a0
Caught exception in looking up space point ptr... 0x417e06a0, 0x417e53d0
Caught exception in looking up space point ptr... 0x5a143940, 0x5a144f50
Caught exception in looking up space point ptr... 0x5a144f50, 0x64cb5c10
Caught exception in looking up space point ptr... 0x3a1e5900, 0x3a1e56a0
Caught exception in looking up space point ptr... 0x3a1e56a0, 0x3a1e5900
Caught exception in looking up space point ptr... 0x3a1e5900, 0x3a1e57d0
Caught exception in looking up space point ptr... 0x3a1e57d0, 0x3a1e5900
Caught exception in looking up space point ptr... 0x6eef9550, 0x6eef9400
Caught exception in looking up space point ptr... 0x6eef9400, 0x6eef9550
Caught exception in looking up space point ptr... 0x3aea07c0, 0x3aea8810
Caught exception in looking up space point ptr... 0x3aea8810, 0x3aea0600
Caught exception in looking up space point ptr... 0x3aea0600, 0x3aea6120
Caught exception in looking up space point ptr... 0x3aea6120, 0x3aea07c0
Caught exception in looking up space point ptr... 0x177cbbb0, 0x5e93abc0
Caught exception in looking up space point ptr... 0x177cd8c0, 0x177cbbb0
Caught exception in looking up space point ptr... 0x32f4e010, 0x68b2dc40
Caught exception in looking up space point ptr... 0x68b2dc40, 0x43218280
Caught exception in looking up space point ptr... 0x38256a30, 0x5e21f900
Caught exception in looking up space point ptr... 0x5e21f900, 0x38256a30
Caught exception in looking up space point ptr... 0x3aeb1270, 0x133860c0
Caught exception in looking up space point ptr... 0x133860c0, 0x3aeb1270
Caught exception in looking up space point ptr... 0x3dec4b60, 0x3dec4a10
Caught exception in looking up space point ptr... 0x3dec4a10, 0x3dec4b60
Caught exception in looking up space point ptr... 0x40760240, 0x40760560
Caught exception in looking up space point ptr... 0x40760560, 0x40760240
Caught exception in looking up space point ptr... 0x40760240, 0x407606f0
Caught exception in looking up space point ptr... 0x407606f0, 0x40760240
Caught exception in looking up space point ptr... 0x30bcdd60, 0x1a056730
Caught exception in looking up space point ptr... 0x1a056730, 0x30bcdd60
Caught exception in looking up space point ptr... 0x30bcdd60, 0x30bc8e50
Caught exception in looking up space point ptr... 0x30bc8e50, 0x30bc86a0
Caught exception in looking up space point ptr... 0x30bc86a0, 0x30bb8fd0
Caught exception in looking up space point ptr... 0x30bb8fd0, 0x30bcdd60
Caught exception in looking up space point ptr... 0x369d6b30, 0x52c67980
Caught exception in looking up space point ptr... 0x52c67980, 0x52c6f1a0
Caught exception in looking up space point ptr... 0x3aeaa2e0, 0x3aea95f0
Caught exception in looking up space point ptr... 0x3aea95f0, 0x3aeadbe0
Caught exception in looking up space point ptr... 0xf1ae820, 0xf1ae580
Caught exception in looking up space point ptr... 0xf1ae580, 0xf1ae820
Caught exception in looking up space point ptr... 0x5acc2e50, 0x195f1f00
Caught exception in looking up space point ptr... 0x195f1f00, 0x5ace7c50
Caught exception in looking up space point ptr... 0x5ace7c50, 0x865dede0
Caught exception in looking up space point ptr... 0x865dede0, 0x5acc2e50
Caught exception in looking up space point ptr... 0x14685eb0, 0x3a4c6800
Caught exception in looking up space point ptr... 0x3a4c6800, 0x14685eb0
Caught exception in looking up space point ptr... 0x380b2960, 0x380b1d90
Caught exception in looking up space point ptr... 0x380b1d90, 0x380b2960
Caught exception in looking up space point ptr... 0x380b2960, 0x380b1f30
Caught exception in looking up space point ptr... 0x380b1f30, 0x380b2960
Caught exception in looking up space point ptr... 0x19df8a40, 0x19df8ce0
Caught exception in looking up space point ptr... 0x19df8ce0, 0x19df8b40
Caught exception in looking up space point ptr... 0x19dfaaa0, 0x19dfaba0
Caught exception in looking up space point ptr... 0x19dfaba0, 0x19dfa9a0
Caught exception in looking up space point ptr... 0x37581320, 0x8001c670
Caught exception in looking up space point ptr... 0x8001c670, 0x8001b970
Caught exception in looking up space point ptr... 0x5c8576d0, 0x5c8574f0
Caught exception in looking up space point ptr... 0x5c8574f0, 0x5c8576d0
++++>>>> total num hits: 95670, num free: 92796
04-Feb-2026 01:16:27 CET  Closed output file "np02vd_raw_run042433_0533_df-s03-d1_dw_0_20260203T220441_reco_stage0_20260204T001627_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         124.237       144.493       168.756       140.902       14.1117        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                        4.555e-05    8.23204e-05   0.000191861    8.544e-05    3.19986e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                5.82107       6.97429       9.2985        6.90365      0.799778        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.0125152     0.0235668     0.0412461     0.0218753    0.00871515       19     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000476723   0.000633274   0.000897704   0.000621043   0.00011763       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.000440362   0.00057023    0.000681183   0.000578983   8.0813e-05       19     
produce:beamevent:BeamEvent                      7.0071e-05    0.000148618   0.000865624   0.00010192    0.000170468      19     
produce:pdvddaphne:DAPHNEReaderPDVD                2.14125       2.78618       3.4689        2.64674      0.390488        19     
produce:ophit:OpHitFinder                         0.0313333     0.038608      0.0495247     0.0376222     0.0044353       19     
produce:wclsdatavd:WireCellToolkit                 88.6004       103.186       119.462       102.909       9.28065        19     
produce:gaushit:GausHitFinder                     0.799857       1.39237       2.10634       1.47282      0.335886        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000270311   0.000403207   0.000523123   0.000412172   7.8854e-05       19     
produce:reco3d:SpacePointSolver                    13.754        22.122        31.6605       22.441        5.03492        19     
produce:hitpdune:DisambigFromSpacePoints          0.110372      0.203235      0.366655      0.160679      0.0800346       19     
produce:cluster3d:Cluster3D                         1.34         2.79084       6.29201       2.62323       1.21112        19     
[art]:TriggerResults:TriggerResultInserter        2.822e-05    3.65412e-05    9.187e-05     3.243e-05    1.38941e-05      19     
end_path:out1:RootOutput                          7.53e-06     1.07738e-05   2.9201e-05     9.74e-06     4.60433e-06      19     
end_path:out1:RootOutput(write)                    4.04872       4.93611       6.95208       4.58132      0.839782        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5409.03 MB
  Peak resident set size usage (VmHWM): 3427.9 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0533_df-s03-d1_dw_0_20260203T220441_reco_stage0_20260204T001627_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0533_df-s03-d1_dw_0_20260203T220441_reco_stage0_20260204T001627_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0533_df-s03-d1_dw_0_20260203T220441_reco_stage0_20260204T001627_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770161403.0,
    "core.end_time": 1770164189.0,
    "core.events": [
      60771,
      60777,
      60783,
      60789,
      60795,
      60801,
      60807,
      60813,
      60819,
      60825,
      60831,
      60837,
      60843,
      60849,
      60855,
      60861,
      60867,
      60873,
      60879
    ],
    "core.event_count": 19,
    "core.first_event_number": 60771,
    "core.last_event_number": 60879,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0533_df-s03-d1_dw_0_20260203T220441.hdf5"
    }
  ]
}
justIN time: 2026-02-04 05:58:11 UTC       justIN version: 01.06.00