justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303646.80@dunegpschedd01.fnal.gov

Jobsub ID303646.80@dunegpschedd01.fnal.gov
Workflow ID12718
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:44:24
SiteUK_Lancaster
EntryUBoone_UK_Lancaster_HEC_grendel_ce02
Last heartbeat2026-02-04 00:14:38
From worker nodeHostnamecomp22-07
cpuinfoIntel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 22:46:44
Input filesvd-protodune:np02vd_raw_run042422_0092_df-s02-d0_dw_0_20260203T160433.hdf5
JobscriptExit code0
Real time1h (4902s)
CPU time1h (3646s = 74%)
Max RSS bytes3480608768 (3319 MiB)
Outputting started2026-02-04 00:08:29
Output files
Finished2026-02-04 00:14:38
Saved logsjustin-logs:303646.80-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

a40, 0x7957b00
Caught exception in looking up space point ptr... 0x7957b00, 0x44d1f5c0
Caught exception in looking up space point ptr... 0x2cd2c720, 0x48d43fe0
Caught exception in looking up space point ptr... 0x25aa7510, 0x2cd2c720
Caught exception in looking up space point ptr... 0x972a840, 0x493a9cc0
Caught exception in looking up space point ptr... 0x493a9cc0, 0x493a9b30
Caught exception in looking up space point ptr... 0x6780c740, 0x6780c640
Caught exception in looking up space point ptr... 0x6780c640, 0x542838e0
Caught exception in looking up space point ptr... 0x2ec127d0, 0x3ce1d020
Caught exception in looking up space point ptr... 0x3ce1d020, 0x2ec127d0
Caught exception in looking up space point ptr... 0xcb6d1b0, 0xcb7f5e0
Caught exception in looking up space point ptr... 0x121630c0, 0xcb6d1b0
Caught exception in looking up space point ptr... 0xcb6d4d0, 0xcb7f900
Caught exception in looking up space point ptr... 0x121633e0, 0xcb6d4d0
Caught exception in looking up space point ptr... 0x3ca7f970, 0x3ca6d6d0
Caught exception in looking up space point ptr... 0x3ca91620, 0x3ca7f970
Caught exception in looking up space point ptr... 0x4c6e8040, 0x2dbf9280
Caught exception in looking up space point ptr... 0x4c6e81d0, 0x4c6e8040
Caught exception in looking up space point ptr... 0x34392150, 0x2a373db0
Caught exception in looking up space point ptr... 0x343a24b0, 0x34392150
Caught exception in looking up space point ptr... 0x28bfdd90, 0x28beb7d0
Caught exception in looking up space point ptr... 0x28c0f8b0, 0x28bfdd90
Caught exception in looking up space point ptr... 0x2bd1b870, 0xcc8dbd0
Caught exception in looking up space point ptr... 0x9264680, 0x2bd1b870
Caught exception in looking up space point ptr... 0x3d76e4d0, 0x24c656d0
Caught exception in looking up space point ptr... 0x54b0dbd0, 0x3d76e4d0
Caught exception in looking up space point ptr... 0xce5f9f0, 0xce5f860
Caught exception in looking up space point ptr... 0xce5f860, 0x719a42b0
Caught exception in looking up space point ptr... 0x10e26d00, 0x25100d00
Caught exception in looking up space point ptr... 0x10e35da0, 0x10e26d00
Caught exception in looking up space point ptr... 0x3f383500, 0x3f3734c0
Caught exception in looking up space point ptr... 0x3f3734c0, 0x3f383370
Caught exception in looking up space point ptr... 0x361b36c0, 0x125bf480
Caught exception in looking up space point ptr... 0x361c5e10, 0x361b36c0
Caught exception in looking up space point ptr... 0x401c2b30, 0x401b3130
Caught exception in looking up space point ptr... 0x401b3130, 0x401a3550
Caught exception in looking up space point ptr... 0x34b006d0, 0x400bf4e0
Caught exception in looking up space point ptr... 0x400bf4e0, 0x34b00540
Caught exception in looking up space point ptr... 0x5eb042d0, 0x5eb163e0
Caught exception in looking up space point ptr... 0x2fb581c0, 0x5eb042d0
Caught exception in looking up space point ptr... 0x43187270, 0x43178810
Caught exception in looking up space point ptr... 0x43178810, 0x431870e0
Caught exception in looking up space point ptr... 0x922e430, 0x921b970
Caught exception in looking up space point ptr... 0x921b970, 0x922e2a0
Caught exception in looking up space point ptr... 0x4317d310, 0x13d0c760
Caught exception in looking up space point ptr... 0x13d0c760, 0x4317d4a0
Caught exception in looking up space point ptr... 0x45752780, 0xef57f60
Caught exception in looking up space point ptr... 0xef57f60, 0x45752920
Caught exception in looking up space point ptr... 0x33f8c880, 0x200d6450
Caught exception in looking up space point ptr... 0x200d6450, 0x33f8c880
Caught exception in looking up space point ptr... 0x52550e40, 0x52550f40
Caught exception in looking up space point ptr... 0x52550f40, 0x52550e40
Caught exception in looking up space point ptr... 0x60811b80, 0x693c51a0
Caught exception in looking up space point ptr... 0x60813560, 0x60811b80
Caught exception in looking up space point ptr... 0xa263a70, 0xa263b70
Caught exception in looking up space point ptr... 0xa263b70, 0xed0b930
Caught exception in looking up space point ptr... 0x60854ae0, 0x22aa3680
Caught exception in looking up space point ptr... 0x22aa3680, 0x60854ae0
Caught exception in looking up space point ptr... 0x46e32ac0, 0x2f8c33e0
Caught exception in looking up space point ptr... 0x2f8c33e0, 0x46e32ac0
Caught exception in looking up space point ptr... 0x360df5f0, 0x4f118ac0
Caught exception in looking up space point ptr... 0x4f118ac0, 0x48c01310
Caught exception in looking up space point ptr... 0x5d14fcf0, 0x25ba33d0
Caught exception in looking up space point ptr... 0x25ba33d0, 0x5d14fcf0
Caught exception in looking up space point ptr... 0x2274dab0, 0x47241270
Caught exception in looking up space point ptr... 0x47241270, 0x36a370b0
++++>>>> total num hits: 319401, num free: 313008
04-Feb-2026 00:08:01 GMT  Closed output file "np02vd_raw_run042422_0092_df-s02-d0_dw_0_20260203T160433_reco_stage0_20260204T000801_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         108.17        177.257       249.975       172.292       40.9632        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       6.7062e-05    0.000143738   0.000254864   0.000143584   2.99126e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                15.112        41.8577       97.752        18.174        31.3314        27     
produce:triggerrawdecoder:PDVDTriggerReader4      0.698637       0.89474       1.35833      0.777991      0.193835        27     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0232826     0.0305748     0.0730279     0.0285663     0.0102242       27     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.116067      0.141958      0.193819      0.150906      0.0234734       27     
produce:beamevent:BeamEvent                      0.000156999   0.000233381   0.00132387    0.000185582   0.000215346      27     
produce:pdvddaphne:DAPHNEReaderPDVD                4.97982       8.46041       15.2924       6.24654       3.64719        27     
produce:ophit:OpHitFinder                         0.0255238     0.0313989     0.0340273     0.0318132    0.00174522       27     
produce:wclsdatavd:WireCellToolkit                  69.65        72.9269       77.0848       73.2136       1.64355        27     
produce:gaushit:GausHitFinder                     0.878163       1.4201        2.25574       1.39473       0.3409         27     
produce:nhitsfilter:NumberOfHitsFilter           0.000286227   0.00038293    0.000543411   0.000354451   7.03327e-05      27     
produce:reco3d:SpacePointSolver                    4.13162       7.31842       12.2919       6.87664       2.35571        27     
produce:hitpdune:DisambigFromSpacePoints          0.0954797     0.241957      0.560266      0.202972      0.123661        27     
produce:cluster3d:Cluster3D                        1.36915       39.0216       122.381       38.2064       26.8883        27     
[art]:TriggerResults:TriggerResultInserter       2.3984e-05    2.83783e-05   6.2074e-05    2.7221e-05    6.74569e-06      27     
end_path:out1:RootOutput                          7.136e-06    8.99963e-06   1.9009e-05     8.734e-06    2.03847e-06      27     
end_path:out1:RootOutput(write)                    3.09733       4.87739       19.6909       4.07907       3.03091        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5435.34 MB
  Peak resident set size usage (VmHWM): 3480.61 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042422_0092_df-s02-d0_dw_0_20260203T160433_reco_stage0_20260204T000801_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042422_0092_df-s02-d0_dw_0_20260203T160433_reco_stage0_20260204T000801_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042422_0092_df-s02-d0_dw_0_20260203T160433_reco_stage0_20260204T000801_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770158811.0,
    "core.end_time": 1770163700.0,
    "core.events": [
      14922,
      14928,
      14934,
      14940,
      14946,
      14952,
      14958,
      14964,
      14970,
      14976,
      14982,
      14988,
      14994,
      15000,
      15006,
      15012,
      15018,
      15024,
      15030,
      15036,
      15042,
      15048,
      15054,
      15060,
      15066,
      15072,
      15078
    ],
    "core.event_count": 27,
    "core.first_event_number": 14922,
    "core.last_event_number": 15078,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42422
    ],
    "core.runs_subruns": [
      4242200001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042422_0092_df-s02-d0_dw_0_20260203T160433.hdf5"
    }
  ]
}
justIN time: 2026-02-04 06:07:40 UTC       justIN version: 01.06.00