justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303646.74@dunegpschedd01.fnal.gov

Jobsub ID303646.74@dunegpschedd01.fnal.gov
Workflow ID12718
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:44:24
SiteUK_Lancaster
EntryUBoone_UK_Lancaster_HEC_grendel_ce02
Last heartbeat2026-02-04 01:40:41
From worker nodeHostnamecomp08-00
cpuinfoIntel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 22:45:53
Input filesvd-protodune:np02vd_raw_run042422_0089_df-s04-d0_dw_0_20260203T160253.hdf5
JobscriptExit code0
Real time2h (10269s)
CPU time41m (2513s = 24%)
Max RSS bytes2584678400 (2464 MiB)
Outputting started2026-02-04 01:37:08
Output files
Finished2026-02-04 01:40:41
Saved logsjustin-logs:303646.74-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

0x2e6b3fa0
Caught exception in looking up space point ptr... 0x128ffc20, 0x3abe81b0
Caught exception in looking up space point ptr... 0x3abe81b0, 0x12900cc0
Caught exception in looking up space point ptr... 0x12904e00, 0x128ffc20
Caught exception in looking up space point ptr... 0x1256b6e0, 0x1256e490
Caught exception in looking up space point ptr... 0x1256e490, 0x12566a60
Caught exception in looking up space point ptr... 0x12566a60, 0x1256bb90
Caught exception in looking up space point ptr... 0x31a10750, 0x2918ed30
Caught exception in looking up space point ptr... 0x2918ed30, 0x236b9180
Caught exception in looking up space point ptr... 0x236b9180, 0x45afa070
Caught exception in looking up space point ptr... 0x490c57c0, 0x490c5210
Caught exception in looking up space point ptr... 0x490c9e10, 0x490c57c0
Caught exception in looking up space point ptr... 0x40c3e4d0, 0x40b8ecc0
Caught exception in looking up space point ptr... 0x40b8ecc0, 0x40b8e9a0
Caught exception in looking up space point ptr... 0x40b8e9a0, 0x40c40590
Caught exception in looking up space point ptr... 0x40c3e1b0, 0x40c3e4d0
Caught exception in looking up space point ptr... 0x40b8a1d0, 0x40b8a040
Caught exception in looking up space point ptr... 0x40b8a040, 0x40b89eb0
Caught exception in looking up space point ptr... 0x40b89eb0, 0x40c31f40
Caught exception in looking up space point ptr... 0x40c32300, 0x40b8a1d0
Caught exception in looking up space point ptr... 0xa3401b0, 0x54440e50
Caught exception in looking up space point ptr... 0x24ee2150, 0xa3401b0
Caught exception in looking up space point ptr... 0x25af6e00, 0x25af6f60
Caught exception in looking up space point ptr... 0x25af6f60, 0x25afd480
Caught exception in looking up space point ptr... 0x75bd6a0, 0x75bd570
Caught exception in looking up space point ptr... 0x75bda30, 0x75bd6a0
Caught exception in looking up space point ptr... 0x31471600, 0x23709b60
Caught exception in looking up space point ptr... 0x23709b60, 0x3a4f5100
Caught exception in looking up space point ptr... 0x155c0300, 0x3b9e0190
Caught exception in looking up space point ptr... 0x3b9e0190, 0x155c0300
Caught exception in looking up space point ptr... 0x3abe6e50, 0x1146c500
Caught exception in looking up space point ptr... 0x2e483120, 0x3abe6e50
Caught exception in looking up space point ptr... 0x40536680, 0x40538180
Caught exception in looking up space point ptr... 0x40538180, 0x40538280
Caught exception in looking up space point ptr... 0x498c8990, 0x498cc810
Caught exception in looking up space point ptr... 0x498cc810, 0x498c8990
Caught exception in looking up space point ptr... 0x566b5480, 0x566b5330
Caught exception in looking up space point ptr... 0x566b5330, 0x566b5480
Caught exception in looking up space point ptr... 0x3a2ea840, 0x25aa13a0
Caught exception in looking up space point ptr... 0x2ef5fa70, 0x3a2ea840
Caught exception in looking up space point ptr... 0x57a28ca0, 0x57a28ba0
Caught exception in looking up space point ptr... 0x29ade7e0, 0x57a28ca0
Caught exception in looking up space point ptr... 0x2dab2590, 0x2dab2190
Caught exception in looking up space point ptr... 0x2dab2190, 0x2dab2590
Caught exception in looking up space point ptr... 0x859fd10, 0x859efa0
Caught exception in looking up space point ptr... 0x859efa0, 0x859fd10
Caught exception in looking up space point ptr... 0x3e76ac80, 0x3e76ab40
Caught exception in looking up space point ptr... 0x3e76ab40, 0x3e76ac80
Caught exception in looking up space point ptr... 0x12841cb0, 0x3e7646d0
Caught exception in looking up space point ptr... 0x3e7646d0, 0x12841cb0
Caught exception in looking up space point ptr... 0x569ec810, 0x85a1540
Caught exception in looking up space point ptr... 0x85a1540, 0x51c00520
Caught exception in looking up space point ptr... 0x2d238130, 0x2d232b40
Caught exception in looking up space point ptr... 0x2d232b40, 0x10171400
Caught exception in looking up space point ptr... 0x498807f0, 0x498804d0
Caught exception in looking up space point ptr... 0x498804d0, 0x498807f0
Caught exception in looking up space point ptr... 0x8ecd1e0, 0x28e96cc0
Caught exception in looking up space point ptr... 0x28e96cc0, 0x8ecd1e0
Caught exception in looking up space point ptr... 0x3c6b5410, 0x3c6b6b60
Caught exception in looking up space point ptr... 0x3c6b6b60, 0x3c6b6dc0
Caught exception in looking up space point ptr... 0x75b4410, 0x75b4130
Caught exception in looking up space point ptr... 0x75b4130, 0x75b4410
Caught exception in looking up space point ptr... 0x3cdadac0, 0x2536ca80
Caught exception in looking up space point ptr... 0x2536ca80, 0x498a6a40
Caught exception in looking up space point ptr... 0x578efec0, 0x56b50b40
Caught exception in looking up space point ptr... 0x5ba59130, 0x578efec0
++++>>>> total num hits: 60181, num free: 58642
04-Feb-2026 01:34:03 GMT  Closed output file "np02vd_raw_run042422_0089_df-s04-d0_dw_0_20260203T160253_reco_stage0_20260204T013403_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         244.309       370.303       779.283       325.67        139.404        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       3.9105e-05    6.36437e-05   0.000200537    4.929e-05    3.78315e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                119.779       215.284       378.016       184.073       71.1164        27     
produce:triggerrawdecoder:PDVDTriggerReader4      0.836559       1.26772       3.18057       1.10346      0.509057        27     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0231506     0.0344136     0.105338      0.0291702     0.019374        27     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.114677       0.15389      0.228517      0.155355      0.0328371       27     
produce:beamevent:BeamEvent                      4.7956e-05    8.84512e-05   0.00074674    6.1205e-05    0.000130197      27     
produce:pdvddaphne:DAPHNEReaderPDVD                16.7836       27.3268       43.254        24.9343       7.93907        27     
produce:ophit:OpHitFinder                         0.0277924     0.0317607     0.0357961     0.031957     0.00177892       27     
produce:wclsdatavd:WireCellToolkit                 64.559        77.4004       230.401       68.0903       32.8381        27     
produce:gaushit:GausHitFinder                     0.354556      0.954031       1.45145       0.9404       0.261845        27     
produce:nhitsfilter:NumberOfHitsFilter           0.000169876   0.000308006   0.000429779   0.000294896   7.63918e-05      27     
produce:reco3d:SpacePointSolver                    2.14023       10.0889       20.6218       9.83632       4.33356        27     
produce:hitpdune:DisambigFromSpacePoints          0.0356578     0.112057      0.367997      0.101798      0.0599493       27     
produce:cluster3d:Cluster3D                       0.242918       2.70817       29.5664       1.44419       5.33147        27     
[art]:TriggerResults:TriggerResultInserter       1.5527e-05    2.22757e-05   6.4095e-05    2.0164e-05    9.31128e-06      27     
end_path:out1:RootOutput                          5.951e-06    7.80341e-06   1.9565e-05     7.071e-06    2.65284e-06      27     
end_path:out1:RootOutput(write)                    3.00002       34.9262       404.036       3.35568       104.201        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4565.34 MB
  Peak resident set size usage (VmHWM): 2584.68 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042422_0089_df-s04-d0_dw_0_20260203T160253_reco_stage0_20260204T013403_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042422_0089_df-s04-d0_dw_0_20260203T160253_reco_stage0_20260204T013403_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042422_0089_df-s04-d0_dw_0_20260203T160253_reco_stage0_20260204T013403_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770158761.0,
    "core.end_time": 1770168937.0,
    "core.events": [
      14434,
      14440,
      14446,
      14452,
      14458,
      14464,
      14470,
      14476,
      14482,
      14488,
      14494,
      14500,
      14506,
      14512,
      14518,
      14524,
      14530,
      14536,
      14542,
      14548,
      14554,
      14560,
      14566,
      14572,
      14578,
      14584,
      14590
    ],
    "core.event_count": 27,
    "core.first_event_number": 14434,
    "core.last_event_number": 14590,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42422
    ],
    "core.runs_subruns": [
      4242200001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042422_0089_df-s04-d0_dw_0_20260203T160253.hdf5"
    }
  ]
}
justIN time: 2026-02-04 05:49:06 UTC       justIN version: 01.06.00