justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297921.125@dunegpschedd02.fnal.gov

Jobsub ID297921.125@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:58:26
SiteUK_RAL-Tier1
EntryLIGO_UK_RAL_arc_ce04
Last heartbeat2026-02-04 02:17:35
From worker nodeHostnamedune001-1520388.0-lcg2351.gridpp.rl.ac.uk
cpuinfoIntel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit216000 (60 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:55:55
Input filesvd-protodune:np02vd_raw_run042433_0292_df-s03-d1_dw_0_20260203T203306.hdf5
JobscriptExit code0
Real time2h (8395s)
CPU time1h (4330s = 51%)
Max RSS bytes3314601984 (3161 MiB)
Outputting started2026-02-04 02:15:52
Output files
Finished2026-02-04 02:17:35
Saved logsjustin-logs:297921.125-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

.. 0x671f2390, 0x4530a0b0
Caught exception in looking up space point ptr... 0x4530a0b0, 0x4530a2b0
Caught exception in looking up space point ptr... 0x4e76b130, 0x4e76d8b0
Caught exception in looking up space point ptr... 0x4e76d8b0, 0x66ebcdd0
Caught exception in looking up space point ptr... 0x204f99f0, 0x204fccb0
Caught exception in looking up space point ptr... 0x204fccb0, 0x204f99f0
Caught exception in looking up space point ptr... 0x1e94a370, 0x1e9527d0
Caught exception in looking up space point ptr... 0x1e9527d0, 0x1e958f00
Caught exception in looking up space point ptr... 0x1e958f00, 0x1e959030
Caught exception in looking up space point ptr... 0x1e959160, 0x1e94a370
Caught exception in looking up space point ptr... 0x7c670180, 0x7c675570
Caught exception in looking up space point ptr... 0x7c675570, 0x7c670180
Caught exception in looking up space point ptr... 0x4eae36e0, 0x675a7340
Caught exception in looking up space point ptr... 0x57baab00, 0x4eae36e0
Caught exception in looking up space point ptr... 0x575c5ff0, 0x862d57d0
Caught exception in looking up space point ptr... 0x862d57d0, 0x575c5ff0
Caught exception in looking up space point ptr... 0x44a8ba90, 0x1ba818d0
Caught exception in looking up space point ptr... 0x1ba818d0, 0x62591a20
Caught exception in looking up space point ptr... 0x4c33bd60, 0x4c330470
Caught exception in looking up space point ptr... 0x4c335f50, 0x4c33bd60
Caught exception in looking up space point ptr... 0x86d58210, 0x86d532c0
Caught exception in looking up space point ptr... 0x86d532c0, 0x86d58210
Caught exception in looking up space point ptr... 0x90b4f4d0, 0x2570c010
Caught exception in looking up space point ptr... 0x2570c010, 0x90b4f4d0
Caught exception in looking up space point ptr... 0x5c3439c0, 0x5c343890
Caught exception in looking up space point ptr... 0x5c343890, 0x5c3439c0
Caught exception in looking up space point ptr... 0x8c9bca50, 0x8c9bc920
Caught exception in looking up space point ptr... 0x8c9bc920, 0x8c9bca50
Caught exception in looking up space point ptr... 0x245cf8e0, 0x245cce60
Caught exception in looking up space point ptr... 0x245cce60, 0x245cbbb0
Caught exception in looking up space point ptr... 0x4e75e6f0, 0x27668f90
Caught exception in looking up space point ptr... 0x24fc33c0, 0x5f3c9890
Caught exception in looking up space point ptr... 0x5f3c9890, 0x4e75e6f0
Caught exception in looking up space point ptr... 0x86d6a1a0, 0x86d74b40
Caught exception in looking up space point ptr... 0x86d74b40, 0x86d72a10
Caught exception in looking up space point ptr... 0x49b603f0, 0x48619bd0
Caught exception in looking up space point ptr... 0x48619bd0, 0x214c96d0
Caught exception in looking up space point ptr... 0x442095a0, 0x44209800
Caught exception in looking up space point ptr... 0x44209800, 0x442095a0
Caught exception in looking up space point ptr... 0x4c33d230, 0x4c33cd80
Caught exception in looking up space point ptr... 0x4c33cd80, 0x4c33d230
Caught exception in looking up space point ptr... 0x7c66e780, 0x7c66e8b0
Caught exception in looking up space point ptr... 0x7c66e8b0, 0x7c66e780
Caught exception in looking up space point ptr... 0x45600ae0, 0x45603cd0
Caught exception in looking up space point ptr... 0x45603cd0, 0x7fbc7a50
Caught exception in looking up space point ptr... 0x44824400, 0x51cb2a20
Caught exception in looking up space point ptr... 0x51cb2a20, 0x1b5b7330
Caught exception in looking up space point ptr... 0x57115c20, 0x57115d70
Caught exception in looking up space point ptr... 0x57115d70, 0x57116010
Caught exception in looking up space point ptr... 0x7c5b99b0, 0x56e99b60
Caught exception in looking up space point ptr... 0x56e99b60, 0x56e99e90
Caught exception in looking up space point ptr... 0x27a4f6d0, 0x27a4c110
Caught exception in looking up space point ptr... 0x27a4c110, 0x27a4f6d0
Caught exception in looking up space point ptr... 0x27a62a60, 0x27a63450
Caught exception in looking up space point ptr... 0x27a63450, 0x27a66cd0
Caught exception in looking up space point ptr... 0x496b8d90, 0x84ea8be0
Caught exception in looking up space point ptr... 0x84ea8be0, 0x496b8d90
Caught exception in looking up space point ptr... 0x496b8d90, 0x724bdf90
Caught exception in looking up space point ptr... 0x724bdf90, 0x496b8d90
Caught exception in looking up space point ptr... 0x7c674990, 0x7c671f50
Caught exception in looking up space point ptr... 0x7c671f50, 0x7c674850
Caught exception in looking up space point ptr... 0x4f732c40, 0x4f731e60
Caught exception in looking up space point ptr... 0x4f731e60, 0x4f732c40
Caught exception in looking up space point ptr... 0x21fd1720, 0x7fbd0570
Caught exception in looking up space point ptr... 0x7fbd0570, 0x21fcd890
Caught exception in looking up space point ptr... 0x245c77f0, 0x25339720
Caught exception in looking up space point ptr... 0x25339720, 0x1f2dd5a0
++++>>>> total num hits: 53314, num free: 50956
04-Feb-2026 02:15:40 UTC  Closed output file "np02vd_raw_run042433_0292_df-s03-d1_dw_0_20260203T203306_reco_stage0_20260204T021540_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         363.193       436.842       540.322       442.667       43.6383        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       9.1976e-05    0.000156898   0.000392938   0.000122702   7.33386e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                147.448       206.048       297.56        202.075       37.0636        19     
produce:triggerrawdecoder:PDVDTriggerReader4       0.57243      0.897614       1.53124      0.857197      0.224235        19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0180737     0.0236147     0.0420375     0.0239254    0.00557009       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.018045      0.0194778     0.0424485     0.0181457    0.00541761       19     
produce:beamevent:BeamEvent                      0.000146393   0.000292925   0.00142826    0.000223302   0.000275839      19     
produce:pdvddaphne:DAPHNEReaderPDVD                14.7081       20.7462       28.1127       20.7202       3.22933        19     
produce:ophit:OpHitFinder                         0.0385217     0.0653851     0.0877996     0.0675202     0.0188083       19     
produce:wclsdatavd:WireCellToolkit                 146.726       179.419       217.427       183.873       17.0324        19     
produce:gaushit:GausHitFinder                      1.45648       1.96068       2.53605       1.95399      0.307661        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000437812   0.000583472   0.000987567   0.00057218    0.000108803      19     
produce:reco3d:SpacePointSolver                    12.5274       17.497        22.9602       16.8821       2.95484        19     
produce:hitpdune:DisambigFromSpacePoints          0.150511      0.318718      0.437323      0.314233      0.072498        19     
produce:cluster3d:Cluster3D                        2.01072       3.01436       4.45218       2.9393       0.735366        19     
[art]:TriggerResults:TriggerResultInserter       2.8326e-05    5.11477e-05   0.000106395   4.7563e-05    1.88954e-05      19     
end_path:out1:RootOutput                          4.421e-06    9.89605e-06   2.8933e-05     8.34e-06     6.12713e-06      19     
end_path:out1:RootOutput(write)                    5.4753        6.76277       8.94189       6.77508      0.904906        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5467.19 MB
  Peak resident set size usage (VmHWM): 3314.6 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0292_df-s03-d1_dw_0_20260203T203306_reco_stage0_20260204T021540_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0292_df-s03-d1_dw_0_20260203T203306_reco_stage0_20260204T021540_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0292_df-s03-d1_dw_0_20260203T203306_reco_stage0_20260204T021540_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770162966.0,
    "core.end_time": 1770171342.0,
    "core.events": [
      33297,
      33303,
      33309,
      33315,
      33321,
      33327,
      33333,
      33339,
      33345,
      33351,
      33357,
      33363,
      33369,
      33375,
      33381,
      33387,
      33393,
      33399,
      33405
    ],
    "core.event_count": 19,
    "core.first_event_number": 33297,
    "core.last_event_number": 33405,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0292_df-s03-d1_dw_0_20260203T203306.hdf5"
    }
  ]
}
justIN time: 2026-02-04 06:08:12 UTC       justIN version: 01.06.00