justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297921.108@dunegpschedd02.fnal.gov

Jobsub ID297921.108@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:58:26
SiteUK_RAL-Tier1
EntryLIGO_UK_RAL_arc_ce05
Last heartbeat2026-02-04 01:01:09
From worker nodeHostnamedune001-1844961.0-lcg2452.gridpp.rl.ac.uk
cpuinfoAMD EPYC 7452 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit216000 (60 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:55:42
Input filesvd-protodune:np02vd_raw_run042433_0016_df-s04-d0_dw_0_20260203T184812.hdf5
JobscriptExit code0
Real time1h (3824s)
CPU time58m (3508s = 91%)
Max RSS bytes3382198272 (3225 MiB)
Outputting started2026-02-04 00:59:27
Output files
Finished2026-02-04 01:01:09
Saved logsjustin-logs:297921.108-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

g up space point ptr... 0x61db6730, 0x61db6860
Caught exception in looking up space point ptr... 0x68195c40, 0x8a443080
Caught exception in looking up space point ptr... 0x8a443080, 0xb016a560
Caught exception in looking up space point ptr... 0xa2d2de40, 0xa31b2700
Caught exception in looking up space point ptr... 0xa31b2700, 0xa31b0f50
Caught exception in looking up space point ptr... 0x46ce1ee0, 0x46ce1d50
Caught exception in looking up space point ptr... 0x46ce1d50, 0x46ce1ee0
Caught exception in looking up space point ptr... 0x46ce1ee0, 0x46ce1b00
Caught exception in looking up space point ptr... 0x46ce1b00, 0x46ce1ee0
Caught exception in looking up space point ptr... 0x7e4f3d20, 0x76c01360
Caught exception in looking up space point ptr... 0x489e5950, 0x7e4f3d20
Caught exception in looking up space point ptr... 0x7382eee0, 0xb052c520
Caught exception in looking up space point ptr... 0xb052c520, 0x7382eee0
Caught exception in looking up space point ptr... 0x6926e130, 0x9eef8ed0
Caught exception in looking up space point ptr... 0x4fa8f310, 0x6926e130
Caught exception in looking up space point ptr... 0xa30f69d0, 0xa30f6ad0
Caught exception in looking up space point ptr... 0xa30f6ad0, 0xa30f68d0
Caught exception in looking up space point ptr... 0x8a479f60, 0x8a46bf10
Caught exception in looking up space point ptr... 0x8a479e30, 0x8a479f60
Caught exception in looking up space point ptr... 0x41d61ee0, 0x41d61fe0
Caught exception in looking up space point ptr... 0x41d61fe0, 0x41d61ee0
Caught exception in looking up space point ptr... 0x6d28ee40, 0x6a84ccd0
Caught exception in looking up space point ptr... 0x7206fca0, 0x6d28ee40
Caught exception in looking up space point ptr... 0xa2d73780, 0xa2d73bb0
Caught exception in looking up space point ptr... 0xa2d73bb0, 0xa2d73780
Caught exception in looking up space point ptr... 0x9dbda930, 0x70071d70
Caught exception in looking up space point ptr... 0x70071d70, 0x9dbda930
Caught exception in looking up space point ptr... 0x812a52f0, 0x812a50a0
Caught exception in looking up space point ptr... 0x812a50a0, 0x812a52f0
Caught exception in looking up space point ptr... 0xa5c0a0e0, 0xa5c062a0
Caught exception in looking up space point ptr... 0xa5c062a0, 0xa5c0a0e0
Caught exception in looking up space point ptr... 0xa7af0390, 0x899a30c0
Caught exception in looking up space point ptr... 0x7f43fc60, 0xa7af0390
Caught exception in looking up space point ptr... 0xa2d37360, 0x71a31df0
Caught exception in looking up space point ptr... 0x71a31df0, 0xa2d37360
Caught exception in looking up space point ptr... 0x4a5f18c0, 0x4a5f2540
Caught exception in looking up space point ptr... 0x4a5f2540, 0x4a5f18c0
Caught exception in looking up space point ptr... 0x6fd0b6e0, 0x6fd0bcb0
Caught exception in looking up space point ptr... 0x6fd0bcb0, 0x6fd0b0c0
Caught exception in looking up space point ptr... 0x7e620090, 0x7e61fd70
Caught exception in looking up space point ptr... 0x7e61fd70, 0x7e620090
Caught exception in looking up space point ptr... 0x858d95e0, 0x858d9450
Caught exception in looking up space point ptr... 0x858d9450, 0x858d95e0
Caught exception in looking up space point ptr... 0x681c16e0, 0x681c1480
Caught exception in looking up space point ptr... 0x681c1480, 0x681c1300
Caught exception in looking up space point ptr... 0x50621fd0, 0x50622120
Caught exception in looking up space point ptr... 0x9e0ff650, 0x50621fd0
Caught exception in looking up space point ptr... 0x400e6fb0, 0x400e7200
Caught exception in looking up space point ptr... 0x400e7200, 0x7991d110
Caught exception in looking up space point ptr... 0x773dc1b0, 0x773dc050
Caught exception in looking up space point ptr... 0x773d8c40, 0x773dc1b0
Caught exception in looking up space point ptr... 0x98c2cb20, 0x8b463330
Caught exception in looking up space point ptr... 0x8b463330, 0x8b4622a0
Caught exception in looking up space point ptr... 0x4fa59aa0, 0x7e6065c0
Caught exception in looking up space point ptr... 0x7e6065c0, 0x4fa4d690
Caught exception in looking up space point ptr... 0xb3af3800, 0xb3af3b20
Caught exception in looking up space point ptr... 0xb3af3b20, 0xb3b03d80
Caught exception in looking up space point ptr... 0xa7a1deb0, 0xa7a1d8c0
Caught exception in looking up space point ptr... 0xa7a1d8c0, 0xa7a1deb0
Caught exception in looking up space point ptr... 0x86cd2630, 0x86cdcd70
Caught exception in looking up space point ptr... 0x86cdcd70, 0x7e62dac0
Caught exception in looking up space point ptr... 0x79200540, 0x683de390
Caught exception in looking up space point ptr... 0x683de390, 0x79200540
Caught exception in looking up space point ptr... 0x79200540, 0x683de1d0
Caught exception in looking up space point ptr... 0x683de1d0, 0x79200540
Caught exception in looking up space point ptr... 0x47dd18f0, 0x47dd1c60
Caught exception in looking up space point ptr... 0x47dd1c60, 0x47dd18f0
++++>>>> total num hits: 87520, num free: 83372
04-Feb-2026 00:59:17 UTC  Closed output file "np02vd_raw_run042433_0016_df-s04-d0_dw_0_20260203T184812_reco_stage0_20260204T005917_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                          161.9        195.688       218.757       197.18        14.9429        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       9.3486e-05    0.000152502   0.000373504   0.000109085   7.28832e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                17.389        18.2285       20.5247       17.922       0.768927        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.694517      0.777422      0.987304      0.760972      0.0711678       19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0230864     0.0258933     0.0370116     0.0234219    0.00408394       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0229766     0.0233247     0.0251747     0.023215     0.000477747      19     
produce:beamevent:BeamEvent                      0.000123503   0.00035701    0.00215735    0.000181672   0.000530347      19     
produce:pdvddaphne:DAPHNEReaderPDVD                3.43355       3.75591       4.25871       3.70602      0.188168        19     
produce:ophit:OpHitFinder                         0.0300206     0.0416748     0.0502254     0.0424788    0.00471371       19     
produce:wclsdatavd:WireCellToolkit                 115.291       141.623       155.271       142.714       10.8614        19     
produce:gaushit:GausHitFinder                      0.95563       1.75964       2.40776       1.82348      0.423103        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000326065   0.000511727   0.000977323   0.000458494   0.000162306      19     
produce:reco3d:SpacePointSolver                    13.0204       19.4652       28.5905       19.4212       4.50537        19     
produce:hitpdune:DisambigFromSpacePoints          0.133025      0.255229       0.38228       0.26526      0.0667868       19     
produce:cluster3d:Cluster3D                        1.48403       3.25447       7.08413       3.0198        1.49756        19     
[art]:TriggerResults:TriggerResultInserter       2.8403e-05    4.4411e-05    0.000151405   3.9995e-05    2.56777e-05      19     
end_path:out1:RootOutput                          5.661e-06    1.12476e-05   3.5998e-05     9.999e-06    6.75308e-06      19     
end_path:out1:RootOutput(write)                    4.59906       6.43318       7.66081       6.62289       0.84604        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5474.56 MB
  Peak resident set size usage (VmHWM): 3382.2 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0016_df-s04-d0_dw_0_20260203T184812_reco_stage0_20260204T005917_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0016_df-s04-d0_dw_0_20260203T184812_reco_stage0_20260204T005917_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0016_df-s04-d0_dw_0_20260203T184812_reco_stage0_20260204T005917_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770162956.0,
    "core.end_time": 1770166760.0,
    "core.events": [
      1834,
      1840,
      1846,
      1852,
      1858,
      1864,
      1870,
      1876,
      1882,
      1888,
      1894,
      1900,
      1906,
      1912,
      1918,
      1924,
      1930,
      1936,
      1942
    ],
    "core.event_count": 19,
    "core.first_event_number": 1834,
    "core.last_event_number": 1942,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0016_df-s04-d0_dw_0_20260203T184812.hdf5"
    }
  ]
}
justIN time: 2026-02-04 10:38:25 UTC       justIN version: 01.06.00