justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303654.69@dunegpschedd01.fnal.gov

Jobsub ID303654.69@dunegpschedd01.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:52:26
SiteUK_Manchester
EntryUBoone_T2_UK_Manchester_ce01
Last heartbeat2026-02-04 02:58:57
From worker nodeHostnamewn1205080.tier2.hep.manchester.ac.uk
cpuinfoAMD EPYC 7702P 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-04 00:02:18
Input filesvd-protodune:np02vd_raw_run042433_0406_df-s02-d1_dw_0_20260203T211625.hdf5
JobscriptExit code0
Real time2h (10396s)
CPU time1h (5595s = 53%)
Max RSS bytes3558625280 (3393 MiB)
Outputting started2026-02-04 02:55:37
Output files
Finished2026-02-04 02:58:57
Saved logsjustin-logs:303654.69-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ace point ptr... 0x40a4baa0, 0x34c956e0
Caught exception in looking up space point ptr... 0x16d71fe0, 0x3ca59a50
Caught exception in looking up space point ptr... 0x3ca59a50, 0x3ca59d70
Caught exception in looking up space point ptr... 0x16da3a70, 0x3577f300
Caught exception in looking up space point ptr... 0x16da3cf0, 0x16da3a70
Caught exception in looking up space point ptr... 0x38c1df20, 0x5e5994c0
Caught exception in looking up space point ptr... 0x6406e710, 0x38c1df20
Caught exception in looking up space point ptr... 0x1a762400, 0x34fc1660
Caught exception in looking up space point ptr... 0x1a762590, 0x1a762400
Caught exception in looking up space point ptr... 0x47919ec0, 0x47908580
Caught exception in looking up space point ptr... 0x4792b670, 0x47919ec0
Caught exception in looking up space point ptr... 0x179c7ce0, 0x5a49fd80
Caught exception in looking up space point ptr... 0x83c1cfa0, 0x179c7ce0
Caught exception in looking up space point ptr... 0x85e21dc0, 0x186cd220
Caught exception in looking up space point ptr... 0x85e23730, 0x85e21dc0
Caught exception in looking up space point ptr... 0x9057de0, 0x8dd06b0
Caught exception in looking up space point ptr... 0x8dd0840, 0x9057de0
Caught exception in looking up space point ptr... 0x169f38d0, 0x10d7f820
Caught exception in looking up space point ptr... 0x8a53300, 0x169f38d0
Caught exception in looking up space point ptr... 0x2f583b50, 0x3305fc40
Caught exception in looking up space point ptr... 0x3305fc40, 0x2f583b50
Caught exception in looking up space point ptr... 0x49203910, 0x49203730
Caught exception in looking up space point ptr... 0x49203730, 0x49203910
Caught exception in looking up space point ptr... 0xf093e60, 0xf084dc0
Caught exception in looking up space point ptr... 0xf084dc0, 0xf075b90
Caught exception in looking up space point ptr... 0x48f9ceb0, 0x52b9c8b0
Caught exception in looking up space point ptr... 0x52b9c8b0, 0x48f9ceb0
Caught exception in looking up space point ptr... 0x59ad6bb0, 0x18f11a90
Caught exception in looking up space point ptr... 0x4105f190, 0x59ad6bb0
Caught exception in looking up space point ptr... 0x3989b0f0, 0x3b8010a0
Caught exception in looking up space point ptr... 0x3b8010a0, 0x3bc22400
Caught exception in looking up space point ptr... 0x62bd0d10, 0x3163cac0
Caught exception in looking up space point ptr... 0x3163cac0, 0x62bd0d10
Caught exception in looking up space point ptr... 0x404f0640, 0x59a7e710
Caught exception in looking up space point ptr... 0x59a7e710, 0x40d342b0
Caught exception in looking up space point ptr... 0x35863e10, 0x156114d0
Caught exception in looking up space point ptr... 0x156114d0, 0x35863e10
Caught exception in looking up space point ptr... 0x4a9c1010, 0x4a9c11f0
Caught exception in looking up space point ptr... 0x4a9c11f0, 0x4a9c0a70
Caught exception in looking up space point ptr... 0x3ae345f0, 0x3ae33420
Caught exception in looking up space point ptr... 0x3ae33420, 0x3ae345f0
Caught exception in looking up space point ptr... 0xa763290, 0xaeaca80
Caught exception in looking up space point ptr... 0xaeaca80, 0x7c9b5210
Caught exception in looking up space point ptr... 0x50c1ffe0, 0x50c20200
Caught exception in looking up space point ptr... 0x50c20200, 0x50c1ffe0
Caught exception in looking up space point ptr... 0x12263080, 0x312dc370
Caught exception in looking up space point ptr... 0x312dc370, 0x12263080
Caught exception in looking up space point ptr... 0x9156bd0, 0x654d6510
Caught exception in looking up space point ptr... 0x654d6510, 0x60fd6040
Caught exception in looking up space point ptr... 0x66d77ba0, 0x3b29b9c0
Caught exception in looking up space point ptr... 0x3b29b9c0, 0x3b29b830
Caught exception in looking up space point ptr... 0x3fef1e80, 0x602700d0
Caught exception in looking up space point ptr... 0x602700d0, 0x60270420
Caught exception in looking up space point ptr... 0x623b4050, 0x623b41a0
Caught exception in looking up space point ptr... 0x623b41a0, 0x623b4050
Caught exception in looking up space point ptr... 0x8bb36fa0, 0x6da08d50
Caught exception in looking up space point ptr... 0x6da08d50, 0x8bb3ab10
Caught exception in looking up space point ptr... 0xef3e2e0, 0xef3e430
Caught exception in looking up space point ptr... 0xef3e430, 0xef3e2e0
Caught exception in looking up space point ptr... 0x34bb5330, 0x4ea26970
Caught exception in looking up space point ptr... 0x4ea26970, 0x34bb5430
Caught exception in looking up space point ptr... 0x623a7570, 0x3d205690
Caught exception in looking up space point ptr... 0x3d205690, 0x59d78690
Caught exception in looking up space point ptr... 0x600edad0, 0x49a36be0
Caught exception in looking up space point ptr... 0x49a36be0, 0x7cd8c680
Caught exception in looking up space point ptr... 0x8188f4e0, 0x1995b380
Caught exception in looking up space point ptr... 0x1995b380, 0x8188f4e0
++++>>>> total num hits: 528994, num free: 519583
04-Feb-2026 02:55:25 GMT  Closed output file "np02vd_raw_run042433_0406_df-s02-d1_dw_0_20260203T211625_reco_stage0_20260204T025525_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         398.225       539.436       702.565       552.393       76.6809        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       9.1633e-05    0.000117866   0.000323489   0.00010049    5.15331e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                153.845       235.241       355.675       222.45        43.0705        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.831008       1.17342       1.69111       1.1804       0.210777        19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0220157     0.0263429     0.036435      0.0260315    0.00435024       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0219409     0.0384249     0.270333      0.0224985     0.0558824       19     
produce:beamevent:BeamEvent                      9.3326e-05    0.000234575    0.0022384    0.000126438   0.000472812      19     
produce:pdvddaphne:DAPHNEReaderPDVD                15.086        20.972        28.1214       21.0443       3.51213        19     
produce:ophit:OpHitFinder                         0.0398261     0.0524826     0.0640792     0.0533805    0.00648538       19     
produce:wclsdatavd:WireCellToolkit                 170.716       182.268       193.678       182.524       6.89294        19     
produce:gaushit:GausHitFinder                      1.98948       3.28434       5.12104       3.11681      0.882356        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000405013   0.000611491   0.00113734    0.000577478   0.000151165      19     
produce:reco3d:SpacePointSolver                    15.8771       24.2726       34.9612       23.9276       5.23692        19     
produce:hitpdune:DisambigFromSpacePoints          0.248705      0.464096      0.702311      0.458672      0.115377        19     
produce:cluster3d:Cluster3D                        3.93752       61.198        110.437       82.5197       43.0925        19     
[art]:TriggerResults:TriggerResultInserter       2.6089e-05    4.37845e-05   0.000132028   3.4846e-05    2.33787e-05      19     
end_path:out1:RootOutput                          3.727e-06    7.10611e-06    3.205e-05     5.149e-06    6.57073e-06      19     
end_path:out1:RootOutput(write)                    8.58449       10.3945       12.9675       10.2652       1.18529        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5541.44 MB
  Peak resident set size usage (VmHWM): 3558.63 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0406_df-s02-d1_dw_0_20260203T211625_reco_stage0_20260204T025525_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0406_df-s02-d1_dw_0_20260203T211625_reco_stage0_20260204T025525_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0406_df-s02-d1_dw_0_20260203T211625_reco_stage0_20260204T025525_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770163355.0,
    "core.end_time": 1770173728.0,
    "core.events": [
      46291,
      46297,
      46303,
      46309,
      46315,
      46321,
      46327,
      46333,
      46339,
      46345,
      46351,
      46357,
      46363,
      46369,
      46375,
      46381,
      46387,
      46393,
      46399
    ],
    "core.event_count": 19,
    "core.first_event_number": 46291,
    "core.last_event_number": 46399,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0406_df-s02-d1_dw_0_20260203T211625.hdf5"
    }
  ]
}
justIN time: 2026-02-04 06:06:31 UTC       justIN version: 01.06.00