justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297920.131@dunegpschedd02.fnal.gov

Jobsub ID297920.131@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:54:27
SiteUK_Manchester
EntryUBoone_T2_UK_Manchester_ce01
Last heartbeat2026-02-04 03:06:02
From worker nodeHostnamewn1206241.tier2.hep.manchester.ac.uk
cpuinfoAMD EPYC 7702P 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:43:08
Input filesvd-protodune:np02vd_raw_run042433_0473_df-s04-d0_dw_0_20260203T214154.hdf5
JobscriptExit code0
Real time3h (11966s)
CPU time1h (6390s = 53%)
Max RSS bytes3533697024 (3369 MiB)
Outputting started2026-02-04 03:02:37
Output files
Finished2026-02-04 03:06:02
Saved logsjustin-logs:297920.131-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

up space point ptr... 0x18dd2520, 0x37aa43b0
Caught exception in looking up space point ptr... 0x563a8a30, 0x563a8bc0
Caught exception in looking up space point ptr... 0x563a8bc0, 0x563a8ee0
Caught exception in looking up space point ptr... 0x18cac210, 0x32280ed0
Caught exception in looking up space point ptr... 0x32280ed0, 0x18cb98e0
Caught exception in looking up space point ptr... 0x8854fd0, 0x59c95240
Caught exception in looking up space point ptr... 0x59c95240, 0xe72cf70
Caught exception in looking up space point ptr... 0x59c955b0, 0x59c95360
Caught exception in looking up space point ptr... 0x59c95360, 0x59c9e8a0
Caught exception in looking up space point ptr... 0x18ca3f70, 0x3227b4c0
Caught exception in looking up space point ptr... 0x3227b4c0, 0x18cb0b40
Caught exception in looking up space point ptr... 0x2bf3a1a0, 0x2bf39f10
Caught exception in looking up space point ptr... 0x2bf39f10, 0x2bf3ac30
Caught exception in looking up space point ptr... 0x79444610, 0x854d4010
Caught exception in looking up space point ptr... 0x854d4010, 0xe76bf80
Caught exception in looking up space point ptr... 0x153c00b0, 0x38b4fe00
Caught exception in looking up space point ptr... 0x38b4fe00, 0x34529de0
Caught exception in looking up space point ptr... 0xb2c0280, 0x59c4e700
Caught exception in looking up space point ptr... 0x59c4e700, 0xb2be860
Caught exception in looking up space point ptr... 0x32ba5750, 0x4bf69350
Caught exception in looking up space point ptr... 0x4bf69350, 0x4bdda840
Caught exception in looking up space point ptr... 0x799d1c30, 0x5a431440
Caught exception in looking up space point ptr... 0x799d1aa0, 0x799d1c30
Caught exception in looking up space point ptr... 0x12025080, 0x15fe8370
Caught exception in looking up space point ptr... 0x366a4430, 0x12025080
Caught exception in looking up space point ptr... 0x8653a8f0, 0x8653a7a0
Caught exception in looking up space point ptr... 0x8653a500, 0x8653a8f0
Caught exception in looking up space point ptr... 0x3e2bf420, 0x3e2bf290
Caught exception in looking up space point ptr... 0x3e2bef70, 0x3e2bf420
Caught exception in looking up space point ptr... 0xb29fe70, 0x59c9fac0
Caught exception in looking up space point ptr... 0xb2a3880, 0xb29fe70
Caught exception in looking up space point ptr... 0x6f89b040, 0x6f89b160
Caught exception in looking up space point ptr... 0x6f89b160, 0x6f89b040
Caught exception in looking up space point ptr... 0x1a7020d0, 0x1a702390
Caught exception in looking up space point ptr... 0x1a702390, 0x1a7020d0
Caught exception in looking up space point ptr... 0x498490e0, 0x4983bdf0
Caught exception in looking up space point ptr... 0x4983bdf0, 0xd6d3f00
Caught exception in looking up space point ptr... 0x42a21e70, 0x42a14d10
Caught exception in looking up space point ptr... 0x42a2efd0, 0x42a21e70
Caught exception in looking up space point ptr... 0x46e3a890, 0x10bb42d0
Caught exception in looking up space point ptr... 0x46e2be30, 0x46e3a890
Caught exception in looking up space point ptr... 0x43541920, 0x46156840
Caught exception in looking up space point ptr... 0x46155b80, 0x43541920
Caught exception in looking up space point ptr... 0x6f54ee10, 0x1a41a410
Caught exception in looking up space point ptr... 0x538961a0, 0x6f54ee10
Caught exception in looking up space point ptr... 0x8623a830, 0x8623a060
Caught exception in looking up space point ptr... 0x8623a060, 0x86239700
Caught exception in looking up space point ptr... 0x3b7cf7d0, 0x3b7cf570
Caught exception in looking up space point ptr... 0x3b7cf570, 0x3b7cf1e0
Caught exception in looking up space point ptr... 0xd3befb0, 0xd3bebf0
Caught exception in looking up space point ptr... 0xd3bebf0, 0xd3be9d0
Caught exception in looking up space point ptr... 0x199bd8a0, 0x199abf60
Caught exception in looking up space point ptr... 0x199abf60, 0x199bda30
Caught exception in looking up space point ptr... 0x58e8aa80, 0x58e782e0
Caught exception in looking up space point ptr... 0x58e9d040, 0x58e8aa80
Caught exception in looking up space point ptr... 0x6f7ca2c0, 0x6f7ca780
Caught exception in looking up space point ptr... 0x6f7ca780, 0x37e64c40
Caught exception in looking up space point ptr... 0xd1c4920, 0x31485950
Caught exception in looking up space point ptr... 0x31485950, 0xd1c5980
Caught exception in looking up space point ptr... 0xd1c5980, 0x794dec30
Caught exception in looking up space point ptr... 0xd1c6020, 0xd1c4920
Caught exception in looking up space point ptr... 0x2f5f11e0, 0x1145e980
Caught exception in looking up space point ptr... 0x1145e980, 0x2f5f11e0
Caught exception in looking up space point ptr... 0x5ee9a4e0, 0x8dafbce0
Caught exception in looking up space point ptr... 0x8dafbce0, 0x8dafe4b0
Caught exception in looking up space point ptr... 0xf227530, 0xf227390
Caught exception in looking up space point ptr... 0xf227390, 0xf227660
++++>>>> total num hits: 545369, num free: 537074
04-Feb-2026 03:02:24 GMT  Closed output file "np02vd_raw_run042433_0473_df-s04-d0_dw_0_20260203T214154_reco_stage0_20260204T030224_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         468.134       622.598        713.1        624.69        61.4169        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       0.000102273   0.000123161   0.000297771   0.000112051   4.19199e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                184.797       272.191       420.221       261.901       57.3616        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.823667       1.17037       1.75594       1.05603      0.259431        19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0221543     0.0298864     0.0746691     0.0255347     0.0130107       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0221264     0.0248619     0.0452002     0.0225818    0.00676389       19     
produce:beamevent:BeamEvent                      7.5171e-05    0.000204869   0.00162551    0.000121218   0.000335985      19     
produce:pdvddaphne:DAPHNEReaderPDVD                17.2241       23.2206       32.2386       22.882        3.64825        19     
produce:ophit:OpHitFinder                         0.0490138     0.0544802     0.067483      0.0531885    0.00490377       19     
produce:wclsdatavd:WireCellToolkit                 173.307       201.081        226.7        199.189       11.3134        19     
produce:gaushit:GausHitFinder                      2.47415       3.71718       4.97552       4.01235      0.771865        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000478602   0.00075286    0.000992089   0.000753591   0.000151186      19     
produce:reco3d:SpacePointSolver                    20.3935       31.8101       50.1105       29.2625       8.51968        19     
produce:hitpdune:DisambigFromSpacePoints          0.328106       0.51231      0.813413      0.506344      0.132786        19     
produce:cluster3d:Cluster3D                        4.49432       77.6187       114.035       101.933       43.2493        19     
[art]:TriggerResults:TriggerResultInserter       2.8483e-05    4.73661e-05   0.000115538   4.0396e-05    1.99554e-05      19     
end_path:out1:RootOutput                          4.459e-06    7.81837e-06   3.0758e-05     5.69e-06     6.26624e-06      19     
end_path:out1:RootOutput(write)                    7.50055       11.1063       12.5606       11.4141       1.43854        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5577.16 MB
  Peak resident set size usage (VmHWM): 3533.7 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0473_df-s04-d0_dw_0_20260203T214154_reco_stage0_20260204T030224_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0473_df-s04-d0_dw_0_20260203T214154_reco_stage0_20260204T030224_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0473_df-s04-d0_dw_0_20260203T214154_reco_stage0_20260204T030224_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770162205.0,
    "core.end_time": 1770174147.0,
    "core.events": [
      53932,
      53938,
      53944,
      53950,
      53956,
      53962,
      53968,
      53974,
      53980,
      53986,
      53992,
      53998,
      54004,
      54010,
      54016,
      54022,
      54028,
      54034,
      54040
    ],
    "core.event_count": 19,
    "core.first_event_number": 53932,
    "core.last_event_number": 54040,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0473_df-s04-d0_dw_0_20260203T214154.hdf5"
    }
  ]
}
justIN time: 2026-02-04 04:32:58 UTC       justIN version: 01.06.00