justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303744.4@dunegpschedd01.fnal.gov

Jobsub ID303744.4@dunegpschedd01.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-04 01:32:31
SiteUK_Manchester
EntryUBoone_T2_UK_Manchester_ce01
Last heartbeat2026-02-04 04:21:47
From worker nodeHostnamewn1205080.tier2.hep.manchester.ac.uk
cpuinfoAMD EPYC 7702P 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-04 01:33:22
Input filesvd-protodune:np02vd_raw_run042433_0505_df-s02-d0_dw_0_20260203T215403.hdf5
JobscriptExit code0
Real time2h (9890s)
CPU time1h (4496s = 45%)
Max RSS bytes3383599104 (3226 MiB)
Outputting started2026-02-04 04:18:14
Output files
Finished2026-02-04 04:21:47
Saved logsjustin-logs:303744.4-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

nt ptr... 0x4f081450, 0xf819060
Caught exception in looking up space point ptr... 0xf819060, 0xf814450
Caught exception in looking up space point ptr... 0xf814450, 0x4f081330
Caught exception in looking up space point ptr... 0x4f081330, 0x4f0815a0
Caught exception in looking up space point ptr... 0x59f5dbb0, 0x59f67100
Caught exception in looking up space point ptr... 0x59f67100, 0x4b502f20
Caught exception in looking up space point ptr... 0x59f5dbb0, 0x59f5ebb0
Caught exception in looking up space point ptr... 0x59f5ebb0, 0x59f5dbb0
Caught exception in looking up space point ptr... 0x4f0b7e90, 0x4f0b81b0
Caught exception in looking up space point ptr... 0x4f0b81b0, 0x4f0b7e90
Caught exception in looking up space point ptr... 0x119e2b10, 0x6090a420
Caught exception in looking up space point ptr... 0x6090a5b0, 0x119e2b10
Caught exception in looking up space point ptr... 0x72016230, 0x3eab2b90
Caught exception in looking up space point ptr... 0x3eab2b90, 0x3eab2a90
Caught exception in looking up space point ptr... 0x3eab2a90, 0x72015cf0
Caught exception in looking up space point ptr... 0x118db040, 0x118daeb0
Caught exception in looking up space point ptr... 0x868c32f0, 0x118db040
Caught exception in looking up space point ptr... 0x573b6130, 0x31a82690
Caught exception in looking up space point ptr... 0x31a82690, 0x573b6130
Caught exception in looking up space point ptr... 0x1796f220, 0x1796f120
Caught exception in looking up space point ptr... 0x1796f120, 0x3ada0340
Caught exception in looking up space point ptr... 0x3ada0340, 0x3ada0490
Caught exception in looking up space point ptr... 0x3ada0490, 0x3ada05e0
Caught exception in looking up space point ptr... 0x31a93760, 0x31a94820
Caught exception in looking up space point ptr... 0x31a94820, 0x31a94970
Caught exception in looking up space point ptr... 0x3ada9170, 0x1795f1d0
Caught exception in looking up space point ptr... 0x1795f1d0, 0x3ada9170
Caught exception in looking up space point ptr... 0x37abff90, 0x3d5baad0
Caught exception in looking up space point ptr... 0x3d5baad0, 0x3d5b8580
Caught exception in looking up space point ptr... 0x525427f0, 0x52542da0
Caught exception in looking up space point ptr... 0x52542da0, 0x52543070
Caught exception in looking up space point ptr... 0x6b840780, 0x6b838a90
Caught exception in looking up space point ptr... 0x6b838a90, 0x6b838960
Caught exception in looking up space point ptr... 0x6b838960, 0x6b840520
Caught exception in looking up space point ptr... 0x6b840520, 0x6b840780
Caught exception in looking up space point ptr... 0x598dd5e0, 0x598d9cb0
Caught exception in looking up space point ptr... 0x598d9cb0, 0x598dd5e0
Caught exception in looking up space point ptr... 0x106e28d0, 0x515c4350
Caught exception in looking up space point ptr... 0x515c4350, 0x106e28d0
Caught exception in looking up space point ptr... 0x7cfe30a0, 0x72037000
Caught exception in looking up space point ptr... 0x72037000, 0x7cdcae10
Caught exception in looking up space point ptr... 0x4a1b2760, 0x4a1b20e0
Caught exception in looking up space point ptr... 0x4a1b20e0, 0x4a1b1f90
Caught exception in looking up space point ptr... 0x61d189d0, 0x61d18cb0
Caught exception in looking up space point ptr... 0x61d18cb0, 0x61d18e40
Caught exception in looking up space point ptr... 0x598efa00, 0x598e8810
Caught exception in looking up space point ptr... 0x598e8810, 0x598efa00
Caught exception in looking up space point ptr... 0x394ec8e0, 0x394ec0c0
Caught exception in looking up space point ptr... 0x394ec0c0, 0x394ec8e0
Caught exception in looking up space point ptr... 0x4bfb2a90, 0x3ec6f160
Caught exception in looking up space point ptr... 0x3ec6f160, 0x4bfb2a90
Caught exception in looking up space point ptr... 0xed34890, 0x61d19ac0
Caught exception in looking up space point ptr... 0x61d19ac0, 0x61d19c50
Caught exception in looking up space point ptr... 0x608f6490, 0x6a9376d0
Caught exception in looking up space point ptr... 0x6a9376d0, 0x608f6490
Caught exception in looking up space point ptr... 0x6b843780, 0x6b843650
Caught exception in looking up space point ptr... 0x6b843650, 0x6b843780
Caught exception in looking up space point ptr... 0x1251ebf0, 0x88e75f0
Caught exception in looking up space point ptr... 0x88e75f0, 0x4b4262f0
Caught exception in looking up space point ptr... 0x573aa270, 0x5739c960
Caught exception in looking up space point ptr... 0x5739c960, 0x573aa590
Caught exception in looking up space point ptr... 0x3ad7af70, 0x868dc890
Caught exception in looking up space point ptr... 0x868dc890, 0x3ec3e1a0
Caught exception in looking up space point ptr... 0x4b513ce0, 0x4b5141a0
Caught exception in looking up space point ptr... 0x4b5141a0, 0x4b513ce0
Caught exception in looking up space point ptr... 0x3d5b63c0, 0x3d5b6110
Caught exception in looking up space point ptr... 0x3d5b6110, 0x3d5b63c0
++++>>>> total num hits: 77586, num free: 74511
04-Feb-2026 04:17:58 GMT  Closed output file "np02vd_raw_run042433_0505_df-s02-d0_dw_0_20260203T215403_reco_stage0_20260204T041758_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         373.233       512.745       557.33        532.102       47.7269        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       8.3256e-05    0.000113592   0.000322288   9.7684e-05    5.33707e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                153.944       264.388       309.554       273.297       38.5615        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.846068       1.29322       1.96821       1.18656      0.302712        19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0220062     0.0254161     0.0344092     0.0230307    0.00409744       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0219634     0.0247325     0.0656361     0.0224577    0.00964538       19     
produce:beamevent:BeamEvent                      0.000104727   0.000192504    0.0012646    0.000130516   0.000253511      19     
produce:pdvddaphne:DAPHNEReaderPDVD                12.9452       22.8607       28.9313       23.3166       3.44247        19     
produce:ophit:OpHitFinder                         0.0368954     0.0523941     0.0581632     0.0539293    0.00545122       19     
produce:wclsdatavd:WireCellToolkit                 166.867       178.254       200.671       172.739       10.6384        19     
produce:gaushit:GausHitFinder                      1.44376       2.27725       3.11843       2.43153      0.476012        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000350981   0.000543985   0.00121385    0.000501705   0.000183168      19     
produce:reco3d:SpacePointSolver                    14.2215       29.0981       46.8385       27.448        8.40676        19     
produce:hitpdune:DisambigFromSpacePoints          0.207309      0.340793      0.447335      0.348618      0.0647726       19     
produce:cluster3d:Cluster3D                        2.55701       4.99062       7.99158       4.71667       1.24336        19     
[art]:TriggerResults:TriggerResultInserter       2.5679e-05    3.93677e-05   0.000121459   3.3604e-05    2.09492e-05      19     
end_path:out1:RootOutput                          3.767e-06    6.34874e-06   3.0758e-05     5.049e-06    5.76948e-06      19     
end_path:out1:RootOutput(write)                    7.46105       9.08638       11.9639       9.12288      0.837888        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5345.46 MB
  Peak resident set size usage (VmHWM): 3383.6 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0505_df-s02-d0_dw_0_20260203T215403_reco_stage0_20260204T041758_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0505_df-s02-d0_dw_0_20260203T215403_reco_stage0_20260204T041758_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0505_df-s02-d0_dw_0_20260203T215403_reco_stage0_20260204T041758_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770168819.0,
    "core.end_time": 1770178682.0,
    "core.events": [
      57576,
      57582,
      57588,
      57594,
      57600,
      57606,
      57612,
      57618,
      57624,
      57630,
      57636,
      57642,
      57648,
      57654,
      57660,
      57666,
      57672,
      57678,
      57684
    ],
    "core.event_count": 19,
    "core.first_event_number": 57576,
    "core.last_event_number": 57684,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0505_df-s02-d0_dw_0_20260203T215403.hdf5"
    }
  ]
}
justIN time: 2026-02-04 04:29:05 UTC       justIN version: 01.06.00