justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297921.130@dunegpschedd02.fnal.gov

Jobsub ID297921.130@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:58:26
SiteUK_RAL-Tier1
EntryLIGO_UK_RAL_arc_ce05
Last heartbeat2026-02-04 01:17:01
From worker nodeHostnamedune001-1845817.0-lcg2535.gridpp.rl.ac.uk
cpuinfoAMD EPYC 7452 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit216000 (60 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:55:56
Input filesvd-protodune:np02vd_raw_run042433_0192_df-s04-d1_dw_0_20260203T195507.hdf5
JobscriptExit code0
Real time1h (4745s)
CPU time1h (4329s = 91%)
Max RSS bytes3397349376 (3239 MiB)
Outputting started2026-02-04 01:15:04
Output files
Finished2026-02-04 01:17:01
Saved logsjustin-logs:297921.130-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ce point ptr... 0x440e7590, 0x7297a060
Caught exception in looking up space point ptr... 0xcbfb0e0, 0x7669e730
Caught exception in looking up space point ptr... 0x7669e730, 0xcbfb0e0
Caught exception in looking up space point ptr... 0xcbfb0e0, 0x58ed3b60
Caught exception in looking up space point ptr... 0x58ed3b60, 0xcbfb0e0
Caught exception in looking up space point ptr... 0x5a601400, 0x5a601260
Caught exception in looking up space point ptr... 0x5a601260, 0x5a600ad0
Caught exception in looking up space point ptr... 0x52877360, 0x52876830
Caught exception in looking up space point ptr... 0x52876830, 0x52879c10
Caught exception in looking up space point ptr... 0x5859d170, 0x5859cfe0
Caught exception in looking up space point ptr... 0x5859cfe0, 0x5859d170
Caught exception in looking up space point ptr... 0x2b210850, 0x18708ef0
Caught exception in looking up space point ptr... 0x18708ef0, 0x2b210850
Caught exception in looking up space point ptr... 0x367ee4c0, 0x59002b80
Caught exception in looking up space point ptr... 0x59002b80, 0x367ee4c0
Caught exception in looking up space point ptr... 0x367ee4c0, 0x4a9abad0
Caught exception in looking up space point ptr... 0x4a9abad0, 0x367ee4c0
Caught exception in looking up space point ptr... 0x367ee4c0, 0x59002f40
Caught exception in looking up space point ptr... 0x59002f40, 0x367ee4c0
Caught exception in looking up space point ptr... 0x5a06f6b0, 0x5a06cd30
Caught exception in looking up space point ptr... 0x5a06cd30, 0x5a06f6b0
Caught exception in looking up space point ptr... 0x31a4f040, 0x31a4eea0
Caught exception in looking up space point ptr... 0x31a4eea0, 0x67a7d960
Caught exception in looking up space point ptr... 0x67a7d960, 0x67a7dd00
Caught exception in looking up space point ptr... 0x67a7dd00, 0x67a7da60
Caught exception in looking up space point ptr... 0x4b6f5440, 0x4b6f3ff0
Caught exception in looking up space point ptr... 0x4b6f3ff0, 0x4b6f5440
Caught exception in looking up space point ptr... 0x65646da0, 0x65647000
Caught exception in looking up space point ptr... 0x65647000, 0x65646da0
Caught exception in looking up space point ptr... 0x62c10370, 0x62c10870
Caught exception in looking up space point ptr... 0x62c10870, 0x62c10be0
Caught exception in looking up space point ptr... 0x3da07a80, 0x3da06270
Caught exception in looking up space point ptr... 0x3da14080, 0x3da07a80
Caught exception in looking up space point ptr... 0xd5f69c0, 0xd5fd710
Caught exception in looking up space point ptr... 0xd5fd710, 0xd5f0610
Caught exception in looking up space point ptr... 0x36cf1370, 0x833916b0
Caught exception in looking up space point ptr... 0x833916b0, 0x36cf1370
Caught exception in looking up space point ptr... 0x5ae632a0, 0x5ae635c0
Caught exception in looking up space point ptr... 0x5ae635c0, 0x5ae632a0
Caught exception in looking up space point ptr... 0x4dc16ff0, 0x60ca3740
Caught exception in looking up space point ptr... 0x60ca3740, 0x4dc16ff0
Caught exception in looking up space point ptr... 0x38043e50, 0xcc9f680
Caught exception in looking up space point ptr... 0xcc9f680, 0x38043e50
Caught exception in looking up space point ptr... 0x29e39c00, 0x29e3ccf0
Caught exception in looking up space point ptr... 0x29e3ccf0, 0x29e39c00
Caught exception in looking up space point ptr... 0x64aec0e0, 0x64aea8b0
Caught exception in looking up space point ptr... 0x64aea8b0, 0x64aebaa0
Caught exception in looking up space point ptr... 0x6b68bf20, 0x58ffdf70
Caught exception in looking up space point ptr... 0x58ffdf70, 0x58ffe100
Caught exception in looking up space point ptr... 0x6986f690, 0x6986f540
Caught exception in looking up space point ptr... 0x6986f540, 0x6986f930
Caught exception in looking up space point ptr... 0x710dedb0, 0x710dec60
Caught exception in looking up space point ptr... 0x710dec60, 0x710dedb0
Caught exception in looking up space point ptr... 0x67a7c560, 0x9a8e3c0
Caught exception in looking up space point ptr... 0x9a8e3c0, 0x67a7c560
Caught exception in looking up space point ptr... 0x1573c8d0, 0x1573cae0
Caught exception in looking up space point ptr... 0x1573cae0, 0x1573c8d0
Caught exception in looking up space point ptr... 0x3d48b520, 0x64537af0
Caught exception in looking up space point ptr... 0x64537af0, 0x3d48b520
Caught exception in looking up space point ptr... 0x71c7d440, 0x71c7d720
Caught exception in looking up space point ptr... 0x71c7d720, 0x71c7d440
Caught exception in looking up space point ptr... 0x722d9ec0, 0x722d9f90
Caught exception in looking up space point ptr... 0x722d9f90, 0x722da2a0
Caught exception in looking up space point ptr... 0xdc6d1a0, 0x4b712480
Caught exception in looking up space point ptr... 0x4b712480, 0xdc6d1a0
Caught exception in looking up space point ptr... 0x58ed3680, 0x78c8c4f0
Caught exception in looking up space point ptr... 0x78c8c4f0, 0x58ed44a0
++++>>>> total num hits: 90701, num free: 86919
04-Feb-2026 01:14:49 UTC  Closed output file "np02vd_raw_run042433_0192_df-s04-d1_dw_0_20260203T195507_reco_stage0_20260204T011449_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         208.029       242.118        272.3        233.12        18.1869        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       0.000106992   0.000141272   0.000392777   0.000125582   6.16089e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                19.5503       23.3691       39.4385       21.3858       4.8346         19     
produce:triggerrawdecoder:PDVDTriggerReader4       0.69274      0.776318       1.04982      0.740744      0.0889974       19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0233174     0.0291755     0.0574756     0.0236053     0.0102615       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0231483     0.0235717     0.0243117     0.0235524    0.000290174      19     
produce:beamevent:BeamEvent                      0.000154443   0.000354149   0.00157731    0.000255565   0.000327483      19     
produce:pdvddaphne:DAPHNEReaderPDVD                4.09589       4.71752       5.28713       4.82167      0.318749        19     
produce:ophit:OpHitFinder                         0.037342      0.0520058     0.0672552     0.0528152    0.00665323       19     
produce:wclsdatavd:WireCellToolkit                 152.736       169.44        187.112       168.893       8.38802        19     
produce:gaushit:GausHitFinder                      1.25697       2.31147       3.62302       2.32877      0.638378        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000396148   0.000681764   0.00139685    0.000632712   0.000235799      19     
produce:reco3d:SpacePointSolver                    14.4835       28.5976       51.6646       26.9587       11.4811        19     
produce:hitpdune:DisambigFromSpacePoints          0.161687      0.352904      0.574256      0.345788      0.102151        19     
produce:cluster3d:Cluster3D                        1.71621       4.45709       9.8072        4.26065       1.80313        19     
[art]:TriggerResults:TriggerResultInserter        2.732e-05    6.16402e-05   0.000145513    4.437e-05    3.54366e-05      19     
end_path:out1:RootOutput                          7.04e-06     3.63238e-05   0.000469329    9.82e-06     0.000102283      19     
end_path:out1:RootOutput(write)                    5.83514       7.90662       9.04061       8.35855       1.01121        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5565.88 MB
  Peak resident set size usage (VmHWM): 3397.35 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0192_df-s04-d1_dw_0_20260203T195507_reco_stage0_20260204T011449_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0192_df-s04-d1_dw_0_20260203T195507_reco_stage0_20260204T011449_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0192_df-s04-d1_dw_0_20260203T195507_reco_stage0_20260204T011449_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770162977.0,
    "core.end_time": 1770167692.0,
    "core.events": [
      21899,
      21905,
      21911,
      21917,
      21923,
      21929,
      21935,
      21941,
      21947,
      21953,
      21959,
      21965,
      21971,
      21977,
      21983,
      21989,
      21995,
      22001,
      22007
    ],
    "core.event_count": 19,
    "core.first_event_number": 21899,
    "core.last_event_number": 22007,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0192_df-s04-d1_dw_0_20260203T195507.hdf5"
    }
  ]
}
justIN time: 2026-02-04 09:16:54 UTC       justIN version: 01.06.00