justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297921.66@dunegpschedd02.fnal.gov

Jobsub ID297921.66@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:58:26
SiteUK_Manchester
EntryUBoone_T2_UK_Manchester_ce01
Last heartbeat2026-02-04 02:59:00
From worker nodeHostnamewn1907291.tier2.hep.manchester.ac.uk
cpuinfoIntel(R) Xeon(R) Gold 5118 CPU @ 2.30GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:52:12
Input filesvd-protodune:np02vd_raw_run042433_0402_df-s04-d0_dw_0_20260203T211455.hdf5
JobscriptExit code0
Real time3h (11029s)
CPU time1h (5992s = 54%)
Max RSS bytes3345723392 (3190 MiB)
Outputting started2026-02-04 02:56:03
Output files
Finished2026-02-04 02:59:00
Saved logsjustin-logs:297921.66-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

r... 0x4294d6d0, 0x38ef7650
Caught exception in looking up space point ptr... 0x4294d8e0, 0x4294d6d0
Caught exception in looking up space point ptr... 0x15b53510, 0x15b531f0
Caught exception in looking up space point ptr... 0x38781920, 0x15b53380
Caught exception in looking up space point ptr... 0x15b53380, 0x15b53510
Caught exception in looking up space point ptr... 0x3791beb0, 0x378c75f0
Caught exception in looking up space point ptr... 0x378c75f0, 0x3791beb0
Caught exception in looking up space point ptr... 0x38ef7af0, 0x10302a90
Caught exception in looking up space point ptr... 0x10302a90, 0x7b1e49c0
Caught exception in looking up space point ptr... 0x38e3d250, 0x467b7b50
Caught exception in looking up space point ptr... 0x467b7b50, 0x3758ea90
Caught exception in looking up space point ptr... 0x38540a20, 0x3b0f5c70
Caught exception in looking up space point ptr... 0x3b0f5c70, 0x9ec2c70
Caught exception in looking up space point ptr... 0x38782040, 0x17ffd7f0
Caught exception in looking up space point ptr... 0x17ffd7f0, 0x38782040
Caught exception in looking up space point ptr... 0x59963840, 0x4bccd3e0
Caught exception in looking up space point ptr... 0x4bccd3e0, 0x3c96c930
Caught exception in looking up space point ptr... 0x3c96c930, 0x42bdd500
Caught exception in looking up space point ptr... 0x42bdd500, 0x52cc4180
Caught exception in looking up space point ptr... 0x52cc4180, 0x59963840
Caught exception in looking up space point ptr... 0x74c5a000, 0x74c57a90
Caught exception in looking up space point ptr... 0x621d72d0, 0x74c5a000
Caught exception in looking up space point ptr... 0x36e18e00, 0x38c83670
Caught exception in looking up space point ptr... 0x36e18cd0, 0x36e18e00
Caught exception in looking up space point ptr... 0x8c33140, 0x4a2ba690
Caught exception in looking up space point ptr... 0x4a2bab30, 0x8c33140
Caught exception in looking up space point ptr... 0x2b7a43c0, 0x2b7d8350
Caught exception in looking up space point ptr... 0x2b7d8350, 0x18763420
Caught exception in looking up space point ptr... 0x18763420, 0x2b7c1050
Caught exception in looking up space point ptr... 0x47d10310, 0x47d104a0
Caught exception in looking up space point ptr... 0x47d104a0, 0x47d15ae0
Caught exception in looking up space point ptr... 0x8134b170, 0x62c158a0
Caught exception in looking up space point ptr... 0x62c158a0, 0x8134b170
Caught exception in looking up space point ptr... 0x6232d9e0, 0x6f5dd700
Caught exception in looking up space point ptr... 0x6f5dd3e0, 0x6232d9e0
Caught exception in looking up space point ptr... 0x3d36c1d0, 0x3d369eb0
Caught exception in looking up space point ptr... 0x5d7342f0, 0x3d36c1d0
Caught exception in looking up space point ptr... 0x3b1cf080, 0x3b1cf190
Caught exception in looking up space point ptr... 0x3b1cf190, 0x3b1cfa60
Caught exception in looking up space point ptr... 0x7996b110, 0x7996b640
Caught exception in looking up space point ptr... 0x7996b640, 0x7996b110
Caught exception in looking up space point ptr... 0x3024e8d0, 0x18cd8c50
Caught exception in looking up space point ptr... 0x18cd8c50, 0x814b9a70
Caught exception in looking up space point ptr... 0x4a26eed0, 0x30312de0
Caught exception in looking up space point ptr... 0x30312de0, 0x30312ce0
Caught exception in looking up space point ptr... 0x62c13d90, 0x62c13ee0
Caught exception in looking up space point ptr... 0x62c13ee0, 0x62c14180
Caught exception in looking up space point ptr... 0x66f3f710, 0x58887a80
Caught exception in looking up space point ptr... 0x58887a80, 0x66f3f710
Caught exception in looking up space point ptr... 0x49e933c0, 0x49e92040
Caught exception in looking up space point ptr... 0x49e92040, 0x49e933c0
Caught exception in looking up space point ptr... 0x36e95470, 0x62c31e90
Caught exception in looking up space point ptr... 0x62c31e90, 0x36e95470
Caught exception in looking up space point ptr... 0x49f2ca60, 0x49f35c90
Caught exception in looking up space point ptr... 0x49f35c90, 0x49f2ca60
Caught exception in looking up space point ptr... 0x5cf6e820, 0x5cf6e380
Caught exception in looking up space point ptr... 0x5cf6e380, 0x2b7e77c0
Caught exception in looking up space point ptr... 0x3b145620, 0x66e4a7d0
Caught exception in looking up space point ptr... 0x66e4a7d0, 0x66e4a8d0
Caught exception in looking up space point ptr... 0x66b4afd0, 0x54de64f0
Caught exception in looking up space point ptr... 0x3b4e3120, 0x66b4afd0
Caught exception in looking up space point ptr... 0x351f6510, 0x351f76b0
Caught exception in looking up space point ptr... 0x351f76b0, 0x351f6510
Caught exception in looking up space point ptr... 0x74ebabd0, 0x74ebb3c0
Caught exception in looking up space point ptr... 0x74ebb3c0, 0x74ebbb40
Caught exception in looking up space point ptr... 0x80acd7c0, 0x80acd660
Caught exception in looking up space point ptr... 0x80acd660, 0x80acd7c0
++++>>>> total num hits: 58039, num free: 55834
04-Feb-2026 02:55:47 GMT  Closed output file "np02vd_raw_run042433_0402_df-s04-d0_dw_0_20260203T211455_reco_stage0_20260204T025547_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         493.941       573.065       674.847       573.315       43.8227        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       8.5938e-05    0.000238607   0.000549759   0.000209393   0.000129482      19     
produce:tpcrawdecoder:PDVDTPCReader                188.016       249.537       364.683       247.975       42.2324        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.840468       1.16968       1.73883       1.12184      0.225111        19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0220347     0.0319389     0.0710862     0.0303766     0.0107177       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.021979      0.0233619     0.0314488     0.0228981    0.00196962       19     
produce:beamevent:BeamEvent                      0.000172445   0.000332603   0.00140613    0.000272891   0.000260277      19     
produce:pdvddaphne:DAPHNEReaderPDVD                20.0095       24.9555       30.6491       24.1257       3.04891        19     
produce:ophit:OpHitFinder                         0.0483572     0.0760741     0.0897016     0.0769533     0.011167        19     
produce:wclsdatavd:WireCellToolkit                 225.938       247.779       281.609       242.946       15.4661        19     
produce:gaushit:GausHitFinder                      1.84163       2.82732       3.76088       2.76467      0.461012        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000599073   0.000952177    0.001544     0.000929835   0.000243027      19     
produce:reco3d:SpacePointSolver                    18.9063       31.5013       42.537        32.4327       6.85474        19     
produce:hitpdune:DisambigFromSpacePoints          0.272064      0.477501      0.652029       0.50699      0.104454        19     
produce:cluster3d:Cluster3D                        2.86972       4.86406       7.21896       4.7159        1.23723        19     
[art]:TriggerResults:TriggerResultInserter        4.387e-05    7.01621e-05   0.000154127   6.0996e-05    2.40487e-05      19     
end_path:out1:RootOutput                          8.73e-06     1.95041e-05   4.7541e-05    1.5629e-05    1.00631e-05      19     
end_path:out1:RootOutput(write)                    7.57638       9.73828       10.9755       9.84741       0.83239        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5321.26 MB
  Peak resident set size usage (VmHWM): 3345.72 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0402_df-s04-d0_dw_0_20260203T211455_reco_stage0_20260204T025547_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0402_df-s04-d0_dw_0_20260203T211455_reco_stage0_20260204T025547_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0402_df-s04-d0_dw_0_20260203T211455_reco_stage0_20260204T025547_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770162747.0,
    "core.end_time": 1770173754.0,
    "core.events": [
      45838,
      45844,
      45850,
      45856,
      45862,
      45868,
      45874,
      45880,
      45886,
      45892,
      45898,
      45904,
      45910,
      45916,
      45922,
      45928,
      45934,
      45940,
      45946
    ],
    "core.event_count": 19,
    "core.first_event_number": 45838,
    "core.last_event_number": 45946,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0402_df-s04-d0_dw_0_20260203T211455.hdf5"
    }
  ]
}
justIN time: 2026-02-04 04:32:57 UTC       justIN version: 01.06.00