justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297921.85@dunegpschedd02.fnal.gov

Jobsub ID297921.85@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:58:26
SiteUK_RAL-Tier1
EntryLIGO_UK_RAL_arc_ce05
Last heartbeat2026-02-04 01:06:14
From worker nodeHostnamedune001-1843717.0-lcg2395.gridpp.rl.ac.uk
cpuinfoAMD EPYC 7452 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit216000 (60 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:54:39
Input filesvd-protodune:np02vd_raw_run042433_0285_df-s02-d1_dw_0_20260203T203027.hdf5
JobscriptExit code0
Real time1h (4185s)
CPU time1h (3838s = 91%)
Max RSS bytes3412680704 (3254 MiB)
Outputting started2026-02-04 01:04:25
Output files
Finished2026-02-04 01:06:14
Saved logsjustin-logs:297921.85-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

. 0x907f40d0, 0x907f43f0
Caught exception in looking up space point ptr... 0x87e3c660, 0x87e3c4d0
Caught exception in looking up space point ptr... 0x87e3c4d0, 0x87e3c980
Caught exception in looking up space point ptr... 0x53641610, 0x53641430
Caught exception in looking up space point ptr... 0x53641430, 0x347d94c0
Caught exception in looking up space point ptr... 0xa99b0600, 0xa99b0060
Caught exception in looking up space point ptr... 0xa99b0060, 0xa99afdd0
Caught exception in looking up space point ptr... 0x35d23b00, 0x35d2ab10
Caught exception in looking up space point ptr... 0x5f081bb0, 0x35d23b00
Caught exception in looking up space point ptr... 0x37e39480, 0x37e439e0
Caught exception in looking up space point ptr... 0x37e439e0, 0x37e39480
Caught exception in looking up space point ptr... 0x7098f830, 0x5a2ee0a0
Caught exception in looking up space point ptr... 0x5a2ee0a0, 0x5a2edfa0
Caught exception in looking up space point ptr... 0x5eae5b70, 0x575e8bf0
Caught exception in looking up space point ptr... 0x575e8bf0, 0x5eae5b70
Caught exception in looking up space point ptr... 0xa99acbf0, 0xa99ac9f0
Caught exception in looking up space point ptr... 0xa99ac9f0, 0xa99acbf0
Caught exception in looking up space point ptr... 0x68e0f2b0, 0x651c6af0
Caught exception in looking up space point ptr... 0x651c6af0, 0x68e0f2b0
Caught exception in looking up space point ptr... 0x37823c90, 0x78939050
Caught exception in looking up space point ptr... 0x78939050, 0x37823c90
Caught exception in looking up space point ptr... 0x92bdcff0, 0x92bcf400
Caught exception in looking up space point ptr... 0x92bcf400, 0x92bdcff0
Caught exception in looking up space point ptr... 0x8f1161c0, 0x5d229f90
Caught exception in looking up space point ptr... 0x5d229f90, 0x8f1161c0
Caught exception in looking up space point ptr... 0xaa2aabe0, 0x5f8b2700
Caught exception in looking up space point ptr... 0x5f8b2700, 0xaa2aabe0
Caught exception in looking up space point ptr... 0x39383360, 0x39383040
Caught exception in looking up space point ptr... 0x39383040, 0x39383360
Caught exception in looking up space point ptr... 0x84da19d0, 0x84da2470
Caught exception in looking up space point ptr... 0x8eefca20, 0x84da19d0
Caught exception in looking up space point ptr... 0xaea0c840, 0xaea0c520
Caught exception in looking up space point ptr... 0xaea0c520, 0xaea0c840
Caught exception in looking up space point ptr... 0x3d15ef80, 0x3d15ed30
Caught exception in looking up space point ptr... 0x3d15ed30, 0x3d15f1d0
Caught exception in looking up space point ptr... 0xa8b1e6f0, 0xa8b18440
Caught exception in looking up space point ptr... 0xa8b18440, 0xa8b1e6f0
Caught exception in looking up space point ptr... 0x84ed3ce0, 0x84ed3b50
Caught exception in looking up space point ptr... 0x84ed3b50, 0x84ed3ce0
Caught exception in looking up space point ptr... 0x61715f10, 0x61716530
Caught exception in looking up space point ptr... 0x61716530, 0x61715e70
Caught exception in looking up space point ptr... 0x3d2eb840, 0x3d2ea480
Caught exception in looking up space point ptr... 0x3d2ea480, 0x3d2eb4e0
Caught exception in looking up space point ptr... 0x3b493ee0, 0x3b49d460
Caught exception in looking up space point ptr... 0x3b49d460, 0x3b4a5330
Caught exception in looking up space point ptr... 0xa8fc3970, 0x907d3d30
Caught exception in looking up space point ptr... 0x907d3d30, 0xa8fc3970
Caught exception in looking up space point ptr... 0x8633f290, 0x8633eff0
Caught exception in looking up space point ptr... 0x8633eff0, 0x8633f290
Caught exception in looking up space point ptr... 0x8633f290, 0x8633eea0
Caught exception in looking up space point ptr... 0x8633eea0, 0x8633f290
Caught exception in looking up space point ptr... 0x33c30c40, 0x95592d70
Caught exception in looking up space point ptr... 0x95592d70, 0x33c30c40
Caught exception in looking up space point ptr... 0x70872850, 0x9550f200
Caught exception in looking up space point ptr... 0x9550f200, 0xa724a580
Caught exception in looking up space point ptr... 0x8be4ba50, 0x7277b8c0
Caught exception in looking up space point ptr... 0x7277b8c0, 0x6f919c20
Caught exception in looking up space point ptr... 0x65069ad0, 0x8c1c4450
Caught exception in looking up space point ptr... 0x8c1c4450, 0x8638ea10
Caught exception in looking up space point ptr... 0xa6bccc80, 0xad46ef20
Caught exception in looking up space point ptr... 0xad46ef20, 0xa6bccc80
Caught exception in looking up space point ptr... 0x8eb12b00, 0x87fc3d70
Caught exception in looking up space point ptr... 0x87fc3d70, 0x8eb12b00
Caught exception in looking up space point ptr... 0x571fb700, 0x8c617ce0
Caught exception in looking up space point ptr... 0x8c617ce0, 0xad71d100
Caught exception in looking up space point ptr... 0x61ed5b40, 0x36088260
Caught exception in looking up space point ptr... 0x36088260, 0x7f530230
++++>>>> total num hits: 89744, num free: 85817
04-Feb-2026 01:04:14 UTC  Closed output file "np02vd_raw_run042433_0285_df-s02-d1_dw_0_20260203T203027_reco_stage0_20260204T010414_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         188.091       214.604       237.677       215.957       11.3641        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       0.000102152   0.000184322   0.000397147   0.000137759   9.7096e-05       19     
produce:tpcrawdecoder:PDVDTPCReader                18.4087       19.2188       20.1507       19.2373      0.478945        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.727901      0.835234      0.976806      0.831359      0.0616483       19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0230758     0.0281003     0.0707897     0.0237287     0.0106651       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0229849     0.0235213     0.0243623     0.0234125    0.000337096      19     
produce:beamevent:BeamEvent                      0.00012754    0.000285579   0.00159449    0.000222108   0.000312838      19     
produce:pdvddaphne:DAPHNEReaderPDVD                3.34361       3.90082       4.42038       3.86281      0.271494        19     
produce:ophit:OpHitFinder                         0.0240548     0.0441996     0.0598873     0.0446659    0.00683118       19     
produce:wclsdatavd:WireCellToolkit                 132.555       147.432       157.156       147.11        5.83787        19     
produce:gaushit:GausHitFinder                      1.31467       2.0702        3.0558        2.0682       0.400764        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000489301   0.000683995   0.00140262    0.000615189   0.000219577      19     
produce:reco3d:SpacePointSolver                    19.2761       29.7291       39.0875       29.4219       5.5538         19     
produce:hitpdune:DisambigFromSpacePoints          0.187572      0.307173      0.464948      0.306393      0.073687        19     
produce:cluster3d:Cluster3D                        1.89192       4.09049       7.72046       3.73035       1.6029         19     
[art]:TriggerResults:TriggerResultInserter       3.1239e-05    4.36717e-05   0.000138831   3.6178e-05    2.35494e-05      19     
end_path:out1:RootOutput                          8.215e-06    1.27741e-05   3.3834e-05    1.1943e-05    5.28365e-06      19     
end_path:out1:RootOutput(write)                    5.56281       6.85447       7.59444       7.0519        0.56759        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5541.97 MB
  Peak resident set size usage (VmHWM): 3412.68 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0285_df-s02-d1_dw_0_20260203T203027_reco_stage0_20260204T010414_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0285_df-s02-d1_dw_0_20260203T203027_reco_stage0_20260204T010414_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0285_df-s02-d1_dw_0_20260203T203027_reco_stage0_20260204T010414_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770162892.0,
    "core.end_time": 1770167057.0,
    "core.events": [
      32497,
      32503,
      32509,
      32515,
      32521,
      32527,
      32533,
      32539,
      32545,
      32551,
      32557,
      32563,
      32569,
      32575,
      32581,
      32587,
      32593,
      32599,
      32605
    ],
    "core.event_count": 19,
    "core.first_event_number": 32497,
    "core.last_event_number": 32605,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0285_df-s02-d1_dw_0_20260203T203027.hdf5"
    }
  ]
}
justIN time: 2026-02-04 09:13:22 UTC       justIN version: 01.06.00