justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297920.124@dunegpschedd02.fnal.gov

Jobsub ID297920.124@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:54:27
SiteUK_Oxford
EntryDUNE_UK_SGrid_Oxford_arc01
Last heartbeat2026-02-04 01:39:25
From worker nodeHostnamet2wn064.physics.ox.ac.uk
cpuinfoAMD EPYC 7763 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:42:59
Input filesvd-protodune:np02vd_raw_run042433_0282_df-s03-d1_dw_0_20260203T202919.hdf5
JobscriptExit code0
Real time1h (6791s)
CPU time1h (3750s = 55%)
Max RSS bytes3364179968 (3208 MiB)
Outputting started2026-02-04 01:36:11
Output files
Finished2026-02-04 01:39:25
Saved logsjustin-logs:297920.124-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

int ptr... 0x79e0080, 0x7d5aa730
Caught exception in looking up space point ptr... 0x6985e020, 0x698651a0
Caught exception in looking up space point ptr... 0x698651a0, 0x65d913d0
Caught exception in looking up space point ptr... 0x12f18090, 0x12f348d0
Caught exception in looking up space point ptr... 0x12f348d0, 0x12f233f0
Caught exception in looking up space point ptr... 0x52cc30d0, 0x52cc68f0
Caught exception in looking up space point ptr... 0x52cc68f0, 0x52cc3900
Caught exception in looking up space point ptr... 0x53bb3f10, 0x53bb2dc0
Caught exception in looking up space point ptr... 0x53bb2dc0, 0x53bb3f10
Caught exception in looking up space point ptr... 0xd140620, 0x79e0690
Caught exception in looking up space point ptr... 0x79e0690, 0xd140620
Caught exception in looking up space point ptr... 0xe930f80, 0x32674690
Caught exception in looking up space point ptr... 0x32674690, 0x3e115220
Caught exception in looking up space point ptr... 0x528ebd30, 0x5f6b7ee0
Caught exception in looking up space point ptr... 0x5f6b7ee0, 0x528ebd30
Caught exception in looking up space point ptr... 0x5efa3480, 0x5efabd20
Caught exception in looking up space point ptr... 0x5efabd20, 0x5efa3480
Caught exception in looking up space point ptr... 0x5433f320, 0x7a59be0
Caught exception in looking up space point ptr... 0x7a59be0, 0x7887d0f0
Caught exception in looking up space point ptr... 0x58cd5770, 0x58cf33e0
Caught exception in looking up space point ptr... 0x58cf33e0, 0x58cd5770
Caught exception in looking up space point ptr... 0x15c11170, 0x6296a770
Caught exception in looking up space point ptr... 0x6296a770, 0x15c11170
Caught exception in looking up space point ptr... 0x4a342c50, 0x4a342b20
Caught exception in looking up space point ptr... 0x4a342b20, 0x4a341ed0
Caught exception in looking up space point ptr... 0x4dbade20, 0x1097f6a0
Caught exception in looking up space point ptr... 0x1097f6a0, 0x29d18630
Caught exception in looking up space point ptr... 0xfe73180, 0xfe732d0
Caught exception in looking up space point ptr... 0x5f5b6900, 0xfe73180
Caught exception in looking up space point ptr... 0x4cfc8b70, 0x4cfc8d00
Caught exception in looking up space point ptr... 0x4cfc8d00, 0x4cfc8b70
Caught exception in looking up space point ptr... 0x53260520, 0x53260ae0
Caught exception in looking up space point ptr... 0x53260ae0, 0x5325fcd0
Caught exception in looking up space point ptr... 0x5f696830, 0x342c15e0
Caught exception in looking up space point ptr... 0x69dae840, 0x5f696830
Caught exception in looking up space point ptr... 0x35e472b0, 0x65cdb3d0
Caught exception in looking up space point ptr... 0x65cdb3d0, 0x65cdaf30
Caught exception in looking up space point ptr... 0x4cedd410, 0x4cee2aa0
Caught exception in looking up space point ptr... 0x4cee2aa0, 0x4cee9fc0
Caught exception in looking up space point ptr... 0x4d07aa70, 0x4d07c830
Caught exception in looking up space point ptr... 0x4d07c830, 0x4d07aa70
Caught exception in looking up space point ptr... 0x1763b790, 0x1763b3a0
Caught exception in looking up space point ptr... 0x1763b3a0, 0x1763b790
Caught exception in looking up space point ptr... 0x594df070, 0x594df390
Caught exception in looking up space point ptr... 0x594df390, 0x594df520
Caught exception in looking up space point ptr... 0x104e4a30, 0x104e4620
Caught exception in looking up space point ptr... 0x104e4620, 0x104e4a30
Caught exception in looking up space point ptr... 0x58dd7b30, 0x58dd60a0
Caught exception in looking up space point ptr... 0x58dd60a0, 0x5fce47c0
Caught exception in looking up space point ptr... 0x5f5c9410, 0x5f5cdd60
Caught exception in looking up space point ptr... 0x5f5cdd60, 0x5f5c6050
Caught exception in looking up space point ptr... 0x7a374eb0, 0x7a374d10
Caught exception in looking up space point ptr... 0x7a374d10, 0x7a374c20
Caught exception in looking up space point ptr... 0x36274570, 0x36274310
Caught exception in looking up space point ptr... 0x36274310, 0x54040f60
Caught exception in looking up space point ptr... 0x558bd5b0, 0x3d4fbec0
Caught exception in looking up space point ptr... 0x3d4fbec0, 0x2bd7fe00
Caught exception in looking up space point ptr... 0x53c825b0, 0x53a29f30
Caught exception in looking up space point ptr... 0x53a29f30, 0x53c95e10
Caught exception in looking up space point ptr... 0x558b5050, 0x558a7740
Caught exception in looking up space point ptr... 0x558a7740, 0x558b5050
Caught exception in looking up space point ptr... 0x4aa6c840, 0x3b1b93a0
Caught exception in looking up space point ptr... 0x3b1b93a0, 0x4aa6c840
Caught exception in looking up space point ptr... 0x75fdd1a0, 0x75fdce80
Caught exception in looking up space point ptr... 0x75fdce80, 0x75fdd1a0
Caught exception in looking up space point ptr... 0x7ae84800, 0x5e255170
Caught exception in looking up space point ptr... 0x5e255170, 0x7ae84800
++++>>>> total num hits: 149506, num free: 143043
04-Feb-2026 01:35:59 GMT  Closed output file "np02vd_raw_run042433_0282_df-s03-d1_dw_0_20260203T202919_reco_stage0_20260204T013559_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         185.227       351.917       537.695       384.978       125.813        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       0.000114524   0.000189398   0.000347169   0.000196678   5.89419e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                15.8791       149.975       330.272       171.867       111.118        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.584365       0.90532       1.48869       0.90671      0.282027        19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0196338     0.023026      0.0469309     0.0199239     0.0065843       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0194941     0.0216146     0.0467539     0.019878     0.00602478       19     
produce:beamevent:BeamEvent                      0.000152625   0.000278169   0.00167302    0.000204482   0.000329438      19     
produce:pdvddaphne:DAPHNEReaderPDVD                3.66617       13.2221       26.1093       15.091         7.874         19     
produce:ophit:OpHitFinder                         0.033418      0.0466706     0.0588068     0.0454431    0.00581436       19     
produce:wclsdatavd:WireCellToolkit                 117.29        141.161       156.567       142.865       11.3269        19     
produce:gaushit:GausHitFinder                      1.18038       2.13226       3.43973       1.87161      0.565553        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000351778   0.000617493   0.00140062    0.000566259   0.000218102      19     
produce:reco3d:SpacePointSolver                    16.4055       30.2475       49.9016       29.8706       9.06535        19     
produce:hitpdune:DisambigFromSpacePoints          0.160154      0.329446      0.581021      0.284356      0.118068        19     
produce:cluster3d:Cluster3D                        2.2981        6.8012        47.1551       3.88603       9.71193        19     
[art]:TriggerResults:TriggerResultInserter       2.8793e-05    4.60796e-05   0.00013922    3.9994e-05    2.35447e-05      19     
end_path:out1:RootOutput                          7.905e-06    1.23024e-05   3.7921e-05     1.107e-05    6.30403e-06      19     
end_path:out1:RootOutput(write)                    4.80965       6.98159       9.12546       7.17703       1.09385        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5458.81 MB
  Peak resident set size usage (VmHWM): 3364.18 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0282_df-s03-d1_dw_0_20260203T202919_reco_stage0_20260204T013559_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0282_df-s03-d1_dw_0_20260203T202919_reco_stage0_20260204T013559_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0282_df-s03-d1_dw_0_20260203T202919_reco_stage0_20260204T013559_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770162197.0,
    "core.end_time": 1770168961.0,
    "core.events": [
      32157,
      32163,
      32169,
      32175,
      32181,
      32187,
      32193,
      32199,
      32205,
      32211,
      32217,
      32223,
      32229,
      32235,
      32241,
      32247,
      32253,
      32259,
      32265
    ],
    "core.event_count": 19,
    "core.first_event_number": 32157,
    "core.last_event_number": 32265,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0282_df-s03-d1_dw_0_20260203T202919.hdf5"
    }
  ]
}
justIN time: 2026-02-04 04:28:39 UTC       justIN version: 01.06.00