justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303775.1@dunegpschedd01.fnal.gov

Jobsub ID303775.1@dunegpschedd01.fnal.gov
Workflow ID12718
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-04 02:18:33
SiteUK_Manchester
EntryUBoone_T2_UK_Manchester_ce02
Last heartbeat2026-02-04 03:40:25
From worker nodeHostnamewn1208091.tier2.hep.manchester.ac.uk
cpuinfoAMD EPYC 7702P 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-04 02:19:05
Input filesvd-protodune:np02vd_raw_run042422_0080_df-s02-d0_dw_0_20260203T155803.hdf5
JobscriptExit code0
Real time1h (4688s)
CPU time1h (3990s = 85%)
Max RSS bytes2584223744 (2464 MiB)
Outputting started2026-02-04 03:37:15
Output files
Finished2026-02-04 03:40:25
Saved logsjustin-logs:303775.1-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

40
Caught exception in looking up space point ptr... 0x61a103b0, 0x60253800
Caught exception in looking up space point ptr... 0x60253800, 0x60253990
Caught exception in looking up space point ptr... 0x60253990, 0x61a106d0
Caught exception in looking up space point ptr... 0x4f7ca740, 0x4f7ca240
Caught exception in looking up space point ptr... 0x4f7ca240, 0x4f7ca740
Caught exception in looking up space point ptr... 0x503be030, 0x503be350
Caught exception in looking up space point ptr... 0x503be350, 0x503be030
Caught exception in looking up space point ptr... 0x4e66eb20, 0x4e678510
Caught exception in looking up space point ptr... 0x4e678510, 0x4e66eb20
Caught exception in looking up space point ptr... 0x503cb320, 0x503cb4b0
Caught exception in looking up space point ptr... 0x503cb4b0, 0x13a86b80
Caught exception in looking up space point ptr... 0xfee9e70, 0xfee9ce0
Caught exception in looking up space point ptr... 0xfee9ce0, 0xfee96e0
Caught exception in looking up space point ptr... 0x3bc799e0, 0x66019310
Caught exception in looking up space point ptr... 0x66019310, 0x3bc799e0
Caught exception in looking up space point ptr... 0x3bc799e0, 0x3bc79680
Caught exception in looking up space point ptr... 0x3bc79680, 0x3bc799e0
Caught exception in looking up space point ptr... 0x13e56240, 0x13e560b0
Caught exception in looking up space point ptr... 0x13e55c00, 0x13e56240
Caught exception in looking up space point ptr... 0x143349e0, 0xf2c0c40
Caught exception in looking up space point ptr... 0xf2c0c40, 0x143349e0
Caught exception in looking up space point ptr... 0x47d86710, 0x142bf9e0
Caught exception in looking up space point ptr... 0x142bf9e0, 0x142bfb70
Caught exception in looking up space point ptr... 0x4f7deca0, 0x4f7e98f0
Caught exception in looking up space point ptr... 0x4f7e98f0, 0x4f7e9760
Caught exception in looking up space point ptr... 0x4e655470, 0x25a96980
Caught exception in looking up space point ptr... 0x25a96980, 0x25a8e860
Caught exception in looking up space point ptr... 0xfee8650, 0xfee70a0
Caught exception in looking up space point ptr... 0xfee70a0, 0xfee8650
Caught exception in looking up space point ptr... 0x5e1a1830, 0x5e1a1730
Caught exception in looking up space point ptr... 0x5e1a1730, 0x5e1a2850
Caught exception in looking up space point ptr... 0x654bb6f0, 0x654f2b30
Caught exception in looking up space point ptr... 0x654f2b30, 0x654eff30
Caught exception in looking up space point ptr... 0x503c8da0, 0x4e676eb0
Caught exception in looking up space point ptr... 0x503c8c10, 0x503c8da0
Caught exception in looking up space point ptr... 0x145b9a90, 0x167601d0
Caught exception in looking up space point ptr... 0x167601d0, 0x65763cf0
Caught exception in looking up space point ptr... 0x4fdd8110, 0x65ba2dc0
Caught exception in looking up space point ptr... 0x65ba2dc0, 0x4fdd8110
Caught exception in looking up space point ptr... 0x5e293820, 0x5e294520
Caught exception in looking up space point ptr... 0x5e294520, 0x5e293820
Caught exception in looking up space point ptr... 0x596b5100, 0x596b4f70
Caught exception in looking up space point ptr... 0xb307c70, 0x596b5100
Caught exception in looking up space point ptr... 0x15af7a70, 0x426a6570
Caught exception in looking up space point ptr... 0x426a6570, 0x15af7a70
Caught exception in looking up space point ptr... 0x4ed08fd0, 0x5e1e66e0
Caught exception in looking up space point ptr... 0x5e1e66e0, 0x4ed08fd0
Caught exception in looking up space point ptr... 0x5738d240, 0x2118e9e0
Caught exception in looking up space point ptr... 0x2118e9e0, 0x5738d240
Caught exception in looking up space point ptr... 0x2398c8d0, 0x4554ad20
Caught exception in looking up space point ptr... 0x4554ad20, 0x2398c8d0
Caught exception in looking up space point ptr... 0x3fbc4da0, 0x124f3410
Caught exception in looking up space point ptr... 0x124f3410, 0x3fbc4da0
Caught exception in looking up space point ptr... 0x4e65bf70, 0x4e659120
Caught exception in looking up space point ptr... 0x4e659120, 0x4e65bf70
Caught exception in looking up space point ptr... 0x16af39c0, 0x54c07640
Caught exception in looking up space point ptr... 0x54c07640, 0x16af39c0
Caught exception in looking up space point ptr... 0x4f7c8d50, 0x4f7c8bc0
Caught exception in looking up space point ptr... 0x4f7c8bc0, 0x4f7c8d50
Caught exception in looking up space point ptr... 0x3fbbc570, 0x3fbbc700
Caught exception in looking up space point ptr... 0x3fbbc700, 0x3fbbc570
Caught exception in looking up space point ptr... 0x4a2a54d0, 0x4a2a5380
Caught exception in looking up space point ptr... 0x4a2a5380, 0x4a2a5090
Caught exception in looking up space point ptr... 0x4e663fe0, 0x4e664110
Caught exception in looking up space point ptr... 0x4e664110, 0x4e664370
++++>>>> total num hits: 301146, num free: 294398
04-Feb-2026 03:37:02 GMT  Closed output file "np02vd_raw_run042422_0080_df-s02-d0_dw_0_20260203T155803_reco_stage0_20260204T033702_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         157.01        168.119       198.157       164.248       9.88842        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                        9.55e-05     0.000114518   0.00032877    0.000104166   4.31455e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                15.473        21.7814       37.2107       21.1264       3.80351        27     
produce:triggerrawdecoder:PDVDTriggerReader4      0.695393      0.749307      0.858924      0.737934      0.0412078       27     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0222978     0.0243984     0.0448969     0.0227135    0.00458515       27     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.109896      0.112267      0.131522       0.11139     0.00386331       27     
produce:beamevent:BeamEvent                      9.2995e-05    0.000175894   0.00140155    0.000117322   0.000243087      27     
produce:pdvddaphne:DAPHNEReaderPDVD                4.85482       5.45362       6.17196       5.41274      0.316434        27     
produce:ophit:OpHitFinder                         0.041158      0.0545973     0.0669361     0.0545431    0.00530101       27     
produce:wclsdatavd:WireCellToolkit                 104.915       109.195       113.601       109.582       1.74778        27     
produce:gaushit:GausHitFinder                      1.25448       1.7369        2.79591       1.7095       0.367585        27     
produce:nhitsfilter:NumberOfHitsFilter           0.00030782    0.000439104   0.000807101   0.000422446   0.000110921      27     
produce:reco3d:SpacePointSolver                    10.945        16.8658       26.2001       15.6438       3.79312        27     
produce:hitpdune:DisambigFromSpacePoints          0.102545      0.187851      0.344697      0.170866      0.0628013       27     
produce:cluster3d:Cluster3D                        1.5711        5.27468       37.861        3.41257       7.12441        27     
[art]:TriggerResults:TriggerResultInserter       2.7511e-05    3.66101e-05   0.000114616   3.3663e-05    1.57776e-05      27     
end_path:out1:RootOutput                          4.128e-06    6.04141e-06   3.0077e-05     5.089e-06    4.75544e-06      27     
end_path:out1:RootOutput(write)                    5.93131       6.65291       8.03724       6.60874      0.452505        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4535.82 MB
  Peak resident set size usage (VmHWM): 2584.22 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042422_0080_df-s02-d0_dw_0_20260203T155803_reco_stage0_20260204T033702_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042422_0080_df-s02-d0_dw_0_20260203T155803_reco_stage0_20260204T033702_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042422_0080_df-s02-d0_dw_0_20260203T155803_reco_stage0_20260204T033702_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770171564.0,
    "core.end_time": 1770176225.0,
    "core.events": [
      12978,
      12984,
      12990,
      12996,
      13002,
      13008,
      13014,
      13020,
      13026,
      13032,
      13038,
      13044,
      13050,
      13056,
      13062,
      13068,
      13074,
      13080,
      13086,
      13092,
      13098,
      13104,
      13110,
      13116,
      13122,
      13128,
      13134
    ],
    "core.event_count": 27,
    "core.first_event_number": 12978,
    "core.last_event_number": 13134,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42422
    ],
    "core.runs_subruns": [
      4242200001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042422_0080_df-s02-d0_dw_0_20260203T155803.hdf5"
    }
  ]
}
justIN time: 2026-02-04 09:54:01 UTC       justIN version: 01.06.00