justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297919.95@dunegpschedd02.fnal.gov

Jobsub ID297919.95@dunegpschedd02.fnal.gov
Workflow ID12717
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:54:25
SiteNL_NIKHEF
EntryVIRGO_NL_NIKHEF_juk_multicore
Last heartbeat2026-02-04 01:33:39
From worker nodeHostnamewn-pijl-009.farm.nikhef.nl
cpuinfoAMD EPYC 9754 128-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:31:23
Input filesvd-protodune:np02vd_raw_run042421_0001_df-s02-d1_dw_0_20260203T133126.hdf5
JobscriptExit code0
Real time1h (7198s)
CPU time47m (2821s = 39%)
Max RSS bytes2599649280 (2479 MiB)
Outputting started2026-02-04 01:31:22
Output files
Finished2026-02-04 01:33:39
Saved logsjustin-logs:297919.95-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ooking up space point ptr... 0x80258350, 0x6da22540
Caught exception in looking up space point ptr... 0x6a0093d0, 0x447d8ee0
Caught exception in looking up space point ptr... 0x447d8ee0, 0x6a0093d0
Caught exception in looking up space point ptr... 0x39e9e950, 0x39e9e7c0
Caught exception in looking up space point ptr... 0x39e9e7c0, 0x39e9e950
Caught exception in looking up space point ptr... 0x3bd9c740, 0x3bd9c610
Caught exception in looking up space point ptr... 0x3bdacad0, 0x3bd9c740
Caught exception in looking up space point ptr... 0x73f0fe70, 0x5d3cd990
Caught exception in looking up space point ptr... 0x5d3cd990, 0x73f0fe70
Caught exception in looking up space point ptr... 0x73f0fe70, 0x5bc49b90
Caught exception in looking up space point ptr... 0x5bc49b90, 0x73f0fe70
Caught exception in looking up space point ptr... 0x3b7045d0, 0x3b706890
Caught exception in looking up space point ptr... 0x3b705350, 0x3b7045d0
Caught exception in looking up space point ptr... 0x6c35bc80, 0x6c35b720
Caught exception in looking up space point ptr... 0x6c35b720, 0x6c35b9f0
Caught exception in looking up space point ptr... 0x739a4470, 0x7399b0f0
Caught exception in looking up space point ptr... 0x7399b0f0, 0x739a47c0
Caught exception in looking up space point ptr... 0x8a62dc30, 0x71dc8730
Caught exception in looking up space point ptr... 0x71dc8730, 0x8a62dc30
Caught exception in looking up space point ptr... 0x3983bdb0, 0x3983a7c0
Caught exception in looking up space point ptr... 0x3983a7c0, 0x39839740
Caught exception in looking up space point ptr... 0x611bbf60, 0x8ca12220
Caught exception in looking up space point ptr... 0x80d2cf40, 0x611bbf60
Caught exception in looking up space point ptr... 0x5c1b7bb0, 0x59cd9970
Caught exception in looking up space point ptr... 0x59cd9970, 0x8c7b9690
Caught exception in looking up space point ptr... 0x3e572720, 0x80d14f50
Caught exception in looking up space point ptr... 0x3e572540, 0x3e572720
Caught exception in looking up space point ptr... 0x5ced8b00, 0x5ced88a0
Caught exception in looking up space point ptr... 0x5ced88a0, 0x5ced8b00
Caught exception in looking up space point ptr... 0x5ced8b00, 0x5ced8c30
Caught exception in looking up space point ptr... 0x5ced8c30, 0x5ced8b00
Caught exception in looking up space point ptr... 0x7c4f5fa0, 0x418e16b0
Caught exception in looking up space point ptr... 0x418e16b0, 0x45ef85f0
Caught exception in looking up space point ptr... 0x45ef85f0, 0x613da800
Caught exception in looking up space point ptr... 0x5d995de0, 0x4080a210
Caught exception in looking up space point ptr... 0x5d995b90, 0x5d995de0
Caught exception in looking up space point ptr... 0x71dd7a60, 0x71dd7bf0
Caught exception in looking up space point ptr... 0x71dd7bf0, 0x71dd7a60
Caught exception in looking up space point ptr... 0x8ac17f40, 0x55c203e0
Caught exception in looking up space point ptr... 0x55c203e0, 0x8ac17f40
Caught exception in looking up space point ptr... 0x51761a50, 0x51761920
Caught exception in looking up space point ptr... 0x51761920, 0x51761a50
Caught exception in looking up space point ptr... 0x90ac66f0, 0x90ac6b00
Caught exception in looking up space point ptr... 0x90ac6b00, 0x90ac6e50
Caught exception in looking up space point ptr... 0x524d1d40, 0x439671a0
Caught exception in looking up space point ptr... 0x56cb6ea0, 0x524d1d40
Caught exception in looking up space point ptr... 0x65f316a0, 0x6cbcef00
Caught exception in looking up space point ptr... 0x6cbcef00, 0x65f316a0
Caught exception in looking up space point ptr... 0x45deb580, 0x45deed60
Caught exception in looking up space point ptr... 0x45deed60, 0x45deb580
Caught exception in looking up space point ptr... 0x762bb890, 0x762bb740
Caught exception in looking up space point ptr... 0x762bb740, 0x762bb890
Caught exception in looking up space point ptr... 0x39a4c000, 0x6f33df80
Caught exception in looking up space point ptr... 0x6f33df80, 0x39a4c000
Caught exception in looking up space point ptr... 0x39a4c000, 0x6f33b230
Caught exception in looking up space point ptr... 0x6f33b230, 0x39a4c000
Caught exception in looking up space point ptr... 0x8253faa0, 0x762eb7b0
Caught exception in looking up space point ptr... 0x762eb7b0, 0x8253faa0
Caught exception in looking up space point ptr... 0x8253faa0, 0x851b0630
Caught exception in looking up space point ptr... 0x851b0630, 0x8253faa0
Caught exception in looking up space point ptr... 0x8253faa0, 0x6824b370
Caught exception in looking up space point ptr... 0x6824b370, 0x8253faa0
Caught exception in looking up space point ptr... 0x61fd8ea0, 0x86455920
Caught exception in looking up space point ptr... 0x86455920, 0x61fd8ea0
Caught exception in looking up space point ptr... 0x5bd7cda0, 0x5bd7cb40
Caught exception in looking up space point ptr... 0x5bd7cb40, 0x5bd7cda0
++++>>>> total num hits: 78421, num free: 76148
04-Feb-2026 02:31:08 CET  Closed output file "np02vd_raw_run042421_0001_df-s02-d1_dw_0_20260203T133126_reco_stage0_20260204T013108_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         205.667       262.402       337.354       256.133       33.7673        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       7.2509e-05    0.000111499   0.000336598   9.8668e-05    4.83979e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                105.02        150.085       219.676       141.982       29.8982        27     
produce:triggerrawdecoder:PDVDTriggerReader4      0.594757      0.804966       1.18683      0.791896      0.143106        27     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0178268     0.0255988     0.0458285     0.0260503    0.00714115       27     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0874689     0.0991903     0.310239      0.0883824     0.0418668       27     
produce:beamevent:BeamEvent                      0.000111428   0.000198613   0.00129117    0.000155694   0.000215674      27     
produce:pdvddaphne:DAPHNEReaderPDVD                11.701        18.6439       24.8736       18.2494       3.06795        27     
produce:ophit:OpHitFinder                         0.0346746     0.0461097     0.0582702     0.0462777    0.00592539       27     
produce:wclsdatavd:WireCellToolkit                 61.1287       70.8196       89.4588       71.109        6.72916        27     
produce:gaushit:GausHitFinder                     0.693149       1.08589       2.23325       1.00814      0.351815        27     
produce:nhitsfilter:NumberOfHitsFilter           0.000305721   0.000971705   0.00174425    0.000900619   0.000384643      27     
produce:reco3d:SpacePointSolver                    8.45845       13.4029       21.326        13.2631       3.11279        27     
produce:hitpdune:DisambigFromSpacePoints          0.0772325     0.133946      0.256534      0.132968      0.0385198       27     
produce:cluster3d:Cluster3D                        1.72539       2.76009       5.00863       2.38514      0.898567        27     
[art]:TriggerResults:TriggerResultInserter       2.5338e-05    3.78588e-05   8.4157e-05    3.1828e-05    1.41474e-05      27     
end_path:out1:RootOutput                          4.557e-06    7.28389e-06   3.4793e-05     5.218e-06    6.32626e-06      27     
end_path:out1:RootOutput(write)                    3.65888       4.43904       5.75053       4.21841       0.71434        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4612.59 MB
  Peak resident set size usage (VmHWM): 2599.65 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042421_0001_df-s02-d1_dw_0_20260203T133126_reco_stage0_20260204T013108_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042421_0001_df-s02-d1_dw_0_20260203T133126_reco_stage0_20260204T013108_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042421_0001_df-s02-d1_dw_0_20260203T133126_reco_stage0_20260204T013108_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770161495.0,
    "core.end_time": 1770168671.0,
    "core.events": [
      163,
      169,
      175,
      181,
      187,
      193,
      199,
      205,
      211,
      217,
      223,
      229,
      235,
      241,
      247,
      253,
      259,
      265,
      271,
      277,
      283,
      289,
      295,
      301,
      307,
      313,
      319
    ],
    "core.event_count": 27,
    "core.first_event_number": 163,
    "core.last_event_number": 319,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42421
    ],
    "core.runs_subruns": [
      4242100001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042421_0001_df-s02-d1_dw_0_20260203T133126.hdf5"
    }
  ]
}
justIN time: 2026-02-04 06:00:38 UTC       justIN version: 01.06.00