justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303840.0@dunegpschedd01.fnal.gov

Jobsub ID303840.0@dunegpschedd01.fnal.gov
Workflow ID12717
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-04 03:40:36
SiteNL_NIKHEF
EntryVIRGO_NL_NIKHEF_dissel
Last heartbeat2026-02-04 05:41:53
From worker nodeHostnamewn-sate-029.farm.nikhef.nl
cpuinfoAMD EPYC 7551P 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-04 03:41:03
Input filesvd-protodune:np02vd_raw_run042421_0127_df-s04-d0_dw_0_20260203T143941.hdf5
JobscriptExit code0
Real time1h (7121s)
CPU time50m (3057s = 42%)
Max RSS bytes2588057600 (2468 MiB)
Outputting started2026-02-04 05:39:46
Output files
Finished2026-02-04 05:41:53
Saved logsjustin-logs:303840.0-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

exception in looking up space point ptr... 0x25b0b2c0, 0x25b0f780
Caught exception in looking up space point ptr... 0x25b0f780, 0x54c2c3c0
Caught exception in looking up space point ptr... 0x54c2c3c0, 0x40e7be70
Caught exception in looking up space point ptr... 0x2c972940, 0x2c972320
Caught exception in looking up space point ptr... 0x2c972320, 0x2c972940
Caught exception in looking up space point ptr... 0x59583d20, 0x54815480
Caught exception in looking up space point ptr... 0x54815480, 0x201d9520
Caught exception in looking up space point ptr... 0x201e2a50, 0x26d22da0
Caught exception in looking up space point ptr... 0x26d22da0, 0x201ee720
Caught exception in looking up space point ptr... 0x41ec58c0, 0x41ec3bd0
Caught exception in looking up space point ptr... 0x41ec3bd0, 0x41ec58c0
Caught exception in looking up space point ptr... 0x1efacf80, 0x36fd83a0
Caught exception in looking up space point ptr... 0x36fd83a0, 0x36fd8600
Caught exception in looking up space point ptr... 0x577fdba0, 0x3fce9640
Caught exception in looking up space point ptr... 0x3fce9640, 0x577fd650
Caught exception in looking up space point ptr... 0x577fd650, 0x577fd9c0
Caught exception in looking up space point ptr... 0x577fd9c0, 0x577fdba0
Caught exception in looking up space point ptr... 0x20a78930, 0x5b8200b0
Caught exception in looking up space point ptr... 0x555aa730, 0x20a78930
Caught exception in looking up space point ptr... 0x26cbd9a0, 0x1f2901d0
Caught exception in looking up space point ptr... 0x1f2901d0, 0x429159f0
Caught exception in looking up space point ptr... 0x5496be00, 0x5b32a850
Caught exception in looking up space point ptr... 0x5b32a850, 0x5496be00
Caught exception in looking up space point ptr... 0x5f2f74f0, 0x6fc2a1b0
Caught exception in looking up space point ptr... 0x6fc2a1b0, 0x5f304ac0
Caught exception in looking up space point ptr... 0x5f304ac0, 0x5f3051f0
Caught exception in looking up space point ptr... 0x5f3051f0, 0x6fc29610
Caught exception in looking up space point ptr... 0x61f38820, 0x61f38f00
Caught exception in looking up space point ptr... 0x61f38f00, 0x61f38950
Caught exception in looking up space point ptr... 0x4a063650, 0x4a063470
Caught exception in looking up space point ptr... 0x4a063470, 0x4a063650
Caught exception in looking up space point ptr... 0x41b44750, 0x206978b0
Caught exception in looking up space point ptr... 0x206978b0, 0x41b44750
Caught exception in looking up space point ptr... 0x41b44750, 0x3f0b67e0
Caught exception in looking up space point ptr... 0x3f0b67e0, 0x41b44750
Caught exception in looking up space point ptr... 0x5f310310, 0x5f2f8100
Caught exception in looking up space point ptr... 0x5f2f8100, 0x5f316840
Caught exception in looking up space point ptr... 0x7ab91720, 0x4a0688a0
Caught exception in looking up space point ptr... 0x4a0688a0, 0x7ab91720
Caught exception in looking up space point ptr... 0x3f76fc10, 0x51787700
Caught exception in looking up space point ptr... 0x51787700, 0x3f76fc10
Caught exception in looking up space point ptr... 0x75294960, 0x75294b60
Caught exception in looking up space point ptr... 0x75294b60, 0x75294960
Caught exception in looking up space point ptr... 0x25d22e60, 0x25d1d460
Caught exception in looking up space point ptr... 0x25d1d460, 0x25d22e60
Caught exception in looking up space point ptr... 0x4a2a7d60, 0x426c3bd0
Caught exception in looking up space point ptr... 0x426c3bd0, 0x4a2a7d60
Caught exception in looking up space point ptr... 0x40367300, 0x474e4b10
Caught exception in looking up space point ptr... 0x474e4b10, 0x40367300
Caught exception in looking up space point ptr... 0x201d41d0, 0x201d11b0
Caught exception in looking up space point ptr... 0x201d3be0, 0x201d41d0
Caught exception in looking up space point ptr... 0x25d0d200, 0x25d18d30
Caught exception in looking up space point ptr... 0x25d18d30, 0x25d0d200
Caught exception in looking up space point ptr... 0x261cd890, 0x261ccee0
Caught exception in looking up space point ptr... 0x261ccee0, 0x261cd890
Caught exception in looking up space point ptr... 0x261cd890, 0x261cc360
Caught exception in looking up space point ptr... 0x261cc360, 0x261cd890
Caught exception in looking up space point ptr... 0x4a25e4e0, 0x4a25e400
Caught exception in looking up space point ptr... 0x4a25e400, 0x4a25e4e0
Caught exception in looking up space point ptr... 0x2d230c80, 0x2d231730
Caught exception in looking up space point ptr... 0x2d231140, 0x2d230c80
Caught exception in looking up space point ptr... 0x2c0b8a00, 0x4a05f060
Caught exception in looking up space point ptr... 0x4a05f060, 0x7ab95ac0
Caught exception in looking up space point ptr... 0x77022200, 0x438c29e0
Caught exception in looking up space point ptr... 0x438c29e0, 0x77022200
++++>>>> total num hits: 46687, num free: 44781
04-Feb-2026 06:39:37 CET  Closed output file "np02vd_raw_run042421_0127_df-s04-d0_dw_0_20260203T143941_reco_stage0_20260204T053937_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         184.144       260.434       319.015       272.74        36.4488        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       6.8028e-05    0.000100909   0.000236905   9.1122e-05    3.38795e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                69.2415       138.536       182.668       150.144       32.5343        27     
produce:triggerrawdecoder:PDVDTriggerReader4      0.577213      0.832925       1.28881      0.810769      0.143838        27     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0174636     0.0223606     0.0755259     0.0177394     0.0124673       27     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0864556     0.0873377     0.0910075     0.0871623    0.000921346      27     
produce:beamevent:BeamEvent                      0.000110047   0.000182453   0.00104974    0.00014919    0.000171528      27     
produce:pdvddaphne:DAPHNEReaderPDVD                10.0889       17.8061       23.1964       18.2092        3.893         27     
produce:ophit:OpHitFinder                         0.0375253     0.0485135     0.0614897     0.0478796    0.00517945       27     
produce:wclsdatavd:WireCellToolkit                 78.7104       82.5238       88.5302       81.9787       2.22941        27     
produce:gaushit:GausHitFinder                     0.761942       1.18957       1.69158       1.17658      0.206213        27     
produce:nhitsfilter:NumberOfHitsFilter           0.000307287   0.000407808   0.000675961   0.000405211   8.24004e-05      27     
produce:reco3d:SpacePointSolver                     6.335        12.7043       20.6183       12.7906       3.29637        27     
produce:hitpdune:DisambigFromSpacePoints          0.0874588     0.131065      0.199891      0.125438       0.02937        27     
produce:cluster3d:Cluster3D                       0.815085       1.94995       3.36643       1.87086       0.53641        27     
[art]:TriggerResults:TriggerResultInserter       2.8714e-05    4.11533e-05   0.000102964   3.4314e-05    1.53047e-05      27     
end_path:out1:RootOutput                          6.121e-06    8.96022e-06   2.6259e-05     7.745e-06    3.83119e-06      27     
end_path:out1:RootOutput(write)                    4.14444       4.56304       4.91138       4.53139      0.181754        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4507.11 MB
  Peak resident set size usage (VmHWM): 2588.06 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042421_0127_df-s04-d0_dw_0_20260203T143941_reco_stage0_20260204T053937_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042421_0127_df-s04-d0_dw_0_20260203T143941_reco_stage0_20260204T053937_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042421_0127_df-s04-d0_dw_0_20260203T143941_reco_stage0_20260204T053937_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770176473.0,
    "core.end_time": 1770183579.0,
    "core.events": [
      20638,
      20644,
      20650,
      20656,
      20662,
      20668,
      20674,
      20680,
      20686,
      20692,
      20698,
      20704,
      20710,
      20716,
      20722,
      20728,
      20734,
      20740,
      20746,
      20752,
      20758,
      20764,
      20770,
      20776,
      20782,
      20788,
      20794
    ],
    "core.event_count": 27,
    "core.first_event_number": 20638,
    "core.last_event_number": 20794,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42421
    ],
    "core.runs_subruns": [
      4242100001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042421_0127_df-s04-d0_dw_0_20260203T143941.hdf5"
    }
  ]
}
justIN time: 2026-02-04 06:00:43 UTC       justIN version: 01.06.00