justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297919.37@dunegpschedd02.fnal.gov

Jobsub ID297919.37@dunegpschedd02.fnal.gov
Workflow ID12717
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:54:25
SiteNL_NIKHEF
EntryVIRGO_NL_NIKHEF_klomp_multicore
Last heartbeat2026-02-04 01:30:18
From worker nodeHostnamewn-pijl-028.farm.nikhef.nl
cpuinfoAMD EPYC 9754 128-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:30:27
Input filesvd-protodune:np02vd_raw_run042421_0048_df-s04-d0_dw_0_20260203T135657.hdf5
JobscriptExit code0
Real time1h (7044s)
CPU time47m (2863s = 40%)
Max RSS bytes2579644416 (2460 MiB)
Outputting started2026-02-04 01:27:53
Output files
Finished2026-02-04 01:30:18
Saved logsjustin-logs:297919.37-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

0x3bda05d0, 0x3bda0440
Caught exception in looking up space point ptr... 0x3bda0440, 0x3bda05d0
Caught exception in looking up space point ptr... 0x57fe6dd0, 0x1aed33d0
Caught exception in looking up space point ptr... 0x1aed33d0, 0x57fe6dd0
Caught exception in looking up space point ptr... 0x64d18d10, 0x64d0fd40
Caught exception in looking up space point ptr... 0x64d0fd40, 0x64d237f0
Caught exception in looking up space point ptr... 0x3a724f70, 0x3a724de0
Caught exception in looking up space point ptr... 0x3a724de0, 0x3a725290
Caught exception in looking up space point ptr... 0x5e5237a0, 0x5e5238d0
Caught exception in looking up space point ptr... 0x5e5238d0, 0x5e5232e0
Caught exception in looking up space point ptr... 0x4b357360, 0x4b354c00
Caught exception in looking up space point ptr... 0x4b354c00, 0x4b355ce0
Caught exception in looking up space point ptr... 0x17822940, 0x4b357360
Caught exception in looking up space point ptr... 0x38264cc0, 0x382652b0
Caught exception in looking up space point ptr... 0x382652b0, 0x38265120
Caught exception in looking up space point ptr... 0x4d5ad7c0, 0x1aed5e90
Caught exception in looking up space point ptr... 0x1aed5e90, 0x439a6570
Caught exception in looking up space point ptr... 0x1aee3f80, 0x1aee4e50
Caught exception in looking up space point ptr... 0x1aee4e50, 0x1aee3e70
Caught exception in looking up space point ptr... 0x64eb4a80, 0x64eb8f30
Caught exception in looking up space point ptr... 0x64eb8f30, 0x64eb9910
Caught exception in looking up space point ptr... 0x64eb9910, 0x3f686860
Caught exception in looking up space point ptr... 0x3f686860, 0x64ebb830
Caught exception in looking up space point ptr... 0x64ebb830, 0x64eb4a80
Caught exception in looking up space point ptr... 0x5d784390, 0x1f3143a0
Caught exception in looking up space point ptr... 0x6051e6b0, 0x5d784390
Caught exception in looking up space point ptr... 0x3bbde540, 0x3d643680
Caught exception in looking up space point ptr... 0x3d643680, 0x3bbde540
Caught exception in looking up space point ptr... 0x63bd0270, 0x63bd0400
Caught exception in looking up space point ptr... 0x63bd0400, 0x63bd0270
Caught exception in looking up space point ptr... 0x40d08320, 0x40d081f0
Caught exception in looking up space point ptr... 0x40d081f0, 0x40d080c0
Caught exception in looking up space point ptr... 0x53583b30, 0x484f0860
Caught exception in looking up space point ptr... 0x484f0860, 0x40bd5620
Caught exception in looking up space point ptr... 0x35660360, 0x35660100
Caught exception in looking up space point ptr... 0x35660100, 0x524f59e0
Caught exception in looking up space point ptr... 0x524f5bc0, 0x254e1eb0
Caught exception in looking up space point ptr... 0x254e1eb0, 0x223e3af0
Caught exception in looking up space point ptr... 0x3bbfdba0, 0x612ca100
Caught exception in looking up space point ptr... 0x612ca100, 0x3bbfdd80
Caught exception in looking up space point ptr... 0x3824d640, 0x3824d2d0
Caught exception in looking up space point ptr... 0x3824d2d0, 0x3824d4b0
Caught exception in looking up space point ptr... 0x386cddd0, 0x25925cf0
Caught exception in looking up space point ptr... 0x25925cf0, 0x326cb640
Caught exception in looking up space point ptr... 0x54f7f0d0, 0x54f7ede0
Caught exception in looking up space point ptr... 0x54f7ede0, 0x54f7ce70
Caught exception in looking up space point ptr... 0x6cb0e7e0, 0x6cb0b6c0
Caught exception in looking up space point ptr... 0x6cb0b6c0, 0x1aef3540
Caught exception in looking up space point ptr... 0x3f6abd50, 0x3f69cff0
Caught exception in looking up space point ptr... 0x3f69cff0, 0x3f69d290
Caught exception in looking up space point ptr... 0x4ab4f420, 0x4ab5e930
Caught exception in looking up space point ptr... 0x4ab5ee50, 0x4ab4f420
Caught exception in looking up space point ptr... 0x4b949dd0, 0x2011db60
Caught exception in looking up space point ptr... 0x2011db60, 0x4b949dd0
Caught exception in looking up space point ptr... 0x23254fb0, 0x6c0c4720
Caught exception in looking up space point ptr... 0x6c0c4720, 0x67c44ad0
Caught exception in looking up space point ptr... 0x3f82d1f0, 0x22f51930
Caught exception in looking up space point ptr... 0x22f51930, 0x3f82d1f0
Caught exception in looking up space point ptr... 0x6cb0cdf0, 0x439a0bb0
Caught exception in looking up space point ptr... 0x439a0bb0, 0x6cb0cdf0
Caught exception in looking up space point ptr... 0x63db5570, 0x63db4f50
Caught exception in looking up space point ptr... 0x63db4f50, 0x63db5570
Caught exception in looking up space point ptr... 0x3185bc50, 0x3185bf70
Caught exception in looking up space point ptr... 0x3185bf70, 0x3185c290
Caught exception in looking up space point ptr... 0x3f69d680, 0x3f69d530
Caught exception in looking up space point ptr... 0x3f69d530, 0x3f6ab0d0
++++>>>> total num hits: 90447, num free: 86345
04-Feb-2026 02:27:38 CET  Closed output file "np02vd_raw_run042421_0048_df-s04-d0_dw_0_20260203T135657_reco_stage0_20260204T012738_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         196.708       256.491       326.505       255.067       33.902         27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       8.3917e-05    0.000145412   0.000428999   0.000138409   6.70602e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                91.3174       144.254       208.76        137.955       30.0144        27     
produce:triggerrawdecoder:PDVDTriggerReader4      0.674359      0.904021       1.46701       0.82515       0.20923        27     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.017805      0.0292354     0.0758764     0.0222549     0.0147487       27     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0874439     0.0944259      0.13252      0.0893168     0.0108477       27     
produce:beamevent:BeamEvent                      0.000109586   0.000293446   0.00222169    0.00021142    0.000384284      27     
produce:pdvddaphne:DAPHNEReaderPDVD                11.4413       17.9698       27.9541       17.8558       3.47588        27     
produce:ophit:OpHitFinder                         0.0320586     0.0503577     0.094859      0.0488252     0.0112862       27     
produce:wclsdatavd:WireCellToolkit                 63.5384       74.8917       93.2464       74.6852       7.10029        27     
produce:gaushit:GausHitFinder                     0.318975       1.04635       1.64817      0.946272      0.356771        27     
produce:nhitsfilter:NumberOfHitsFilter           0.000222406   0.00105772    0.00261189    0.000999081   0.000593573      27     
produce:reco3d:SpacePointSolver                    2.96919       9.95608       15.7405       9.17424       3.1015         27     
produce:hitpdune:DisambigFromSpacePoints          0.0485221     0.125303      0.197726       0.11426      0.0390543       27     
produce:cluster3d:Cluster3D                        0.6604        2.45051       4.57215       2.3501       0.902454        27     
[art]:TriggerResults:TriggerResultInserter       2.3705e-05    5.41347e-05   0.000147083    4.609e-05    2.87392e-05      27     
end_path:out1:RootOutput                          4.007e-06    2.09966e-05   0.000361356    6.479e-06    6.69403e-05      27     
end_path:out1:RootOutput(write)                    3.33806       4.65521       6.63717       4.39568      0.813108        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4557.58 MB
  Peak resident set size usage (VmHWM): 2579.64 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042421_0048_df-s04-d0_dw_0_20260203T135657_reco_stage0_20260204T012738_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042421_0048_df-s04-d0_dw_0_20260203T135657_reco_stage0_20260204T012738_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042421_0048_df-s04-d0_dw_0_20260203T135657_reco_stage0_20260204T012738_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770161441.0,
    "core.end_time": 1770168462.0,
    "core.events": [
      7822,
      7828,
      7834,
      7840,
      7846,
      7852,
      7858,
      7864,
      7870,
      7876,
      7882,
      7888,
      7894,
      7900,
      7906,
      7912,
      7918,
      7924,
      7930,
      7936,
      7942,
      7948,
      7954,
      7960,
      7966,
      7972,
      7978
    ],
    "core.event_count": 27,
    "core.first_event_number": 7822,
    "core.last_event_number": 7978,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42421
    ],
    "core.runs_subruns": [
      4242100001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042421_0048_df-s04-d0_dw_0_20260203T135657.hdf5"
    }
  ]
}
justIN time: 2026-02-04 06:07:00 UTC       justIN version: 01.06.00