justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297923.30@dunegpschedd02.fnal.gov

Jobsub ID297923.30@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 23:00:26
SiteNL_NIKHEF
EntryVIRGO_NL_NIKHEF_klomp_multicore
Last heartbeat2026-02-04 02:11:06
From worker nodeHostnamewn-pijl-011.farm.nikhef.nl
cpuinfoAMD EPYC 9754 128-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:58:00
Input filesvd-protodune:np02vd_raw_run042433_0343_df-s02-d1_dw_0_20260203T205229.hdf5
JobscriptExit code0
Real time2h (7845s)
CPU time1h (3751s = 47%)
Max RSS bytes3384115200 (3227 MiB)
Outputting started2026-02-04 02:08:46
Output files
Finished2026-02-04 02:11:06
Saved logsjustin-logs:297923.30-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

. 0x441570c0, 0x44154480
Caught exception in looking up space point ptr... 0x8064edb0, 0x8064a740
Caught exception in looking up space point ptr... 0x80653150, 0x8064edb0
Caught exception in looking up space point ptr... 0x88024dc0, 0x9e301a90
Caught exception in looking up space point ptr... 0x6f89b190, 0x88024dc0
Caught exception in looking up space point ptr... 0x4876b080, 0x4f723a10
Caught exception in looking up space point ptr... 0xb439a840, 0x4876b080
Caught exception in looking up space point ptr... 0x46ff6c20, 0x46fedda0
Caught exception in looking up space point ptr... 0x46fedda0, 0x46ffe180
Caught exception in looking up space point ptr... 0x7707f420, 0x7707f020
Caught exception in looking up space point ptr... 0x46ed7b60, 0x7707f420
Caught exception in looking up space point ptr... 0x69f323f0, 0x725c6520
Caught exception in looking up space point ptr... 0x725c6520, 0x69f323f0
Caught exception in looking up space point ptr... 0x69f323f0, 0x6e77c190
Caught exception in looking up space point ptr... 0x6e77c190, 0x69f323f0
Caught exception in looking up space point ptr... 0x9e4e6a80, 0x8929ca10
Caught exception in looking up space point ptr... 0x8929ca10, 0x89f8e2f0
Caught exception in looking up space point ptr... 0x7eca2900, 0x7ec92c60
Caught exception in looking up space point ptr... 0x7ec92c60, 0x7ec927c0
Caught exception in looking up space point ptr... 0x7ec927c0, 0x7eca23a0
Caught exception in looking up space point ptr... 0x7eca2630, 0x7eca2900
Caught exception in looking up space point ptr... 0x6adf7f30, 0x4fee8f10
Caught exception in looking up space point ptr... 0x4fee8f10, 0x4fee2550
Caught exception in looking up space point ptr... 0x6be834c0, 0x6be867c0
Caught exception in looking up space point ptr... 0x6be867c0, 0x6be834c0
Caught exception in looking up space point ptr... 0x43b2dd00, 0x43b2db00
Caught exception in looking up space point ptr... 0x43b2db00, 0x43b2dd00
Caught exception in looking up space point ptr... 0x5053fce0, 0x3ff156a0
Caught exception in looking up space point ptr... 0x3ff156a0, 0x5053fce0
Caught exception in looking up space point ptr... 0x47c95900, 0x47c95a90
Caught exception in looking up space point ptr... 0x47c95a90, 0x47c95900
Caught exception in looking up space point ptr... 0x883d83c0, 0x883d54e0
Caught exception in looking up space point ptr... 0x883d54e0, 0x883d0410
Caught exception in looking up space point ptr... 0x88990660, 0xba02bf70
Caught exception in looking up space point ptr... 0xba02bf70, 0x8f77e5e0
Caught exception in looking up space point ptr... 0x486cf1a0, 0x6cae6450
Caught exception in looking up space point ptr... 0x6cae6450, 0x486cf1a0
Caught exception in looking up space point ptr... 0x6dc8fbb0, 0x3ea18140
Caught exception in looking up space point ptr... 0x3ea18140, 0x6dc8fbb0
Caught exception in looking up space point ptr... 0xafd1a260, 0xafd15a00
Caught exception in looking up space point ptr... 0xafd15a00, 0xafd091b0
Caught exception in looking up space point ptr... 0x94c9c940, 0x94c9c090
Caught exception in looking up space point ptr... 0x94c9c090, 0x94ca0ca0
Caught exception in looking up space point ptr... 0x6ade7ba0, 0x6add9480
Caught exception in looking up space point ptr... 0x6add9480, 0x6add7490
Caught exception in looking up space point ptr... 0x7713f320, 0x77191200
Caught exception in looking up space point ptr... 0x77191200, 0x78f7d3c0
Caught exception in looking up space point ptr... 0x9fce5fc0, 0x9fce67f0
Caught exception in looking up space point ptr... 0x9fce67f0, 0x9fce5fc0
Caught exception in looking up space point ptr... 0x6afdda80, 0x6afda680
Caught exception in looking up space point ptr... 0x6afda680, 0x6afdda80
Caught exception in looking up space point ptr... 0x6afdda80, 0x6afd8960
Caught exception in looking up space point ptr... 0x6afd8960, 0x6afdda80
Caught exception in looking up space point ptr... 0x9fce1fa0, 0x9fce1ea0
Caught exception in looking up space point ptr... 0x9fce1ea0, 0x4c7ddc60
Caught exception in looking up space point ptr... 0xb750ad70, 0xb75082c0
Caught exception in looking up space point ptr... 0xb75082c0, 0xb750ad70
Caught exception in looking up space point ptr... 0xb750ad70, 0xb750aeb0
Caught exception in looking up space point ptr... 0xb750aeb0, 0xb750ad70
Caught exception in looking up space point ptr... 0x94eec240, 0x94ee6d00
Caught exception in looking up space point ptr... 0x94ee6d00, 0x94eec240
Caught exception in looking up space point ptr... 0x4f941910, 0x8f872e70
Caught exception in looking up space point ptr... 0x8f872e70, 0x4f941910
Caught exception in looking up space point ptr... 0x80621da0, 0x806199e0
Caught exception in looking up space point ptr... 0x806199e0, 0x80621da0
Caught exception in looking up space point ptr... 0x50fec7b0, 0x50fe8af0
Caught exception in looking up space point ptr... 0x50fe8af0, 0x50fec7b0
++++>>>> total num hits: 96406, num free: 92519
04-Feb-2026 03:08:36 CET  Closed output file "np02vd_raw_run042433_0343_df-s02-d1_dw_0_20260203T205229_reco_stage0_20260204T020836_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         340.602       406.699       489.046       399.005       38.8914        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       9.2199e-05    0.000120964   0.000341316   0.00010641    5.33046e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                144.839       206.067       279.85        201.27        36.2278        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.651984       1.02973       1.37745       1.01816      0.198915        19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0177633     0.0354097     0.154197      0.0221471     0.0338034       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0175857     0.0310495     0.256561      0.0178966     0.0531981       19     
produce:beamevent:BeamEvent                      0.000138189   0.000251506   0.00160424    0.000166131   0.000320734      19     
produce:pdvddaphne:DAPHNEReaderPDVD                14.529        18.0793       22.6607       18.162        2.15687        19     
produce:ophit:OpHitFinder                         0.038361      0.0492045     0.0585008     0.0493623    0.00478529       19     
produce:wclsdatavd:WireCellToolkit                 131.706       142.696       156.276       142.734       7.14612        19     
produce:gaushit:GausHitFinder                      1.13147       1.90882       3.04033       1.8857       0.449842        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000351751    0.0011818    0.00307553    0.00105053    0.000657111      19     
produce:reco3d:SpacePointSolver                    14.8675       24.3345       44.594        22.8675       7.07846        19     
produce:hitpdune:DisambigFromSpacePoints          0.158817      0.313411      0.489721      0.314958      0.0766481       19     
produce:cluster3d:Cluster3D                        2.19123       4.57789       6.96434       4.80911       1.29528        19     
[art]:TriggerResults:TriggerResultInserter       2.8543e-05     4.786e-05    0.000119631   4.1131e-05    2.13847e-05      19     
end_path:out1:RootOutput                          4.336e-06    8.96189e-06   2.8834e-05     6.93e-06     5.83306e-06      19     
end_path:out1:RootOutput(write)                    6.0331        7.48443       9.12748       7.68961      0.829194        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5391.29 MB
  Peak resident set size usage (VmHWM): 3384.12 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0343_df-s02-d1_dw_0_20260203T205229_reco_stage0_20260204T020836_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0343_df-s02-d1_dw_0_20260203T205229_reco_stage0_20260204T020836_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0343_df-s02-d1_dw_0_20260203T205229_reco_stage0_20260204T020836_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770163095.0,
    "core.end_time": 1770170918.0,
    "core.events": [
      39109,
      39115,
      39121,
      39127,
      39133,
      39139,
      39145,
      39151,
      39157,
      39163,
      39169,
      39175,
      39181,
      39187,
      39193,
      39199,
      39205,
      39211,
      39217
    ],
    "core.event_count": 19,
    "core.first_event_number": 39109,
    "core.last_event_number": 39217,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0343_df-s02-d1_dw_0_20260203T205229.hdf5"
    }
  ]
}
justIN time: 2026-02-04 06:06:09 UTC       justIN version: 01.06.00