justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 298092.2@dunegpschedd02.fnal.gov

Jobsub ID298092.2@dunegpschedd02.fnal.gov
Workflow ID12718
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-04 03:20:35
SiteNL_NIKHEF
EntryVIRGO_NL_NIKHEF_klomp
Last heartbeat2026-02-04 05:14:26
From worker nodeHostnamewn-pijl-012.farm.nikhef.nl
cpuinfoAMD EPYC 9754 128-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-04 03:20:56
Input filesvd-protodune:np02vd_raw_run042422_0105_df-s03-d1_dw_0_20260203T161131.hdf5
JobscriptExit code0
Real time1h (6669s)
CPU time49m (2979s = 44%)
Max RSS bytes2627067904 (2505 MiB)
Outputting started2026-02-04 05:12:06
Output files
Finished2026-02-04 05:14:26
Saved logsjustin-logs:298092.2-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

t exception in looking up space point ptr... 0x341f1040, 0x4f4687e0
Caught exception in looking up space point ptr... 0x3423ed40, 0x408f0410
Caught exception in looking up space point ptr... 0x408f0410, 0x4f871ed0
Caught exception in looking up space point ptr... 0x3246cc00, 0x5c9c0a50
Caught exception in looking up space point ptr... 0x53f37180, 0x3246cc00
Caught exception in looking up space point ptr... 0x541580c0, 0x54157ff0
Caught exception in looking up space point ptr... 0x54157ff0, 0x45b1da80
Caught exception in looking up space point ptr... 0x4b4c6920, 0x256b99c0
Caught exception in looking up space point ptr... 0x256b99c0, 0x256b97a0
Caught exception in looking up space point ptr... 0x1f136820, 0x1f136ac0
Caught exception in looking up space point ptr... 0x1f136ac0, 0x1f136c10
Caught exception in looking up space point ptr... 0x4d39e730, 0x46170610
Caught exception in looking up space point ptr... 0x46170610, 0x3fe4e7d0
Caught exception in looking up space point ptr... 0x3baafdb0, 0x211cd3b0
Caught exception in looking up space point ptr... 0x211cd3b0, 0x18cc10c0
Caught exception in looking up space point ptr... 0x3312f5d0, 0x6ea62b10
Caught exception in looking up space point ptr... 0x6ea62b10, 0x340f0780
Caught exception in looking up space point ptr... 0x74a3a840, 0x74a39260
Caught exception in looking up space point ptr... 0x74a39260, 0x74a3a840
Caught exception in looking up space point ptr... 0x4573c710, 0x4573ec30
Caught exception in looking up space point ptr... 0x4573ec30, 0x4573c610
Caught exception in looking up space point ptr... 0x47bd5c40, 0x47bd5ab0
Caught exception in looking up space point ptr... 0x47bd5ab0, 0x47bd5c40
Caught exception in looking up space point ptr... 0x3c7b7590, 0x3c7bbc30
Caught exception in looking up space point ptr... 0x3c7bbc30, 0x3c7b7590
Caught exception in looking up space point ptr... 0x4f0d49f0, 0x3e304a90
Caught exception in looking up space point ptr... 0x3e304a90, 0x4f0d49f0
Caught exception in looking up space point ptr... 0x57d09bc0, 0x3dda3230
Caught exception in looking up space point ptr... 0x3dda3230, 0x57d09bc0
Caught exception in looking up space point ptr... 0x1e37ccd0, 0x1e37cbd0
Caught exception in looking up space point ptr... 0x1e37cbd0, 0x1e37cdd0
Caught exception in looking up space point ptr... 0x1e391560, 0x255ae440
Caught exception in looking up space point ptr... 0x255ae440, 0x1e391560
Caught exception in looking up space point ptr... 0x40ba3ce0, 0x4b38b830
Caught exception in looking up space point ptr... 0x4b38b830, 0x40ba3ce0
Caught exception in looking up space point ptr... 0x619dba60, 0x619e1e90
Caught exception in looking up space point ptr... 0x619e1e90, 0x619dba60
Caught exception in looking up space point ptr... 0x1d88d670, 0x4b4c6730
Caught exception in looking up space point ptr... 0x4b4c6730, 0x3568d230
Caught exception in looking up space point ptr... 0x23e08030, 0x5cf27d90
Caught exception in looking up space point ptr... 0x5cf27d90, 0x23e08030
Caught exception in looking up space point ptr... 0x5390c980, 0x4c4bc730
Caught exception in looking up space point ptr... 0x4c4bc730, 0x4c4c2980
Caught exception in looking up space point ptr... 0x20d981a0, 0x259da130
Caught exception in looking up space point ptr... 0x259da130, 0x20d981a0
Caught exception in looking up space point ptr... 0x619f04e0, 0x32770390
Caught exception in looking up space point ptr... 0x32770390, 0x619f04e0
Caught exception in looking up space point ptr... 0x47bd76e0, 0x47be0af0
Caught exception in looking up space point ptr... 0x47be0af0, 0x4d3c9f70
Caught exception in looking up space point ptr... 0x4da1e360, 0x57dd7a70
Caught exception in looking up space point ptr... 0x57dd7a70, 0x4ed9c350
Caught exception in looking up space point ptr... 0x5470e280, 0x419658c0
Caught exception in looking up space point ptr... 0x419658c0, 0x5470e280
Caught exception in looking up space point ptr... 0x233e9840, 0x233e96f0
Caught exception in looking up space point ptr... 0x233e96f0, 0x233e9840
Caught exception in looking up space point ptr... 0x6da51ea0, 0x4952b780
Caught exception in looking up space point ptr... 0x4952b780, 0x6da51ea0
Caught exception in looking up space point ptr... 0x4db049b0, 0x4db059a0
Caught exception in looking up space point ptr... 0x4db059a0, 0x4db04e90
Caught exception in looking up space point ptr... 0x4f5c93e0, 0x1e389620
Caught exception in looking up space point ptr... 0x1e389620, 0x4f5c93e0
Caught exception in looking up space point ptr... 0x63b8c180, 0x63b8c2b0
Caught exception in looking up space point ptr... 0x63b8c2b0, 0x63b8c050
Caught exception in looking up space point ptr... 0x47802f30, 0x33190ae0
Caught exception in looking up space point ptr... 0x33190ae0, 0x47803030
++++>>>> total num hits: 71440, num free: 68456
04-Feb-2026 06:11:57 CET  Closed output file "np02vd_raw_run042422_0105_df-s03-d1_dw_0_20260203T161131_reco_stage0_20260204T051157_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         172.763       243.306       294.622       250.236       36.2398        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       8.3787e-05    0.000105337   0.000295606   9.6526e-05    3.92769e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                74.2363       126.744       177.435       134.578       28.4992        27     
produce:triggerrawdecoder:PDVDTriggerReader4      0.578899       0.81684       1.2064       0.784816      0.155247        27     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0175118     0.0198466     0.041707      0.0178534    0.00513568       27     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0866786     0.088753      0.105263      0.0873137     0.0046168       27     
produce:beamevent:BeamEvent                      0.00010619    0.000200573   0.00133448    0.000153181   0.00022398       27     
produce:pdvddaphne:DAPHNEReaderPDVD                9.17513       15.9383       22.874        16.3846       3.13903        27     
produce:ophit:OpHitFinder                         0.0337092     0.0469396     0.0544232     0.0481985    0.00480274       27     
produce:wclsdatavd:WireCellToolkit                 61.4271       77.2731       89.2472       78.7716       7.60515        27     
produce:gaushit:GausHitFinder                     0.664213       1.25989       2.09439       1.20689      0.382419        27     
produce:nhitsfilter:NumberOfHitsFilter           0.000261254   0.000743008   0.00171873    0.000602482   0.000384353      27     
produce:reco3d:SpacePointSolver                    6.60618        12.14        24.0663       11.9143       3.8892         27     
produce:hitpdune:DisambigFromSpacePoints          0.0662818     0.142602      0.278658      0.130007      0.0560834       27     
produce:cluster3d:Cluster3D                        1.39406       2.99575       6.4734        2.49082       1.26436        27     
[art]:TriggerResults:TriggerResultInserter       2.5619e-05    4.49789e-05   0.000114783   3.8719e-05    1.85318e-05      27     
end_path:out1:RootOutput                          4.547e-06    7.15078e-06   2.7281e-05     5.839e-06    4.47654e-06      27     
end_path:out1:RootOutput(write)                    4.90111       5.78785       6.95705       5.74898      0.521145        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4582.7 MB
  Peak resident set size usage (VmHWM): 2627.07 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042422_0105_df-s03-d1_dw_0_20260203T161131_reco_stage0_20260204T051157_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042422_0105_df-s03-d1_dw_0_20260203T161131_reco_stage0_20260204T051157_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042422_0105_df-s03-d1_dw_0_20260203T161131_reco_stage0_20260204T051157_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770175269.0,
    "core.end_time": 1770181919.0,
    "core.events": [
      17025,
      17031,
      17037,
      17043,
      17049,
      17055,
      17061,
      17067,
      17073,
      17079,
      17085,
      17091,
      17097,
      17103,
      17109,
      17115,
      17121,
      17127,
      17133,
      17139,
      17145,
      17151,
      17157,
      17163,
      17169,
      17175,
      17181
    ],
    "core.event_count": 27,
    "core.first_event_number": 17025,
    "core.last_event_number": 17181,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42422
    ],
    "core.runs_subruns": [
      4242200001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042422_0105_df-s03-d1_dw_0_20260203T161131.hdf5"
    }
  ]
}
justIN time: 2026-02-04 06:00:00 UTC       justIN version: 01.06.00