justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297921.12@dunegpschedd02.fnal.gov

Jobsub ID297921.12@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:58:26
SiteNL_NIKHEF
EntryVIRGO_NL_NIKHEF_juk_multicore
Last heartbeat2026-02-04 01:37:49
From worker nodeHostnamewn-pijl-008.farm.nikhef.nl
cpuinfoAMD EPYC 9754 128-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:44:56
Input filesvd-protodune:np02vd_raw_run042433_0158_df-s04-d1_dw_0_20260203T194212.hdf5
JobscriptExit code0
Real time1h (6640s)
CPU time47m (2857s = 43%)
Max RSS bytes3395989504 (3238 MiB)
Outputting started2026-02-04 01:35:38
Output files
Finished2026-02-04 01:37:49
Saved logsjustin-logs:297921.12-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

0x87ea95d0, 0x87e9f790
Caught exception in looking up space point ptr... 0x87e9f790, 0x87e9a870
Caught exception in looking up space point ptr... 0x87e9a870, 0x87e9f790
Caught exception in looking up space point ptr... 0x6990ad20, 0x6990b1d0
Caught exception in looking up space point ptr... 0x6990b1d0, 0x6990ad20
Caught exception in looking up space point ptr... 0x957918e0, 0x95794ef0
Caught exception in looking up space point ptr... 0x95794ef0, 0x957918e0
Caught exception in looking up space point ptr... 0x957918e0, 0x957962f0
Caught exception in looking up space point ptr... 0x957962f0, 0x957918e0
Caught exception in looking up space point ptr... 0x330c5560, 0x330d0310
Caught exception in looking up space point ptr... 0x330d0310, 0x330b2330
Caught exception in looking up space point ptr... 0x9fd2ce10, 0x9fd2bb50
Caught exception in looking up space point ptr... 0x9fd2bb50, 0x9fd2ba50
Caught exception in looking up space point ptr... 0x688f9140, 0x4f30dd80
Caught exception in looking up space point ptr... 0x4f30dd80, 0x688f9140
Caught exception in looking up space point ptr... 0x427fa890, 0x7ed6bbc0
Caught exception in looking up space point ptr... 0x7ed6bbc0, 0x9f0d6dd0
Caught exception in looking up space point ptr... 0x9f0d6dd0, 0x9f0ea030
Caught exception in looking up space point ptr... 0x9f0e95a0, 0x427fa890
Caught exception in looking up space point ptr... 0x6aea5510, 0x42aecc80
Caught exception in looking up space point ptr... 0x42aecc80, 0x6aea5510
Caught exception in looking up space point ptr... 0x7e6ed5b0, 0x698fa250
Caught exception in looking up space point ptr... 0x698fa250, 0x7e6ed5b0
Caught exception in looking up space point ptr... 0x2ab686a0, 0x5cfc4ea0
Caught exception in looking up space point ptr... 0x5cfc4ea0, 0x2ab686a0
Caught exception in looking up space point ptr... 0x2ab686a0, 0x283b7c30
Caught exception in looking up space point ptr... 0x283b7c30, 0x2ab686a0
Caught exception in looking up space point ptr... 0x2c193630, 0x2c19ac90
Caught exception in looking up space point ptr... 0x2c19ac90, 0x2c19aa30
Caught exception in looking up space point ptr... 0x42809c50, 0x42805120
Caught exception in looking up space point ptr... 0x427f8e40, 0x42809c50
Caught exception in looking up space point ptr... 0x687fda70, 0x687fd8e0
Caught exception in looking up space point ptr... 0x687fd8e0, 0x687fda70
Caught exception in looking up space point ptr... 0x5675b940, 0x5675bdc0
Caught exception in looking up space point ptr... 0x5675bdc0, 0x5675ca70
Caught exception in looking up space point ptr... 0x33c59ad0, 0xa1392790
Caught exception in looking up space point ptr... 0xa1392790, 0xa13bb950
Caught exception in looking up space point ptr... 0x2e34e4d0, 0x2e34e1b0
Caught exception in looking up space point ptr... 0x2e34e1b0, 0x2e34e4d0
Caught exception in looking up space point ptr... 0x7df88eb0, 0x7df85340
Caught exception in looking up space point ptr... 0x2d8c5240, 0x7df88eb0
Caught exception in looking up space point ptr... 0x4a5b2830, 0x6ac1ed90
Caught exception in looking up space point ptr... 0x6ac1ed90, 0xa13c4480
Caught exception in looking up space point ptr... 0x782ac810, 0x782af130
Caught exception in looking up space point ptr... 0x782af130, 0x782ac810
Caught exception in looking up space point ptr... 0x75e46290, 0x75e39cf0
Caught exception in looking up space point ptr... 0x75e39cf0, 0x75e12c40
Caught exception in looking up space point ptr... 0x69a107d0, 0x69a109b0
Caught exception in looking up space point ptr... 0x69a109b0, 0x69a107d0
Caught exception in looking up space point ptr... 0x9f0d78a0, 0x9f0d84a0
Caught exception in looking up space point ptr... 0x9f0d84a0, 0x9f0d7710
Caught exception in looking up space point ptr... 0x506db920, 0x506c5990
Caught exception in looking up space point ptr... 0x506c5990, 0x506ce560
Caught exception in looking up space point ptr... 0x68804770, 0x6881eb60
Caught exception in looking up space point ptr... 0x6881eb60, 0x68821890
Caught exception in looking up space point ptr... 0x33e20a40, 0x33e018a0
Caught exception in looking up space point ptr... 0x33e018a0, 0x33e20a40
Caught exception in looking up space point ptr... 0x33e20a40, 0x7e6ecf00
Caught exception in looking up space point ptr... 0x7e6ecf00, 0x33e20a40
Caught exception in looking up space point ptr... 0x4291b650, 0x9ab3ee80
Caught exception in looking up space point ptr... 0x9ab3ee80, 0x4291b650
Caught exception in looking up space point ptr... 0x2adc83e0, 0x86758570
Caught exception in looking up space point ptr... 0x86758570, 0x2adc83e0
Caught exception in looking up space point ptr... 0x782b9730, 0x2fd02410
Caught exception in looking up space point ptr... 0x2fd02410, 0x782b9730
Caught exception in looking up space point ptr... 0x21defba0, 0x21defe00
Caught exception in looking up space point ptr... 0x21defe00, 0x21defba0
++++>>>> total num hits: 100029, num free: 95509
04-Feb-2026 02:35:28 CET  Closed output file "np02vd_raw_run042433_0158_df-s04-d1_dw_0_20260203T194212_reco_stage0_20260204T013528_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         274.902       344.562       488.591       331.023       54.2147        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                        8.639e-05    0.000121072   0.000405692   0.000107192   6.82899e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                129.075       191.568       316.324       183.903       48.858         19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.673515      0.989531       1.49094      0.990012       0.21677        19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0176543     0.035939      0.258684      0.0216582      0.05303        19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0176389     0.0179892     0.0187277     0.0179402    0.000261914      19     
produce:beamevent:BeamEvent                      0.000109325   0.00024897    0.00130515    0.000155404   0.000283316      19     
produce:pdvddaphne:DAPHNEReaderPDVD                11.5773       16.1679       23.2275       15.4715       3.10029        19     
produce:ophit:OpHitFinder                         0.0326201     0.0435005     0.0523843     0.044667     0.00532834       19     
produce:wclsdatavd:WireCellToolkit                 95.2044       108.933       130.689       108.605       10.7596        19     
produce:gaushit:GausHitFinder                     0.838125       1.28216       2.02653       1.2313       0.322911        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000357539   0.00104841    0.00211518    0.00104866    0.000489899      19     
produce:reco3d:SpacePointSolver                    9.7779         16.75        24.7033       15.7356       4.02426        19     
produce:hitpdune:DisambigFromSpacePoints          0.113359      0.200954      0.277477       0.18677      0.0475311       19     
produce:cluster3d:Cluster3D                        1.59955       3.01739       4.52248       2.94306      0.803721        19     
[art]:TriggerResults:TriggerResultInserter        2.61e-05     4.1663e-05    9.3171e-05    3.4893e-05    1.81059e-05      19     
end_path:out1:RootOutput                          4.296e-06    1.99206e-05   0.000253973    5.569e-06    5.53069e-05      19     
end_path:out1:RootOutput(write)                    4.41316       5.46721       7.09992       5.17064      0.867089        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5352.05 MB
  Peak resident set size usage (VmHWM): 3395.99 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0158_df-s04-d1_dw_0_20260203T194212_reco_stage0_20260204T013528_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0158_df-s04-d1_dw_0_20260203T194212_reco_stage0_20260204T013528_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0158_df-s04-d1_dw_0_20260203T194212_reco_stage0_20260204T013528_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770162310.0,
    "core.end_time": 1770168930.0,
    "core.events": [
      18023,
      18029,
      18035,
      18041,
      18047,
      18053,
      18059,
      18065,
      18071,
      18077,
      18083,
      18089,
      18095,
      18101,
      18107,
      18113,
      18119,
      18125,
      18131
    ],
    "core.event_count": 19,
    "core.first_event_number": 18023,
    "core.last_event_number": 18131,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0158_df-s04-d1_dw_0_20260203T194212.hdf5"
    }
  ]
}
justIN time: 2026-02-04 06:00:38 UTC       justIN version: 01.06.00