justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303728.7@dunegpschedd01.fnal.gov

Jobsub ID303728.7@dunegpschedd01.fnal.gov
Workflow ID12717
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-04 01:20:30
SiteNL_NIKHEF
EntryVIRGO_NL_NIKHEF_juk
Last heartbeat2026-02-04 03:12:16
From worker nodeHostnamewn-pep-008.farm.nikhef.nl
cpuinfoIntel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-04 01:20:44
Input filesvd-protodune:np02vd_raw_run042421_0148_df-s02-d1_dw_0_20260203T145103.hdf5
JobscriptExit code0
Real time1h (6565s)
CPU time50m (3058s = 46%)
Max RSS bytes2633646080 (2511 MiB)
Outputting started2026-02-04 03:10:11
Output files
Finished2026-02-04 03:12:16
Saved logsjustin-logs:303728.7-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

exception in looking up space point ptr... 0x5c75f2a0, 0x5c75f4a0
Caught exception in looking up space point ptr... 0x5c75f4a0, 0x5c75f5a0
Caught exception in looking up space point ptr... 0x6111a6c0, 0x6111a530
Caught exception in looking up space point ptr... 0x6111a530, 0x6111a6c0
Caught exception in looking up space point ptr... 0x48dd2680, 0x60b2ac90
Caught exception in looking up space point ptr... 0x60b2ac90, 0x3b042180
Caught exception in looking up space point ptr... 0x69a0edd0, 0x69a0f320
Caught exception in looking up space point ptr... 0x69a0f320, 0x69a04310
Caught exception in looking up space point ptr... 0x69a04310, 0x69a03ce0
Caught exception in looking up space point ptr... 0x69a03ce0, 0x69a0de00
Caught exception in looking up space point ptr... 0x69a0de00, 0x69a0edd0
Caught exception in looking up space point ptr... 0x55f4c670, 0x27816c40
Caught exception in looking up space point ptr... 0x27816c40, 0x55f4c670
Caught exception in looking up space point ptr... 0x5cd21410, 0x48d8c390
Caught exception in looking up space point ptr... 0x48d8c390, 0x5cd21410
Caught exception in looking up space point ptr... 0x5610bdf0, 0x5610bc60
Caught exception in looking up space point ptr... 0x56114b50, 0x5610bdf0
Caught exception in looking up space point ptr... 0x43819e80, 0x4381a330
Caught exception in looking up space point ptr... 0x4381a330, 0x15bc5d00
Caught exception in looking up space point ptr... 0x3ad9db40, 0x2da25f10
Caught exception in looking up space point ptr... 0x2da25f10, 0x3ad9db40
Caught exception in looking up space point ptr... 0x44716540, 0x447163e0
Caught exception in looking up space point ptr... 0x447163e0, 0x4471af10
Caught exception in looking up space point ptr... 0x600b7f70, 0x4536f450
Caught exception in looking up space point ptr... 0x4536f450, 0x4ddd5470
Caught exception in looking up space point ptr... 0x60cc39d0, 0x60c79850
Caught exception in looking up space point ptr... 0x60c79850, 0x60cd3d50
Caught exception in looking up space point ptr... 0x4f21d920, 0x64f843b0
Caught exception in looking up space point ptr... 0x64f843b0, 0x27817240
Caught exception in looking up space point ptr... 0x4471d0b0, 0x447142a0
Caught exception in looking up space point ptr... 0x447142a0, 0x4471d4b0
Caught exception in looking up space point ptr... 0x6f39bd20, 0x599f6ee0
Caught exception in looking up space point ptr... 0x599f6ee0, 0x36e35520
Caught exception in looking up space point ptr... 0x600b4b70, 0x600b48d0
Caught exception in looking up space point ptr... 0x367f4f80, 0x600b4b70
Caught exception in looking up space point ptr... 0x5c46aa00, 0x5c46a870
Caught exception in looking up space point ptr... 0x5c46a870, 0x5c46a550
Caught exception in looking up space point ptr... 0x36a27170, 0x63b81ef0
Caught exception in looking up space point ptr... 0x3b028310, 0x36a27170
Caught exception in looking up space point ptr... 0x36a27170, 0x3b028220
Caught exception in looking up space point ptr... 0x30ecd290, 0x36a27170
Caught exception in looking up space point ptr... 0x4cce5ee0, 0x47ecb350
Caught exception in looking up space point ptr... 0x47ecb350, 0x4cce5ee0
Caught exception in looking up space point ptr... 0x44d1a3a0, 0x44d11c70
Caught exception in looking up space point ptr... 0x508818e0, 0x44d1a3a0
Caught exception in looking up space point ptr... 0x14ac0a60, 0x14abfe40
Caught exception in looking up space point ptr... 0x14abfe40, 0x14ac0a60
Caught exception in looking up space point ptr... 0x375d8160, 0x3af6a290
Caught exception in looking up space point ptr... 0x3af6a290, 0x375d8160
Caught exception in looking up space point ptr... 0x4ddc6430, 0x5c745140
Caught exception in looking up space point ptr... 0x5c745140, 0x4ddc6430
Caught exception in looking up space point ptr... 0x55cb6ee0, 0x55cb6490
Caught exception in looking up space point ptr... 0x55cb6490, 0x55cb6ee0
Caught exception in looking up space point ptr... 0x6cb85140, 0x48d50050
Caught exception in looking up space point ptr... 0x48d50050, 0x6cb85140
Caught exception in looking up space point ptr... 0x44d31090, 0x44d31b20
Caught exception in looking up space point ptr... 0x44d31b20, 0x44d31090
Caught exception in looking up space point ptr... 0x590055b0, 0x59005940
Caught exception in looking up space point ptr... 0x58fedb10, 0x590055b0
Caught exception in looking up space point ptr... 0x1a38e1f0, 0x1a38e510
Caught exception in looking up space point ptr... 0x1a38e510, 0x1a38e6a0
Caught exception in looking up space point ptr... 0x4fed9570, 0x3cae7180
Caught exception in looking up space point ptr... 0x3cae7180, 0x4fed9570
Caught exception in looking up space point ptr... 0x369ed500, 0x369f9c90
Caught exception in looking up space point ptr... 0x369f9c90, 0x369ed500
++++>>>> total num hits: 69139, num free: 66920
04-Feb-2026 04:10:04 CET  Closed output file "np02vd_raw_run042421_0148_df-s02-d1_dw_0_20260203T145103_reco_stage0_20260204T031004_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         197.976       239.999       284.343       237.194       21.0199        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                        8.408e-05    0.000195262   0.000463548   0.000168111   0.000100326      27     
produce:tpcrawdecoder:PDVDTPCReader                85.1816       119.525       153.807       119.598       18.9138        27     
produce:triggerrawdecoder:PDVDTriggerReader4      0.569048      0.750285       1.20278      0.695525      0.160025        27     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0172043     0.0206021     0.0471626     0.017714     0.00634155       27     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0850622     0.0875348     0.103821      0.0863487    0.00464568       27     
produce:beamevent:BeamEvent                      0.000118692   0.000288605   0.00143328    0.000229577   0.000238499      27     
produce:pdvddaphne:DAPHNEReaderPDVD                13.5646       17.1755       22.5324       16.7166       2.12697        27     
produce:ophit:OpHitFinder                         0.0322518     0.0379616     0.0435187     0.0379246    0.00241117       27     
produce:wclsdatavd:WireCellToolkit                 74.1677       76.3922       79.6696       76.5429       1.11183        27     
produce:gaushit:GausHitFinder                     0.852822       1.23646       1.88507       1.20051      0.244004        27     
produce:nhitsfilter:NumberOfHitsFilter           0.000447746   0.000586897   0.000923576   0.00057638    9.68844e-05      27     
produce:reco3d:SpacePointSolver                    9.94426       14.8264       23.6618       14.8074       3.16594        27     
produce:hitpdune:DisambigFromSpacePoints          0.113007      0.156403      0.246004      0.147748      0.0347706       27     
produce:cluster3d:Cluster3D                        1.32443       2.60538       5.11524       2.45685      0.816355        27     
[art]:TriggerResults:TriggerResultInserter       2.3368e-05    3.5132e-05    7.8595e-05    3.3266e-05    1.27253e-05      27     
end_path:out1:RootOutput                          7.661e-06    9.66222e-06    2.177e-05     9.063e-06    2.62975e-06      27     
end_path:out1:RootOutput(write)                    5.97663       7.14362       7.74586       7.25671      0.448318        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4553.83 MB
  Peak resident set size usage (VmHWM): 2633.65 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042421_0148_df-s02-d1_dw_0_20260203T145103_reco_stage0_20260204T031004_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042421_0148_df-s02-d1_dw_0_20260203T145103_reco_stage0_20260204T031004_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042421_0148_df-s02-d1_dw_0_20260203T145103_reco_stage0_20260204T031004_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770168053.0,
    "core.end_time": 1770174605.0,
    "core.events": [
      24043,
      24049,
      24055,
      24061,
      24067,
      24073,
      24079,
      24085,
      24091,
      24097,
      24103,
      24109,
      24115,
      24121,
      24127,
      24133,
      24139,
      24145,
      24151,
      24157,
      24163,
      24169,
      24175,
      24181,
      24187,
      24193,
      24199
    ],
    "core.event_count": 27,
    "core.first_event_number": 24043,
    "core.last_event_number": 24199,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42421
    ],
    "core.runs_subruns": [
      4242100001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042421_0148_df-s02-d1_dw_0_20260203T145103.hdf5"
    }
  ]
}
justIN time: 2026-02-04 06:07:34 UTC       justIN version: 01.06.00