justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303656.14@dunegpschedd01.fnal.gov

Jobsub ID303656.14@dunegpschedd01.fnal.gov
Workflow ID12717
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:56:25
SiteUK_RAL-Tier1
EntryLIGO_UK_RAL_arc_ce04
Last heartbeat2026-02-04 00:59:10
From worker nodeHostnamedune001-1520746.0-lcg2719.gridpp.rl.ac.uk
cpuinfoAMD EPYC 9654 96-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit216000 (60 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-04 00:11:40
Input filesvd-protodune:np02vd_raw_run042421_0070_df-s02-d1_dw_0_20260203T140852.hdf5
JobscriptExit code0
Real time45m (2744s)
CPU time39m (2369s = 86%)
Max RSS bytes2543259648 (2425 MiB)
Outputting started2026-02-04 00:57:26
Output files
Finished2026-02-04 00:59:10
Saved logsjustin-logs:303656.14-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

exception in looking up space point ptr... 0x647857e0, 0x64782e90
Caught exception in looking up space point ptr... 0x3efaff40, 0x3efb3cf0
Caught exception in looking up space point ptr... 0x3efb3cf0, 0x3efb3f60
Caught exception in looking up space point ptr... 0x3efb3f60, 0x3efb3de0
Caught exception in looking up space point ptr... 0x3efb0370, 0x3efb00c0
Caught exception in looking up space point ptr... 0x3efb00c0, 0x3efaff40
Caught exception in looking up space point ptr... 0x59400170, 0x59400040
Caught exception in looking up space point ptr... 0x59400040, 0x593fb490
Caught exception in looking up space point ptr... 0x28026920, 0x28023420
Caught exception in looking up space point ptr... 0x28023420, 0x28026920
Caught exception in looking up space point ptr... 0x707c59b0, 0x707c64c0
Caught exception in looking up space point ptr... 0x707c64c0, 0x707c5750
Caught exception in looking up space point ptr... 0x652de370, 0x652de2d0
Caught exception in looking up space point ptr... 0x652de2d0, 0x652de370
Caught exception in looking up space point ptr... 0x57597aa0, 0x67b79150
Caught exception in looking up space point ptr... 0x67b79150, 0x57597aa0
Caught exception in looking up space point ptr... 0x71f2cae0, 0x3e1648c0
Caught exception in looking up space point ptr... 0x3e1648c0, 0x71f2cae0
Caught exception in looking up space point ptr... 0x3ef9ae50, 0x3ef9ac90
Caught exception in looking up space point ptr... 0x3ef9ac90, 0x3ef9ae50
Caught exception in looking up space point ptr... 0x25c051f0, 0x72a38ab0
Caught exception in looking up space point ptr... 0x72a38ab0, 0x26887ad0
Caught exception in looking up space point ptr... 0x5f1cf150, 0x5f1c8a20
Caught exception in looking up space point ptr... 0x5f1c8a20, 0x5f1c8d00
Caught exception in looking up space point ptr... 0x517aeed0, 0x664ccff0
Caught exception in looking up space point ptr... 0x664ccff0, 0x664cd1d0
Caught exception in looking up space point ptr... 0x664cd1d0, 0x664cd360
Caught exception in looking up space point ptr... 0x21fe8fe0, 0x473adbf0
Caught exception in looking up space point ptr... 0x473adbf0, 0x6945b7d0
Caught exception in looking up space point ptr... 0x593812f0, 0x45b25440
Caught exception in looking up space point ptr... 0x45b25440, 0x593812f0
Caught exception in looking up space point ptr... 0x6d111c90, 0x6d1119f0
Caught exception in looking up space point ptr... 0x6d1119f0, 0x6d1118a0
Caught exception in looking up space point ptr... 0x24c9d7f0, 0x24c9d940
Caught exception in looking up space point ptr... 0x24c9d940, 0x24c9d7f0
Caught exception in looking up space point ptr... 0x5e4a3630, 0x5e4a5ae0
Caught exception in looking up space point ptr... 0x5e4a5990, 0x5e4a3630
Caught exception in looking up space point ptr... 0x45088cc0, 0x45088b30
Caught exception in looking up space point ptr... 0x45088b30, 0x45088cc0
Caught exception in looking up space point ptr... 0x1e973c40, 0x1e970d60
Caught exception in looking up space point ptr... 0x1e970d60, 0x1e971490
Caught exception in looking up space point ptr... 0x471538a0, 0x47153710
Caught exception in looking up space point ptr... 0x47153710, 0x47153a30
Caught exception in looking up space point ptr... 0x5f1c91f0, 0x5f1c93f0
Caught exception in looking up space point ptr... 0x5f1c93f0, 0x5f1c91f0
Caught exception in looking up space point ptr... 0x39947c60, 0x399478f0
Caught exception in looking up space point ptr... 0x399478f0, 0x39947c60
Caught exception in looking up space point ptr... 0x6d1132d0, 0x6d113440
Caught exception in looking up space point ptr... 0x6d113440, 0x6d113760
Caught exception in looking up space point ptr... 0x24cb8f90, 0x24cb85d0
Caught exception in looking up space point ptr... 0x24cb85d0, 0x24cb8e40
Caught exception in looking up space point ptr... 0x35e70280, 0x35e700f0
Caught exception in looking up space point ptr... 0x35e700f0, 0x35e70280
Caught exception in looking up space point ptr... 0x45d01380, 0x45d01510
Caught exception in looking up space point ptr... 0x45d01510, 0x45d01380
Caught exception in looking up space point ptr... 0x35e6a120, 0x35e69e00
Caught exception in looking up space point ptr... 0x35e69e00, 0x35e69ae0
Caught exception in looking up space point ptr... 0x57302e50, 0x66135520
Caught exception in looking up space point ptr... 0x66135520, 0x68f20b40
Caught exception in looking up space point ptr... 0x22f95240, 0x22f96530
Caught exception in looking up space point ptr... 0x22f96530, 0x22f95240
Caught exception in looking up space point ptr... 0x22f95240, 0x22f95f20
Caught exception in looking up space point ptr... 0x22f95f20, 0x22f95240
Caught exception in looking up space point ptr... 0x5bb72570, 0x69747b30
Caught exception in looking up space point ptr... 0x69747b30, 0x46b73f80
++++>>>> total num hits: 100303, num free: 96418
04-Feb-2026 00:57:15 UTC  Closed output file "np02vd_raw_run042421_0070_df-s02-d1_dw_0_20260203T140852_reco_stage0_20260204T005715_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         83.1622       98.4326       109.674       97.5035       6.94871        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       7.5394e-05    0.000129904   0.000341535   0.000100131   6.37788e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                12.8515       13.4251       15.7626       13.231       0.654433        27     
produce:triggerrawdecoder:PDVDTriggerReader4       0.68347      0.714815      0.814529      0.713168      0.0244668       27     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0229753     0.0242787     0.0368896     0.023318     0.00285089       27     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.113491      0.117438      0.138696      0.115045     0.00727825       27     
produce:beamevent:BeamEvent                      0.000102465   0.000202374   0.00100979    0.00017909    0.000161686      27     
produce:pdvddaphne:DAPHNEReaderPDVD                3.18647       3.62559       3.98961       3.72534      0.231846        27     
produce:ophit:OpHitFinder                         0.0331183     0.0418903     0.0556536     0.0416524    0.00511297       27     
produce:wclsdatavd:WireCellToolkit                 53.8903       63.5765       71.8276       64.0383       5.85314        27     
produce:gaushit:GausHitFinder                     0.519222       1.00809       1.49148      0.963978      0.274813        27     
produce:nhitsfilter:NumberOfHitsFilter           0.000137758   0.000262345   0.000593044   0.000251319   9.56941e-05      27     
produce:reco3d:SpacePointSolver                    4.56091       9.29601       14.0429       9.11953       2.35453        27     
produce:hitpdune:DisambigFromSpacePoints          0.0504128      0.10438      0.153208      0.106471      0.0279825       27     
produce:cluster3d:Cluster3D                        0.88298       2.45053       18.3052       1.79522       3.15389        27     
[art]:TriggerResults:TriggerResultInserter       2.7291e-05    3.84004e-05   7.0716e-05    3.6856e-05    9.62121e-06      27     
end_path:out1:RootOutput                          3.455e-06    1.02848e-05   3.2669e-05     8.963e-06    5.90265e-06      27     
end_path:out1:RootOutput(write)                    2.87051       4.01214       4.69464       4.16338      0.489477        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4821.45 MB
  Peak resident set size usage (VmHWM): 2543.26 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042421_0070_df-s02-d1_dw_0_20260203T140852_reco_stage0_20260204T005715_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042421_0070_df-s02-d1_dw_0_20260203T140852_reco_stage0_20260204T005715_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042421_0070_df-s02-d1_dw_0_20260203T140852_reco_stage0_20260204T005715_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770163911.0,
    "core.end_time": 1770166637.0,
    "core.events": [
      11395,
      11401,
      11407,
      11413,
      11419,
      11425,
      11431,
      11437,
      11443,
      11449,
      11455,
      11461,
      11467,
      11473,
      11479,
      11485,
      11491,
      11497,
      11503,
      11509,
      11515,
      11521,
      11527,
      11533,
      11539,
      11545,
      11551
    ],
    "core.event_count": 27,
    "core.first_event_number": 11395,
    "core.last_event_number": 11551,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42421
    ],
    "core.runs_subruns": [
      4242100001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042421_0070_df-s02-d1_dw_0_20260203T140852.hdf5"
    }
  ]
}
justIN time: 2026-02-04 04:28:01 UTC       justIN version: 01.06.00