justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297920.48@dunegpschedd02.fnal.gov

Jobsub ID297920.48@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:54:27
SiteUK_Imperial
EntryDUNE_T2_UK_London_IC_ceprod03
Last heartbeat2026-02-04 01:03:40
From worker nodeHostnamewm14.grid.hep.ph.ic.ac.uk
cpuinfoAMD EPYC 7742 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:37:52
Input filesvd-protodune:np02vd_raw_run042433_0334_df-s02-d1_dw_0_20260203T204903.hdf5
JobscriptExit code0
Real time1h (4990s)
CPU time1h (4356s = 87%)
Max RSS bytes3397742592 (3240 MiB)
Outputting started2026-02-04 01:01:04
Output files
Finished2026-02-04 01:03:40
Saved logsjustin-logs:297920.48-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

g up space point ptr... 0xfc70cf0, 0xfc8b650
Caught exception in looking up space point ptr... 0x1a895150, 0x1a8953f0
Caught exception in looking up space point ptr... 0x1a8953f0, 0x1a895150
Caught exception in looking up space point ptr... 0x1a895150, 0x1a8952a0
Caught exception in looking up space point ptr... 0x1a8952a0, 0x1a895150
Caught exception in looking up space point ptr... 0xedd98e0, 0xedd03c0
Caught exception in looking up space point ptr... 0xedd03c0, 0x58dd0a40
Caught exception in looking up space point ptr... 0x1a88ca90, 0x1a89af30
Caught exception in looking up space point ptr... 0x1a89af30, 0x1a88c8d0
Caught exception in looking up space point ptr... 0xfe46f30, 0xfe43d80
Caught exception in looking up space point ptr... 0xfe43d80, 0xfe43f10
Caught exception in looking up space point ptr... 0x91e1fc0, 0x42e017c0
Caught exception in looking up space point ptr... 0x42e017c0, 0x91e1fc0
Caught exception in looking up space point ptr... 0x34041580, 0x68d6c1a0
Caught exception in looking up space point ptr... 0x34041710, 0x34041580
Caught exception in looking up space point ptr... 0x1513cb70, 0x14fb73f0
Caught exception in looking up space point ptr... 0x14fb73f0, 0x1513cb70
Caught exception in looking up space point ptr... 0x3c469400, 0x3c46a910
Caught exception in looking up space point ptr... 0x3c46a910, 0x3c46bbd0
Caught exception in looking up space point ptr... 0x66c5cfa0, 0x66c5cda0
Caught exception in looking up space point ptr... 0x66c4e940, 0x66c5cfa0
Caught exception in looking up space point ptr... 0x483053b0, 0x1697b480
Caught exception in looking up space point ptr... 0x1697b480, 0xb2e4e50
Caught exception in looking up space point ptr... 0x1a8ad470, 0x1a8b2c40
Caught exception in looking up space point ptr... 0x1a8b2c40, 0x1a8b2d40
Caught exception in looking up space point ptr... 0x55065b90, 0x55064e20
Caught exception in looking up space point ptr... 0x55064e20, 0x55065b90
Caught exception in looking up space point ptr... 0x8415ca20, 0x84153550
Caught exception in looking up space point ptr... 0x84153550, 0x84157010
Caught exception in looking up space point ptr... 0x65178320, 0x65190780
Caught exception in looking up space point ptr... 0x65190780, 0x651905f0
Caught exception in looking up space point ptr... 0x7cf4e130, 0x1b138b50
Caught exception in looking up space point ptr... 0x1b138b50, 0x477b0810
Caught exception in looking up space point ptr... 0x56032050, 0x2b133580
Caught exception in looking up space point ptr... 0x2b133580, 0x56032050
Caught exception in looking up space point ptr... 0xfe293b0, 0xfe2bfc0
Caught exception in looking up space point ptr... 0xfe29060, 0xfe293b0
Caught exception in looking up space point ptr... 0x786e7ac0, 0x498667f0
Caught exception in looking up space point ptr... 0x498667f0, 0xfb39050
Caught exception in looking up space point ptr... 0x7b68be00, 0x62493d90
Caught exception in looking up space point ptr... 0x62493d90, 0x7b68be00
Caught exception in looking up space point ptr... 0x3be0be00, 0x3be07920
Caught exception in looking up space point ptr... 0x3be07920, 0x34011800
Caught exception in looking up space point ptr... 0x1b110f50, 0x1b110e00
Caught exception in looking up space point ptr... 0x1b110e00, 0x3bee72b0
Caught exception in looking up space point ptr... 0x1a8b1340, 0x1a8b14d0
Caught exception in looking up space point ptr... 0x1a8b14d0, 0x1a8b17f0
Caught exception in looking up space point ptr... 0x178da840, 0x178dab30
Caught exception in looking up space point ptr... 0x178dab30, 0x178da840
Caught exception in looking up space point ptr... 0xfe159b0, 0xfe15b50
Caught exception in looking up space point ptr... 0xfe15b50, 0xfe159b0
Caught exception in looking up space point ptr... 0x42ef4ac0, 0x1882f550
Caught exception in looking up space point ptr... 0x1882f550, 0x42ef4ac0
Caught exception in looking up space point ptr... 0xfb3a090, 0x56a3f020
Caught exception in looking up space point ptr... 0x56a3f020, 0xfb3a190
Caught exception in looking up space point ptr... 0x5f912490, 0x87c36a80
Caught exception in looking up space point ptr... 0x87c36a80, 0xee12bb0
Caught exception in looking up space point ptr... 0x340371c0, 0x34036f30
Caught exception in looking up space point ptr... 0x34036f30, 0x340371c0
Caught exception in looking up space point ptr... 0x7ac08370, 0x7ac06760
Caught exception in looking up space point ptr... 0x7ac06760, 0x7ac05540
Caught exception in looking up space point ptr... 0x39119b30, 0x49a9a510
Caught exception in looking up space point ptr... 0x49a9a510, 0x39119b30
Caught exception in looking up space point ptr... 0x7284f4b0, 0x7284cd00
Caught exception in looking up space point ptr... 0x7284cd00, 0x7284f4b0
Caught exception in looking up space point ptr... 0x2b153ed0, 0x2b1542c0
Caught exception in looking up space point ptr... 0x2b1542c0, 0x2b153ed0
++++>>>> total num hits: 120955, num free: 117162
04-Feb-2026 01:00:55 GMT  Closed output file "np02vd_raw_run042433_0334_df-s02-d1_dw_0_20260203T204903_reco_stage0_20260204T010055_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         208.084       256.712       280.85        260.531       19.0692        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       7.2937e-05    0.000151615   0.000281341   0.000110528   6.55596e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                15.6129       18.8018       21.6335       18.7615       1.46171        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.470402      0.502413      0.536321      0.504511      0.0141412       19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0163359     0.016593      0.0167955      0.0166      0.000154781      19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0162531     0.0165908     0.0168817     0.0165965    0.000206018      19     
produce:beamevent:BeamEvent                      6.8489e-05    0.000228377   0.00137332    0.00017519    0.000274205      19     
produce:pdvddaphne:DAPHNEReaderPDVD                3.62062       4.72976       5.77978       4.86088      0.511363        19     
produce:ophit:OpHitFinder                         0.0363098     0.0502083     0.0571088     0.0517149    0.00586273       19     
produce:wclsdatavd:WireCellToolkit                 148.003       178.208       190.055       181.743       10.6484        19     
produce:gaushit:GausHitFinder                      1.37808       2.34531       3.22932       2.53697      0.502207        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000418429   0.000719105   0.00126999    0.000688819   0.000180392      19     
produce:reco3d:SpacePointSolver                    22.6073       35.9409       48.1704       35.8562       8.06511        19     
produce:hitpdune:DisambigFromSpacePoints          0.212755      0.357092      0.520474       0.32903      0.0857449       19     
produce:cluster3d:Cluster3D                        2.87154       5.36313       8.66129       5.28534       1.52684        19     
[art]:TriggerResults:TriggerResultInserter       2.8524e-05    4.00418e-05   0.000123884   3.6098e-05    2.01978e-05      19     
end_path:out1:RootOutput                          6.232e-06    1.08083e-05   3.5427e-05     8.796e-06    6.27489e-06      19     
end_path:out1:RootOutput(write)                    7.5325        10.3309       11.7163       10.5007       1.09971        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5336.32 MB
  Peak resident set size usage (VmHWM): 3397.74 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0334_df-s02-d1_dw_0_20260203T204903_reco_stage0_20260204T010055_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0334_df-s02-d1_dw_0_20260203T204903_reco_stage0_20260204T010055_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0334_df-s02-d1_dw_0_20260203T204903_reco_stage0_20260204T010055_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770161889.0,
    "core.end_time": 1770166857.0,
    "core.events": [
      38083,
      38089,
      38095,
      38101,
      38107,
      38113,
      38119,
      38125,
      38131,
      38137,
      38143,
      38149,
      38155,
      38161,
      38167,
      38173,
      38179,
      38185,
      38191
    ],
    "core.event_count": 19,
    "core.first_event_number": 38083,
    "core.last_event_number": 38191,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0334_df-s02-d1_dw_0_20260203T204903.hdf5"
    }
  ]
}
justIN time: 2026-02-04 04:31:56 UTC       justIN version: 01.06.00