justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297917.29@dunegpschedd02.fnal.gov

Jobsub ID297917.29@dunegpschedd02.fnal.gov
Workflow ID12717
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:48:25
SiteUK_Imperial
EntryDUNE_T2_UK_London_IC_ceprod02
Last heartbeat2026-02-04 00:18:59
From worker nodeHostnamewl00.grid.hep.ph.ic.ac.uk
cpuinfoAMD EPYC 7742 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:10:18
Input filesvd-protodune:np02vd_raw_run042421_0067_df-s03-d1_dw_0_20260203T140717.hdf5
JobscriptExit code0
Real time1h (3964s)
CPU time56m (3374s = 85%)
Max RSS bytes2613755904 (2492 MiB)
Outputting started2026-02-04 00:16:24
Output files
Finished2026-02-04 00:18:59
Saved logsjustin-logs:297917.29-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

64c53c10
Caught exception in looking up space point ptr... 0x64c53c10, 0x64c53f30
Caught exception in looking up space point ptr... 0x346d5270, 0x346d4f50
Caught exception in looking up space point ptr... 0x346d4f50, 0x346d5270
Caught exception in looking up space point ptr... 0x64c51370, 0x64c511e0
Caught exception in looking up space point ptr... 0x64c511e0, 0x64c51500
Caught exception in looking up space point ptr... 0x64c59b60, 0x64c599d0
Caught exception in looking up space point ptr... 0x64c599d0, 0x64c59cf0
Caught exception in looking up space point ptr... 0x375992d0, 0x37598fb0
Caught exception in looking up space point ptr... 0x37598fb0, 0x37599140
Caught exception in looking up space point ptr... 0x31ad80f0, 0x4c047980
Caught exception in looking up space point ptr... 0x30d2b520, 0x31ad80f0
Caught exception in looking up space point ptr... 0x44259450, 0x2e299940
Caught exception in looking up space point ptr... 0x2e299940, 0x30f59ee0
Caught exception in looking up space point ptr... 0x9f90980, 0x9f906e0
Caught exception in looking up space point ptr... 0x9f906e0, 0x9f90980
Caught exception in looking up space point ptr... 0x5180b960, 0x2df6fca0
Caught exception in looking up space point ptr... 0x2df6fca0, 0x5180b960
Caught exception in looking up space point ptr... 0x2df6f110, 0x25ab73c0
Caught exception in looking up space point ptr... 0x25ab73c0, 0x25ab7290
Caught exception in looking up space point ptr... 0x64c564b0, 0x64c56640
Caught exception in looking up space point ptr... 0x64c56640, 0x64c56320
Caught exception in looking up space point ptr... 0x4484abc0, 0x27776a00
Caught exception in looking up space point ptr... 0x27776a00, 0x4484abc0
Caught exception in looking up space point ptr... 0x4484abc0, 0x38d1fb90
Caught exception in looking up space point ptr... 0x38d1fb90, 0x4484abc0
Caught exception in looking up space point ptr... 0x4484abc0, 0x27776b90
Caught exception in looking up space point ptr... 0x27776b90, 0x4484abc0
Caught exception in looking up space point ptr... 0x4d014950, 0x1479d160
Caught exception in looking up space point ptr... 0x1479d160, 0x4d014950
Caught exception in looking up space point ptr... 0x64c50ec0, 0x64c50ba0
Caught exception in looking up space point ptr... 0x64c50ba0, 0x64c50d30
Caught exception in looking up space point ptr... 0x64c543e0, 0x64c540c0
Caught exception in looking up space point ptr... 0x64c540c0, 0x64c54250
Caught exception in looking up space point ptr... 0x12370fe0, 0x12370e50
Caught exception in looking up space point ptr... 0x12370e50, 0x27785370
Caught exception in looking up space point ptr... 0x38d21280, 0xa801380
Caught exception in looking up space point ptr... 0xa801380, 0xa801510
Caught exception in looking up space point ptr... 0x3b8a3a30, 0x2676b940
Caught exception in looking up space point ptr... 0x2676b940, 0x3b8a3a30
Caught exception in looking up space point ptr... 0x27777cc0, 0x27777fe0
Caught exception in looking up space point ptr... 0x27777fe0, 0x27778170
Caught exception in looking up space point ptr... 0x4dd1c980, 0xfd5f050
Caught exception in looking up space point ptr... 0xfd5f050, 0x4dd1c980
Caught exception in looking up space point ptr... 0x1a561830, 0x1a556fd0
Caught exception in looking up space point ptr... 0x1a556fd0, 0x1a561830
Caught exception in looking up space point ptr... 0x3759abd0, 0x3759aa40
Caught exception in looking up space point ptr... 0x3759aa40, 0x3759abd0
Caught exception in looking up space point ptr... 0x1af9fa60, 0xc98f930
Caught exception in looking up space point ptr... 0xc98f930, 0xc98fa30
Caught exception in looking up space point ptr... 0x29b6f200, 0x29b6f070
Caught exception in looking up space point ptr... 0x29b6f070, 0x29b6f200
Caught exception in looking up space point ptr... 0x346ee350, 0x346ee1c0
Caught exception in looking up space point ptr... 0x346ee1c0, 0x346ee350
Caught exception in looking up space point ptr... 0x64c59390, 0x64c59200
Caught exception in looking up space point ptr... 0x64c59200, 0x64c59390
Caught exception in looking up space point ptr... 0x1a56f8f0, 0x1a56fa80
Caught exception in looking up space point ptr... 0x1a56fa80, 0x1a56f8f0
Caught exception in looking up space point ptr... 0x50734ce0, 0x402a9980
Caught exception in looking up space point ptr... 0x402a9980, 0x50734ce0
Caught exception in looking up space point ptr... 0x29b797a0, 0x29b79610
Caught exception in looking up space point ptr... 0x29b79610, 0x29b797a0
Caught exception in looking up space point ptr... 0xa8113f0, 0x25a6dea0
Caught exception in looking up space point ptr... 0x25a6dea0, 0xa8113f0
Caught exception in looking up space point ptr... 0xfe1b750, 0x12b81910
Caught exception in looking up space point ptr... 0x12b81910, 0x5d8b98c0
++++>>>> total num hits: 98882, num free: 95139
04-Feb-2026 00:16:12 GMT  Closed output file "np02vd_raw_run042421_0067_df-s03-d1_dw_0_20260203T140717_reco_stage0_20260204T001612_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         114.904       141.87        163.564       140.571       11.3882        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       6.5682e-05    0.000106772   0.000281769   9.5192e-05    4.38057e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                12.3372       12.9992       14.5339       13.0348      0.466198        27     
produce:triggerrawdecoder:PDVDTriggerReader4       0.49158      0.552767      0.710383      0.532643      0.0519894       27     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0163163     0.0206578     0.114539      0.0167831     0.0184369       27     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0804013     0.0850115     0.0995008     0.0823777     0.0063338       27     
produce:beamevent:BeamEvent                      9.0922e-05    0.000188186   0.00123526    0.000147174   0.00021002       27     
produce:pdvddaphne:DAPHNEReaderPDVD                4.08238       4.70095       5.92814       4.62209       0.48203        27     
produce:ophit:OpHitFinder                         0.0401966     0.0504801     0.0583081     0.0506671    0.00487566       27     
produce:wclsdatavd:WireCellToolkit                 74.9304       95.7515       108.328       97.107        9.00274        27     
produce:gaushit:GausHitFinder                     0.747298       1.44967       2.15976       1.42629       0.3982         27     
produce:nhitsfilter:NumberOfHitsFilter           0.000235887   0.000392616   0.000793104   0.000389432   0.000112592      27     
produce:reco3d:SpacePointSolver                    6.06777       14.4518       21.7644       15.1283       4.48743        27     
produce:hitpdune:DisambigFromSpacePoints          0.0686802     0.167342      0.373319      0.179135      0.0607147       27     
produce:cluster3d:Cluster3D                        1.67876       4.06668       12.0488       3.21977       2.11073        27     
[art]:TriggerResults:TriggerResultInserter       2.1841e-05    3.39628e-05   8.0423e-05    3.1151e-05    1.1755e-05       27     
end_path:out1:RootOutput                          4.21e-06     1.01224e-05   3.1071e-05     8.351e-06    6.29205e-06      27     
end_path:out1:RootOutput(write)                    5.23484       7.53975       11.0589       7.49007       1.27075        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4520.34 MB
  Peak resident set size usage (VmHWM): 2613.76 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042421_0067_df-s03-d1_dw_0_20260203T140717_reco_stage0_20260204T001612_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042421_0067_df-s03-d1_dw_0_20260203T140717_reco_stage0_20260204T001612_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042421_0067_df-s03-d1_dw_0_20260203T140717_reco_stage0_20260204T001612_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770160238.0,
    "core.end_time": 1770164174.0,
    "core.events": [
      10917,
      10923,
      10929,
      10935,
      10941,
      10947,
      10953,
      10959,
      10965,
      10971,
      10977,
      10983,
      10989,
      10995,
      11001,
      11007,
      11013,
      11019,
      11025,
      11031,
      11037,
      11043,
      11049,
      11055,
      11061,
      11067,
      11073
    ],
    "core.event_count": 27,
    "core.first_event_number": 10917,
    "core.last_event_number": 11073,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42421
    ],
    "core.runs_subruns": [
      4242100001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042421_0067_df-s03-d1_dw_0_20260203T140717.hdf5"
    }
  ]
}
justIN time: 2026-02-04 05:58:03 UTC       justIN version: 01.06.00