justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297921.20@dunegpschedd02.fnal.gov

Jobsub ID297921.20@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:58:26
SiteUK_RAL-PPD
EntryCMSHTPC_T2_UK_SGrid_RALPP_hep206
Last heartbeat2026-02-04 00:39:51
From worker nodeHostnameheplnc171.pp.rl.ac.uk
cpuinfoAMD EPYC 9534 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:45:00
Input filesvd-protodune:np02vd_raw_run042433_0373_df-s04-d0_dw_0_20260203T210353.hdf5
JobscriptExit code0
Real time52m (3144s)
CPU time48m (2893s = 92%)
Max RSS bytes3371839488 (3215 MiB)
Outputting started2026-02-04 00:37:26
Output files
Finished2026-02-04 00:39:51
Saved logsjustin-logs:297921.20-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

nt ptr... 0x53fd68e0, 0x53fd6a70
Caught exception in looking up space point ptr... 0x3a136eb0, 0x3a130490
Caught exception in looking up space point ptr... 0x3a130490, 0x3a136eb0
Caught exception in looking up space point ptr... 0x3fabc070, 0x3fabbed0
Caught exception in looking up space point ptr... 0x3fabbed0, 0x3fabc070
Caught exception in looking up space point ptr... 0x652b2500, 0x652b21e0
Caught exception in looking up space point ptr... 0x652b21e0, 0x652b2500
Caught exception in looking up space point ptr... 0x7d119ae0, 0x7d11b350
Caught exception in looking up space point ptr... 0x7d11b350, 0x7d119ae0
Caught exception in looking up space point ptr... 0x474248c0, 0x47424a50
Caught exception in looking up space point ptr... 0x47424a50, 0x47424be0
Caught exception in looking up space point ptr... 0x70576bd0, 0x7058c8a0
Caught exception in looking up space point ptr... 0x7058c8a0, 0x3cd896c0
Caught exception in looking up space point ptr... 0x2baa9940, 0x2baa9cb0
Caught exception in looking up space point ptr... 0x2baa9cb0, 0x6c8ebef0
Caught exception in looking up space point ptr... 0x2baa9e40, 0x2baaa1b0
Caught exception in looking up space point ptr... 0x2baaa1b0, 0x6c8ec040
Caught exception in looking up space point ptr... 0x14eabe00, 0x14eab130
Caught exception in looking up space point ptr... 0x14eab130, 0x14eabe00
Caught exception in looking up space point ptr... 0x482000e0, 0x482003f0
Caught exception in looking up space point ptr... 0x482003f0, 0x482000e0
Caught exception in looking up space point ptr... 0x6c7fb8f0, 0x6c7f9ec0
Caught exception in looking up space point ptr... 0x6c7f9ec0, 0x6c7ffce0
Caught exception in looking up space point ptr... 0x6c7ffce0, 0x6c7fff80
Caught exception in looking up space point ptr... 0x6c7fff80, 0x6c7fb8f0
Caught exception in looking up space point ptr... 0x30300a20, 0x6e74f7b0
Caught exception in looking up space point ptr... 0x6e74f7b0, 0x30300a20
Caught exception in looking up space point ptr... 0x6e303e40, 0x6e303530
Caught exception in looking up space point ptr... 0x6e303530, 0x6e3023d0
Caught exception in looking up space point ptr... 0x705169f0, 0x7051a720
Caught exception in looking up space point ptr... 0x7051a720, 0x70518f30
Caught exception in looking up space point ptr... 0x68611c70, 0x6861b410
Caught exception in looking up space point ptr... 0x686173b0, 0x68611c70
Caught exception in looking up space point ptr... 0x343b6010, 0x343b5cf0
Caught exception in looking up space point ptr... 0x343b5cf0, 0x343b5b60
Caught exception in looking up space point ptr... 0x5d3a30e0, 0x5d3a2f50
Caught exception in looking up space point ptr... 0x5d3a2f50, 0x5d3a30e0
Caught exception in looking up space point ptr... 0x67ccee00, 0x67cd8440
Caught exception in looking up space point ptr... 0x67cd8440, 0x67ccee00
Caught exception in looking up space point ptr... 0x1a702110, 0x1a702260
Caught exception in looking up space point ptr... 0x1a702260, 0x1a702110
Caught exception in looking up space point ptr... 0xb046510, 0xb0461f0
Caught exception in looking up space point ptr... 0xb0461f0, 0xb046080
Caught exception in looking up space point ptr... 0x5493af20, 0x5493ad90
Caught exception in looking up space point ptr... 0x5493ad90, 0x5493af20
Caught exception in looking up space point ptr... 0x5493af20, 0x5493aa70
Caught exception in looking up space point ptr... 0x5493aa70, 0x5493af20
Caught exception in looking up space point ptr... 0x70521df0, 0x70518090
Caught exception in looking up space point ptr... 0x70518090, 0x70521df0
Caught exception in looking up space point ptr... 0x39f622c0, 0x7ae5f30
Caught exception in looking up space point ptr... 0x7ae5f30, 0x39f622c0
Caught exception in looking up space point ptr... 0x3801b040, 0x18cc6d20
Caught exception in looking up space point ptr... 0x18cc6d20, 0x3801b040
Caught exception in looking up space point ptr... 0x685f2f80, 0x685f3160
Caught exception in looking up space point ptr... 0x685f3160, 0x685f2f80
Caught exception in looking up space point ptr... 0xda05850, 0xda05530
Caught exception in looking up space point ptr... 0xda05530, 0xda05850
Caught exception in looking up space point ptr... 0x61c145e0, 0x39747b10
Caught exception in looking up space point ptr... 0x39747b10, 0x61c145e0
Caught exception in looking up space point ptr... 0x55ba2590, 0x55bc6aa0
Caught exception in looking up space point ptr... 0x55bc6aa0, 0x55ba2590
Caught exception in looking up space point ptr... 0x652d7580, 0x34167070
Caught exception in looking up space point ptr... 0x34167070, 0x652d7580
Caught exception in looking up space point ptr... 0x652d7580, 0x652d7a30
Caught exception in looking up space point ptr... 0x652d7a30, 0x652d7580
Caught exception in looking up space point ptr... 0x652d7580, 0x705fb300
Caught exception in looking up space point ptr... 0x705fb300, 0x652d7580
++++>>>> total num hits: 170624, num free: 164056
04-Feb-2026 00:37:18 GMT  Closed output file "np02vd_raw_run042433_0373_df-s04-d0_dw_0_20260203T210353_reco_stage0_20260204T003718_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         141.458       161.301       185.492       159.282       11.0571        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       8.5401e-05    0.000128329   0.000234624   0.000104652   4.75841e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                13.1839       13.7632       16.2493       13.5884      0.670062        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.514041      0.544611      0.584958      0.539737      0.0170837       19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0176544     0.0179904     0.0187372     0.0179833    0.000226623      19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0175761     0.0180467     0.0202514     0.0179219    0.000540282      19     
produce:beamevent:BeamEvent                      0.000102231   0.000205244   0.000975053   0.000170833   0.000184197      19     
produce:pdvddaphne:DAPHNEReaderPDVD                2.85109       3.22668       3.58916       3.20222      0.204924        19     
produce:ophit:OpHitFinder                         0.0287654     0.0397478     0.0455559     0.0413662     0.0045887       19     
produce:wclsdatavd:WireCellToolkit                 102.657       113.331       123.066       112.126       6.02684        19     
produce:gaushit:GausHitFinder                      1.09298       1.65349       2.23507       1.68975       0.35772        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000234963   0.000438697   0.000917613   0.000412576   0.000164414      19     
produce:reco3d:SpacePointSolver                    9.13117       19.1339       31.2942       17.8884       6.59387        19     
produce:hitpdune:DisambigFromSpacePoints          0.133573      0.241452       0.36247      0.241275      0.0640739       19     
produce:cluster3d:Cluster3D                        1.53996       3.3296        6.5198        3.11246       1.24432        19     
[art]:TriggerResults:TriggerResultInserter        2.624e-05    4.30265e-05   0.000116282   3.6481e-05    1.96206e-05      19     
end_path:out1:RootOutput                          5.01e-06     1.04905e-05   4.7501e-05     8.65e-06     8.80648e-06      19     
end_path:out1:RootOutput(write)                    4.35192       5.94576       6.86006       6.07377      0.686405        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5384.47 MB
  Peak resident set size usage (VmHWM): 3371.84 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0373_df-s04-d0_dw_0_20260203T210353_reco_stage0_20260204T003718_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0373_df-s04-d0_dw_0_20260203T210353_reco_stage0_20260204T003718_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0373_df-s04-d0_dw_0_20260203T210353_reco_stage0_20260204T003718_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770162312.0,
    "core.end_time": 1770165440.0,
    "core.events": [
      42532,
      42538,
      42544,
      42550,
      42556,
      42562,
      42568,
      42574,
      42580,
      42586,
      42592,
      42598,
      42604,
      42610,
      42616,
      42622,
      42628,
      42634,
      42640
    ],
    "core.event_count": 19,
    "core.first_event_number": 42532,
    "core.last_event_number": 42640,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0373_df-s04-d0_dw_0_20260203T210353.hdf5"
    }
  ]
}
justIN time: 2026-02-04 09:14:23 UTC       justIN version: 01.06.00