justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297950.0@dunegpschedd02.fnal.gov

Jobsub ID297950.0@dunegpschedd02.fnal.gov
Workflow ID12718
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 23:50:26
SiteUK_RAL-PPD
EntryCMSHTPC_T2_UK_SGrid_RALPP_hep207
Last heartbeat2026-02-04 01:08:24
From worker nodeHostnameheplnc167.pp.rl.ac.uk
cpuinfoAMD EPYC 7763 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-04 00:13:55
Input filesvd-protodune:np02vd_raw_run042422_0122_df-s04-d1_dw_0_20260203T162044.hdf5
JobscriptExit code0
Real time52m (3124s)
CPU time45m (2754s = 88%)
Max RSS bytes2546458624 (2428 MiB)
Outputting started2026-02-04 01:06:01
Output files
Finished2026-02-04 01:08:24
Saved logsjustin-logs:297950.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

xe4193b0
Caught exception in looking up space point ptr... 0xe4193b0, 0xe40b490
Caught exception in looking up space point ptr... 0xe40b490, 0xe418930
Caught exception in looking up space point ptr... 0xe418930, 0x3d2c7010
Caught exception in looking up space point ptr... 0x3d2c7010, 0x3acbd1e0
Caught exception in looking up space point ptr... 0x3acbd1e0, 0x3acb97c0
Caught exception in looking up space point ptr... 0x3acb97c0, 0x3acbcd20
Caught exception in looking up space point ptr... 0x5af7e9a0, 0x4a5f3610
Caught exception in looking up space point ptr... 0x4a5f3610, 0x2b5df470
Caught exception in looking up space point ptr... 0x2b5df470, 0x27e16b40
Caught exception in looking up space point ptr... 0x27e16b40, 0x27e16da0
Caught exception in looking up space point ptr... 0x27e16da0, 0x27e17000
Caught exception in looking up space point ptr... 0x27e17000, 0x27e17390
Caught exception in looking up space point ptr... 0x27e17390, 0x5af7e9a0
Caught exception in looking up space point ptr... 0x108c6540, 0x5e515150
Caught exception in looking up space point ptr... 0x5e515150, 0x4ebc6900
Caught exception in looking up space point ptr... 0x4ebc6900, 0x1400f010
Caught exception in looking up space point ptr... 0x47e432e0, 0x533f5490
Caught exception in looking up space point ptr... 0x533f5490, 0x250a0ca0
Caught exception in looking up space point ptr... 0x250a0ca0, 0x23b33560
Caught exception in looking up space point ptr... 0x23b33560, 0x23b33a20
Caught exception in looking up space point ptr... 0x23b33a20, 0x23b33b20
Caught exception in looking up space point ptr... 0x23b33b20, 0x29a8cda0
Caught exception in looking up space point ptr... 0x29a8cda0, 0x47e432e0
Caught exception in looking up space point ptr... 0x44bbc710, 0x287b34b0
Caught exception in looking up space point ptr... 0x287b34b0, 0xa7b8de0
Caught exception in looking up space point ptr... 0x12ecb580, 0x12ecc8e0
Caught exception in looking up space point ptr... 0x12ecc8e0, 0x572aba00
Caught exception in looking up space point ptr... 0x12eca2c0, 0x12ecb4e0
Caught exception in looking up space point ptr... 0x12ecb4e0, 0x12ecc740
Caught exception in looking up space point ptr... 0x8b125a0, 0x3c0f49b0
Caught exception in looking up space point ptr... 0x3c0f49b0, 0x5b3aa750
Caught exception in looking up space point ptr... 0x4a5f2d80, 0x4a5f2ae0
Caught exception in looking up space point ptr... 0x4a5f2ae0, 0x4a5f2d80
Caught exception in looking up space point ptr... 0x3d7b2430, 0x5dbd63e0
Caught exception in looking up space point ptr... 0x5dbd63e0, 0x3d7b2430
Caught exception in looking up space point ptr... 0x2735abd0, 0x3accdb30
Caught exception in looking up space point ptr... 0x2735c220, 0x2735abd0
Caught exception in looking up space point ptr... 0x2955d560, 0x2955d3d0
Caught exception in looking up space point ptr... 0x4a456d90, 0x2955d560
Caught exception in looking up space point ptr... 0x51a3fd60, 0x51a3fbc0
Caught exception in looking up space point ptr... 0x47d692e0, 0xe3fb810
Caught exception in looking up space point ptr... 0xe3fb810, 0xe3fb950
Caught exception in looking up space point ptr... 0xe3fb950, 0x51a3fd60
Caught exception in looking up space point ptr... 0x42030f40, 0x2530d770
Caught exception in looking up space point ptr... 0x2530d770, 0x42030f40
Caught exception in looking up space point ptr... 0x14b08f40, 0x3d148b80
Caught exception in looking up space point ptr... 0x3d148b80, 0x14b06850
Caught exception in looking up space point ptr... 0x3d12c030, 0x3d12d1d0
Caught exception in looking up space point ptr... 0x3d12d1d0, 0x3d12e3b0
Caught exception in looking up space point ptr... 0x3acba4c0, 0x3acba650
Caught exception in looking up space point ptr... 0x3acba650, 0x3d2c6340
Caught exception in looking up space point ptr... 0x14b03540, 0x14b02f90
Caught exception in looking up space point ptr... 0x14b02f90, 0x14b03540
Caught exception in looking up space point ptr... 0x14b03540, 0x14b03110
Caught exception in looking up space point ptr... 0x14b03110, 0x14b03540
Caught exception in looking up space point ptr... 0x14b03540, 0x14b03410
Caught exception in looking up space point ptr... 0x14b03410, 0x14b03540
Caught exception in looking up space point ptr... 0xd90d5d0, 0xd90d3f0
Caught exception in looking up space point ptr... 0xd90d3f0, 0xd90d5d0
Caught exception in looking up space point ptr... 0x2e3835f0, 0x2e386780
Caught exception in looking up space point ptr... 0x2e386780, 0x2e3835f0
Caught exception in looking up space point ptr... 0x14b0bd20, 0x14b0be50
Caught exception in looking up space point ptr... 0x14b0be50, 0x14b0c0b0
Caught exception in looking up space point ptr... 0x300e03a0, 0x3da17530
Caught exception in looking up space point ptr... 0x3da17530, 0x300e03a0
++++>>>> total num hits: 38350, num free: 37009
04-Feb-2026 01:05:51 GMT  Closed output file "np02vd_raw_run042422_0122_df-s04-d1_dw_0_20260203T162044_reco_stage0_20260204T010551_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         95.8081       112.124       128.033       112.139       8.96441        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       8.0691e-05    0.000129365   0.000291766   0.000108914   4.96012e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                12.0673       12.708        20.1793       12.3869       1.4838         27     
produce:triggerrawdecoder:PDVDTriggerReader4      0.524015      0.540962      0.552859      0.543741     0.00774886       27     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0176023     0.0178087     0.0185448     0.0177827    0.000177786      27     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0869021     0.0905887     0.109687      0.0879569    0.00651555       27     
produce:beamevent:BeamEvent                      0.000106891   0.000217806   0.00191016    0.000158987   0.000333305      27     
produce:pdvddaphne:DAPHNEReaderPDVD                2.61716       3.59865       4.00628       3.63187      0.261157        27     
produce:ophit:OpHitFinder                         0.0340421     0.0449567     0.0533213     0.0449376    0.00426187       27     
produce:wclsdatavd:WireCellToolkit                 60.6733       76.2064       86.6665       77.1517       7.09431        27     
produce:gaushit:GausHitFinder                     0.659226        1.089        1.6945       0.969266      0.287673        27     
produce:nhitsfilter:NumberOfHitsFilter           0.000188833   0.000286951   0.000765303   0.000255228   0.000107362      27     
produce:reco3d:SpacePointSolver                    5.1652        11.0154       18.7037       10.7116       3.57309        27     
produce:hitpdune:DisambigFromSpacePoints          0.0580852     0.116039      0.198124      0.115522      0.0355142       27     
produce:cluster3d:Cluster3D                       0.854233       2.07442       3.64219       2.09182      0.741504        27     
[art]:TriggerResults:TriggerResultInserter       2.1791e-05    3.97786e-05   0.000150071   3.3873e-05    2.4964e-05       27     
end_path:out1:RootOutput                          5.089e-06    1.03542e-05   7.5061e-05     6.993e-06    1.30259e-05      27     
end_path:out1:RootOutput(write)                    3.30959       4.59272       5.66227       4.85565      0.756864        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4543.57 MB
  Peak resident set size usage (VmHWM): 2546.46 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042422_0122_df-s04-d1_dw_0_20260203T162044_reco_stage0_20260204T010551_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042422_0122_df-s04-d1_dw_0_20260203T162044_reco_stage0_20260204T010551_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042422_0122_df-s04-d1_dw_0_20260203T162044_reco_stage0_20260204T010551_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770164051.0,
    "core.end_time": 1770167153.0,
    "core.events": [
      19787,
      19793,
      19799,
      19805,
      19811,
      19817,
      19823,
      19829,
      19835,
      19841,
      19847,
      19853,
      19859,
      19865,
      19871,
      19877,
      19883,
      19889,
      19895,
      19901,
      19907,
      19913,
      19919,
      19925,
      19931,
      19937,
      19943
    ],
    "core.event_count": 27,
    "core.first_event_number": 19787,
    "core.last_event_number": 19943,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42422
    ],
    "core.runs_subruns": [
      4242200001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042422_0122_df-s04-d1_dw_0_20260203T162044.hdf5"
    }
  ]
}
justIN time: 2026-02-04 04:28:32 UTC       justIN version: 01.06.00