justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303656.39@dunegpschedd01.fnal.gov

Jobsub ID303656.39@dunegpschedd01.fnal.gov
Workflow ID12717
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:56:25
SiteUK_RAL-PPD
EntryCMSHTPC_T2_UK_SGrid_RALPP_hep207
Last heartbeat2026-02-04 01:06:31
From worker nodeHostnameheplnc147.pp.rl.ac.uk
cpuinfoAMD EPYC 7452 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-04 00:12:04
Input filesvd-protodune:np02vd_raw_run042421_0040_df-s04-d0_dw_0_20260203T135239.hdf5
JobscriptExit code0
Real time52m (3126s)
CPU time45m (2719s = 86%)
Max RSS bytes2591117312 (2471 MiB)
Outputting started2026-02-04 01:04:11
Output files
Finished2026-02-04 01:06:31
Saved logsjustin-logs:303656.39-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

pace point ptr... 0x15114490, 0x15114170
Caught exception in looking up space point ptr... 0x656c7370, 0x4992cd70
Caught exception in looking up space point ptr... 0x4992cd70, 0x656c7370
Caught exception in looking up space point ptr... 0x2e07b560, 0x2e084700
Caught exception in looking up space point ptr... 0x41237b30, 0x2e07b560
Caught exception in looking up space point ptr... 0x607e74d0, 0x607dbca0
Caught exception in looking up space point ptr... 0x607dbca0, 0x607e7920
Caught exception in looking up space point ptr... 0x98db1a0, 0x11809170
Caught exception in looking up space point ptr... 0x229c4710, 0x5737a030
Caught exception in looking up space point ptr... 0x5737a030, 0x98db1a0
Caught exception in looking up space point ptr... 0x474dc2e0, 0x2a33ff90
Caught exception in looking up space point ptr... 0x2a33ff90, 0x474dc2e0
Caught exception in looking up space point ptr... 0x43028f30, 0x430290c0
Caught exception in looking up space point ptr... 0x430290c0, 0x43038dd0
Caught exception in looking up space point ptr... 0x445c0750, 0x445c0650
Caught exception in looking up space point ptr... 0x445c0650, 0x445c0580
Caught exception in looking up space point ptr... 0x9512090, 0x41235ee0
Caught exception in looking up space point ptr... 0x41235ee0, 0x9512090
Caught exception in looking up space point ptr... 0x656c6ed0, 0x1ffcfd50
Caught exception in looking up space point ptr... 0x1ffcfd50, 0x656c6ed0
Caught exception in looking up space point ptr... 0x3051ad50, 0x363c6750
Caught exception in looking up space point ptr... 0xe765980, 0x3051ad50
Caught exception in looking up space point ptr... 0x607f71a0, 0x607f4ed0
Caught exception in looking up space point ptr... 0x9a28a30, 0x607f71a0
Caught exception in looking up space point ptr... 0x59721580, 0x4431cd80
Caught exception in looking up space point ptr... 0x4431cd80, 0x310d4260
Caught exception in looking up space point ptr... 0x225bd3f0, 0x225bb250
Caught exception in looking up space point ptr... 0x225bb250, 0x225bd3f0
Caught exception in looking up space point ptr... 0x22c7fa10, 0x22c7fd30
Caught exception in looking up space point ptr... 0x22c7fd30, 0x22c7fa10
Caught exception in looking up space point ptr... 0x33efcf20, 0x33f00ee0
Caught exception in looking up space point ptr... 0x33f00ee0, 0x24ff2100
Caught exception in looking up space point ptr... 0x66aff10, 0x43d15e90
Caught exception in looking up space point ptr... 0x43d15e90, 0x66aff10
Caught exception in looking up space point ptr... 0x229d3b90, 0x229d4c10
Caught exception in looking up space point ptr... 0x229d4c10, 0x229d97e0
Caught exception in looking up space point ptr... 0x15128c00, 0x15128f70
Caught exception in looking up space point ptr... 0x15128f70, 0x151284e0
Caught exception in looking up space point ptr... 0x155adb50, 0x155ad4f0
Caught exception in looking up space point ptr... 0x155ad4f0, 0x155adde0
Caught exception in looking up space point ptr... 0x2bfc5b10, 0x2bfc5db0
Caught exception in looking up space point ptr... 0x2bfc5db0, 0x2bfc5c60
Caught exception in looking up space point ptr... 0xa0b7760, 0x21e33140
Caught exception in looking up space point ptr... 0x21e33140, 0xa0b7760
Caught exception in looking up space point ptr... 0x661e6500, 0x58a95fc0
Caught exception in looking up space point ptr... 0x58a95fc0, 0x661e6500
Caught exception in looking up space point ptr... 0x2b4b4b30, 0x948fab0
Caught exception in looking up space point ptr... 0x948fab0, 0x3870bd30
Caught exception in looking up space point ptr... 0xccf1d20, 0xccf2040
Caught exception in looking up space point ptr... 0xccf2040, 0xccf1d20
Caught exception in looking up space point ptr... 0x2a46a360, 0x469c3d80
Caught exception in looking up space point ptr... 0x469c3d80, 0x667a6d0
Caught exception in looking up space point ptr... 0x227500b0, 0x1fa65dd0
Caught exception in looking up space point ptr... 0x1fa65dd0, 0x227500b0
Caught exception in looking up space point ptr... 0x22c7d2b0, 0x22c7cd60
Caught exception in looking up space point ptr... 0x22c7cd60, 0x22c7cf40
Caught exception in looking up space point ptr... 0x12b1c530, 0x10a108d0
Caught exception in looking up space point ptr... 0x10a108d0, 0x12b1c530
Caught exception in looking up space point ptr... 0x12b1c530, 0x288d3840
Caught exception in looking up space point ptr... 0x288d3840, 0x12b1c530
Caught exception in looking up space point ptr... 0x30930060, 0x2c0a2d40
Caught exception in looking up space point ptr... 0x2c0a2d40, 0x30930060
Caught exception in looking up space point ptr... 0x30930060, 0x4022ff50
Caught exception in looking up space point ptr... 0x4022ff50, 0x30930060
Caught exception in looking up space point ptr... 0x30930060, 0x15184080
Caught exception in looking up space point ptr... 0x15184080, 0x30930060
++++>>>> total num hits: 76550, num free: 73705
04-Feb-2026 01:04:02 GMT  Closed output file "np02vd_raw_run042421_0040_df-s04-d0_dw_0_20260203T135239_reco_stage0_20260204T010402_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         94.7412       111.902       132.357       112.573       9.19587        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       8.1865e-05    0.000127875   0.00025386    9.8796e-05    5.00783e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                11.956        13.1788       15.4516       12.928       0.942661        27     
produce:triggerrawdecoder:PDVDTriggerReader4      0.530414      0.573587      0.698407      0.561339      0.0359725       27     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0173688     0.0202903     0.0399362     0.0179818    0.00596505       27     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0867484     0.0896322     0.106086      0.0885484    0.00443303       27     
produce:beamevent:BeamEvent                       7.424e-05    0.000201842   0.00163647    0.000136909   0.000284908      27     
produce:pdvddaphne:DAPHNEReaderPDVD                3.11477       3.61488       4.09585       3.59215      0.212171        27     
produce:ophit:OpHitFinder                         0.0318334     0.0433529     0.0511068     0.0432683    0.00447131       27     
produce:wclsdatavd:WireCellToolkit                 62.2641       72.8839       87.8096       71.4553       7.25536        27     
produce:gaushit:GausHitFinder                     0.556409       1.09904       1.8423         1.07        0.316786        27     
produce:nhitsfilter:NumberOfHitsFilter           0.000273235   0.000390832   0.000629558   0.000377623   8.23516e-05      27     
produce:reco3d:SpacePointSolver                    7.89793       13.7003       19.6579       14.2871       2.95892        27     
produce:hitpdune:DisambigFromSpacePoints          0.0747571     0.131566      0.190509       0.13552      0.0359761       27     
produce:cluster3d:Cluster3D                       0.963193       2.2667        4.62533       2.17137      0.891428        27     
[art]:TriggerResults:TriggerResultInserter       2.3915e-05    3.74848e-05   0.000195569   2.7662e-05    3.17371e-05      27     
end_path:out1:RootOutput                          3.938e-06     1.115e-05    4.9363e-05     9.808e-06    8.06491e-06      27     
end_path:out1:RootOutput(write)                    3.31402       4.27465       5.56497       4.0585       0.709237        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4548.4 MB
  Peak resident set size usage (VmHWM): 2591.12 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042421_0040_df-s04-d0_dw_0_20260203T135239_reco_stage0_20260204T010402_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042421_0040_df-s04-d0_dw_0_20260203T135239_reco_stage0_20260204T010402_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042421_0040_df-s04-d0_dw_0_20260203T135239_reco_stage0_20260204T010402_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770163937.0,
    "core.end_time": 1770167044.0,
    "core.events": [
      6526,
      6532,
      6538,
      6544,
      6550,
      6556,
      6562,
      6568,
      6574,
      6580,
      6586,
      6592,
      6598,
      6604,
      6610,
      6616,
      6622,
      6628,
      6634,
      6640,
      6646,
      6652,
      6658,
      6664,
      6670,
      6676,
      6682
    ],
    "core.event_count": 27,
    "core.first_event_number": 6526,
    "core.last_event_number": 6682,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42421
    ],
    "core.runs_subruns": [
      4242100001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042421_0040_df-s04-d0_dw_0_20260203T135239.hdf5"
    }
  ]
}
justIN time: 2026-02-04 07:36:57 UTC       justIN version: 01.06.00