justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303656.36@dunegpschedd01.fnal.gov

Jobsub ID303656.36@dunegpschedd01.fnal.gov
Workflow ID12717
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:56:25
SiteUK_RAL-PPD
EntryCMSHTPC_T2_UK_SGrid_RALPP_hep207
Last heartbeat2026-02-04 01:39:46
From worker nodeHostnameheplnc147.pp.rl.ac.uk
cpuinfoAMD EPYC 7452 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-04 00:11:38
Input filesvd-protodune:np02vd_raw_run042421_0079_df-s04-d1_dw_0_20260203T141345.hdf5
JobscriptExit code0
Real time1h (5102s)
CPU time1h (4622s = 90%)
Max RSS bytes4770603008 (4549 MiB)
Outputting started2026-02-04 01:36:43
Output files
Finished2026-02-04 01:39:46
Saved logsjustin-logs:303656.36-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

b0f90
Caught exception in looking up space point ptr... 0x621bc5a0, 0x19727e10
Caught exception in looking up space point ptr... 0x19728a40, 0x522c0d50
Caught exception in looking up space point ptr... 0x522c0d50, 0x42c8f860
Caught exception in looking up space point ptr... 0x42c8f860, 0x621bc5a0
Caught exception in looking up space point ptr... 0x459f28c0, 0x2fb7c730
Caught exception in looking up space point ptr... 0x2fb7c730, 0x4df32730
Caught exception in looking up space point ptr... 0x4df32730, 0x15b6fda0
Caught exception in looking up space point ptr... 0x42829ba0, 0x636a3a30
Caught exception in looking up space point ptr... 0x636a3a30, 0x276c2bf0
Caught exception in looking up space point ptr... 0x31b434c0, 0x31c94210
Caught exception in looking up space point ptr... 0x31c94210, 0x17b4f350
Caught exception in looking up space point ptr... 0x312157a0, 0x31203500
Caught exception in looking up space point ptr... 0x31203370, 0x312157a0
Caught exception in looking up space point ptr... 0x395a3f90, 0x59ae05d0
Caught exception in looking up space point ptr... 0x59ae05d0, 0x3e669bd0
Caught exception in looking up space point ptr... 0x4645d570, 0x37932840
Caught exception in looking up space point ptr... 0x430778e0, 0x4645d570
Caught exception in looking up space point ptr... 0x39810070, 0x398102c0
Caught exception in looking up space point ptr... 0x398102c0, 0xdab0440
Caught exception in looking up space point ptr... 0x4a669190, 0x3af3c500
Caught exception in looking up space point ptr... 0x3af2b200, 0x4a669190
Caught exception in looking up space point ptr... 0x57e291b0, 0x9324790
Caught exception in looking up space point ptr... 0x57e29400, 0x57e291b0
Caught exception in looking up space point ptr... 0x36e838f0, 0xeebeda0
Caught exception in looking up space point ptr... 0xeebeda0, 0x39431820
Caught exception in looking up space point ptr... 0x3384d260, 0x49fe40b0
Caught exception in looking up space point ptr... 0x49fe40b0, 0x2cd2c4f0
Caught exception in looking up space point ptr... 0x1146f530, 0x2a5b0f90
Caught exception in looking up space point ptr... 0x2a5b0f90, 0x1145cf70
Caught exception in looking up space point ptr... 0x9487ad0, 0x9499d70
Caught exception in looking up space point ptr... 0x9499d70, 0x9499f00
Caught exception in looking up space point ptr... 0x35562310, 0x273912a0
Caught exception in looking up space point ptr... 0x273912a0, 0x3f49d080
Caught exception in looking up space point ptr... 0x18bed090, 0x37d13b50
Caught exception in looking up space point ptr... 0x5c850ad0, 0x18bed090
Caught exception in looking up space point ptr... 0x2bbbf580, 0x50bf9f70
Caught exception in looking up space point ptr... 0x50bf9f70, 0x2bbbf580
Caught exception in looking up space point ptr... 0x1803ea30, 0x1803ec10
Caught exception in looking up space point ptr... 0x1803ec10, 0x1803ea30
Caught exception in looking up space point ptr... 0x14bfb1b0, 0x2daf8880
Caught exception in looking up space point ptr... 0x2daf8880, 0x2daf89e0
Caught exception in looking up space point ptr... 0xeebf250, 0x5dd10de0
Caught exception in looking up space point ptr... 0x5dd10de0, 0xeebf250
Caught exception in looking up space point ptr... 0x5dc1c1a0, 0x3e6f9da0
Caught exception in looking up space point ptr... 0x3416fcf0, 0x5dc1c1a0
Caught exception in looking up space point ptr... 0x4a4e0450, 0x4e7c4110
Caught exception in looking up space point ptr... 0x4a4e05e0, 0x4a4e0450
Caught exception in looking up space point ptr... 0xe1f0c50, 0x2a2ff980
Caught exception in looking up space point ptr... 0xe1f0de0, 0xe1f0c50
Caught exception in looking up space point ptr... 0x2ccf2b90, 0x3eae6580
Caught exception in looking up space point ptr... 0x2ccf1460, 0x2ccf2b90
Caught exception in looking up space point ptr... 0x14b15310, 0x14b1cc60
Caught exception in looking up space point ptr... 0x14b1cc60, 0x51ab72c0
Caught exception in looking up space point ptr... 0x42636900, 0x3f5e46c0
Caught exception in looking up space point ptr... 0x42644870, 0x42636900
Caught exception in looking up space point ptr... 0x4250fc20, 0x424ff8c0
Caught exception in looking up space point ptr... 0x424ff8c0, 0x467aa200
Caught exception in looking up space point ptr... 0x45b17c20, 0x33d985a0
Caught exception in looking up space point ptr... 0x33d985a0, 0x45b17c20
Caught exception in looking up space point ptr... 0x3a8732f0, 0x3a872f10
Caught exception in looking up space point ptr... 0x3a872f10, 0x3a8732f0
Caught exception in looking up space point ptr... 0x3a8732f0, 0x3a873090
Caught exception in looking up space point ptr... 0x3a873090, 0x3a8732f0
Caught exception in looking up space point ptr... 0x339c2600, 0x339c2420
Caught exception in looking up space point ptr... 0x339c2420, 0x339c2920
++++>>>> total num hits: 303619, num free: 296713
04-Feb-2026 01:36:33 GMT  Closed output file "np02vd_raw_run042421_0079_df-s04-d1_dw_0_20260203T141345_reco_stage0_20260204T013633_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         97.8697       185.433       460.509       167.497       78.8509        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       9.2535e-05    0.000135907   0.000329261   0.000112111   4.95954e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                11.3131       13.6533       16.0513       13.8315       1.22873        27     
produce:triggerrawdecoder:PDVDTriggerReader4      0.526055      0.547267      0.585992      0.548465      0.0111749       27     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0176402     0.0181357     0.0196454     0.0180519    0.000389258      27     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0867654     0.0901262     0.106467      0.0884144    0.00499196       27     
produce:beamevent:BeamEvent                      0.000126018   0.000225841   0.00149135    0.000174429   0.000250258      27     
produce:pdvddaphne:DAPHNEReaderPDVD                3.54843       3.98143       4.52056       4.00112      0.220047        27     
produce:ophit:OpHitFinder                         0.0300452     0.0407997     0.052751      0.0415641     0.0059725       27     
produce:wclsdatavd:WireCellToolkit                 66.0634       82.4318       100.635       81.6991       9.21358        27     
produce:gaushit:GausHitFinder                     0.614657       1.97242       6.77339       1.33931       1.42075        27     
produce:nhitsfilter:NumberOfHitsFilter           0.000227389   0.000453267   0.000774713   0.000424761   0.000136979      27     
produce:reco3d:SpacePointSolver                    5.44077       13.3418       41.0486       11.3745       7.57997        27     
produce:hitpdune:DisambigFromSpacePoints          0.0662064     0.492337       2.93059      0.289539      0.619218        27     
produce:cluster3d:Cluster3D                        1.33394       62.7538       298.488       54.8279       67.1902        27     
[art]:TriggerResults:TriggerResultInserter       2.4456e-05    4.30731e-05   0.000106532   4.3021e-05    1.56475e-05      27     
end_path:out1:RootOutput                          6.583e-06    1.27903e-05   4.3683e-05    1.0771e-05    7.69189e-06      27     
end_path:out1:RootOutput(write)                    3.61825       6.06648       11.508        5.75984       1.71158        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6973.01 MB
  Peak resident set size usage (VmHWM): 4770.6 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042421_0079_df-s04-d1_dw_0_20260203T141345_reco_stage0_20260204T013633_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042421_0079_df-s04-d1_dw_0_20260203T141345_reco_stage0_20260204T013633_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042421_0079_df-s04-d1_dw_0_20260203T141345_reco_stage0_20260204T013633_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770163911.0,
    "core.end_time": 1770168995.0,
    "core.events": [
      12863,
      12869,
      12875,
      12881,
      12887,
      12893,
      12899,
      12905,
      12911,
      12917,
      12923,
      12929,
      12935,
      12941,
      12947,
      12953,
      12959,
      12965,
      12971,
      12977,
      12983,
      12989,
      12995,
      13001,
      13007,
      13013,
      13019
    ],
    "core.event_count": 27,
    "core.first_event_number": 12863,
    "core.last_event_number": 13019,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42421
    ],
    "core.runs_subruns": [
      4242100001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042421_0079_df-s04-d1_dw_0_20260203T141345.hdf5"
    }
  ]
}
justIN time: 2026-02-04 04:31:43 UTC       justIN version: 01.06.00