justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297923.52@dunegpschedd02.fnal.gov

Jobsub ID297923.52@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 23:00:26
SiteUK_RAL-PPD
EntryCMSHTPC_T2_UK_SGrid_RALPP_hep206
Last heartbeat2026-02-04 01:06:49
From worker nodeHostnameheplnc153.pp.rl.ac.uk
cpuinfoAMD EPYC 7763 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:59:28
Input filesvd-protodune:np02vd_raw_run042433_0564_df-s02-d1_dw_0_20260203T221628.hdf5
JobscriptExit code0
Real time1h (3884s)
CPU time58m (3537s = 91%)
Max RSS bytes3387043840 (3230 MiB)
Outputting started2026-02-04 01:04:13
Output files
Finished2026-02-04 01:06:49
Saved logsjustin-logs:297923.52-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

point ptr... 0x61a6a2d0, 0x61a73410
Caught exception in looking up space point ptr... 0x29c346b0, 0x29c35b90
Caught exception in looking up space point ptr... 0x29c35b90, 0x29c31550
Caught exception in looking up space point ptr... 0x46a622d0, 0x46a62180
Caught exception in looking up space point ptr... 0x46a62180, 0x46a622d0
Caught exception in looking up space point ptr... 0x18f75ba0, 0x62732220
Caught exception in looking up space point ptr... 0x62732220, 0x18f75ba0
Caught exception in looking up space point ptr... 0x18f75ba0, 0xdae5320
Caught exception in looking up space point ptr... 0xdae5320, 0x18f75ba0
Caught exception in looking up space point ptr... 0x3e111e80, 0x3e1102f0
Caught exception in looking up space point ptr... 0x3e111d90, 0x3e111e80
Caught exception in looking up space point ptr... 0x7cfc9170, 0x7cfc9300
Caught exception in looking up space point ptr... 0x7cfc9300, 0x7cfc9170
Caught exception in looking up space point ptr... 0x79a731d0, 0x79a73560
Caught exception in looking up space point ptr... 0x79a73560, 0x79a731d0
Caught exception in looking up space point ptr... 0x7cf87fd0, 0x7cf89190
Caught exception in looking up space point ptr... 0x7cf89190, 0x7cf87fd0
Caught exception in looking up space point ptr... 0x3a8c2e60, 0x3b08e900
Caught exception in looking up space point ptr... 0x3b08e900, 0x3a8c2e60
Caught exception in looking up space point ptr... 0x8bccdc0, 0x8bccc30
Caught exception in looking up space point ptr... 0x8bccc30, 0x8bccf50
Caught exception in looking up space point ptr... 0x17065170, 0x17064fe0
Caught exception in looking up space point ptr... 0x17064fe0, 0x17065170
Caught exception in looking up space point ptr... 0x594448d0, 0x457e9340
Caught exception in looking up space point ptr... 0x457e9340, 0x594448d0
Caught exception in looking up space point ptr... 0x3b100b20, 0x3b100990
Caught exception in looking up space point ptr... 0x3b100990, 0x3b100b20
Caught exception in looking up space point ptr... 0x457df7b0, 0x59460d90
Caught exception in looking up space point ptr... 0x59460d90, 0x457df7b0
Caught exception in looking up space point ptr... 0x457df7b0, 0x594609f0
Caught exception in looking up space point ptr... 0x594609f0, 0x457df7b0
Caught exception in looking up space point ptr... 0x457df7b0, 0x457e9930
Caught exception in looking up space point ptr... 0x457e9930, 0x457df7b0
Caught exception in looking up space point ptr... 0x457df7b0, 0x62e9fa10
Caught exception in looking up space point ptr... 0x62e9fa10, 0x457df7b0
Caught exception in looking up space point ptr... 0x61a95e40, 0x19922250
Caught exception in looking up space point ptr... 0x7bc213d0, 0x61a95e40
Caught exception in looking up space point ptr... 0x79a73d80, 0x79a6f410
Caught exception in looking up space point ptr... 0x79a6f410, 0x79a6f970
Caught exception in looking up space point ptr... 0x2975d970, 0x2975ff70
Caught exception in looking up space point ptr... 0x2975ff70, 0x2975d970
Caught exception in looking up space point ptr... 0x7cf62720, 0x7cf62590
Caught exception in looking up space point ptr... 0x7cf62590, 0x7cf62720
Caught exception in looking up space point ptr... 0x190a1150, 0x585d7e60
Caught exception in looking up space point ptr... 0x585d7e60, 0x374d7ba0
Caught exception in looking up space point ptr... 0x457a9730, 0x457a5f70
Caught exception in looking up space point ptr... 0x457a5f70, 0x457a9730
Caught exception in looking up space point ptr... 0x895eb20, 0x895e300
Caught exception in looking up space point ptr... 0x895e300, 0x895e490
Caught exception in looking up space point ptr... 0x508e1840, 0x508deed0
Caught exception in looking up space point ptr... 0x5090bd40, 0x508e1840
Caught exception in looking up space point ptr... 0x61424cd0, 0x6142a890
Caught exception in looking up space point ptr... 0x6142a890, 0x6142cd40
Caught exception in looking up space point ptr... 0xdda80d0, 0x42202ac0
Caught exception in looking up space point ptr... 0x42202ac0, 0xdda80d0
Caught exception in looking up space point ptr... 0x61ca91f0, 0x61cafc90
Caught exception in looking up space point ptr... 0x61cafc90, 0x61ca91f0
Caught exception in looking up space point ptr... 0x79a79080, 0x79a79c20
Caught exception in looking up space point ptr... 0x79a79c20, 0x79a79080
Caught exception in looking up space point ptr... 0x79a79080, 0x79a78360
Caught exception in looking up space point ptr... 0x79a78360, 0x79a79080
Caught exception in looking up space point ptr... 0x61b8b1d0, 0x61b8b0a0
Caught exception in looking up space point ptr... 0x61b8b0a0, 0x61b8ae40
Caught exception in looking up space point ptr... 0x2975ab10, 0x8d671d0
Caught exception in looking up space point ptr... 0x8d671d0, 0x2975ab10
Caught exception in looking up space point ptr... 0x36695a10, 0x457984e0
Caught exception in looking up space point ptr... 0x457984e0, 0x36695a10
++++>>>> total num hits: 141879, num free: 136107
04-Feb-2026 01:04:04 GMT  Closed output file "np02vd_raw_run042433_0564_df-s02-d1_dw_0_20260203T221628_reco_stage0_20260204T010404_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         176.829       199.828       223.024       200.012       12.5789        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       0.00010141    0.000158081   0.00021909    0.000172453   4.71123e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                14.7044       15.6978       17.6207       15.5448      0.620216        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.513665      0.677122       2.40366      0.570915      0.408974        19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0177768     0.0198599     0.0319014     0.0181929    0.00403396       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0176633     0.018825      0.0294105     0.0181606    0.00253075       19     
produce:beamevent:BeamEvent                      0.000135393   0.000237374   0.00109062    0.000194103   0.000203875      19     
produce:pdvddaphne:DAPHNEReaderPDVD                3.40592       3.77875       4.19593       3.81718      0.245353        19     
produce:ophit:OpHitFinder                         0.0369948     0.0454897     0.0524283     0.0467691    0.00418152       19     
produce:wclsdatavd:WireCellToolkit                 118.605       138.659       151.744       142.248       9.62061        19     
produce:gaushit:GausHitFinder                      1.16145       1.8454        2.79141       1.88771      0.461345        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000377066   0.000565863   0.00185268    0.000487072   0.000311319      19     
produce:reco3d:SpacePointSolver                    20.8713       28.6748       37.9739       28.6452       5.58953        19     
produce:hitpdune:DisambigFromSpacePoints          0.164523      0.272449      0.383835      0.290143      0.087918        19     
produce:cluster3d:Cluster3D                        2.47351       3.69779       5.72687       3.62646      0.989691        19     
[art]:TriggerResults:TriggerResultInserter       2.7892e-05    4.17645e-05   0.000151023   3.4284e-05    2.66296e-05      19     
end_path:out1:RootOutput                          6.933e-06    1.35997e-05   4.5025e-05    1.2023e-05    8.21201e-06      19     
end_path:out1:RootOutput(write)                    5.03707       6.37558       8.22328       6.08581       1.03227        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5408.19 MB
  Peak resident set size usage (VmHWM): 3387.04 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0564_df-s02-d1_dw_0_20260203T221628_reco_stage0_20260204T010404_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0564_df-s02-d1_dw_0_20260203T221628_reco_stage0_20260204T010404_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0564_df-s02-d1_dw_0_20260203T221628_reco_stage0_20260204T010404_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770163183.0,
    "core.end_time": 1770167046.0,
    "core.events": [
      64303,
      64309,
      64315,
      64321,
      64327,
      64333,
      64339,
      64345,
      64351,
      64357,
      64363,
      64369,
      64375,
      64381,
      64387,
      64393,
      64399,
      64405,
      64411
    ],
    "core.event_count": 19,
    "core.first_event_number": 64303,
    "core.last_event_number": 64411,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0564_df-s02-d1_dw_0_20260203T221628.hdf5"
    }
  ]
}
justIN time: 2026-02-04 09:14:27 UTC       justIN version: 01.06.00