justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303730.8@dunegpschedd01.fnal.gov

Jobsub ID303730.8@dunegpschedd01.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-04 01:20:32
SiteUK_Manchester
EntryUBoone_T2_UK_Manchester_ce02
Last heartbeat2026-02-04 05:32:33
From worker nodeHostnamewn5917301.tier2.hep.manchester.ac.uk
cpuinfoAMD EPYC 7702P 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-04 01:21:28
Input filesvd-protodune:np02vd_raw_run042433_0627_df-s04-d1_dw_0_20260203T224026.hdf5
JobscriptExit code0
Real time4h (14775s)
CPU time2h (9769s = 66%)
Max RSS bytes8492359680 (8098 MiB)
Outputting started2026-02-04 05:27:55
Output files
Finished2026-02-04 05:32:33
Saved logsjustin-logs:303730.8-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

. 0x7e789eb0, 0x7e78a040
Caught exception in looking up space point ptr... 0x5fef5e20, 0x5ff08570
Caught exception in looking up space point ptr... 0x5fee6d80, 0x5fef5e20
Caught exception in looking up space point ptr... 0x816d9b40, 0x816d98f0
Caught exception in looking up space point ptr... 0x816d98f0, 0x816e3710
Caught exception in looking up space point ptr... 0x8a61600, 0x17cc0a90
Caught exception in looking up space point ptr... 0x17cc0a90, 0x8a623e0
Caught exception in looking up space point ptr... 0x4ed224a0, 0x4ed22250
Caught exception in looking up space point ptr... 0x4ed22250, 0x4ed25530
Caught exception in looking up space point ptr... 0x579d9d20, 0x96f06310
Caught exception in looking up space point ptr... 0x96f06310, 0x5790dc90
Caught exception in looking up space point ptr... 0x17eb1080, 0x17eb0fa0
Caught exception in looking up space point ptr... 0x17eb0fa0, 0xa95b6a0
Caught exception in looking up space point ptr... 0x972ea2a0, 0x972ea010
Caught exception in looking up space point ptr... 0x972ea010, 0x972f35f0
Caught exception in looking up space point ptr... 0x10aeab940, 0x32004df0
Caught exception in looking up space point ptr... 0x32004df0, 0x16744930
Caught exception in looking up space point ptr... 0x5ac8d790, 0x33c856c0
Caught exception in looking up space point ptr... 0x17f3ea60, 0x5ac8d790
Caught exception in looking up space point ptr... 0x9c2f2730, 0x33746770
Caught exception in looking up space point ptr... 0x33746770, 0x33746130
Caught exception in looking up space point ptr... 0x3e9b0bc0, 0x2a997a70
Caught exception in looking up space point ptr... 0x2a997a70, 0x370f1ac0
Caught exception in looking up space point ptr... 0x46d79af0, 0x37651c00
Caught exception in looking up space point ptr... 0x37651c00, 0x4bdb57c0
Caught exception in looking up space point ptr... 0x42fc4490, 0x42fc4940
Caught exception in looking up space point ptr... 0x42fc4940, 0x42fc4490
Caught exception in looking up space point ptr... 0x96f13210, 0x58beb5d0
Caught exception in looking up space point ptr... 0x58beb5d0, 0x16cc9a60
Caught exception in looking up space point ptr... 0x87e12810, 0x42fbc710
Caught exception in looking up space point ptr... 0x42fbc710, 0x42fbc270
Caught exception in looking up space point ptr... 0x8c642f70, 0x8c6314a0
Caught exception in looking up space point ptr... 0x8c6314a0, 0x8c61fb60
Caught exception in looking up space point ptr... 0x6816ea20, 0x3679b7c0
Caught exception in looking up space point ptr... 0x6816ebb0, 0x6816ea20
Caught exception in looking up space point ptr... 0xdbe67880, 0xdbe676f0
Caught exception in looking up space point ptr... 0xdbe676f0, 0xdbe50f10
Caught exception in looking up space point ptr... 0x1a6cff90, 0x1a6cfc70
Caught exception in looking up space point ptr... 0x1a6cfc70, 0x1a69dfa0
Caught exception in looking up space point ptr... 0xbc939580, 0xbc939710
Caught exception in looking up space point ptr... 0xea19b5f0, 0xbc939580
Caught exception in looking up space point ptr... 0x99797300, 0x96f48d10
Caught exception in looking up space point ptr... 0x997826c0, 0x99797300
Caught exception in looking up space point ptr... 0x35189e90, 0x35177a10
Caught exception in looking up space point ptr... 0x35177a10, 0x35165900
Caught exception in looking up space point ptr... 0x37843fe0, 0xd8506b90
Caught exception in looking up space point ptr... 0xd8506b90, 0x37843fe0
Caught exception in looking up space point ptr... 0x609c5160, 0x1332eb70
Caught exception in looking up space point ptr... 0x1332eb70, 0x680dcb00
Caught exception in looking up space point ptr... 0xc45b5a10, 0xc45b5bf0
Caught exception in looking up space point ptr... 0xc45b5bf0, 0xc45b5f60
Caught exception in looking up space point ptr... 0x3e96b4f0, 0x17e4d250
Caught exception in looking up space point ptr... 0x17e4d250, 0x3e96b4f0
Caught exception in looking up space point ptr... 0x9966b5b0, 0x9966bf10
Caught exception in looking up space point ptr... 0x9966bf10, 0x9966b5b0
Caught exception in looking up space point ptr... 0x1459a690, 0x972469d0
Caught exception in looking up space point ptr... 0x972469d0, 0x97246bb0
Caught exception in looking up space point ptr... 0xa718d460, 0xa2c41480
Caught exception in looking up space point ptr... 0xa2c41480, 0x167433c0
Caught exception in looking up space point ptr... 0x40061100, 0x40061200
Caught exception in looking up space point ptr... 0x40061200, 0x40061100
Caught exception in looking up space point ptr... 0x7e651ad0, 0x7e651c70
Caught exception in looking up space point ptr... 0x7e651c70, 0x7e651ad0
Caught exception in looking up space point ptr... 0xbad24c60, 0xbad24f60
Caught exception in looking up space point ptr... 0xbad24f60, 0xbad24c60
Caught exception in looking up space point ptr... 0x6ec38c40, 0x6ec39030
Caught exception in looking up space point ptr... 0x6ec39030, 0x6ec38c40
++++>>>> total num hits: 528721, num free: 519277
04-Feb-2026 05:27:29 GMT  Closed output file "np02vd_raw_run042433_0627_df-s04-d1_dw_0_20260203T224026_reco_stage0_20260204T052729_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         413.896       769.169       1751.36       633.367       343.169        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       9.9959e-05    0.00012246    0.000303443   0.000105429   4.49685e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                154.087       238.332       307.898       240.455       44.8022        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.842131       1.14871       1.71042       1.12025      0.223137        19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0222189     0.0286461     0.075738      0.0261266     0.0115873       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0217186     0.0225216     0.023012      0.0225838    0.000301763      19     
produce:beamevent:BeamEvent                      8.1904e-05    0.000199185   0.00131164     0.0001431    0.000263435      19     
produce:pdvddaphne:DAPHNEReaderPDVD                14.9408       21.2141       26.0324       21.0167       2.87828        19     
produce:ophit:OpHitFinder                         0.0377976     0.0520835     0.0642487     0.0534049    0.00651545       19     
produce:wclsdatavd:WireCellToolkit                 163.93        203.209       221.499       205.654       14.5093        19     
produce:gaushit:GausHitFinder                      2.51956       4.65884       9.13534       3.92666       2.00104        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000413891   0.000734769   0.00126624    0.000600743   0.000283277      19     
produce:reco3d:SpacePointSolver                    14.8084       35.1007       91.8262       26.332        20.396         19     
produce:hitpdune:DisambigFromSpacePoints          0.348614       1.18274       3.4607       0.551379       1.07355        19     
produce:cluster3d:Cluster3D                        5.21913       249.759       1140.19       101.647       299.52         19     
[art]:TriggerResults:TriggerResultInserter       2.3424e-05    4.59397e-05   0.000126879   3.8523e-05    2.4694e-05       19     
end_path:out1:RootOutput                          4.018e-06    8.89574e-06   3.4235e-05       5e-06      8.70661e-06      19     
end_path:out1:RootOutput(write)                    8.57154       14.3833       29.6473       11.793        5.69682        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 10533.8 MB
  Peak resident set size usage (VmHWM): 8492.36 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0627_df-s04-d1_dw_0_20260203T224026_reco_stage0_20260204T052729_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0627_df-s04-d1_dw_0_20260203T224026_reco_stage0_20260204T052729_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0627_df-s04-d1_dw_0_20260203T224026_reco_stage0_20260204T052729_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770168106.0,
    "core.end_time": 1770182855.0,
    "core.events": [
      71489,
      71495,
      71501,
      71507,
      71513,
      71519,
      71525,
      71531,
      71537,
      71543,
      71549,
      71555,
      71561,
      71567,
      71573,
      71579,
      71585,
      71591,
      71597
    ],
    "core.event_count": 19,
    "core.first_event_number": 71489,
    "core.last_event_number": 71597,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0627_df-s04-d1_dw_0_20260203T224026.hdf5"
    }
  ]
}
justIN time: 2026-02-04 07:04:25 UTC       justIN version: 01.06.00