justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303654.21@dunegpschedd01.fnal.gov

Jobsub ID303654.21@dunegpschedd01.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:52:26
SiteUK_Manchester
EntryUBoone_T2_UK_Manchester_ce01
Last heartbeat2026-02-04 02:59:16
From worker nodeHostnamewn2208251.tier2.hep.manchester.ac.uk
cpuinfoAMD EPYC 7513 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:50:42
Input filesvd-protodune:np02vd_raw_run042433_0012_df-s04-d1_dw_0_20260203T184641.hdf5
JobscriptExit code0
Real time3h (11110s)
CPU time1h (5183s = 46%)
Max RSS bytes3568766976 (3403 MiB)
Outputting started2026-02-04 02:55:58
Output files
Finished2026-02-04 02:59:16
Saved logsjustin-logs:303654.21-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ing up space point ptr... 0x724f9fc0, 0x724f9e70
Caught exception in looking up space point ptr... 0x6853d890, 0x90adc20
Caught exception in looking up space point ptr... 0x90adc20, 0x6852f7f0
Caught exception in looking up space point ptr... 0x14330b00, 0x38b567a0
Caught exception in looking up space point ptr... 0x7560de40, 0x14330b00
Caught exception in looking up space point ptr... 0x5fa1fb10, 0x87af49c0
Caught exception in looking up space point ptr... 0x87af49c0, 0x87af0be0
Caught exception in looking up space point ptr... 0x4a215c40, 0x436b9ef0
Caught exception in looking up space point ptr... 0x436b9ef0, 0x436ba210
Caught exception in looking up space point ptr... 0x436ba210, 0x436b9c40
Caught exception in looking up space point ptr... 0x436ba080, 0x4a215c40
Caught exception in looking up space point ptr... 0x463925d0, 0x46392760
Caught exception in looking up space point ptr... 0x463801a0, 0x463925d0
Caught exception in looking up space point ptr... 0x643294d0, 0x162d2220
Caught exception in looking up space point ptr... 0x6430d3c0, 0x643294d0
Caught exception in looking up space point ptr... 0x724fa630, 0x6927cdf0
Caught exception in looking up space point ptr... 0x6927cdf0, 0x724fa630
Caught exception in looking up space point ptr... 0x797518c0, 0x79751aa0
Caught exception in looking up space point ptr... 0x79751aa0, 0x797518c0
Caught exception in looking up space point ptr... 0x38e71950, 0x81b1dbd0
Caught exception in looking up space point ptr... 0x81b1dbd0, 0x38e71950
Caught exception in looking up space point ptr... 0x81b41660, 0x19347080
Caught exception in looking up space point ptr... 0x19347080, 0x13643f60
Caught exception in looking up space point ptr... 0x17171210, 0x1715ede0
Caught exception in looking up space point ptr... 0x1715ede0, 0x1714c820
Caught exception in looking up space point ptr... 0x61a70900, 0x61a77700
Caught exception in looking up space point ptr... 0x61a77700, 0x61a737e0
Caught exception in looking up space point ptr... 0x34673170, 0x392de010
Caught exception in looking up space point ptr... 0x34673300, 0x34673170
Caught exception in looking up space point ptr... 0x3ea4f7f0, 0x3ea4f660
Caught exception in looking up space point ptr... 0x3ea4f660, 0x316530f0
Caught exception in looking up space point ptr... 0x15f9f850, 0x6597cc40
Caught exception in looking up space point ptr... 0x15f9f950, 0x15f9f850
Caught exception in looking up space point ptr... 0x2b12e740, 0x2b11d480
Caught exception in looking up space point ptr... 0x2b12e8d0, 0x2b12e740
Caught exception in looking up space point ptr... 0x35f1ff60, 0x35f1fdd0
Caught exception in looking up space point ptr... 0x35f1fdd0, 0x48719360
Caught exception in looking up space point ptr... 0x46ca0030, 0x46c8dd90
Caught exception in looking up space point ptr... 0x46c8dd90, 0x46c7b640
Caught exception in looking up space point ptr... 0x6bc2b280, 0x6bc3c710
Caught exception in looking up space point ptr... 0x6bc3c710, 0x6bc4a680
Caught exception in looking up space point ptr... 0x192a6cc0, 0x192b6320
Caught exception in looking up space point ptr... 0x192b6320, 0x192b64c0
Caught exception in looking up space point ptr... 0x7c3018b0, 0x7c301720
Caught exception in looking up space point ptr... 0x7c301720, 0x11930e40
Caught exception in looking up space point ptr... 0x7e1b3060, 0x7e1b33d0
Caught exception in looking up space point ptr... 0x7e1b33d0, 0x7e1b3060
Caught exception in looking up space point ptr... 0x142aa5e0, 0x1429bd50
Caught exception in looking up space point ptr... 0x1429bd50, 0x142aa2c0
Caught exception in looking up space point ptr... 0x2f21c130, 0x2f21c310
Caught exception in looking up space point ptr... 0x2f21c310, 0x2f21c130
Caught exception in looking up space point ptr... 0xbeda840, 0x11972ef0
Caught exception in looking up space point ptr... 0x11972ef0, 0xbeda840
Caught exception in looking up space point ptr... 0x2f165d20, 0x2f165e20
Caught exception in looking up space point ptr... 0x2f165e20, 0x2f165f20
Caught exception in looking up space point ptr... 0x38e735d0, 0x38e733f0
Caught exception in looking up space point ptr... 0x38e733f0, 0x38e735d0
Caught exception in looking up space point ptr... 0x38e735d0, 0x38e73760
Caught exception in looking up space point ptr... 0x38e73760, 0x38e735d0
Caught exception in looking up space point ptr... 0x38e735d0, 0x38e730a0
Caught exception in looking up space point ptr... 0x38e730a0, 0x38e735d0
Caught exception in looking up space point ptr... 0x3040b000, 0x6a3d22a0
Caught exception in looking up space point ptr... 0x6a3d22a0, 0xbedf340
Caught exception in looking up space point ptr... 0x2f41f440, 0x1855d3f0
Caught exception in looking up space point ptr... 0x1855d3f0, 0x2f41f440
Caught exception in looking up space point ptr... 0x42f32900, 0x42f71cc0
Caught exception in looking up space point ptr... 0x42f71cc0, 0x5f93d810
++++>>>> total num hits: 568508, num free: 558646
04-Feb-2026 02:55:43 GMT  Closed output file "np02vd_raw_run042433_0012_df-s04-d1_dw_0_20260203T184641_reco_stage0_20260204T025543_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         457.344       579.146       741.27        584.287       62.6151        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                        8.504e-05    0.000154307   0.000304482   0.00012246    6.30671e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                233.14        291.635       421.883       280.53        52.4249        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.826949       1.15677       1.80551       1.07191      0.258634        19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0220726     0.0310872     0.075681      0.0262182     0.013599        19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0219757     0.0224601     0.0234024     0.0224614    0.000321838      19     
produce:beamevent:BeamEvent                      0.000138771   0.000244406   0.00103232    0.000187192   0.000191977      19     
produce:pdvddaphne:DAPHNEReaderPDVD                15.5623       23.373        31.0214       22.649        3.93728        19     
produce:ophit:OpHitFinder                         0.0349563     0.0475446     0.061708      0.0472231    0.00651007       19     
produce:wclsdatavd:WireCellToolkit                 117.625       157.01        176.725       158.408       13.4722        19     
produce:gaushit:GausHitFinder                      1.27487       2.84435       3.93747       2.85501      0.620809        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000391816   0.000543954   0.000951708   0.000506592   0.000136728      19     
produce:reco3d:SpacePointSolver                    17.506        26.2248       36.1264       24.2481       5.00215        19     
produce:hitpdune:DisambigFromSpacePoints          0.200821      0.411681      0.607185      0.417779      0.0989182       19     
produce:cluster3d:Cluster3D                        3.04458       67.0843       82.9055       77.5775       22.9862        19     
[art]:TriggerResults:TriggerResultInserter       2.7021e-05    4.70773e-05   0.000142538   4.1198e-05    2.48793e-05      19     
end_path:out1:RootOutput                          5.491e-06    1.09209e-05    3.773e-05     9.427e-06    6.95106e-06      19     
end_path:out1:RootOutput(write)                    7.19023       9.24863       10.9857       9.5524        1.02649        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5547.76 MB
  Peak resident set size usage (VmHWM): 3568.77 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0012_df-s04-d1_dw_0_20260203T184641_reco_stage0_20260204T025543_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0012_df-s04-d1_dw_0_20260203T184641_reco_stage0_20260204T025543_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0012_df-s04-d1_dw_0_20260203T184641_reco_stage0_20260204T025543_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770162653.0,
    "core.end_time": 1770173745.0,
    "core.events": [
      1379,
      1385,
      1391,
      1397,
      1403,
      1409,
      1415,
      1421,
      1427,
      1433,
      1439,
      1445,
      1451,
      1457,
      1463,
      1469,
      1475,
      1481,
      1487
    ],
    "core.event_count": 19,
    "core.first_event_number": 1379,
    "core.last_event_number": 1487,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0012_df-s04-d1_dw_0_20260203T184641.hdf5"
    }
  ]
}
justIN time: 2026-02-04 04:28:03 UTC       justIN version: 01.06.00