justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303718.4@dunegpschedd01.fnal.gov

Jobsub ID303718.4@dunegpschedd01.fnal.gov
Workflow ID12717
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-04 00:52:28
SiteUK_Manchester
EntryUBoone_T2_UK_Manchester_ce01
Last heartbeat2026-02-04 03:03:24
From worker nodeHostnamewn2208251.tier2.hep.manchester.ac.uk
cpuinfoAMD EPYC 7513 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-04 00:53:03
Input filesvd-protodune:np02vd_raw_run042421_0053_df-s02-d1_dw_0_20260203T135939.hdf5
JobscriptExit code0
Real time2h (7656s)
CPU time49m (2976s = 38%)
Max RSS bytes2563706880 (2444 MiB)
Outputting started2026-02-04 03:00:41
Output files
Finished2026-02-04 03:03:24
Saved logsjustin-logs:303718.4-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

nt ptr... 0x552e2d30, 0x552e2900
Caught exception in looking up space point ptr... 0x552e2900, 0x552e2d30
Caught exception in looking up space point ptr... 0x124d0ab0, 0x32bb93d0
Caught exception in looking up space point ptr... 0x32bb93d0, 0x32bb9240
Caught exception in looking up space point ptr... 0x308d44b0, 0x2b7ec5f0
Caught exception in looking up space point ptr... 0x2b7ec5f0, 0x308d44b0
Caught exception in looking up space point ptr... 0x60ffdd70, 0x60fff5d0
Caught exception in looking up space point ptr... 0x60fff5d0, 0x61002ed0
Caught exception in looking up space point ptr... 0x14caed40, 0x14cb3430
Caught exception in looking up space point ptr... 0x14cb3430, 0x14cb2d10
Caught exception in looking up space point ptr... 0x56101870, 0x561017b0
Caught exception in looking up space point ptr... 0x561017b0, 0x598409d0
Caught exception in looking up space point ptr... 0x2f29eff0, 0x2f2a3ab0
Caught exception in looking up space point ptr... 0x2f2a3ab0, 0x2f2a5a70
Caught exception in looking up space point ptr... 0x2f2a5a70, 0x2f2a6400
Caught exception in looking up space point ptr... 0x29f51200, 0x29f51570
Caught exception in looking up space point ptr... 0x29f51570, 0x29f51200
Caught exception in looking up space point ptr... 0x151d2060, 0x153ee580
Caught exception in looking up space point ptr... 0x153ee6d0, 0x151d2060
Caught exception in looking up space point ptr... 0x4a4b7980, 0x4a4b5520
Caught exception in looking up space point ptr... 0x2741e4c0, 0x4a4b7980
Caught exception in looking up space point ptr... 0x2cc63860, 0x2cc63420
Caught exception in looking up space point ptr... 0x2cc63420, 0x2cc62670
Caught exception in looking up space point ptr... 0x532748a0, 0x53274b90
Caught exception in looking up space point ptr... 0x53274b90, 0x532748a0
Caught exception in looking up space point ptr... 0x456ffd40, 0x56108270
Caught exception in looking up space point ptr... 0x56108270, 0x45704b20
Caught exception in looking up space point ptr... 0x39027bf0, 0x3910e220
Caught exception in looking up space point ptr... 0x3910e220, 0x636c2df0
Caught exception in looking up space point ptr... 0x636c2df0, 0x3910e220
Caught exception in looking up space point ptr... 0x3910e220, 0x39115e50
Caught exception in looking up space point ptr... 0x46b403c0, 0x59871ce0
Caught exception in looking up space point ptr... 0x59871ce0, 0x46b403c0
Caught exception in looking up space point ptr... 0x3466bb70, 0x421dc400
Caught exception in looking up space point ptr... 0x421dc400, 0x3466bb70
Caught exception in looking up space point ptr... 0x3b8564a0, 0x5f888f20
Caught exception in looking up space point ptr... 0x5f888f20, 0x5f8873a0
Caught exception in looking up space point ptr... 0x60f57250, 0x60f56f00
Caught exception in looking up space point ptr... 0x60f56f00, 0x60f57250
Caught exception in looking up space point ptr... 0x9dab610, 0x12fc9660
Caught exception in looking up space point ptr... 0x12fc9660, 0x9dab610
Caught exception in looking up space point ptr... 0x2eb016c0, 0x2eb019e0
Caught exception in looking up space point ptr... 0x2eb019e0, 0x591b5710
Caught exception in looking up space point ptr... 0x2eb00d40, 0x2eb016c0
Caught exception in looking up space point ptr... 0xcd00410, 0xccf97a0
Caught exception in looking up space point ptr... 0xccf97a0, 0xcd00410
Caught exception in looking up space point ptr... 0x560f8340, 0x560fccf0
Caught exception in looking up space point ptr... 0x560fccf0, 0x560f89e0
Caught exception in looking up space point ptr... 0x58f89a70, 0x50548bc0
Caught exception in looking up space point ptr... 0x308aa180, 0x58f89a70
Caught exception in looking up space point ptr... 0x34a79610, 0x5b3cf790
Caught exception in looking up space point ptr... 0x5b3cf790, 0x34a79610
Caught exception in looking up space point ptr... 0x48a7d360, 0x2abd06b0
Caught exception in looking up space point ptr... 0x2abd06b0, 0x48a7d360
Caught exception in looking up space point ptr... 0xcd02580, 0xcd02260
Caught exception in looking up space point ptr... 0xcd02260, 0xcd023f0
Caught exception in looking up space point ptr... 0x308c4cd0, 0x38ec5c40
Caught exception in looking up space point ptr... 0x38ec5c40, 0x38ec59b0
Caught exception in looking up space point ptr... 0x12b6b880, 0x25b71df0
Caught exception in looking up space point ptr... 0x25b71df0, 0x12b63ca0
Caught exception in looking up space point ptr... 0x3e00e840, 0x570b34d0
Caught exception in looking up space point ptr... 0x570b34d0, 0x3e00e840
Caught exception in looking up space point ptr... 0x38b50110, 0x38b50370
Caught exception in looking up space point ptr... 0x38b50370, 0x38b50110
Caught exception in looking up space point ptr... 0x4a4b7d00, 0x4a4b7e00
Caught exception in looking up space point ptr... 0x4a4b7e00, 0x4a4b7b00
++++>>>> total num hits: 59795, num free: 57563
04-Feb-2026 03:00:32 GMT  Closed output file "np02vd_raw_run042421_0053_df-s02-d1_dw_0_20260203T135939_reco_stage0_20260204T030032_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         226.754       278.902       326.674       277.178       25.6323        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       9.2363e-05    0.000163001   0.000366439   0.000121438   7.65922e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                114.101       158.325       191.704       157.569       20.6848        27     
produce:triggerrawdecoder:PDVDTriggerReader4      0.723562      0.917841       1.14392      0.900367      0.125589        27     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0220595     0.0250311     0.0604953     0.0228911    0.00727325       27     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.108965      0.117296      0.136599      0.111392      0.0102199       27     
produce:beamevent:BeamEvent                      0.000106951   0.000266878   0.00150149    0.000202811   0.000252628      27     
produce:pdvddaphne:DAPHNEReaderPDVD                14.1269       19.4273       24.9127       19.8697       2.89619        27     
produce:ophit:OpHitFinder                         0.0331021     0.044265      0.0570456     0.0440552     0.0052822       27     
produce:wclsdatavd:WireCellToolkit                 65.9603       80.4641       92.7584       81.668        6.63706        27     
produce:gaushit:GausHitFinder                     0.455813       1.21428       2.1509        1.18823      0.357803        27     
produce:nhitsfilter:NumberOfHitsFilter           0.000211568   0.000326354   0.000663337   0.000317656   9.05332e-05      27     
produce:reco3d:SpacePointSolver                    3.83837       10.3014       18.884        10.0361       2.97328        27     
produce:hitpdune:DisambigFromSpacePoints          0.0464446     0.136509      0.564626      0.124487       0.09008        27     
produce:cluster3d:Cluster3D                       0.639168       2.89131       13.0377       2.31278       2.20977        27     
[art]:TriggerResults:TriggerResultInserter       3.1219e-05    5.25527e-05   0.000129664   4.7218e-05    2.07165e-05      27     
end_path:out1:RootOutput                          4.008e-06    8.99022e-06   3.5838e-05     6.723e-06    6.4633e-06       27     
end_path:out1:RootOutput(write)                    3.48229       4.99878       6.33931       5.16446      0.600037        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4521.35 MB
  Peak resident set size usage (VmHWM): 2563.71 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042421_0053_df-s02-d1_dw_0_20260203T135939_reco_stage0_20260204T030032_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042421_0053_df-s02-d1_dw_0_20260203T135939_reco_stage0_20260204T030032_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042421_0053_df-s02-d1_dw_0_20260203T135939_reco_stage0_20260204T030032_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770166399.0,
    "core.end_time": 1770174033.0,
    "core.events": [
      8635,
      8641,
      8647,
      8653,
      8659,
      8665,
      8671,
      8677,
      8683,
      8689,
      8695,
      8701,
      8707,
      8713,
      8719,
      8725,
      8731,
      8737,
      8743,
      8749,
      8755,
      8761,
      8767,
      8773,
      8779,
      8785,
      8791
    ],
    "core.event_count": 27,
    "core.first_event_number": 8635,
    "core.last_event_number": 8791,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42421
    ],
    "core.runs_subruns": [
      4242100001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042421_0053_df-s02-d1_dw_0_20260203T135939.hdf5"
    }
  ]
}
justIN time: 2026-02-04 04:33:50 UTC       justIN version: 01.06.00