justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303789.3@dunegpschedd01.fnal.gov

Jobsub ID303789.3@dunegpschedd01.fnal.gov
Workflow ID12717
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-04 02:40:34
SiteUK_Manchester
EntryUBoone_T2_UK_Manchester_ce01
Last heartbeat2026-02-04 05:06:08
From worker nodeHostnamewn1208080.tier2.hep.manchester.ac.uk
cpuinfoAMD EPYC 7702P 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-04 02:40:57
Input filesvd-protodune:np02vd_raw_run042421_0068_df-s02-d0_dw_0_20260203T140748.hdf5
JobscriptExit code0
Real time2h (8541s)
CPU time1h (3696s = 43%)
Max RSS bytes2531442688 (2414 MiB)
Outputting started2026-02-04 05:03:20
Output files
Finished2026-02-04 05:06:08
Saved logsjustin-logs:303789.3-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

e50
Caught exception in looking up space point ptr... 0x29ef5e50, 0x15a7d780
Caught exception in looking up space point ptr... 0x4508fe70, 0x45090190
Caught exception in looking up space point ptr... 0x45090190, 0x4508fe70
Caught exception in looking up space point ptr... 0x4612f540, 0x461375e0
Caught exception in looking up space point ptr... 0x461375e0, 0x4612e9f0
Caught exception in looking up space point ptr... 0x4612e9f0, 0x4f511790
Caught exception in looking up space point ptr... 0xb271ad0, 0xb271940
Caught exception in looking up space point ptr... 0xb271940, 0xb271ad0
Caught exception in looking up space point ptr... 0x3ccbd940, 0x29c91d60
Caught exception in looking up space point ptr... 0x29c91d60, 0x3ccbd940
Caught exception in looking up space point ptr... 0x21602e10, 0x21602c80
Caught exception in looking up space point ptr... 0x216072e0, 0x21602e10
Caught exception in looking up space point ptr... 0x4624fd50, 0x29979a10
Caught exception in looking up space point ptr... 0x29979a10, 0x21604400
Caught exception in looking up space point ptr... 0x62b68680, 0x3ccb1720
Caught exception in looking up space point ptr... 0x3ccb1720, 0x62b68580
Caught exception in looking up space point ptr... 0x4624f1d0, 0x25e8b350
Caught exception in looking up space point ptr... 0x25e8b350, 0x46266ed0
Caught exception in looking up space point ptr... 0x24d92df0, 0x2a7137a0
Caught exception in looking up space point ptr... 0x2a7137a0, 0x450808b0
Caught exception in looking up space point ptr... 0x37b8a000, 0x2997b1d0
Caught exception in looking up space point ptr... 0x2997b1d0, 0x2995c7a0
Caught exception in looking up space point ptr... 0x29972540, 0x46251cf0
Caught exception in looking up space point ptr... 0x46251cf0, 0x37b6b250
Caught exception in looking up space point ptr... 0x302eefc0, 0x2e0724f0
Caught exception in looking up space point ptr... 0x2e0724f0, 0x302eefc0
Caught exception in looking up space point ptr... 0x302eefc0, 0x62ca74c0
Caught exception in looking up space point ptr... 0x62ca74c0, 0x302eefc0
Caught exception in looking up space point ptr... 0x302eefc0, 0x31d04090
Caught exception in looking up space point ptr... 0x31d04090, 0x302eefc0
Caught exception in looking up space point ptr... 0x4ad18e60, 0x4ad152c0
Caught exception in looking up space point ptr... 0x4ad152c0, 0x4ad12d20
Caught exception in looking up space point ptr... 0x4ad12d20, 0x4ad18bc0
Caught exception in looking up space point ptr... 0x15e53f70, 0x15e56110
Caught exception in looking up space point ptr... 0x15e56110, 0x15e53f70
Caught exception in looking up space point ptr... 0x3d1e41b0, 0x2cdae0e0
Caught exception in looking up space point ptr... 0x2cdae0e0, 0x3d1e41b0
Caught exception in looking up space point ptr... 0x572eaac0, 0x572eb010
Caught exception in looking up space point ptr... 0x572eb010, 0x572eaac0
Caught exception in looking up space point ptr... 0x14fd0eb0, 0x14fd0c00
Caught exception in looking up space point ptr... 0x14fd0c00, 0x14fd0eb0
Caught exception in looking up space point ptr... 0x59dad330, 0x59da4050
Caught exception in looking up space point ptr... 0x59da4050, 0x59dad330
Caught exception in looking up space point ptr... 0x4ad1fb30, 0x4ad1f810
Caught exception in looking up space point ptr... 0x4ad1f810, 0x4ad1fb30
Caught exception in looking up space point ptr... 0x357300b0, 0x3572ff40
Caught exception in looking up space point ptr... 0x3572ff40, 0x357300b0
Caught exception in looking up space point ptr... 0x16cdb970, 0xa5eb750
Caught exception in looking up space point ptr... 0xa5eb750, 0x16cdb970
Caught exception in looking up space point ptr... 0x5f323010, 0x5f3218d0
Caught exception in looking up space point ptr... 0x5f3218d0, 0x5f323010
Caught exception in looking up space point ptr... 0x5063b270, 0x3115a530
Caught exception in looking up space point ptr... 0x3115a530, 0x5063b270
Caught exception in looking up space point ptr... 0x398df410, 0x398df0f0
Caught exception in looking up space point ptr... 0x398df0f0, 0x398df410
Caught exception in looking up space point ptr... 0xb292c30, 0xb292a50
Caught exception in looking up space point ptr... 0xb292a50, 0xb292c30
Caught exception in looking up space point ptr... 0x450a44d0, 0x3981cd40
Caught exception in looking up space point ptr... 0x3981cd40, 0x450a44d0
Caught exception in looking up space point ptr... 0x6607d660, 0x518b2dc0
Caught exception in looking up space point ptr... 0x518b2dc0, 0x6607d660
Caught exception in looking up space point ptr... 0x6607d660, 0x6607d820
Caught exception in looking up space point ptr... 0x6607d820, 0x6607d660
Caught exception in looking up space point ptr... 0x22526d10, 0x175c6af0
Caught exception in looking up space point ptr... 0x175c6af0, 0x22526d10
++++>>>> total num hits: 61913, num free: 59419
04-Feb-2026 05:03:08 GMT  Closed output file "np02vd_raw_run042421_0068_df-s02-d0_dw_0_20260203T140748_reco_stage0_20260204T050308_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         225.281       310.166       373.595       322.951       41.279         27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       7.7486e-05    0.000111829   0.000304414   0.00010054    4.23464e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                92.4279       160.696       217.654       164.164       33.3664        27     
produce:triggerrawdecoder:PDVDTriggerReader4      0.681244      0.931397       1.57406      0.920326      0.187926        27     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0218723     0.0251719     0.052482      0.0223865    0.00607341       27     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.108965      0.119257      0.132323      0.112145      0.0101988       27     
produce:beamevent:BeamEvent                      9.3917e-05    0.000193356   0.00191934    0.000124334   0.000339471      27     
produce:pdvddaphne:DAPHNEReaderPDVD                13.2925       21.7935       49.7493       20.9871       6.69828        27     
produce:ophit:OpHitFinder                         0.0376697     0.0514059     0.0593068     0.0533824    0.00532919       27     
produce:wclsdatavd:WireCellToolkit                 79.5306       102.256       112.507       106.171       8.58322        27     
produce:gaushit:GausHitFinder                     0.800809       1.46533       2.21399       1.4443       0.370986        27     
produce:nhitsfilter:NumberOfHitsFilter           0.000218602   0.000387821   0.00119983    0.000370217   0.000175641      27     
produce:reco3d:SpacePointSolver                    7.62148       13.4393       26.3049       13.082        4.08025        27     
produce:hitpdune:DisambigFromSpacePoints          0.0883754     0.152454      0.257106       0.15581      0.038501        27     
produce:cluster3d:Cluster3D                        1.47358       3.24136       9.77973       2.95054       1.68738        27     
[art]:TriggerResults:TriggerResultInserter       2.2522e-05    3.34091e-05   8.9749e-05    2.9115e-05    1.3018e-05       27     
end_path:out1:RootOutput                          3.797e-06    6.26041e-06   4.9934e-05     4.569e-06    8.57761e-06      27     
end_path:out1:RootOutput(write)                    4.73708       5.96429       6.94698       6.20681      0.694225        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4486.69 MB
  Peak resident set size usage (VmHWM): 2531.44 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042421_0068_df-s02-d0_dw_0_20260203T140748_reco_stage0_20260204T050308_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042421_0068_df-s02-d0_dw_0_20260203T140748_reco_stage0_20260204T050308_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042421_0068_df-s02-d0_dw_0_20260203T140748_reco_stage0_20260204T050308_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770172875.0,
    "core.end_time": 1770181390.0,
    "core.events": [
      11076,
      11082,
      11088,
      11094,
      11100,
      11106,
      11112,
      11118,
      11124,
      11130,
      11136,
      11142,
      11148,
      11154,
      11160,
      11166,
      11172,
      11178,
      11184,
      11190,
      11196,
      11202,
      11208,
      11214,
      11220,
      11226,
      11232
    ],
    "core.event_count": 27,
    "core.first_event_number": 11076,
    "core.last_event_number": 11232,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42421
    ],
    "core.runs_subruns": [
      4242100001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042421_0068_df-s02-d0_dw_0_20260203T140748.hdf5"
    }
  ]
}
justIN time: 2026-02-04 10:33:56 UTC       justIN version: 01.06.00