justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303699.0@dunegpschedd01.fnal.gov

Jobsub ID303699.0@dunegpschedd01.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-04 00:28:28
SiteUK_Manchester
EntryUBoone_T2_UK_Manchester_ce02
Last heartbeat2026-02-04 02:58:25
From worker nodeHostnamewn2208250.tier2.hep.manchester.ac.uk
cpuinfoAMD EPYC 7513 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-04 00:28:58
Input filesvd-protodune:np02vd_raw_run042433_0205_df-s03-d1_dw_0_20260203T200002.hdf5
JobscriptExit code0
Real time2h (8803s)
CPU time54m (3283s = 37%)
Max RSS bytes3253866496 (3103 MiB)
Outputting started2026-02-04 02:55:43
Output files
Finished2026-02-04 02:58:25
Saved logsjustin-logs:303699.0-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

t ptr... 0xf45b280, 0xf45af80
Caught exception in looking up space point ptr... 0x3d3088f0, 0x3d308760
Caught exception in looking up space point ptr... 0x3d308760, 0x3d3088f0
Caught exception in looking up space point ptr... 0x39e56450, 0x1b289d30
Caught exception in looking up space point ptr... 0x1b289d30, 0x39e56450
Caught exception in looking up space point ptr... 0x7d295590, 0x7d29e830
Caught exception in looking up space point ptr... 0x7d29e830, 0x7d295590
Caught exception in looking up space point ptr... 0x3ac700b0, 0x3877ffa0
Caught exception in looking up space point ptr... 0x3877ffa0, 0x3877fe30
Caught exception in looking up space point ptr... 0x39e81ae0, 0x831cc550
Caught exception in looking up space point ptr... 0x831cc550, 0x39e81900
Caught exception in looking up space point ptr... 0x3c4cf680, 0x3da5a610
Caught exception in looking up space point ptr... 0x4a1065d0, 0x3c4cf680
Caught exception in looking up space point ptr... 0x4afdd8d0, 0x4afdda20
Caught exception in looking up space point ptr... 0x4afdda20, 0x4afdd8d0
Caught exception in looking up space point ptr... 0x4afdd8d0, 0x509f92b0
Caught exception in looking up space point ptr... 0x509f92b0, 0x4afdd8d0
Caught exception in looking up space point ptr... 0x834bbc40, 0x12fffb80
Caught exception in looking up space point ptr... 0x12fffb80, 0x834bbc40
Caught exception in looking up space point ptr... 0x5a03cd80, 0x1895db40
Caught exception in looking up space point ptr... 0x1895db40, 0x5a03cd80
Caught exception in looking up space point ptr... 0x6f5c23f0, 0x3c4d2bf0
Caught exception in looking up space point ptr... 0x3c4d2bf0, 0xea01180
Caught exception in looking up space point ptr... 0x4f19a5b0, 0x4f19a4e0
Caught exception in looking up space point ptr... 0x13008ec0, 0x4f19a5b0
Caught exception in looking up space point ptr... 0x3e556620, 0x399041e0
Caught exception in looking up space point ptr... 0x399041e0, 0x180c4be0
Caught exception in looking up space point ptr... 0x3d1f1c80, 0x3d1f1b50
Caught exception in looking up space point ptr... 0x3d1f1b50, 0x3d1f11d0
Caught exception in looking up space point ptr... 0x394fe310, 0x39504f10
Caught exception in looking up space point ptr... 0x39504f10, 0x394fe310
Caught exception in looking up space point ptr... 0x619ccf60, 0x619cce10
Caught exception in looking up space point ptr... 0x619cce10, 0x619ccf60
Caught exception in looking up space point ptr... 0x7d287fe0, 0xa0b55e0
Caught exception in looking up space point ptr... 0xa0b55e0, 0x7d287fe0
Caught exception in looking up space point ptr... 0x511f87b0, 0x159ba580
Caught exception in looking up space point ptr... 0x159ba580, 0x1a942150
Caught exception in looking up space point ptr... 0x619c7980, 0x619c7660
Caught exception in looking up space point ptr... 0x619c7660, 0x619c7980
Caught exception in looking up space point ptr... 0x3bb25f50, 0x111d9530
Caught exception in looking up space point ptr... 0x111d9530, 0x3bb25f50
Caught exception in looking up space point ptr... 0x36f03310, 0x4ad357a0
Caught exception in looking up space point ptr... 0x4ad357a0, 0x36f03310
Caught exception in looking up space point ptr... 0x13489130, 0x3831ac40
Caught exception in looking up space point ptr... 0x3831ac40, 0x13489130
Caught exception in looking up space point ptr... 0x13489130, 0x14505970
Caught exception in looking up space point ptr... 0x14505970, 0x13489130
Caught exception in looking up space point ptr... 0xd692f00, 0x72974310
Caught exception in looking up space point ptr... 0x72974310, 0xd692f00
Caught exception in looking up space point ptr... 0x159d3950, 0x159d3630
Caught exception in looking up space point ptr... 0x159d3630, 0x159d3950
Caught exception in looking up space point ptr... 0x1272ce80, 0x664aae10
Caught exception in looking up space point ptr... 0x664aae10, 0x1272ce80
Caught exception in looking up space point ptr... 0x409574b0, 0x40957700
Caught exception in looking up space point ptr... 0x40957700, 0x409574b0
Caught exception in looking up space point ptr... 0x130319a0, 0x13035a70
Caught exception in looking up space point ptr... 0x13035a70, 0x1302d200
Caught exception in looking up space point ptr... 0x3dc83f70, 0x7dd59040
Caught exception in looking up space point ptr... 0x7dd59040, 0x3dc83f70
Caught exception in looking up space point ptr... 0x47809730, 0x390d8ba0
Caught exception in looking up space point ptr... 0x390d8ba0, 0x47809730
Caught exception in looking up space point ptr... 0x13859010, 0x13858790
Caught exception in looking up space point ptr... 0x13858790, 0x138593a0
Caught exception in looking up space point ptr... 0x18ab84b0, 0x59ee59c0
Caught exception in looking up space point ptr... 0x59ee59c0, 0x18ab84b0
Caught exception in looking up space point ptr... 0x4a9a5aa0, 0x4e6dc6a0
Caught exception in looking up space point ptr... 0x4e6dc6a0, 0x4a9a5aa0
++++>>>> total num hits: 96064, num free: 91678
04-Feb-2026 02:55:33 GMT  Closed output file "np02vd_raw_run042433_0205_df-s03-d1_dw_0_20260203T200002_reco_stage0_20260204T025533_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         359.24        457.796       611.813       467.381       61.7029        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       8.6162e-05    0.000170927   0.000309341   0.000169608   7.5348e-05       19     
produce:tpcrawdecoder:PDVDTPCReader                204.29        273.308       408.699       267.525       49.4965        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.752976       1.13482       1.76409       1.05878       0.2347         19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0219139     0.0289112     0.0614311     0.0229333     0.0114966       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0219967     0.0251013     0.073109      0.0223898     0.0113219       19     
produce:beamevent:BeamEvent                      0.000135805   0.000258655   0.00116577    0.000202601   0.000217996      19     
produce:pdvddaphne:DAPHNEReaderPDVD                16.9685       22.8425       28.5347       22.1238       3.57441        19     
produce:ophit:OpHitFinder                         0.0293592     0.0439925     0.0582671     0.0446492     0.0071625       19     
produce:wclsdatavd:WireCellToolkit                 107.069       132.457       157.091       131.369       14.672         19     
produce:gaushit:GausHitFinder                     0.831788       1.52605       2.7983        1.41715      0.432729        19     
produce:nhitsfilter:NumberOfHitsFilter           0.00031939    0.000545326   0.00301174    0.000398548   0.000586256      19     
produce:reco3d:SpacePointSolver                    9.63445       16.2355       33.8252       15.5121       5.18011        19     
produce:hitpdune:DisambigFromSpacePoints          0.144197      0.219181      0.432406      0.195355      0.0741828       19     
produce:cluster3d:Cluster3D                        1.75106       3.38763       7.69852       2.91429       1.4722         19     
[art]:TriggerResults:TriggerResultInserter       2.7321e-05    4.09221e-05   0.000108894   3.6147e-05    1.7736e-05       19     
end_path:out1:RootOutput                          5.53e-06     1.13128e-05   2.8834e-05     9.528e-06    6.28549e-06      19     
end_path:out1:RootOutput(write)                    4.63014       6.53695       8.05645       6.89287       1.03323        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5355.92 MB
  Peak resident set size usage (VmHWM): 3253.87 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0205_df-s03-d1_dw_0_20260203T200002_reco_stage0_20260204T025533_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0205_df-s03-d1_dw_0_20260203T200002_reco_stage0_20260204T025533_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0205_df-s03-d1_dw_0_20260203T200002_reco_stage0_20260204T025533_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770164951.0,
    "core.end_time": 1770173735.0,
    "core.events": [
      23379,
      23385,
      23391,
      23397,
      23403,
      23409,
      23415,
      23421,
      23427,
      23433,
      23439,
      23445,
      23451,
      23457,
      23463,
      23469,
      23475,
      23481,
      23487
    ],
    "core.event_count": 19,
    "core.first_event_number": 23379,
    "core.last_event_number": 23487,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0205_df-s03-d1_dw_0_20260203T200002.hdf5"
    }
  ]
}
justIN time: 2026-02-04 04:26:39 UTC       justIN version: 01.06.00