justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303744.1@dunegpschedd01.fnal.gov

Jobsub ID303744.1@dunegpschedd01.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-04 01:32:31
SiteUK_Manchester
EntryUBoone_T2_UK_Manchester_ce01
Last heartbeat2026-02-04 04:07:31
From worker nodeHostnamewn1905250.tier2.hep.manchester.ac.uk
cpuinfoAMD EPYC 7513 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-04 01:32:48
Input filesvd-protodune:np02vd_raw_run042433_0351_df-s04-d1_dw_0_20260203T205531.hdf5
JobscriptExit code0
Real time2h (9118s)
CPU time1h (3683s = 40%)
Max RSS bytes3393150976 (3235 MiB)
Outputting started2026-02-04 04:04:48
Output files
Finished2026-02-04 04:07:31
Saved logsjustin-logs:303744.1-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

space point ptr... 0xf0e4d70, 0xf0e4f00
Caught exception in looking up space point ptr... 0xf0e4f00, 0xf0e0150
Caught exception in looking up space point ptr... 0x6fc6f410, 0x91119c0
Caught exception in looking up space point ptr... 0x91119c0, 0x6fc6f410
Caught exception in looking up space point ptr... 0x61db7f00, 0x61db8090
Caught exception in looking up space point ptr... 0x61db8090, 0x61d9eaf0
Caught exception in looking up space point ptr... 0x62383f50, 0x19be8d70
Caught exception in looking up space point ptr... 0x19be8d70, 0x62382d20
Caught exception in looking up space point ptr... 0x37c8f390, 0x37c92380
Caught exception in looking up space point ptr... 0x37c92380, 0x37c925d0
Caught exception in looking up space point ptr... 0x6e87fc90, 0x662cde40
Caught exception in looking up space point ptr... 0x662cde40, 0x80c3c100
Caught exception in looking up space point ptr... 0x39ede3d0, 0x4ea49050
Caught exception in looking up space point ptr... 0x4ea49050, 0x4ea47ff0
Caught exception in looking up space point ptr... 0x81b196b0, 0x81b026b0
Caught exception in looking up space point ptr... 0x81b026b0, 0x81b027e0
Caught exception in looking up space point ptr... 0x6281c830, 0x62818240
Caught exception in looking up space point ptr... 0x62818240, 0x6281ba90
Caught exception in looking up space point ptr... 0x41cb3110, 0x41cb2f10
Caught exception in looking up space point ptr... 0x41cb2f10, 0x41cb3260
Caught exception in looking up space point ptr... 0x16f074c0, 0x114571e0
Caught exception in looking up space point ptr... 0x114571e0, 0x11450060
Caught exception in looking up space point ptr... 0x43438e80, 0x6fcbb5c0
Caught exception in looking up space point ptr... 0x6fcbb5c0, 0x43438e80
Caught exception in looking up space point ptr... 0x52b43a10, 0xe634440
Caught exception in looking up space point ptr... 0xe634440, 0xe6345d0
Caught exception in looking up space point ptr... 0x1860a330, 0x18609fd0
Caught exception in looking up space point ptr... 0x18609fd0, 0x1860a330
Caught exception in looking up space point ptr... 0xf4e4eb0, 0xf4e3170
Caught exception in looking up space point ptr... 0xf4e3170, 0xf4e4eb0
Caught exception in looking up space point ptr... 0xe615800, 0xe615b20
Caught exception in looking up space point ptr... 0xe615b20, 0xe617070
Caught exception in looking up space point ptr... 0xe617070, 0xe615e40
Caught exception in looking up space point ptr... 0x61dd2e60, 0x49e3f900
Caught exception in looking up space point ptr... 0x49e3f900, 0x4a95a310
Caught exception in looking up space point ptr... 0x66da19f0, 0x60dd6bb0
Caught exception in looking up space point ptr... 0x60dd6bb0, 0x60dd7230
Caught exception in looking up space point ptr... 0x60dd7230, 0x6e454d80
Caught exception in looking up space point ptr... 0x6e454d80, 0x3ca645b0
Caught exception in looking up space point ptr... 0x17d44d00, 0x17d34f70
Caught exception in looking up space point ptr... 0x17d34f70, 0x17d44d00
Caught exception in looking up space point ptr... 0x328613e0, 0x5466e040
Caught exception in looking up space point ptr... 0x5466e040, 0x328613e0
Caught exception in looking up space point ptr... 0x16554ea0, 0x16554b80
Caught exception in looking up space point ptr... 0x16554b80, 0x16554ea0
Caught exception in looking up space point ptr... 0x5629e540, 0x387b7300
Caught exception in looking up space point ptr... 0x387b7300, 0x11dd7eb0
Caught exception in looking up space point ptr... 0x146c5510, 0x2c3a5590
Caught exception in looking up space point ptr... 0x2c3a5590, 0x146c5510
Caught exception in looking up space point ptr... 0x4a638a20, 0x5ded10c0
Caught exception in looking up space point ptr... 0x5ded10c0, 0x1653e100
Caught exception in looking up space point ptr... 0x6e415b30, 0x625ea290
Caught exception in looking up space point ptr... 0x620e8fe0, 0x6e415b30
Caught exception in looking up space point ptr... 0x4a63e230, 0x4a63dd80
Caught exception in looking up space point ptr... 0x4a63dd80, 0x4a63e230
Caught exception in looking up space point ptr... 0x11d685f0, 0x66f41df0
Caught exception in looking up space point ptr... 0x66f41df0, 0x66f41990
Caught exception in looking up space point ptr... 0x6e60a1f0, 0x12c2dcb0
Caught exception in looking up space point ptr... 0x12c2dcb0, 0x6e6c6250
Caught exception in looking up space point ptr... 0x6e6c6250, 0x12c2dcb0
Caught exception in looking up space point ptr... 0x12c2dcb0, 0x6fd9a880
Caught exception in looking up space point ptr... 0x6fd9a880, 0x12c2dcb0
Caught exception in looking up space point ptr... 0x12c2dcb0, 0x6e60a1f0
Caught exception in looking up space point ptr... 0x6280dac0, 0x6280d920
Caught exception in looking up space point ptr... 0x6280d920, 0x6280ddb0
Caught exception in looking up space point ptr... 0x6cb49050, 0x6610a190
Caught exception in looking up space point ptr... 0x6610a190, 0x6cb37380
++++>>>> total num hits: 110942, num free: 107796
04-Feb-2026 04:04:36 GMT  Closed output file "np02vd_raw_run042433_0351_df-s04-d1_dw_0_20260203T205531_reco_stage0_20260204T040436_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         359.787       473.939       542.188       483.004       45.2579        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       7.6745e-05    0.000138759   0.000338106   0.000122891   5.62701e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                156.353       267.186       335.057       283.16        46.4988        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.844354       1.23142       1.86886       1.17134       0.23979        19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0222381     0.030904      0.0839488     0.028514      0.0137214       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0219564     0.0228107     0.0303678     0.0223362    0.00180569       19     
produce:beamevent:BeamEvent                      9.9658e-05    0.000262613   0.00177336    0.000158238   0.000361979      19     
produce:pdvddaphne:DAPHNEReaderPDVD                14.022        22.4616       27.3937       23.8662       4.00178        19     
produce:ophit:OpHitFinder                         0.0337698     0.0440378     0.0581199     0.0435664    0.00612929       19     
produce:wclsdatavd:WireCellToolkit                 124.461       147.39        168.343       147.304       12.722         19     
produce:gaushit:GausHitFinder                     0.921359       1.9395        2.49369       2.01269      0.390458        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000268486   0.000450429   0.000835423   0.000409852   0.000149819      19     
produce:reco3d:SpacePointSolver                    11.4724       22.2084       39.266        20.522        6.55935        19     
produce:hitpdune:DisambigFromSpacePoints          0.122374      0.253705       0.33842      0.268021      0.0593264       19     
produce:cluster3d:Cluster3D                        1.97366       4.3303        6.14014       4.16606       1.11431        19     
[art]:TriggerResults:TriggerResultInserter       2.5648e-05    4.15183e-05   0.000124835   3.6028e-05    2.05363e-05      19     
end_path:out1:RootOutput                          4.097e-06    9.87232e-06   4.1558e-05     7.655e-06    8.3932e-06       19     
end_path:out1:RootOutput(write)                    4.64186       6.78045       8.06557       6.95193      0.791813        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5355.95 MB
  Peak resident set size usage (VmHWM): 3393.15 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0351_df-s04-d1_dw_0_20260203T205531_reco_stage0_20260204T040436_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0351_df-s04-d1_dw_0_20260203T205531_reco_stage0_20260204T040436_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0351_df-s04-d1_dw_0_20260203T205531_reco_stage0_20260204T040436_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770168781.0,
    "core.end_time": 1770177879.0,
    "core.events": [
      40025,
      40031,
      40037,
      40043,
      40049,
      40055,
      40061,
      40067,
      40073,
      40079,
      40085,
      40091,
      40097,
      40103,
      40109,
      40115,
      40121,
      40127,
      40133
    ],
    "core.event_count": 19,
    "core.first_event_number": 40025,
    "core.last_event_number": 40133,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0351_df-s04-d1_dw_0_20260203T205531.hdf5"
    }
  ]
}
justIN time: 2026-02-04 04:28:39 UTC       justIN version: 01.06.00