justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297920.166@dunegpschedd02.fnal.gov

Jobsub ID297920.166@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:54:27
SiteUK_Manchester
EntryUBoone_T2_UK_Manchester_ce02
Last heartbeat2026-02-04 03:03:51
From worker nodeHostnamewn1906250.tier2.hep.manchester.ac.uk
cpuinfoIntel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:43:36
Input filesvd-protodune:np02vd_raw_run042433_0229_df-s03-d1_dw_0_20260203T200910.hdf5
JobscriptExit code0
Real time3h (11821s)
CPU time1h (7140s = 60%)
Max RSS bytes3448037376 (3288 MiB)
Outputting started2026-02-04 03:00:40
Output files
Finished2026-02-04 03:03:41
Saved logsjustin-logs:297920.166-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

nt ptr... 0x41223550, 0x55652b90
Caught exception in looking up space point ptr... 0x5557db80, 0x4a732ba0
Caught exception in looking up space point ptr... 0x4a732ba0, 0x11780810
Caught exception in looking up space point ptr... 0x6c01b150, 0x445b3a10
Caught exception in looking up space point ptr... 0x445b41c0, 0x6c01b150
Caught exception in looking up space point ptr... 0x38edaeb0, 0x9a26d50
Caught exception in looking up space point ptr... 0x9a26d50, 0x38edaeb0
Caught exception in looking up space point ptr... 0x38edaeb0, 0x6bab6de0
Caught exception in looking up space point ptr... 0x6bab6de0, 0x38edaeb0
Caught exception in looking up space point ptr... 0x57c3b580, 0x78768e20
Caught exception in looking up space point ptr... 0x78768e20, 0x57c3b580
Caught exception in looking up space point ptr... 0x3e4ba580, 0xe0b2b60
Caught exception in looking up space point ptr... 0xe0b2b60, 0x158d5260
Caught exception in looking up space point ptr... 0x7db18ae0, 0x112ddd70
Caught exception in looking up space point ptr... 0x112ddd70, 0x7db18ae0
Caught exception in looking up space point ptr... 0x39bc56e0, 0x3da22f70
Caught exception in looking up space point ptr... 0x3da22f70, 0x39bc56e0
Caught exception in looking up space point ptr... 0xda72fc0, 0x19d059b0
Caught exception in looking up space point ptr... 0x19d059b0, 0x19d057d0
Caught exception in looking up space point ptr... 0x19d057d0, 0x19d055f0
Caught exception in looking up space point ptr... 0xda72e30, 0xda72fc0
Caught exception in looking up space point ptr... 0x2fea5890, 0x2fea5760
Caught exception in looking up space point ptr... 0x2fea5760, 0x2fea5890
Caught exception in looking up space point ptr... 0x59c57e60, 0x59c58350
Caught exception in looking up space point ptr... 0x59c58350, 0x59c57e60
Caught exception in looking up space point ptr... 0x15faeb70, 0x79584d30
Caught exception in looking up space point ptr... 0x3eb793b0, 0x15faeb70
Caught exception in looking up space point ptr... 0x59a850d0, 0x59a84c20
Caught exception in looking up space point ptr... 0x59a84c20, 0x59a850d0
Caught exception in looking up space point ptr... 0xcc38e30, 0x3a5e90c0
Caught exception in looking up space point ptr... 0x3a5e90c0, 0x47490fe0
Caught exception in looking up space point ptr... 0x2a04dfb0, 0x2a04a170
Caught exception in looking up space point ptr... 0x2a04a170, 0x2a04dfb0
Caught exception in looking up space point ptr... 0x120ad090, 0x120acf70
Caught exception in looking up space point ptr... 0x120acf70, 0x120ad090
Caught exception in looking up space point ptr... 0x3075fd40, 0x3075ff90
Caught exception in looking up space point ptr... 0x3075ff90, 0x3075fb50
Caught exception in looking up space point ptr... 0x12cb8970, 0x474f56b0
Caught exception in looking up space point ptr... 0x474f56b0, 0x1101a1a0
Caught exception in looking up space point ptr... 0x11b30bd0, 0x11b30aa0
Caught exception in looking up space point ptr... 0x11b30aa0, 0x11b30970
Caught exception in looking up space point ptr... 0x3835cd20, 0x78774b90
Caught exception in looking up space point ptr... 0x78774b90, 0x3835cd20
Caught exception in looking up space point ptr... 0x3835cd20, 0x78775070
Caught exception in looking up space point ptr... 0x78775070, 0x3835cd20
Caught exception in looking up space point ptr... 0x4762fc00, 0x36cea5c0
Caught exception in looking up space point ptr... 0x36cea5c0, 0x37b25d60
Caught exception in looking up space point ptr... 0x4325d400, 0x129fe000
Caught exception in looking up space point ptr... 0x129fe000, 0x129fde40
Caught exception in looking up space point ptr... 0x60d55770, 0x60d55b30
Caught exception in looking up space point ptr... 0x60d55b30, 0x60d55950
Caught exception in looking up space point ptr... 0x3bd81450, 0x3bd816b0
Caught exception in looking up space point ptr... 0x3bd816b0, 0x3bd817e0
Caught exception in looking up space point ptr... 0x3589d7e0, 0x3589d4c0
Caught exception in looking up space point ptr... 0x3589d4c0, 0x3589bda0
Caught exception in looking up space point ptr... 0x78af8fd0, 0x78af8e60
Caught exception in looking up space point ptr... 0x78af8e60, 0x78af8fd0
Caught exception in looking up space point ptr... 0x15252810, 0x13294060
Caught exception in looking up space point ptr... 0x13294060, 0x15252810
Caught exception in looking up space point ptr... 0x5566d8f0, 0x38c5b3b0
Caught exception in looking up space point ptr... 0x38c5b3b0, 0x38c5b230
Caught exception in looking up space point ptr... 0x29796050, 0x4a8e9c70
Caught exception in looking up space point ptr... 0x4a8e9c70, 0x29796050
Caught exception in looking up space point ptr... 0x5b7f8150, 0x381ed850
Caught exception in looking up space point ptr... 0x381ed850, 0x5b7f8150
Caught exception in looking up space point ptr... 0x420346c0, 0x4bfd8600
Caught exception in looking up space point ptr... 0x4bfd8600, 0x420346c0
++++>>>> total num hits: 76986, num free: 73532
04-Feb-2026 03:00:14 GMT  Closed output file "np02vd_raw_run042433_0229_df-s03-d1_dw_0_20260203T200910_reco_stage0_20260204T030014_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         476.237       612.562       768.897       622.623       69.4718        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       0.00012001    0.000219524   0.000293249   0.00024151    5.75769e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                163.952       226.352       312.993       231.711       38.6352        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.859195       1.17929       1.92761       1.10925      0.278391        19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0224297     0.0312004     0.0565079     0.029362     0.00925265       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0220203     0.0227965     0.0288673     0.0224994    0.00144475       19     
produce:beamevent:BeamEvent                      0.000160824   0.000306259   0.00131646    0.000256862   0.000240536      19     
produce:pdvddaphne:DAPHNEReaderPDVD                17.9786       22.6817       27.1636       22.2722       2.22026        19     
produce:ophit:OpHitFinder                         0.0482454     0.0734637     0.100807       0.07435      0.0118989       19     
produce:wclsdatavd:WireCellToolkit                 205.896       251.953       318.125       248.445       25.6276        19     
produce:gaushit:GausHitFinder                      2.01893       3.70226       5.13096       3.58321      0.794675        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000573257   0.000842699   0.00144167    0.000785652   0.000212015      19     
produce:reco3d:SpacePointSolver                    15.3961       21.9345       34.0691       21.1004       4.48441        19     
produce:hitpdune:DisambigFromSpacePoints          0.171401      0.547822       0.84394      0.575085       0.19025        19     
produce:cluster3d:Cluster3D                        2.47633       70.4422       127.14        101.743       50.5049        19     
[art]:TriggerResults:TriggerResultInserter       4.6119e-05    7.73215e-05   0.00015117     6.811e-05    2.59011e-05      19     
end_path:out1:RootOutput                          8.398e-06    1.73061e-05    4.18e-05     1.4546e-05    8.89633e-06      19     
end_path:out1:RootOutput(write)                    7.15035       13.5549       19.6671       13.6693       3.46652        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5513.36 MB
  Peak resident set size usage (VmHWM): 3448.04 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0229_df-s03-d1_dw_0_20260203T200910_reco_stage0_20260204T030014_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0229_df-s03-d1_dw_0_20260203T200910_reco_stage0_20260204T030014_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0229_df-s03-d1_dw_0_20260203T200910_reco_stage0_20260204T030014_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770162228.0,
    "core.end_time": 1770174030.0,
    "core.events": [
      26115,
      26121,
      26127,
      26133,
      26139,
      26145,
      26151,
      26157,
      26163,
      26169,
      26175,
      26181,
      26187,
      26193,
      26199,
      26205,
      26211,
      26217,
      26223
    ],
    "core.event_count": 19,
    "core.first_event_number": 26115,
    "core.last_event_number": 26223,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0229_df-s03-d1_dw_0_20260203T200910.hdf5"
    }
  ]
}
justIN time: 2026-02-04 04:33:05 UTC       justIN version: 01.06.00