justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297921.67@dunegpschedd02.fnal.gov

Jobsub ID297921.67@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:58:26
SiteUK_Manchester
EntryUBoone_T2_UK_Manchester_ce01
Last heartbeat2026-02-04 02:58:57
From worker nodeHostnamewn1907291.tier2.hep.manchester.ac.uk
cpuinfoIntel(R) Xeon(R) Gold 5118 CPU @ 2.30GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:52:02
Input filesvd-protodune:np02vd_raw_run042433_0364_df-s02-d1_dw_0_20260203T210029.hdf5
JobscriptExit code0
Real time3h (11034s)
CPU time1h (6404s = 58%)
Max RSS bytes3381837824 (3225 MiB)
Outputting started2026-02-04 02:55:58
Output files
Finished2026-02-04 02:58:57
Saved logsjustin-logs:297921.67-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ce point ptr... 0x2d31b0d0, 0x2d318760
Caught exception in looking up space point ptr... 0x168f2570, 0x1690be20
Caught exception in looking up space point ptr... 0x1690be20, 0x168f2970
Caught exception in looking up space point ptr... 0x4a191e20, 0x4a191cd0
Caught exception in looking up space point ptr... 0x4a191cd0, 0x4a191e20
Caught exception in looking up space point ptr... 0x5afc7730, 0x5afc2f10
Caught exception in looking up space point ptr... 0x5afc2f10, 0x5afc7730
Caught exception in looking up space point ptr... 0x6b27af80, 0x6b2698b0
Caught exception in looking up space point ptr... 0x6b2698b0, 0x6b27af80
Caught exception in looking up space point ptr... 0x2d31fee0, 0x2d31fd50
Caught exception in looking up space point ptr... 0x2d31fd50, 0x2d31fee0
Caught exception in looking up space point ptr... 0x301db2b0, 0x95d5430
Caught exception in looking up space point ptr... 0x95d5430, 0x301db2b0
Caught exception in looking up space point ptr... 0x14705550, 0x32b84200
Caught exception in looking up space point ptr... 0x32b84200, 0x14705550
Caught exception in looking up space point ptr... 0xf70ccc0, 0xf70cfb0
Caught exception in looking up space point ptr... 0xf70cfb0, 0xf70ccc0
Caught exception in looking up space point ptr... 0x4f1e5350, 0x4f1dc3a0
Caught exception in looking up space point ptr... 0x4f1dc3a0, 0x4f1e5350
Caught exception in looking up space point ptr... 0x3f17b3a0, 0x3f178b10
Caught exception in looking up space point ptr... 0x3f178b10, 0x3f17b3a0
Caught exception in looking up space point ptr... 0x3b3c6270, 0x3b3b64f0
Caught exception in looking up space point ptr... 0x3b3b64f0, 0x3b3c6270
Caught exception in looking up space point ptr... 0x38e401d0, 0x38e3fbf0
Caught exception in looking up space point ptr... 0x38e3fbf0, 0x38e40300
Caught exception in looking up space point ptr... 0x5663c210, 0x5663c650
Caught exception in looking up space point ptr... 0x5663c650, 0x5663c210
Caught exception in looking up space point ptr... 0x5b2df1c0, 0x5b2f8560
Caught exception in looking up space point ptr... 0x5b2f8560, 0x5b2df1c0
Caught exception in looking up space point ptr... 0x95cdce0, 0x95cd5a0
Caught exception in looking up space point ptr... 0x95cd5a0, 0x95cdce0
Caught exception in looking up space point ptr... 0x95cdce0, 0x95cd770
Caught exception in looking up space point ptr... 0x95cd770, 0x95cdce0
Caught exception in looking up space point ptr... 0x329b77e0, 0x345789b0
Caught exception in looking up space point ptr... 0x345789b0, 0x329b59e0
Caught exception in looking up space point ptr... 0x2c3e3a70, 0x2c3e3d70
Caught exception in looking up space point ptr... 0x2c3e3d70, 0x2c3e3c70
Caught exception in looking up space point ptr... 0x55a1b360, 0x55a1ab60
Caught exception in looking up space point ptr... 0x55a1ab60, 0x55a1b570
Caught exception in looking up space point ptr... 0x3f17f770, 0x3f17f1f0
Caught exception in looking up space point ptr... 0x3f17f1f0, 0x3f196e40
Caught exception in looking up space point ptr... 0x53fdbcb0, 0x53fd9870
Caught exception in looking up space point ptr... 0x53fd9870, 0x53fe5fe0
Caught exception in looking up space point ptr... 0x62454620, 0x45f69610
Caught exception in looking up space point ptr... 0x45f69610, 0x56682160
Caught exception in looking up space point ptr... 0x16389f40, 0x16388b80
Caught exception in looking up space point ptr... 0x16388b80, 0x16388540
Caught exception in looking up space point ptr... 0x8ebe330, 0x55a34110
Caught exception in looking up space point ptr... 0x55a34110, 0x8ebe330
Caught exception in looking up space point ptr... 0x5a7c4d50, 0x5a7c4bc0
Caught exception in looking up space point ptr... 0x5a7c4bc0, 0x5a7c4d50
Caught exception in looking up space point ptr... 0x585d7680, 0x585d79a0
Caught exception in looking up space point ptr... 0x585d79a0, 0x585d7680
Caught exception in looking up space point ptr... 0x4f7c8b30, 0x4f7c8870
Caught exception in looking up space point ptr... 0x4f7c8870, 0x4f7c8b30
Caught exception in looking up space point ptr... 0x82c6be40, 0x82c72340
Caught exception in looking up space point ptr... 0x82c72340, 0x82c72520
Caught exception in looking up space point ptr... 0x82c29e60, 0x82c29d20
Caught exception in looking up space point ptr... 0x82c29d20, 0x82c29e60
Caught exception in looking up space point ptr... 0x35a820e0, 0x35a76f40
Caught exception in looking up space point ptr... 0x35a76f40, 0x35a820e0
Caught exception in looking up space point ptr... 0x18e2d310, 0x18e23830
Caught exception in looking up space point ptr... 0x18e23830, 0x18e2d310
Caught exception in looking up space point ptr... 0x84191f30, 0x39561640
Caught exception in looking up space point ptr... 0x39561640, 0x84191f30
Caught exception in looking up space point ptr... 0x774f1cf0, 0x774f1f10
Caught exception in looking up space point ptr... 0x774f1f10, 0x774f1cf0
++++>>>> total num hits: 196539, num free: 189627
04-Feb-2026 02:55:44 GMT  Closed output file "np02vd_raw_run042433_0364_df-s02-d1_dw_0_20260203T210029_reco_stage0_20260204T025544_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         493.745       572.896       670.007       567.13        42.1135        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       9.8014e-05    0.000214532   0.000415678   0.000182503   8.57416e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                178.93        228.209       317.919       215.158       35.5269        19     
produce:triggerrawdecoder:PDVDTriggerReader4       0.82407       1.45246       7.25929       1.10914       1.38234        19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0226058     0.113394       1.6403       0.0263276     0.359948        19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0220105     0.0244435     0.0528864     0.0229149    0.00671713       19     
produce:beamevent:BeamEvent                      0.000152804   0.000301879   0.00188914    0.000217418   0.000375789      19     
produce:pdvddaphne:DAPHNEReaderPDVD                16.5721       22.5933       28.7577       22.2564       2.90325        19     
produce:ophit:OpHitFinder                         0.0536856     0.079986      0.0889631     0.0824563     0.0096511       19     
produce:wclsdatavd:WireCellToolkit                 227.504       268.388       288.563       273.248       15.3069        19     
produce:gaushit:GausHitFinder                      1.26855       3.16564       4.60068       3.20799      0.751593        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000494644   0.000849134   0.00144185    0.000809933   0.000206853      19     
produce:reco3d:SpacePointSolver                    14.414        32.4602       60.7573       29.6775       12.0377        19     
produce:hitpdune:DisambigFromSpacePoints          0.212638      0.518499       1.06261      0.501659      0.199687        19     
produce:cluster3d:Cluster3D                        1.97188       5.45103       11.5594       4.99106       2.31787        19     
[art]:TriggerResults:TriggerResultInserter        4.277e-05    6.47782e-05   0.000202712   5.2988e-05    3.39505e-05      19     
end_path:out1:RootOutput                          4.997e-06    1.55138e-05   4.1678e-05    1.2302e-05    9.11443e-06      19     
end_path:out1:RootOutput(write)                    7.57764       10.3446       13.1491       10.0154       1.1657         19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5355.52 MB
  Peak resident set size usage (VmHWM): 3381.84 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0364_df-s02-d1_dw_0_20260203T210029_reco_stage0_20260204T025544_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0364_df-s02-d1_dw_0_20260203T210029_reco_stage0_20260204T025544_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0364_df-s02-d1_dw_0_20260203T210029_reco_stage0_20260204T025544_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770162737.0,
    "core.end_time": 1770173750.0,
    "core.events": [
      41503,
      41509,
      41515,
      41521,
      41527,
      41533,
      41539,
      41545,
      41551,
      41557,
      41563,
      41569,
      41575,
      41581,
      41587,
      41593,
      41599,
      41605,
      41611
    ],
    "core.event_count": 19,
    "core.first_event_number": 41503,
    "core.last_event_number": 41611,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0364_df-s02-d1_dw_0_20260203T210029.hdf5"
    }
  ]
}
justIN time: 2026-02-04 04:28:04 UTC       justIN version: 01.06.00