justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297919.10@dunegpschedd02.fnal.gov

Jobsub ID297919.10@dunegpschedd02.fnal.gov
Workflow ID12717
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:54:25
SiteUK_Imperial
EntryDUNE_T2_UK_London_IC_ceprod02
Last heartbeat2026-02-04 00:52:10
From worker nodeHostnamewk00.grid.hep.ph.ic.ac.uk
cpuinfoIntel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:29:06
Input filesvd-protodune:np02vd_raw_run042421_0077_df-s02-d0_dw_0_20260203T141240.hdf5
JobscriptExit code0
Real time1h (4825s)
CPU time1h (4379s = 90%)
Max RSS bytes2631348224 (2509 MiB)
Outputting started2026-02-04 00:49:35
Output files
Finished2026-02-04 00:52:10
Saved logsjustin-logs:297919.10-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

13b0, 0x29b4bd40
Caught exception in looking up space point ptr... 0xb5d5de0, 0x5b6413b0
Caught exception in looking up space point ptr... 0x5b693ea0, 0x5b693660
Caught exception in looking up space point ptr... 0x5b693660, 0x5b693ea0
Caught exception in looking up space point ptr... 0x2e0e2bf0, 0x2e0e2950
Caught exception in looking up space point ptr... 0x2e0e2950, 0x2e0e2aa0
Caught exception in looking up space point ptr... 0x601bc9e0, 0x601bb510
Caught exception in looking up space point ptr... 0x601bb510, 0x601bc9e0
Caught exception in looking up space point ptr... 0x29d92fe0, 0x29d93670
Caught exception in looking up space point ptr... 0x29d93670, 0x29d92fe0
Caught exception in looking up space point ptr... 0x31b92f80, 0x13a8a7d0
Caught exception in looking up space point ptr... 0x13a8a7d0, 0x31b92f80
Caught exception in looking up space point ptr... 0xee3e7a0, 0xee59ff0
Caught exception in looking up space point ptr... 0xee59ff0, 0xee53250
Caught exception in looking up space point ptr... 0x4e5c6500, 0x4e5c6370
Caught exception in looking up space point ptr... 0x4e5c6370, 0x4e5c6500
Caught exception in looking up space point ptr... 0x40ccdb40, 0x20f0a9d0
Caught exception in looking up space point ptr... 0x20f0a9d0, 0x40ccdb40
Caught exception in looking up space point ptr... 0x280fa170, 0x280fa030
Caught exception in looking up space point ptr... 0x280fa030, 0x280fa0d0
Caught exception in looking up space point ptr... 0x3cff79f0, 0x3cff78f0
Caught exception in looking up space point ptr... 0x3cff78f0, 0x3cff5af0
Caught exception in looking up space point ptr... 0x601af6d0, 0x601ae530
Caught exception in looking up space point ptr... 0x601ae530, 0x601af6d0
Caught exception in looking up space point ptr... 0x31e10520, 0x31e10260
Caught exception in looking up space point ptr... 0x31e10260, 0x2aa3adf0
Caught exception in looking up space point ptr... 0x98a3930, 0x98a3b90
Caught exception in looking up space point ptr... 0x98a3b90, 0x98a3930
Caught exception in looking up space point ptr... 0x16195a70, 0x16195750
Caught exception in looking up space point ptr... 0x16195750, 0x16195a70
Caught exception in looking up space point ptr... 0x623a42e0, 0x623a3fc0
Caught exception in looking up space point ptr... 0x623a3fc0, 0x623a42e0
Caught exception in looking up space point ptr... 0xa49d530, 0x35af21b0
Caught exception in looking up space point ptr... 0x35af21b0, 0xa49d530
Caught exception in looking up space point ptr... 0x473f4710, 0x3d757340
Caught exception in looking up space point ptr... 0x3d757340, 0x473f4710
Caught exception in looking up space point ptr... 0x28fbd050, 0x28fbcdb0
Caught exception in looking up space point ptr... 0x28fbcdb0, 0x28fbd050
Caught exception in looking up space point ptr... 0x2a9a31d0, 0x2a9a32c0
Caught exception in looking up space point ptr... 0x2a9a32c0, 0x2a9a31d0
Caught exception in looking up space point ptr... 0x511475e0, 0x51147260
Caught exception in looking up space point ptr... 0x51147260, 0x511475e0
Caught exception in looking up space point ptr... 0xd5dec00, 0xd5df240
Caught exception in looking up space point ptr... 0xd5df240, 0xd5df3d0
Caught exception in looking up space point ptr... 0x36416d00, 0x2681a120
Caught exception in looking up space point ptr... 0x2681a120, 0x36416d00
Caught exception in looking up space point ptr... 0x3d752140, 0x3d751fb0
Caught exception in looking up space point ptr... 0x3d751fb0, 0x3d752140
Caught exception in looking up space point ptr... 0x15cf9ec0, 0x15cf0ca0
Caught exception in looking up space point ptr... 0x15cf0ca0, 0x15cf9d90
Caught exception in looking up space point ptr... 0x161a7140, 0x161a6fb0
Caught exception in looking up space point ptr... 0x161a6fb0, 0x161a7140
Caught exception in looking up space point ptr... 0x601b03d0, 0x601b0240
Caught exception in looking up space point ptr... 0x601b0240, 0x601b03d0
Caught exception in looking up space point ptr... 0xd5df560, 0xd5df6f0
Caught exception in looking up space point ptr... 0xd5df6f0, 0xd5dfba0
Caught exception in looking up space point ptr... 0x5b12a260, 0x2ac3b030
Caught exception in looking up space point ptr... 0x2ac3b030, 0x2a0c10e0
Caught exception in looking up space point ptr... 0x2adfc930, 0x2adfb370
Caught exception in looking up space point ptr... 0x2adfb370, 0x2adfc930
Caught exception in looking up space point ptr... 0x35c58d80, 0x168d6860
Caught exception in looking up space point ptr... 0x168d6860, 0x35c58d80
Caught exception in looking up space point ptr... 0xd5d6f20, 0xd5d6af0
Caught exception in looking up space point ptr... 0xd5d6af0, 0xd5d6f20
Caught exception in looking up space point ptr... 0x149a13b0, 0x149af6d0
Caught exception in looking up space point ptr... 0x149af6d0, 0x149a9110
++++>>>> total num hits: 79888, num free: 76438
04-Feb-2026 00:49:20 GMT  Closed output file "np02vd_raw_run042421_0077_df-s02-d0_dw_0_20260203T141240_reco_stage0_20260204T004920_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         126.621       174.392       206.237       181.86        25.1885        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       5.9335e-05    0.00012361    0.000307574   9.1294e-05    6.52469e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                12.448        13.6392       16.0645       13.5239      0.770834        27     
produce:triggerrawdecoder:PDVDTriggerReader4       0.48347      0.495938      0.527585      0.491981      0.0099334       27     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0161791     0.0164496     0.0169058     0.0164549    0.000124946      27     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0805197     0.0874905     0.097687      0.0813702     0.0077026       27     
produce:beamevent:BeamEvent                      9.1948e-05    0.000182174   0.00112183    0.000135952   0.000186906      27     
produce:pdvddaphne:DAPHNEReaderPDVD                6.25822       7.60785       8.39921       7.66361      0.604642        27     
produce:ophit:OpHitFinder                         0.0361109     0.0649436     0.084151      0.0666813     0.0161533       27     
produce:wclsdatavd:WireCellToolkit                 82.3411        122.6        144.295       134.925       21.8962        27     
produce:gaushit:GausHitFinder                     0.814815       1.75627       2.96577       1.74697      0.549593        27     
produce:nhitsfilter:NumberOfHitsFilter           0.000261909   0.000480962   0.00102125    0.000453348   0.000139515      27     
produce:reco3d:SpacePointSolver                    6.45499       14.2189       23.5348       13.4245       4.23989        27     
produce:hitpdune:DisambigFromSpacePoints          0.0769731     0.210785      0.441231      0.185961      0.0921053       27     
produce:cluster3d:Cluster3D                        1.46374       4.33538       18.1292       3.08424       3.61552        27     
[art]:TriggerResults:TriggerResultInserter       2.3553e-05    3.72544e-05   0.000121397   3.2526e-05    1.83341e-05      27     
end_path:out1:RootOutput                          5.885e-06    1.00758e-05   3.4439e-05     9.363e-06    5.05521e-06      27     
end_path:out1:RootOutput(write)                    6.85622       9.33077       11.9403       9.78956       1.37888        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4532.45 MB
  Peak resident set size usage (VmHWM): 2631.35 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042421_0077_df-s02-d0_dw_0_20260203T141240_reco_stage0_20260204T004920_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042421_0077_df-s02-d0_dw_0_20260203T141240_reco_stage0_20260204T004920_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042421_0077_df-s02-d0_dw_0_20260203T141240_reco_stage0_20260204T004920_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770161360.0,
    "core.end_time": 1770166164.0,
    "core.events": [
      12534,
      12540,
      12546,
      12552,
      12558,
      12564,
      12570,
      12576,
      12582,
      12588,
      12594,
      12600,
      12606,
      12612,
      12618,
      12624,
      12630,
      12636,
      12642,
      12648,
      12654,
      12660,
      12666,
      12672,
      12678,
      12684,
      12690
    ],
    "core.event_count": 27,
    "core.first_event_number": 12534,
    "core.last_event_number": 12690,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42421
    ],
    "core.runs_subruns": [
      4242100001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042421_0077_df-s02-d0_dw_0_20260203T141240.hdf5"
    }
  ]
}
justIN time: 2026-02-04 04:33:16 UTC       justIN version: 01.06.00