justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303646.175@dunegpschedd01.fnal.gov

Jobsub ID303646.175@dunegpschedd01.fnal.gov
Workflow ID12718
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:44:24
SiteUK_Lancaster
EntryUBoone_UK_Lancaster_HEC_grendel_ce02
Last heartbeat2026-02-04 01:39:18
From worker nodeHostnamecomp22-07
cpuinfoIntel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 22:46:47
Input filesvd-protodune:np02vd_raw_run042422_0068_df-s04-d1_dw_0_20260203T155134.hdf5
JobscriptExit code0
Real time2h (9990s)
CPU time40m (2422s = 24%)
Max RSS bytes2607226880 (2486 MiB)
Outputting started2026-02-04 01:33:19
Output files
Finished2026-02-04 01:39:18
Saved logsjustin-logs:303646.175-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

b440
Caught exception in looking up space point ptr... 0x5078a340, 0x5078b690
Caught exception in looking up space point ptr... 0x102e4e90, 0x4e44d880
Caught exception in looking up space point ptr... 0x4e44d880, 0x102e4e90
Caught exception in looking up space point ptr... 0x40c0fe70, 0x40c06180
Caught exception in looking up space point ptr... 0x40c06180, 0x40c0b110
Caught exception in looking up space point ptr... 0x4fb535d0, 0x4fb536d0
Caught exception in looking up space point ptr... 0x4fb536d0, 0x404d9cc0
Caught exception in looking up space point ptr... 0x404d9cc0, 0x404d9ac0
Caught exception in looking up space point ptr... 0x404d9ac0, 0x4fb535d0
Caught exception in looking up space point ptr... 0x14260840, 0x3fa2d190
Caught exception in looking up space point ptr... 0x3fa2d190, 0x14260840
Caught exception in looking up space point ptr... 0x4a6cdec0, 0x4a6ca290
Caught exception in looking up space point ptr... 0x4a6cdfc0, 0x4a6cdec0
Caught exception in looking up space point ptr... 0x32e94080, 0x2b9e6700
Caught exception in looking up space point ptr... 0x2b9e6700, 0x32e94080
Caught exception in looking up space point ptr... 0x4f6d7280, 0x4f6d70f0
Caught exception in looking up space point ptr... 0x4f6d70f0, 0x4f6d7280
Caught exception in looking up space point ptr... 0x16855410, 0x168579d0
Caught exception in looking up space point ptr... 0x168579d0, 0x16857b00
Caught exception in looking up space point ptr... 0x13894d90, 0x13894cc0
Caught exception in looking up space point ptr... 0x13894380, 0x13894d90
Caught exception in looking up space point ptr... 0x31919070, 0x3a6b10e0
Caught exception in looking up space point ptr... 0x3a6b10e0, 0x31919070
Caught exception in looking up space point ptr... 0x13e61f30, 0x347fe520
Caught exception in looking up space point ptr... 0x347fe520, 0x13e61f30
Caught exception in looking up space point ptr... 0x3bc0acc0, 0x3bc0b030
Caught exception in looking up space point ptr... 0x3bc0d710, 0x3bc0acc0
Caught exception in looking up space point ptr... 0x11c5b400, 0x46c13b00
Caught exception in looking up space point ptr... 0x46c13b00, 0x11c5b400
Caught exception in looking up space point ptr... 0x60e919d0, 0x60e85be0
Caught exception in looking up space point ptr... 0x60e85be0, 0x60e92560
Caught exception in looking up space point ptr... 0x48be2310, 0x48be0cd0
Caught exception in looking up space point ptr... 0x48be0cd0, 0x48be2310
Caught exception in looking up space point ptr... 0x4a49ba70, 0x4a49b8b0
Caught exception in looking up space point ptr... 0x4a49b8b0, 0x4a49ba70
Caught exception in looking up space point ptr... 0x32e3c220, 0x32e3c3b0
Caught exception in looking up space point ptr... 0x32e3c3b0, 0x32e3c220
Caught exception in looking up space point ptr... 0x16847d50, 0x3abb2d60
Caught exception in looking up space point ptr... 0x3abb2d60, 0x60e8eee0
Caught exception in looking up space point ptr... 0x3abd5230, 0x60e80900
Caught exception in looking up space point ptr... 0x60e80900, 0x60e39b20
Caught exception in looking up space point ptr... 0x3bc07f70, 0x426ba690
Caught exception in looking up space point ptr... 0x426bb040, 0x3bc07f70
Caught exception in looking up space point ptr... 0x3bbdb9d0, 0x3bbdbec0
Caught exception in looking up space point ptr... 0x3bbdbec0, 0x3bbdb9d0
Caught exception in looking up space point ptr... 0x7aa7e90, 0x7aa77e0
Caught exception in looking up space point ptr... 0x7aa77e0, 0x7aa7e90
Caught exception in looking up space point ptr... 0x41d0dd30, 0x41608880
Caught exception in looking up space point ptr... 0x41608880, 0x41d0dd30
Caught exception in looking up space point ptr... 0x98257a0, 0xc6bf950
Caught exception in looking up space point ptr... 0xc6bf950, 0x98257a0
Caught exception in looking up space point ptr... 0x3d561e60, 0x3d5651e0
Caught exception in looking up space point ptr... 0x3d5651e0, 0x3d562c40
Caught exception in looking up space point ptr... 0x144282f0, 0x14427ff0
Caught exception in looking up space point ptr... 0x14427ff0, 0x144282f0
Caught exception in looking up space point ptr... 0x426b4a40, 0x426b46f0
Caught exception in looking up space point ptr... 0x426b46f0, 0x426b4a40
Caught exception in looking up space point ptr... 0x349cd6f0, 0x65e47d40
Caught exception in looking up space point ptr... 0x65e47d40, 0x349cd6f0
Caught exception in looking up space point ptr... 0x375f4860, 0x42cc68b0
Caught exception in looking up space point ptr... 0x42cc68b0, 0x3760cd20
Caught exception in looking up space point ptr... 0x3bdd1750, 0x451cb430
Caught exception in looking up space point ptr... 0x451cb430, 0x2de3b510
Caught exception in looking up space point ptr... 0x74d74e0, 0x74d4790
Caught exception in looking up space point ptr... 0x74d4790, 0x39dcdac0
++++>>>> total num hits: 46906, num free: 44942
04-Feb-2026 01:29:36 GMT  Closed output file "np02vd_raw_run042422_0068_df-s04-d1_dw_0_20260203T155134_reco_stage0_20260204T012936_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         229.059       358.548       723.778       322.68        105.754        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       9.9833e-05    0.000125701   0.000268211   0.00013164    3.2244e-05       27     
produce:tpcrawdecoder:PDVDTPCReader                115.02        227.807       375.767       208.212       71.2764        27     
produce:triggerrawdecoder:PDVDTriggerReader4      0.864106       1.54566       8.43263       1.14062       1.4073         27     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0237136     0.0664705     0.896469      0.0318283      0.16339        27     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.116015       0.14341      0.250589      0.129348      0.0334283       27     
produce:beamevent:BeamEvent                      0.000128869   0.000193375   0.00134093    0.000150841   0.000225696      27     
produce:pdvddaphne:DAPHNEReaderPDVD                17.0237       31.0853        87.72        26.7022       14.633         27     
produce:ophit:OpHitFinder                         0.0288666     0.0314372     0.0336194     0.0313788    0.00132442       27     
produce:wclsdatavd:WireCellToolkit                 61.3228       65.3742       72.9681       65.0069       2.60868        27     
produce:gaushit:GausHitFinder                     0.575868      0.898793       1.25174       0.89975      0.152372        27     
produce:nhitsfilter:NumberOfHitsFilter           0.000214551   0.000286648   0.000552595   0.000275156   6.49272e-05      27     
produce:reco3d:SpacePointSolver                    5.56339       8.70484       14.4457       8.52295       1.94322        27     
produce:hitpdune:DisambigFromSpacePoints          0.0595499     0.0938428     0.140034      0.0934189     0.0191842       27     
produce:cluster3d:Cluster3D                       0.590899       1.27605       2.03359       1.27115       0.35957        27     
[art]:TriggerResults:TriggerResultInserter       1.6954e-05    2.44927e-05    6.555e-05    2.2438e-05    8.97586e-06      27     
end_path:out1:RootOutput                          5.578e-06    7.72448e-06   1.8232e-05     7.121e-06    2.61994e-06      27     
end_path:out1:RootOutput(write)                    2.88206       21.4955       490.859       3.10497       92.054         27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4514.66 MB
  Peak resident set size usage (VmHWM): 2607.23 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042422_0068_df-s04-d1_dw_0_20260203T155134_reco_stage0_20260204T012936_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042422_0068_df-s04-d1_dw_0_20260203T155134_reco_stage0_20260204T012936_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042422_0068_df-s04-d1_dw_0_20260203T155134_reco_stage0_20260204T012936_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770158815.0,
    "core.end_time": 1770168669.0,
    "core.events": [
      11027,
      11033,
      11039,
      11045,
      11051,
      11057,
      11063,
      11069,
      11075,
      11081,
      11087,
      11093,
      11099,
      11105,
      11111,
      11117,
      11123,
      11129,
      11135,
      11141,
      11147,
      11153,
      11159,
      11165,
      11171,
      11177,
      11183
    ],
    "core.event_count": 27,
    "core.first_event_number": 11027,
    "core.last_event_number": 11183,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42422
    ],
    "core.runs_subruns": [
      4242200001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042422_0068_df-s04-d1_dw_0_20260203T155134.hdf5"
    }
  ]
}
justIN time: 2026-02-04 04:29:54 UTC       justIN version: 01.06.00