justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303651.37@dunegpschedd01.fnal.gov

Jobsub ID303651.37@dunegpschedd01.fnal.gov
Workflow ID12717
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:50:24
SiteUK_Lancaster
EntryUBoone_UK_Lancaster_HEC_grendel_ce02
Last heartbeat2026-02-04 02:08:23
From worker nodeHostnamecomp22-07
cpuinfoIntel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:18:03
Input filesvd-protodune:np02vd_raw_run042421_0102_df-s04-d0_dw_0_20260203T142610.hdf5
JobscriptExit code0
Real time2h (10003s)
CPU time40m (2452s = 24%)
Max RSS bytes2610532352 (2489 MiB)
Outputting started2026-02-04 02:04:54
Output files
Finished2026-02-04 02:08:23
Saved logsjustin-logs:303651.37-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

r... 0x3aed9420, 0x261a5fc0
Caught exception in looking up space point ptr... 0x261a5fc0, 0x2ff186f0
Caught exception in looking up space point ptr... 0x2ff186f0, 0x3aedb760
Caught exception in looking up space point ptr... 0x3aedb760, 0x3aec4e60
Caught exception in looking up space point ptr... 0x531d6fd0, 0x531d6c60
Caught exception in looking up space point ptr... 0x531d6c60, 0x531d68f0
Caught exception in looking up space point ptr... 0x93ceda0, 0x93daff0
Caught exception in looking up space point ptr... 0x93daff0, 0x93daec0
Caught exception in looking up space point ptr... 0x261a29a0, 0x261a2610
Caught exception in looking up space point ptr... 0x261a2610, 0x261a2110
Caught exception in looking up space point ptr... 0xe3a94c0, 0x5425d8c0
Caught exception in looking up space point ptr... 0x5425d8c0, 0x524b5ce0
Caught exception in looking up space point ptr... 0x524b5ce0, 0xe3a8bb0
Caught exception in looking up space point ptr... 0x2619e650, 0x3c54b4a0
Caught exception in looking up space point ptr... 0x3c54b4a0, 0x2619e650
Caught exception in looking up space point ptr... 0x40d194e0, 0x3aedbb00
Caught exception in looking up space point ptr... 0x3aedbb00, 0x40d194e0
Caught exception in looking up space point ptr... 0x27d352a0, 0x27d35620
Caught exception in looking up space point ptr... 0x27d35620, 0x27d352a0
Caught exception in looking up space point ptr... 0x49b62a50, 0x49b62940
Caught exception in looking up space point ptr... 0x49b61a50, 0x49b62a50
Caught exception in looking up space point ptr... 0x35614d10, 0x121c5830
Caught exception in looking up space point ptr... 0x121c5830, 0x27c9e7e0
Caught exception in looking up space point ptr... 0x9c2af30, 0x9c2b0c0
Caught exception in looking up space point ptr... 0x9c2b0c0, 0x27d1e6d0
Caught exception in looking up space point ptr... 0x2a1fcbf0, 0x2a1fd410
Caught exception in looking up space point ptr... 0x2a1fd410, 0x2a1fd5a0
Caught exception in looking up space point ptr... 0x2a1fd5a0, 0x2a1fda50
Caught exception in looking up space point ptr... 0x10411d70, 0x104127d0
Caught exception in looking up space point ptr... 0x104127d0, 0x10411d70
Caught exception in looking up space point ptr... 0x58ec2700, 0x58ebdec0
Caught exception in looking up space point ptr... 0x58ebdec0, 0x58ec4c20
Caught exception in looking up space point ptr... 0xd67f550, 0xd67f450
Caught exception in looking up space point ptr... 0xd67f450, 0xd67f550
Caught exception in looking up space point ptr... 0x351797c0, 0x351794c0
Caught exception in looking up space point ptr... 0x351794c0, 0x35179630
Caught exception in looking up space point ptr... 0x8a00160, 0x89ffe40
Caught exception in looking up space point ptr... 0x89ffe40, 0x8a00160
Caught exception in looking up space point ptr... 0x40d243e0, 0x40d24570
Caught exception in looking up space point ptr... 0x40d24570, 0x40d243e0
Caught exception in looking up space point ptr... 0x32abee50, 0x32aba4f0
Caught exception in looking up space point ptr... 0x32aba4f0, 0x32aba3a0
Caught exception in looking up space point ptr... 0x38990790, 0x456d2b60
Caught exception in looking up space point ptr... 0x456d2b60, 0x38990790
Caught exception in looking up space point ptr... 0x12115bc0, 0x14d819f0
Caught exception in looking up space point ptr... 0x14d819f0, 0x12115bc0
Caught exception in looking up space point ptr... 0x8a02880, 0x8a1d770
Caught exception in looking up space point ptr... 0x8a1d770, 0x8a026f0
Caught exception in looking up space point ptr... 0x27cf19a0, 0x27cf1680
Caught exception in looking up space point ptr... 0x27cf1680, 0x27cf19a0
Caught exception in looking up space point ptr... 0x27cf19a0, 0x27cf1810
Caught exception in looking up space point ptr... 0x27cf1810, 0x27cf19a0
Caught exception in looking up space point ptr... 0x9c21250, 0x9c20f30
Caught exception in looking up space point ptr... 0x9c20f30, 0x9c210c0
Caught exception in looking up space point ptr... 0xd689000, 0xd688e70
Caught exception in looking up space point ptr... 0xd688e70, 0xd689000
Caught exception in looking up space point ptr... 0xd689000, 0xd688ce0
Caught exception in looking up space point ptr... 0xd688ce0, 0xd689000
Caught exception in looking up space point ptr... 0x40fa45d0, 0x532e4c40
Caught exception in looking up space point ptr... 0x532e4c40, 0x2b1c0de0
Caught exception in looking up space point ptr... 0x458f11f0, 0x45923030
Caught exception in looking up space point ptr... 0x45923030, 0x540bc720
Caught exception in looking up space point ptr... 0x4971ea00, 0x3264de90
Caught exception in looking up space point ptr... 0x3264de90, 0x4971ea00
Caught exception in looking up space point ptr... 0x57f8da60, 0x27d1eb80
Caught exception in looking up space point ptr... 0x27d1eb80, 0x9c18c70
++++>>>> total num hits: 52605, num free: 51436
04-Feb-2026 02:03:54 GMT  Closed output file "np02vd_raw_run042421_0102_df-s04-d0_dw_0_20260203T142610_reco_stage0_20260204T020354_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         229.182       365.102       755.855       343.341       106.567        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       9.9405e-05    0.000116232   0.000176096   0.000109428   1.75243e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                132.895       222.404       382.49        204.445       66.1756        27     
produce:triggerrawdecoder:PDVDTriggerReader4      0.749083       1.16976       1.74586       1.11885      0.279006        27     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0235913     0.0359007     0.114107      0.0325773     0.0202795       27     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.116268       0.14252      0.231939      0.139908      0.0303392       27     
produce:beamevent:BeamEvent                      0.000128513   0.000192507   0.00130939    0.000144979   0.000219704      27     
produce:pdvddaphne:DAPHNEReaderPDVD                14.0199       34.0665       113.968       25.5971       22.116         27     
produce:ophit:OpHitFinder                         0.023854      0.0304762     0.0340637     0.0305324    0.00207346       27     
produce:wclsdatavd:WireCellToolkit                 61.5783       66.2381       86.4972       65.0191       5.38759        27     
produce:gaushit:GausHitFinder                     0.665653      0.991034       1.36661      0.971556      0.170179        27     
produce:nhitsfilter:NumberOfHitsFilter           0.000214867   0.000318468   0.000563883   0.000303166   6.5032e-05       27     
produce:reco3d:SpacePointSolver                    4.62374       12.5485       57.1376       9.81519       9.72487        27     
produce:hitpdune:DisambigFromSpacePoints          0.0677271     0.107155      0.160173      0.104788      0.0215069       27     
produce:cluster3d:Cluster3D                       0.822883       1.58648       2.88116       1.43333      0.528105        27     
[art]:TriggerResults:TriggerResultInserter       1.6839e-05    2.3865e-05    7.6256e-05    2.1003e-05    1.09463e-05      27     
end_path:out1:RootOutput                          5.654e-06    9.95867e-06   2.1415e-05     8.735e-06    3.24245e-06      27     
end_path:out1:RootOutput(write)                    2.90837       25.7574       463.319       3.20394       87.2801        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4515.88 MB
  Peak resident set size usage (VmHWM): 2610.53 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042421_0102_df-s04-d0_dw_0_20260203T142610_reco_stage0_20260204T020354_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042421_0102_df-s04-d0_dw_0_20260203T142610_reco_stage0_20260204T020354_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042421_0102_df-s04-d0_dw_0_20260203T142610_reco_stage0_20260204T020354_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770160692.0,
    "core.end_time": 1770170680.0,
    "core.events": [
      16588,
      16594,
      16600,
      16606,
      16612,
      16618,
      16624,
      16630,
      16636,
      16642,
      16648,
      16654,
      16660,
      16666,
      16672,
      16678,
      16684,
      16690,
      16696,
      16702,
      16708,
      16714,
      16720,
      16726,
      16732,
      16738,
      16744
    ],
    "core.event_count": 27,
    "core.first_event_number": 16588,
    "core.last_event_number": 16744,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42421
    ],
    "core.runs_subruns": [
      4242100001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042421_0102_df-s04-d0_dw_0_20260203T142610.hdf5"
    }
  ]
}
justIN time: 2026-02-04 04:33:14 UTC       justIN version: 01.06.00