justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297923.97@dunegpschedd02.fnal.gov

Jobsub ID297923.97@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 23:00:26
SiteUK_RAL-Tier1
EntryLIGO_UK_RAL_arc_ce03
Last heartbeat2026-02-04 01:03:59
From worker nodeHostnamedune001-1534683.0-lcg2412.gridpp.rl.ac.uk
cpuinfoAMD EPYC 7452 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit216000 (60 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-04 00:01:39
Input filesvd-protodune:np02vd_raw_run042433_0081_df-s02-d1_dw_0_20260203T191254.hdf5
JobscriptExit code0
Real time1h (3634s)
CPU time54m (3280s = 90%)
Max RSS bytes3433779200 (3274 MiB)
Outputting started2026-02-04 01:02:14
Output files
Finished2026-02-04 01:03:59
Saved logsjustin-logs:297923.97-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

up space point ptr... 0x9801b8a0, 0x6872a620
Caught exception in looking up space point ptr... 0x7e08f0f0, 0x7e08efa0
Caught exception in looking up space point ptr... 0x7e08efa0, 0x7e08f0f0
Caught exception in looking up space point ptr... 0x60d374f0, 0x60d330c0
Caught exception in looking up space point ptr... 0x60d330c0, 0x60d2a200
Caught exception in looking up space point ptr... 0xb5547df0, 0x41e530e0
Caught exception in looking up space point ptr... 0x41e530e0, 0x41e53400
Caught exception in looking up space point ptr... 0xb6635470, 0xb664e3b0
Caught exception in looking up space point ptr... 0x7d81b3e0, 0xb6635470
Caught exception in looking up space point ptr... 0x484e84c0, 0x484e7dd0
Caught exception in looking up space point ptr... 0x484e7dd0, 0x484e8020
Caught exception in looking up space point ptr... 0x958c4c50, 0x3d041e20
Caught exception in looking up space point ptr... 0x3d041e20, 0x958c4c50
Caught exception in looking up space point ptr... 0x97e50a10, 0x5daa8e10
Caught exception in looking up space point ptr... 0x5daa8e10, 0x5daa7570
Caught exception in looking up space point ptr... 0x62bd97f0, 0x62bd3830
Caught exception in looking up space point ptr... 0x62bd3830, 0x62bd97f0
Caught exception in looking up space point ptr... 0x3f2ad440, 0x3f2aa550
Caught exception in looking up space point ptr... 0x3f2aa550, 0x3f2ad440
Caught exception in looking up space point ptr... 0xa9777a30, 0xa97778a0
Caught exception in looking up space point ptr... 0xa97778a0, 0xa9777530
Caught exception in looking up space point ptr... 0x815a3f30, 0xa48714c0
Caught exception in looking up space point ptr... 0xa48714c0, 0x60f30bb0
Caught exception in looking up space point ptr... 0x7f1c7bf0, 0x42d76c80
Caught exception in looking up space point ptr... 0x42d76c80, 0xb7a19e40
Caught exception in looking up space point ptr... 0x64c67c50, 0x6970b230
Caught exception in looking up space point ptr... 0x6970b230, 0x64c67c50
Caught exception in looking up space point ptr... 0x64129130, 0x6d6e1f80
Caught exception in looking up space point ptr... 0x6d6e1f80, 0x3cfc6b50
Caught exception in looking up space point ptr... 0xa4502b70, 0x3d063370
Caught exception in looking up space point ptr... 0x3d063370, 0xa4502b70
Caught exception in looking up space point ptr... 0x48318810, 0x98745f30
Caught exception in looking up space point ptr... 0x98745f30, 0x48318810
Caught exception in looking up space point ptr... 0x4ce75f80, 0x4ce76110
Caught exception in looking up space point ptr... 0x4ce76110, 0x4ce75f80
Caught exception in looking up space point ptr... 0x667c04e0, 0x98a4cff0
Caught exception in looking up space point ptr... 0x98a4cff0, 0x667c04e0
Caught exception in looking up space point ptr... 0x6f5cd270, 0x62bc3d40
Caught exception in looking up space point ptr... 0x62bc3d40, 0x6f5cd270
Caught exception in looking up space point ptr... 0x601aeec0, 0x97a35d60
Caught exception in looking up space point ptr... 0x97a35d60, 0x601aeec0
Caught exception in looking up space point ptr... 0x601aeec0, 0x97a359a0
Caught exception in looking up space point ptr... 0x97a359a0, 0x601aeec0
Caught exception in looking up space point ptr... 0x601aeec0, 0x97a35b80
Caught exception in looking up space point ptr... 0x97a35b80, 0x601aeec0
Caught exception in looking up space point ptr... 0x90f0d810, 0x90f0b1a0
Caught exception in looking up space point ptr... 0x90f0b1a0, 0x90f0b550
Caught exception in looking up space point ptr... 0x5e2ac8d0, 0x5e2adc10
Caught exception in looking up space point ptr... 0x5e2adc10, 0x5e2ac8d0
Caught exception in looking up space point ptr... 0xb79fc8f0, 0x6970b8d0
Caught exception in looking up space point ptr... 0x6970b8d0, 0xb79fc8f0
Caught exception in looking up space point ptr... 0x3facc880, 0x97a65300
Caught exception in looking up space point ptr... 0x97a65300, 0x3facc880
Caught exception in looking up space point ptr... 0x99578ce0, 0x99578b00
Caught exception in looking up space point ptr... 0x99578b00, 0x99578ce0
Caught exception in looking up space point ptr... 0x7e08a110, 0x7e08e2b0
Caught exception in looking up space point ptr... 0x7e08e2b0, 0x7e08e520
Caught exception in looking up space point ptr... 0x6f772c70, 0x644bd3d0
Caught exception in looking up space point ptr... 0x644bd3d0, 0x3f29e8c0
Caught exception in looking up space point ptr... 0x6ebccc20, 0x6ebc6fd0
Caught exception in looking up space point ptr... 0x6ebc6fd0, 0x6ebcc9d0
Caught exception in looking up space point ptr... 0x416e7050, 0x8de12c30
Caught exception in looking up space point ptr... 0x8de12c30, 0x416e7050
Caught exception in looking up space point ptr... 0x90c56380, 0x90c56830
Caught exception in looking up space point ptr... 0x90c56830, 0x90c566a0
Caught exception in looking up space point ptr... 0x90c28520, 0xb08ce860
Caught exception in looking up space point ptr... 0xb08ce860, 0x90c28520
++++>>>> total num hits: 84900, num free: 80855
04-Feb-2026 01:02:03 UTC  Closed output file "np02vd_raw_run042433_0081_df-s02-d1_dw_0_20260203T191254_reco_stage0_20260204T010203_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         158.729       185.933       212.02        185.905       13.9303        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       9.6602e-05    0.000148182   0.000273816   0.00013251    5.56473e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                16.796        20.4806       23.8202       20.8171       2.30003        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.683864      0.696148      0.718572      0.694346     0.00969643       19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0230618     0.0233349     0.0237323     0.0232698    0.00017106       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.023082      0.0233673     0.0239235     0.0232954    0.000269957      19     
produce:beamevent:BeamEvent                      0.000126638   0.000273713   0.00122327    0.000200197   0.000237522      19     
produce:pdvddaphne:DAPHNEReaderPDVD                3.41956       3.97297       4.77796       4.00732      0.373905        19     
produce:ophit:OpHitFinder                         0.0321016     0.0409612     0.0488877     0.0429469    0.00501027       19     
produce:wclsdatavd:WireCellToolkit                 109.568       127.231       145.54        127.368       9.78001        19     
produce:gaushit:GausHitFinder                     0.780193       1.51107       2.5824        1.43475      0.469603        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000388332   0.000526367   0.00102579    0.000481006   0.000161562      19     
produce:reco3d:SpacePointSolver                    11.3318       22.5982       39.3718       22.3891       6.97043        19     
produce:hitpdune:DisambigFromSpacePoints          0.126071      0.234258      0.433892      0.236663      0.0774397       19     
produce:cluster3d:Cluster3D                        1.28574       3.03854       6.24364       2.75197       1.23871        19     
[art]:TriggerResults:TriggerResultInserter       2.8333e-05    4.54287e-05   0.000115958   3.7721e-05    2.21715e-05      19     
end_path:out1:RootOutput                          4.499e-06    2.66892e-05   0.000343798    7.734e-06    7.50139e-05      19     
end_path:out1:RootOutput(write)                    4.86598       6.02282       7.52521       5.8826       0.775027        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5568.05 MB
  Peak resident set size usage (VmHWM): 3433.78 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0081_df-s02-d1_dw_0_20260203T191254_reco_stage0_20260204T010203_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0081_df-s02-d1_dw_0_20260203T191254_reco_stage0_20260204T010203_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0081_df-s02-d1_dw_0_20260203T191254_reco_stage0_20260204T010203_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770163312.0,
    "core.end_time": 1770166925.0,
    "core.events": [
      9241,
      9247,
      9253,
      9259,
      9265,
      9271,
      9277,
      9283,
      9289,
      9295,
      9301,
      9307,
      9313,
      9319,
      9325,
      9331,
      9337,
      9343,
      9349
    ],
    "core.event_count": 19,
    "core.first_event_number": 9241,
    "core.last_event_number": 9349,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0081_df-s02-d1_dw_0_20260203T191254.hdf5"
    }
  ]
}
justIN time: 2026-02-04 07:41:49 UTC       justIN version: 01.06.00