justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 295859.0@dunegpschedd02.fnal.gov

Jobsub ID295859.0@dunegpschedd02.fnal.gov
Workflow ID12428
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2026-01-27 08:49:10
SiteFR_CCIN2P3
EntryDUNE_FR_CCIN2P3_cccondorce02
Last heartbeat2026-01-27 09:57:41
From worker nodeHostnameccwcondor0004
cpuinfoAMD EPYC 9334 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit106200 (29 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-01-27 08:49:56
Input filesvd-protodune:np02vd_raw_run039636_0245_df-s04-d0_dw_0_20250926T004622.hdf5
JobscriptExit code0
Real time1h (3687s)
CPU time58m (3480s = 94%)
Max RSS bytes2860306432 (2727 MiB)
Outputting started2026-01-27 09:51:25
Output files
Finished2026-01-27 09:57:41
Saved logsjustin-logs:295859.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

up space point ptr... 0x92695e0, 0x9268eb0
Caught exception in looking up space point ptr... 0x9269270, 0x92695e0
Caught exception in looking up space point ptr... 0x10d96190, 0x10d9bbe0
Caught exception in looking up space point ptr... 0x10d9b520, 0x10d96190
Caught exception in looking up space point ptr... 0x120d5850, 0x120d5a50
Caught exception in looking up space point ptr... 0x120d5a50, 0x120d5850
Caught exception in looking up space point ptr... 0x8fb0f70, 0x2b11e230
Caught exception in looking up space point ptr... 0x2b11e230, 0x8fb0f70
Caught exception in looking up space point ptr... 0x1b2f2a30, 0x1e4b6190
Caught exception in looking up space point ptr... 0x1e4b6190, 0x1c34e330
Caught exception in looking up space point ptr... 0x374ccf10, 0x2fb1bdf0
Caught exception in looking up space point ptr... 0x2fb1c040, 0x374ccf10
Caught exception in looking up space point ptr... 0xd040ac0, 0x26335ec0
Caught exception in looking up space point ptr... 0x26335ec0, 0xd040ac0
Caught exception in looking up space point ptr... 0xad0ab20, 0x242064f0
Caught exception in looking up space point ptr... 0x242064f0, 0xad0a740
Caught exception in looking up space point ptr... 0xda0dbf0, 0x282d5540
Caught exception in looking up space point ptr... 0x282d5540, 0xda0dbf0
Caught exception in looking up space point ptr... 0x9c38e00, 0x9c3b060
Caught exception in looking up space point ptr... 0x9c3b060, 0x9c3af60
Caught exception in looking up space point ptr... 0x9c3af60, 0x9c388b0
Caught exception in looking up space point ptr... 0x12018e10, 0x204da040
Caught exception in looking up space point ptr... 0x204da040, 0x1f92cae0
Caught exception in looking up space point ptr... 0x259ff450, 0x25bc18d0
Caught exception in looking up space point ptr... 0x25bc18d0, 0x259ff450
Caught exception in looking up space point ptr... 0x25b09560, 0x25671640
Caught exception in looking up space point ptr... 0x128d40d0, 0x25b09560
Caught exception in looking up space point ptr... 0x1c34f220, 0x256f2ed0
Caught exception in looking up space point ptr... 0x256f2ed0, 0x1f602f20
Caught exception in looking up space point ptr... 0x1c65c160, 0x8010ec0
Caught exception in looking up space point ptr... 0x8010ec0, 0x2cdbc4f0
Caught exception in looking up space point ptr... 0x2b8787a0, 0x122116a0
Caught exception in looking up space point ptr... 0x122116a0, 0x12210030
Caught exception in looking up space point ptr... 0x12210030, 0x1220ee30
Caught exception in looking up space point ptr... 0x10f63420, 0x1b1180b0
Caught exception in looking up space point ptr... 0x1b1180b0, 0x10f63420
Caught exception in looking up space point ptr... 0x80bd750, 0xe1b17c0
Caught exception in looking up space point ptr... 0xe1b17c0, 0x80bd750
Caught exception in looking up space point ptr... 0x945ba10, 0x1bd9a710
Caught exception in looking up space point ptr... 0x1bd9a710, 0x2e875c90
Caught exception in looking up space point ptr... 0xe683aa0, 0xe683ff0
Caught exception in looking up space point ptr... 0xe683ff0, 0xe684180
Caught exception in looking up space point ptr... 0x36714380, 0x2de92db0
Caught exception in looking up space point ptr... 0x2de92db0, 0x36714380
Caught exception in looking up space point ptr... 0x25598eb0, 0x2b888210
Caught exception in looking up space point ptr... 0x2b888210, 0x25598eb0
Caught exception in looking up space point ptr... 0x220e66a0, 0x13e2e000
Caught exception in looking up space point ptr... 0x13e2e000, 0x220e66a0
Caught exception in looking up space point ptr... 0x117b6e30, 0xa41b6e0
Caught exception in looking up space point ptr... 0xa41b6e0, 0x117b6e30
Caught exception in looking up space point ptr... 0x292260b0, 0x1f119ae0
Caught exception in looking up space point ptr... 0x1f119ae0, 0x292260b0
Caught exception in looking up space point ptr... 0xdbd7770, 0x3134c640
Caught exception in looking up space point ptr... 0x3134c640, 0xdbd7770
Caught exception in looking up space point ptr... 0x2de82e30, 0x29255a80
Caught exception in looking up space point ptr... 0x29255a80, 0x20c34ef0
Caught exception in looking up space point ptr... 0x1ddf6210, 0x1ddf60c0
Caught exception in looking up space point ptr... 0x1ddf60c0, 0x1ddf6210
Caught exception in looking up space point ptr... 0x21cbc870, 0x21cbc6e0
Caught exception in looking up space point ptr... 0x21cbc6e0, 0x21cbc550
Caught exception in looking up space point ptr... 0x13e204b0, 0x13e202d0
Caught exception in looking up space point ptr... 0x13e202d0, 0x13e204b0
++++>>>> total num hits: 71602, num free: 68926
27-Jan-2026 10:51:08 CET  Closed output file "np02vd_raw_run039636_0245_df-s04-d0_dw_0_20250926T004622_reco_stage0_20260127T095108_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         52.6251       74.1821       254.264       62.3313       34.7625        48     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       8.9254e-05    0.000151579   0.000463324   0.000114076   8.28777e-05      48     
produce:tpcrawdecoder:PDVDTPCReader                2.7468        3.33235       5.87059       3.20768      0.523982        48     
produce:triggerrawdecoder:PDVDTriggerReader4      0.0617065     0.074106       0.10176      0.0705973    0.00991725       48     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.00375062    0.00461894     0.0115989    0.00419089    0.00128682       48     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0179041     0.020284      0.0318278     0.0192448    0.00267226       48     
produce:beamevent:BeamEvent                      0.000109003   0.000227755   0.00143641    0.000177416   0.00019176       48     
produce:pdvddaphne:DAPHNEReaderPDVD              0.000263214   0.000369745   0.00099541    0.000345598   0.000109744      48     
produce:ophit:OpHitFinder                        7.5322e-05    0.000198545   0.00154682    0.000137972   0.000235781      48     
produce:wclsdatavd:WireCellToolkit                 39.7267       45.662        50.3257       45.7715       2.14363        48     
produce:gaushit:GausHitFinder                     0.332431      0.686189       2.43144      0.549216      0.392062        48     
produce:nhitsfilter:NumberOfHitsFilter           0.000108953   0.000416487    0.0109678    0.000166008   0.00154118       48     
produce:reco3d:SpacePointSolver                    2.10282       7.28638       47.3691       4.66475       8.38775        48     
produce:hitpdune:DisambigFromSpacePoints          0.0253913     0.157548       1.53245      0.0766677     0.266501        48     
produce:cluster3d:Cluster3D                        0.63162       14.108        186.806       4.84017       29.1446        48     
[art]:TriggerResults:TriggerResultInserter       3.2398e-05    4.91985e-05   0.000154211   4.2744e-05    2.02479e-05      48     
end_path:out1:RootOutput                          4.567e-06    1.06599e-05   4.0711e-05    1.01805e-05   5.49141e-06      48     
end_path:out1:RootOutput(write)                    2.09962       2.81875       5.73432       2.54784      0.838959        48     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4839.1 MB
  Peak resident set size usage (VmHWM): 2860.31 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run039636_0245_df-s04-d0_dw_0_20250926T004622_reco_stage0_20260127T095108_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run039636_0245_df-s04-d0_dw_0_20250926T004622_reco_stage0_20260127T095108_offline.root
Ran successfully
{
  "name": "np02vd_raw_run039636_0245_df-s04-d0_dw_0_20250926T004622_reco_stage0_20260127T095108_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_neutrino.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1769503821.0,
    "core.end_time": 1769507471.0,
    "core.events": [
      46832,
      46836,
      46840,
      46844,
      46848,
      46852,
      46856,
      46860,
      46864,
      46868,
      46872,
      46876,
      46880,
      46884,
      46888,
      46892,
      46896,
      46900,
      46904,
      46908,
      46912,
      46916,
      46920,
      46924,
      46928,
      46932,
      46936,
      46940,
      46944,
      46948,
      46952,
      46956,
      46960,
      46964,
      46968,
      46972,
      46976,
      46980,
      46984,
      46988,
      46992,
      46996,
      47000,
      47004,
      47008,
      47012,
      47016,
      47020
    ],
    "core.event_count": 48,
    "core.first_event_number": 46832,
    "core.last_event_number": 47020,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      39636
    ],
    "core.runs_subruns": [
      3963600001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run039636_0245_df-s04-d0_dw_0_20250926T004622.hdf5"
    }
  ]
}
justIN time: 2026-02-04 07:43:42 UTC       justIN version: 01.06.00