justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303646.179@dunegpschedd01.fnal.gov

Jobsub ID303646.179@dunegpschedd01.fnal.gov
Workflow ID12718
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:44:24
SiteUK_Lancaster
EntryUBoone_UK_Lancaster_HEC_grendel_ce02
Last heartbeat2026-02-03 23:39:42
From worker nodeHostnamecomp22-07
cpuinfoIntel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 22:46:03
Input filesvd-protodune:np02vd_raw_run042422_0079_df-s03-d1_dw_0_20260203T155730.hdf5
JobscriptExit code0
Real time51m (3064s)
CPU time39m (2382s = 77%)
Max RSS bytes2591629312 (2471 MiB)
Outputting started2026-02-03 23:37:10
Output files
Finished2026-02-03 23:39:42
Saved logsjustin-logs:303646.179-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

Caught exception in looking up space point ptr... 0x2c9d92b0, 0x21663de0
Caught exception in looking up space point ptr... 0x21663de0, 0x2c9d92b0
Caught exception in looking up space point ptr... 0x2efbd6b0, 0x2efbd790
Caught exception in looking up space point ptr... 0x2efbd790, 0x2efbd570
Caught exception in looking up space point ptr... 0x2e053700, 0x2e053570
Caught exception in looking up space point ptr... 0x2e053570, 0x2e053700
Caught exception in looking up space point ptr... 0x4e7fbe50, 0x4e7f95d0
Caught exception in looking up space point ptr... 0x4e7f95d0, 0x4e7fbe50
Caught exception in looking up space point ptr... 0x4e7fbe50, 0x4e7fbd00
Caught exception in looking up space point ptr... 0x4e7fbd00, 0x4e7fbe50
Caught exception in looking up space point ptr... 0x30fa34b0, 0x30fa3780
Caught exception in looking up space point ptr... 0x30fa3780, 0x30fa34b0
Caught exception in looking up space point ptr... 0x13b317c0, 0x33fdf700
Caught exception in looking up space point ptr... 0x33fdf700, 0x13b317c0
Caught exception in looking up space point ptr... 0x4e7fb910, 0x4e7fb7f0
Caught exception in looking up space point ptr... 0x4e7fb7f0, 0x4e7fb910
Caught exception in looking up space point ptr... 0x46f7bb60, 0x11a3f740
Caught exception in looking up space point ptr... 0x11a3f740, 0x46f7bb60
Caught exception in looking up space point ptr... 0x47bc3d00, 0x47bc3e00
Caught exception in looking up space point ptr... 0x47bc3e00, 0x47bc3f00
Caught exception in looking up space point ptr... 0x56a414a0, 0x56a41860
Caught exception in looking up space point ptr... 0x56a41860, 0x56a414a0
Caught exception in looking up space point ptr... 0x3c2dd0a0, 0x3c2dcce0
Caught exception in looking up space point ptr... 0x3520bb60, 0x3c2dd0a0
Caught exception in looking up space point ptr... 0x3f551270, 0x3f552dc0
Caught exception in looking up space point ptr... 0x3f552dc0, 0x3f551270
Caught exception in looking up space point ptr... 0x3002e050, 0x3002ecc0
Caught exception in looking up space point ptr... 0x3002ecc0, 0x3002e050
Caught exception in looking up space point ptr... 0x2ac49270, 0x3ebc81c0
Caught exception in looking up space point ptr... 0x3ebc81c0, 0x2ac48450
Caught exception in looking up space point ptr... 0x2c4c7090, 0x562e63c0
Caught exception in looking up space point ptr... 0x562e63c0, 0x2c4c56e0
Caught exception in looking up space point ptr... 0x299a7060, 0x299a6380
Caught exception in looking up space point ptr... 0x299a6380, 0x299a7060
Caught exception in looking up space point ptr... 0x3c8e2930, 0x26586550
Caught exception in looking up space point ptr... 0x26586550, 0x26584b60
Caught exception in looking up space point ptr... 0x139716d0, 0x13973700
Caught exception in looking up space point ptr... 0x1396def0, 0x139716d0
Caught exception in looking up space point ptr... 0x33fd6000, 0x33fd29d0
Caught exception in looking up space point ptr... 0x33fd29d0, 0x33fd6000
Caught exception in looking up space point ptr... 0x45d32270, 0x45d32010
Caught exception in looking up space point ptr... 0x45d32010, 0x45d32270
Caught exception in looking up space point ptr... 0x26f2d720, 0x26f2ca20
Caught exception in looking up space point ptr... 0x26f2d9c0, 0x26f2d720
Caught exception in looking up space point ptr... 0x229f51b0, 0x229f5710
Caught exception in looking up space point ptr... 0x229f5710, 0x229f5960
Caught exception in looking up space point ptr... 0x229f5960, 0x229f5810
Caught exception in looking up space point ptr... 0xe4b4450, 0x3e388700
Caught exception in looking up space point ptr... 0x3e388700, 0x41c2d0e0
Caught exception in looking up space point ptr... 0x41c2d0e0, 0x4e984590
Caught exception in looking up space point ptr... 0x4e984590, 0xe4b4450
Caught exception in looking up space point ptr... 0x2efb9eb0, 0x2efb9dd0
Caught exception in looking up space point ptr... 0x2efb9dd0, 0x2efb9eb0
Caught exception in looking up space point ptr... 0x1567ddb0, 0x1567df40
Caught exception in looking up space point ptr... 0x1567df40, 0x1567ddb0
Caught exception in looking up space point ptr... 0x2efbb630, 0x46f63bc0
Caught exception in looking up space point ptr... 0x46f63bc0, 0x46f64640
Caught exception in looking up space point ptr... 0x4dac84d0, 0x4dac8770
Caught exception in looking up space point ptr... 0x4dac8770, 0x4dac8690
Caught exception in looking up space point ptr... 0x22a01d80, 0x22a07a10
Caught exception in looking up space point ptr... 0x22a07a10, 0x22a01d80
Caught exception in looking up space point ptr... 0x22a01d80, 0x22a07880
Caught exception in looking up space point ptr... 0x22a07880, 0x22a01d80
Caught exception in looking up space point ptr... 0x96f18b0, 0xd0a8ba0
Caught exception in looking up space point ptr... 0xd0a8ba0, 0x96f18b0
++++>>>> total num hits: 87372, num free: 84000
03-Feb-2026 23:36:56 GMT  Closed output file "np02vd_raw_run042422_0079_df-s03-d1_dw_0_20260203T155730_reco_stage0_20260203T233656_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         94.818        109.973       176.731       102.388       22.4004        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       7.6994e-05    0.000144096   0.000259107   0.000135734   3.30814e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                14.8586       23.424        84.9449       16.188        18.8436        27     
produce:triggerrawdecoder:PDVDTriggerReader4      0.705239       0.82814       1.42485      0.771543      0.153424        27     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0233888     0.0300438     0.0492084     0.0261413    0.00805965       27     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.115212      0.127918      0.182603      0.118881      0.018632        27     
produce:beamevent:BeamEvent                      8.2119e-05    0.000223756   0.00134784    0.000165014   0.00022487       27     
produce:pdvddaphne:DAPHNEReaderPDVD                4.78528       6.23332       12.3568       5.49719       2.1268         27     
produce:ophit:OpHitFinder                         0.0262021     0.0317047     0.0361712     0.032036     0.00200496       27     
produce:wclsdatavd:WireCellToolkit                 61.9661       64.4824       66.6078       64.3917       1.16514        27     
produce:gaushit:GausHitFinder                     0.553258      0.856559       1.19793      0.790984      0.184933        27     
produce:nhitsfilter:NumberOfHitsFilter           0.000166207   0.000290589   0.000468721   0.000276977   6.69099e-05      27     
produce:reco3d:SpacePointSolver                    3.97582       8.40784       17.5901       7.13308       3.10597        27     
produce:hitpdune:DisambigFromSpacePoints          0.0466521     0.0905307      0.14951      0.0820238     0.0269169       27     
produce:cluster3d:Cluster3D                       0.479744       2.22814       26.6091       1.07442       4.81243        27     
[art]:TriggerResults:TriggerResultInserter        2.151e-05    2.67309e-05   6.1597e-05    2.5569e-05    7.1653e-06       27     
end_path:out1:RootOutput                          6.894e-06    9.6857e-06    2.0602e-05     8.785e-06    2.59549e-06      27     
end_path:out1:RootOutput(write)                    2.81157       3.20889       5.62576       3.09814      0.499837        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4430.91 MB
  Peak resident set size usage (VmHWM): 2591.63 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042422_0079_df-s03-d1_dw_0_20260203T155730_reco_stage0_20260203T233656_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042422_0079_df-s03-d1_dw_0_20260203T155730_reco_stage0_20260203T233656_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042422_0079_df-s03-d1_dw_0_20260203T155730_reco_stage0_20260203T233656_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770158772.0,
    "core.end_time": 1770161822.0,
    "core.events": [
      12813,
      12819,
      12825,
      12831,
      12837,
      12843,
      12849,
      12855,
      12861,
      12867,
      12873,
      12879,
      12885,
      12891,
      12897,
      12903,
      12909,
      12915,
      12921,
      12927,
      12933,
      12939,
      12945,
      12951,
      12957,
      12963,
      12969
    ],
    "core.event_count": 27,
    "core.first_event_number": 12813,
    "core.last_event_number": 12969,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42422
    ],
    "core.runs_subruns": [
      4242200001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042422_0079_df-s03-d1_dw_0_20260203T155730.hdf5"
    }
  ]
}
justIN time: 2026-02-04 04:29:43 UTC       justIN version: 01.06.00