justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297923.1@dunegpschedd02.fnal.gov

Jobsub ID297923.1@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 23:00:26
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce505
Last heartbeat2026-02-04 00:34:41
From worker nodeHostnameb9p35p0031.cern.ch
cpuinfoAMD EPYC 9655P 96-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:56:42
Input filesvd-protodune:np02vd_raw_run042433_0040_df-s04-d1_dw_0_20260203T185718.hdf5
JobscriptExit code0
Real time35m (2152s)
CPU time34m (2081s = 96%)
Max RSS bytes3387416576 (3230 MiB)
Outputting started2026-02-04 00:32:36
Output files
Finished2026-02-04 00:34:41
Saved logsjustin-logs:297923.1-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ooking up space point ptr... 0x66013200, 0x1778c540
Caught exception in looking up space point ptr... 0x1778c540, 0x1a8a2b90
Caught exception in looking up space point ptr... 0x2ecf66f0, 0x48c68ef0
Caught exception in looking up space point ptr... 0x48c68ef0, 0x2ecf66f0
Caught exception in looking up space point ptr... 0x4b08c7d0, 0x4b08c680
Caught exception in looking up space point ptr... 0x4b08c970, 0x4b08c7d0
Caught exception in looking up space point ptr... 0x37ef7c90, 0x37ee8f60
Caught exception in looking up space point ptr... 0x37ee8f60, 0x37ef7c90
Caught exception in looking up space point ptr... 0x34e5fa30, 0x34e60f30
Caught exception in looking up space point ptr... 0x34e6ce80, 0x34e5fa30
Caught exception in looking up space point ptr... 0x7b194880, 0x7b1949b0
Caught exception in looking up space point ptr... 0x7b1949b0, 0x7b194880
Caught exception in looking up space point ptr... 0x154a20f0, 0x154a2340
Caught exception in looking up space point ptr... 0x154a8de0, 0x154a20f0
Caught exception in looking up space point ptr... 0xf0a89c0, 0xf0a8870
Caught exception in looking up space point ptr... 0xf0a8870, 0xf0a8b10
Caught exception in looking up space point ptr... 0x34f5dd60, 0x34f67780
Caught exception in looking up space point ptr... 0x34f67780, 0x34f675f0
Caught exception in looking up space point ptr... 0x172e96b0, 0xa67b1e0
Caught exception in looking up space point ptr... 0xa67b1e0, 0x172e96b0
Caught exception in looking up space point ptr... 0x2ecbf840, 0x3aaa52d0
Caught exception in looking up space point ptr... 0x3aaa52d0, 0x2ecbf840
Caught exception in looking up space point ptr... 0x313d4f20, 0x313d6780
Caught exception in looking up space point ptr... 0x313d4ad0, 0x313d4f20
Caught exception in looking up space point ptr... 0x38e93ff0, 0x38e941f0
Caught exception in looking up space point ptr... 0x38e941f0, 0x38e93ff0
Caught exception in looking up space point ptr... 0x69f10870, 0x69f147a0
Caught exception in looking up space point ptr... 0x69f147a0, 0x69f11080
Caught exception in looking up space point ptr... 0x38d2bf50, 0x38b508b0
Caught exception in looking up space point ptr... 0x38b508b0, 0x38d2bf50
Caught exception in looking up space point ptr... 0x34f68600, 0x48c94010
Caught exception in looking up space point ptr... 0x48c94010, 0x154aeec0
Caught exception in looking up space point ptr... 0x4f3a2f60, 0x312e1fd0
Caught exception in looking up space point ptr... 0x312e1fd0, 0x312e1e40
Caught exception in looking up space point ptr... 0xe8e61f0, 0x355fb440
Caught exception in looking up space point ptr... 0x355fb440, 0x8897690
Caught exception in looking up space point ptr... 0x4b4fcc00, 0x4b4fcd00
Caught exception in looking up space point ptr... 0x4b4fcd00, 0x4b4fcfa0
Caught exception in looking up space point ptr... 0x633ca060, 0x633ca4b0
Caught exception in looking up space point ptr... 0x633ca4b0, 0x633ca060
Caught exception in looking up space point ptr... 0x69f188b0, 0x69f18520
Caught exception in looking up space point ptr... 0x69f18520, 0x69f175b0
Caught exception in looking up space point ptr... 0x69f16d80, 0x69f16c90
Caught exception in looking up space point ptr... 0x69f16c90, 0x69f16d80
Caught exception in looking up space point ptr... 0x3729e030, 0x48ca1ae0
Caught exception in looking up space point ptr... 0x48ca1ae0, 0x3729e030
Caught exception in looking up space point ptr... 0x4b26f090, 0x4b2685d0
Caught exception in looking up space point ptr... 0x4b2685d0, 0x4b26f090
Caught exception in looking up space point ptr... 0x57c1cc20, 0x57c1cec0
Caught exception in looking up space point ptr... 0x57c1cec0, 0x74172890
Caught exception in looking up space point ptr... 0x3bf9dfa0, 0x3d21c150
Caught exception in looking up space point ptr... 0x3d21c150, 0x3bf9dfa0
Caught exception in looking up space point ptr... 0x800f8bf0, 0x800f72c0
Caught exception in looking up space point ptr... 0x800f72c0, 0x800fb6a0
Caught exception in looking up space point ptr... 0x3867fc00, 0x4f641ad0
Caught exception in looking up space point ptr... 0x4f641ad0, 0x3867fc00
Caught exception in looking up space point ptr... 0x89aa8240, 0x38effb70
Caught exception in looking up space point ptr... 0x38effb70, 0x4e451da0
Caught exception in looking up space point ptr... 0x38bbe9b0, 0x38be7d40
Caught exception in looking up space point ptr... 0x38be7d40, 0x38bbe9b0
Caught exception in looking up space point ptr... 0x69b76080, 0x3d0b1230
Caught exception in looking up space point ptr... 0x3d0b1230, 0x69b76080
Caught exception in looking up space point ptr... 0x3a5f6e70, 0x372641b0
Caught exception in looking up space point ptr... 0x372641b0, 0x3a5f6e70
Caught exception in looking up space point ptr... 0x34f39080, 0x34f38ef0
Caught exception in looking up space point ptr... 0x34f38ef0, 0x34f38bd0
Caught exception in looking up space point ptr... 0x34f38bd0, 0x34f375a0
++++>>>> total num hits: 105384, num free: 101127
04-Feb-2026 01:32:29 CET  Closed output file "np02vd_raw_run042433_0040_df-s04-d1_dw_0_20260203T185718_reco_stage0_20260204T003229_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         97.0477       110.192       137.891       109.729       8.82132        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                        5.845e-05    8.09211e-05   0.00017001     7.702e-05    2.37052e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                5.3636        5.97571        6.467        5.93864      0.279389        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.0102347     0.0233134     0.0617259     0.0221988     0.0125399       19     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000382911   0.00373718     0.0202503    0.000705991    0.0060134       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.00033843    0.000584306   0.00123115    0.00056599    0.000231053      19     
produce:beamevent:BeamEvent                       6.41e-05     0.000128701   0.000852591    8.578e-05    0.000171571      19     
produce:pdvddaphne:DAPHNEReaderPDVD                1.54479       2.08168       2.39187       2.09913      0.219345        19     
produce:ophit:OpHitFinder                         0.0247545     0.0342364     0.0417197     0.0354966    0.00492059       19     
produce:wclsdatavd:WireCellToolkit                 69.9582       79.4259       91.3293       78.1538       5.86537        19     
produce:gaushit:GausHitFinder                     0.646284       1.13382       1.89561       1.05076      0.301785        19     
produce:nhitsfilter:NumberOfHitsFilter           0.00018481    0.000263436   0.00037512    0.00024996    5.58473e-05      19     
produce:reco3d:SpacePointSolver                    8.9414        14.4672       27.0223       13.5507       3.99505        19     
produce:hitpdune:DisambigFromSpacePoints          0.0843042     0.147945      0.222545      0.134348      0.0382868       19     
produce:cluster3d:Cluster3D                        1.00022       1.90213       3.07709       1.93439      0.557778        19     
[art]:TriggerResults:TriggerResultInserter        2.459e-05    3.32953e-05    7.267e-05     3.102e-05    1.01738e-05      19     
end_path:out1:RootOutput                          4.84e-06     6.53211e-06    1.726e-05     5.95e-06     2.6349e-06       19     
end_path:out1:RootOutput(write)                    3.70298       4.94885       6.11842       4.96307      0.566886        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5411.95 MB
  Peak resident set size usage (VmHWM): 3387.42 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0040_df-s04-d1_dw_0_20260203T185718_reco_stage0_20260204T003229_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0040_df-s04-d1_dw_0_20260203T185718_reco_stage0_20260204T003229_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0040_df-s04-d1_dw_0_20260203T185718_reco_stage0_20260204T003229_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770163011.0,
    "core.end_time": 1770165151.0,
    "core.events": [
      4571,
      4577,
      4583,
      4589,
      4595,
      4601,
      4607,
      4613,
      4619,
      4625,
      4631,
      4637,
      4643,
      4649,
      4655,
      4661,
      4667,
      4673,
      4679
    ],
    "core.event_count": 19,
    "core.first_event_number": 4571,
    "core.last_event_number": 4679,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0040_df-s04-d1_dw_0_20260203T185718.hdf5"
    }
  ]
}
justIN time: 2026-02-04 07:42:28 UTC       justIN version: 01.06.00