justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303652.1@dunegpschedd01.fnal.gov

Jobsub ID303652.1@dunegpschedd01.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:50:26
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce515
Last heartbeat2026-02-04 00:05:28
From worker nodeHostnameb9p12p0201.cern.ch
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:24:49
Input filesvd-protodune:np02vd_raw_run042433_0121_df-s03-d1_dw_0_20260203T192806.hdf5
JobscriptExit code0
Real time38m (2306s)
CPU time37m (2238s = 97%)
Max RSS bytes3414118400 (3255 MiB)
Outputting started2026-02-04 00:03:17
Output files
Finished2026-02-04 00:05:28
Saved logsjustin-logs:303652.1-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ng up space point ptr... 0x566042b0, 0x56604440
Caught exception in looking up space point ptr... 0x565e7600, 0xdf30840
Caught exception in looking up space point ptr... 0xdf30840, 0x565e7600
Caught exception in looking up space point ptr... 0xbc8a1b0, 0xbc868b0
Caught exception in looking up space point ptr... 0xbc868b0, 0xbc8a1b0
Caught exception in looking up space point ptr... 0x31c7bf20, 0x31cb5f10
Caught exception in looking up space point ptr... 0x31cb5f10, 0x31c7bf20
Caught exception in looking up space point ptr... 0x1964ed40, 0x1964dd40
Caught exception in looking up space point ptr... 0x1964dd40, 0x1964ed40
Caught exception in looking up space point ptr... 0x4c8c5fa0, 0x4c8c5dd0
Caught exception in looking up space point ptr... 0x4c8c5dd0, 0x4c8c5fa0
Caught exception in looking up space point ptr... 0x80f83530, 0x80f83350
Caught exception in looking up space point ptr... 0x80f83350, 0x80f91fc0
Caught exception in looking up space point ptr... 0x45983390, 0x45983820
Caught exception in looking up space point ptr... 0x45983820, 0x45983390
Caught exception in looking up space point ptr... 0x88da26f0, 0x88da2ce0
Caught exception in looking up space point ptr... 0x88da2ce0, 0x88da26f0
Caught exception in looking up space point ptr... 0xbc72fe0, 0xbc72730
Caught exception in looking up space point ptr... 0xbc72730, 0xbc72fe0
Caught exception in looking up space point ptr... 0x39d19600, 0x31c93770
Caught exception in looking up space point ptr... 0x31c93770, 0x39d19600
Caught exception in looking up space point ptr... 0x67d13730, 0x1b7bedb0
Caught exception in looking up space point ptr... 0x1b7bedb0, 0x1b7bf630
Caught exception in looking up space point ptr... 0x42602830, 0x42602510
Caught exception in looking up space point ptr... 0x42602510, 0x426021f0
Caught exception in looking up space point ptr... 0x42781b70, 0x427818e0
Caught exception in looking up space point ptr... 0x427818e0, 0x42781650
Caught exception in looking up space point ptr... 0x88d8c1c0, 0x88d903b0
Caught exception in looking up space point ptr... 0x88d903b0, 0x88d8c1c0
Caught exception in looking up space point ptr... 0x3f3dbaa0, 0x86bc2fa0
Caught exception in looking up space point ptr... 0x3f3e1b90, 0x3f3dbaa0
Caught exception in looking up space point ptr... 0x30b773f0, 0x30b75f80
Caught exception in looking up space point ptr... 0x30b75f80, 0x30b773f0
Caught exception in looking up space point ptr... 0xf7e4850, 0xf7e46c0
Caught exception in looking up space point ptr... 0xf7e46c0, 0xf7e49e0
Caught exception in looking up space point ptr... 0x695c1550, 0x695c13c0
Caught exception in looking up space point ptr... 0x695c13c0, 0x695c1550
Caught exception in looking up space point ptr... 0x3ccde9f0, 0x1af7d3c0
Caught exception in looking up space point ptr... 0x1af7d3c0, 0x3ccde6d0
Caught exception in looking up space point ptr... 0x3bb1c940, 0x55716120
Caught exception in looking up space point ptr... 0x55716120, 0x3bb1c940
Caught exception in looking up space point ptr... 0x4f525c30, 0x4f525fc0
Caught exception in looking up space point ptr... 0x4f525fc0, 0x4f525c30
Caught exception in looking up space point ptr... 0xbc72120, 0xbc71cb0
Caught exception in looking up space point ptr... 0xbc71cb0, 0xbc72120
Caught exception in looking up space point ptr... 0x17122c90, 0x864b1c20
Caught exception in looking up space point ptr... 0x864b1c20, 0x17122c90
Caught exception in looking up space point ptr... 0x182a6d90, 0x182a5c30
Caught exception in looking up space point ptr... 0x182a5c30, 0x182a6d90
Caught exception in looking up space point ptr... 0x384e53b0, 0x686b0f30
Caught exception in looking up space point ptr... 0x686b0f30, 0x384e53b0
Caught exception in looking up space point ptr... 0x11c525d0, 0x11c552b0
Caught exception in looking up space point ptr... 0x11c552b0, 0x11c61cc0
Caught exception in looking up space point ptr... 0x193e66c0, 0x193de3d0
Caught exception in looking up space point ptr... 0x193de3d0, 0x193e66c0
Caught exception in looking up space point ptr... 0x480fd8c0, 0x480fc6c0
Caught exception in looking up space point ptr... 0x480fc6c0, 0x480fd8c0
Caught exception in looking up space point ptr... 0x11c51ff0, 0x11c534f0
Caught exception in looking up space point ptr... 0x11c534f0, 0x11c51210
Caught exception in looking up space point ptr... 0x4c6cf250, 0x4c6d8830
Caught exception in looking up space point ptr... 0x4c6d8830, 0x4c6cf250
Caught exception in looking up space point ptr... 0xadb47e0, 0x873e8460
Caught exception in looking up space point ptr... 0x873e8460, 0xadb47e0
Caught exception in looking up space point ptr... 0x681cf470, 0x681cef70
Caught exception in looking up space point ptr... 0x681cef70, 0x681cf470
Caught exception in looking up space point ptr... 0xe9b3150, 0xe9ad0f0
Caught exception in looking up space point ptr... 0xe9ad0f0, 0xe9b3150
++++>>>> total num hits: 134827, num free: 128628
04-Feb-2026 01:03:11 CET  Closed output file "np02vd_raw_run042433_0121_df-s03-d1_dw_0_20260203T192806_reco_stage0_20260204T000311_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         104.846       118.312       155.829       114.995       14.0944        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                        4.204e-05    6.56801e-05   0.00016655    4.5671e-05    3.40656e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                3.57025       3.97496       5.65949       3.76205      0.536657        19     
produce:triggerrawdecoder:PDVDTriggerReader4     0.00958738     0.0110759     0.0132371     0.0107206    0.00108818       19     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000370252   0.000481153   0.000733142   0.000438441   9.5291e-05       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.000345561   0.000445034   0.000730812   0.000427802   9.33963e-05      19     
produce:beamevent:BeamEvent                       5.628e-05    0.00012498    0.00105412     7.43e-05     0.000219388      19     
produce:pdvddaphne:DAPHNEReaderPDVD                1.96891       2.23731       2.83215       2.20102      0.216837        19     
produce:ophit:OpHitFinder                         0.0218637     0.031566      0.0366445     0.032346      0.0036981       19     
produce:wclsdatavd:WireCellToolkit                 81.5756       87.7622       102.494       84.9196       6.72363        19     
produce:gaushit:GausHitFinder                     0.845792       1.13037       2.23395       1.03559      0.330937        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000195041   0.000302612   0.000673232   0.00029744    0.00010525       19     
produce:reco3d:SpacePointSolver                    9.19709       16.5243       33.5364       15.9961       6.31368        19     
produce:hitpdune:DisambigFromSpacePoints           0.1111       0.156117      0.253502       0.15085      0.036319        19     
produce:cluster3d:Cluster3D                        1.30036       2.15852       3.73498       2.13845      0.728611        19     
[art]:TriggerResults:TriggerResultInserter        1.94e-05     2.88417e-05    8.691e-05    2.5181e-05     1.426e-05       19     
end_path:out1:RootOutput                          4.87e-06      8.099e-06     2.542e-05     7.23e-06     4.18332e-06      19     
end_path:out1:RootOutput(write)                    3.79336       4.2997        5.49777       4.19365      0.483417        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5386.3 MB
  Peak resident set size usage (VmHWM): 3414.12 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0121_df-s03-d1_dw_0_20260203T192806_reco_stage0_20260204T000311_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0121_df-s03-d1_dw_0_20260203T192806_reco_stage0_20260204T000311_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0121_df-s03-d1_dw_0_20260203T192806_reco_stage0_20260204T000311_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770161100.0,
    "core.end_time": 1770163392.0,
    "core.events": [
      13803,
      13809,
      13815,
      13821,
      13827,
      13833,
      13839,
      13845,
      13851,
      13857,
      13863,
      13869,
      13875,
      13881,
      13887,
      13893,
      13899,
      13905,
      13911
    ],
    "core.event_count": 19,
    "core.first_event_number": 13803,
    "core.last_event_number": 13911,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0121_df-s03-d1_dw_0_20260203T192806.hdf5"
    }
  ]
}
justIN time: 2026-02-04 10:45:18 UTC       justIN version: 01.06.00