justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303647.59@dunegpschedd01.fnal.gov

Jobsub ID303647.59@dunegpschedd01.fnal.gov
Workflow ID12717
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:46:24
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce506
Last heartbeat2026-02-03 23:27:59
From worker nodeHostnameb9p06p2822.cern.ch
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 22:47:36
Input filesvd-protodune:np02vd_raw_run042421_0149_df-s04-d1_dw_0_20260203T145136.hdf5
JobscriptExit code0
Real time37m (2273s)
CPU time35m (2132s = 93%)
Max RSS bytes2616180736 (2494 MiB)
Outputting started2026-02-03 23:25:31
Output files
Finished2026-02-03 23:27:59
Saved logsjustin-logs:303647.59-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

410750, 0x4c0aa6f0
Caught exception in looking up space point ptr... 0x2f2476c0, 0x2f2471c0
Caught exception in looking up space point ptr... 0x2f247da0, 0x2f247a30
Caught exception in looking up space point ptr... 0x2f247a30, 0x2f2476c0
Caught exception in looking up space point ptr... 0xffcbb50, 0xc391460
Caught exception in looking up space point ptr... 0xc3fc140, 0xffcbb50
Caught exception in looking up space point ptr... 0x282f55d0, 0x282f5440
Caught exception in looking up space point ptr... 0x282f5440, 0x282f55d0
Caught exception in looking up space point ptr... 0x3318aec0, 0x29cfcb70
Caught exception in looking up space point ptr... 0x29cfcb70, 0x17641130
Caught exception in looking up space point ptr... 0x2f2555f0, 0x2f255720
Caught exception in looking up space point ptr... 0x2f255720, 0x2f2555f0
Caught exception in looking up space point ptr... 0x52edee00, 0x34d79650
Caught exception in looking up space point ptr... 0x421efa20, 0x52edee00
Caught exception in looking up space point ptr... 0x120fd310, 0x120fd970
Caught exception in looking up space point ptr... 0x120fd070, 0x120fd310
Caught exception in looking up space point ptr... 0x29d3c4c0, 0x56fadf40
Caught exception in looking up space point ptr... 0x56fadf40, 0x56fade40
Caught exception in looking up space point ptr... 0x408b4fe0, 0x2380d010
Caught exception in looking up space point ptr... 0x44ddf3d0, 0x408b4fe0
Caught exception in looking up space point ptr... 0x3e23eb30, 0x4bcaf7e0
Caught exception in looking up space point ptr... 0x4804cab0, 0x3e23eb30
Caught exception in looking up space point ptr... 0x464987e0, 0x46498a70
Caught exception in looking up space point ptr... 0x46498320, 0x464987e0
Caught exception in looking up space point ptr... 0x3ebab5a0, 0x38e92900
Caught exception in looking up space point ptr... 0x472fc900, 0x3ebab5a0
Caught exception in looking up space point ptr... 0x1117fd20, 0x1117be80
Caught exception in looking up space point ptr... 0x1117be80, 0x1117fe50
Caught exception in looking up space point ptr... 0xca35010, 0xca35110
Caught exception in looking up space point ptr... 0xca35110, 0xca2ef10
Caught exception in looking up space point ptr... 0xc156120, 0x2b9a7ad0
Caught exception in looking up space point ptr... 0xc3a6c50, 0xc156120
Caught exception in looking up space point ptr... 0x3bb0f890, 0x3bb0f4d0
Caught exception in looking up space point ptr... 0x3bb0f4d0, 0x3bb0f890
Caught exception in looking up space point ptr... 0x235ab050, 0x235aace0
Caught exception in looking up space point ptr... 0x235aace0, 0x235ab050
Caught exception in looking up space point ptr... 0xc9e9310, 0xc9e8ff0
Caught exception in looking up space point ptr... 0xc9e8ff0, 0x55f5ff00
Caught exception in looking up space point ptr... 0x49843290, 0x498435b0
Caught exception in looking up space point ptr... 0x498435b0, 0x49843290
Caught exception in looking up space point ptr... 0x4452bad0, 0x4452b9d0
Caught exception in looking up space point ptr... 0x4452b9d0, 0xc0d2820
Caught exception in looking up space point ptr... 0xffbb6c0, 0xffb9320
Caught exception in looking up space point ptr... 0xffbbdd0, 0xffbb6c0
Caught exception in looking up space point ptr... 0x235c2730, 0x235c2440
Caught exception in looking up space point ptr... 0x235c2440, 0x235c2730
Caught exception in looking up space point ptr... 0xc3ad2d0, 0xc3ace60
Caught exception in looking up space point ptr... 0xc3ace60, 0xc3ad2d0
Caught exception in looking up space point ptr... 0x489fec00, 0x489fe8e0
Caught exception in looking up space point ptr... 0x489fe8e0, 0x489fec00
Caught exception in looking up space point ptr... 0x4e0901f0, 0x22637fc0
Caught exception in looking up space point ptr... 0x3a49a460, 0x4e0901f0
Caught exception in looking up space point ptr... 0x4804e340, 0xa6e9b20
Caught exception in looking up space point ptr... 0xffc47e0, 0x4804e340
Caught exception in looking up space point ptr... 0x2987d990, 0x29879bc0
Caught exception in looking up space point ptr... 0x29879bc0, 0x2987dbf0
Caught exception in looking up space point ptr... 0x2987dbf0, 0x29881c90
Caught exception in looking up space point ptr... 0x2987cd20, 0x2987d990
Caught exception in looking up space point ptr... 0x52edef50, 0x117a83f0
Caught exception in looking up space point ptr... 0x117a83f0, 0x52edef50
Caught exception in looking up space point ptr... 0x3bb1d6e0, 0x3bb1bf40
Caught exception in looking up space point ptr... 0x3bb1bf40, 0x3bb1d6e0
Caught exception in looking up space point ptr... 0x10341080, 0x10340580
Caught exception in looking up space point ptr... 0x10340580, 0x10341080
Caught exception in looking up space point ptr... 0x235b3960, 0x235b0550
Caught exception in looking up space point ptr... 0x235b0550, 0x235b3960
++++>>>> total num hits: 59046, num free: 57322
04-Feb-2026 00:25:25 CET  Closed output file "np02vd_raw_run042421_0149_df-s04-d1_dw_0_20260203T145136_reco_stage0_20260203T232525_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         70.2858       82.1198       97.8348       81.7004       6.36968        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       4.0581e-05    9.51067e-05   0.000184671   8.4861e-05    3.99948e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                4.93959       5.57128       13.5364       5.23084       1.57524        27     
produce:triggerrawdecoder:PDVDTriggerReader4      0.0104055     0.0283261     0.087685      0.0188148     0.0212062       27     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000394842   0.000553498   0.00112725    0.000510543   0.000150432      27     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.00148363    0.00187267    0.00267658    0.00176627    0.000328464      27     
produce:beamevent:BeamEvent                       5.728e-05    0.000131928   0.00119587    8.7281e-05    0.000209779      27     
produce:pdvddaphne:DAPHNEReaderPDVD                2.37375       2.93769       3.32713       2.92714      0.196968        27     
produce:ophit:OpHitFinder                         0.0276839     0.0379442     0.0465572     0.0390435    0.00497625       27     
produce:wclsdatavd:WireCellToolkit                 52.2449       60.0728       70.8358       60.6093       4.61644        27     
produce:gaushit:GausHitFinder                     0.456951      0.842274       1.52982      0.849193      0.220033        27     
produce:nhitsfilter:NumberOfHitsFilter           0.000138151   0.000230612   0.000624583   0.000206651   9.29703e-05      27     
produce:reco3d:SpacePointSolver                    3.63452       7.73293       12.1093       7.58565       1.95448        27     
produce:hitpdune:DisambigFromSpacePoints          0.0381835     0.079017      0.141853      0.0714048     0.0244332       27     
produce:cluster3d:Cluster3D                       0.634146       1.40012       2.32709       1.35716       0.48261        27     
[art]:TriggerResults:TriggerResultInserter        1.693e-05    2.9209e-05    0.000118331    2.529e-05    1.80747e-05      27     
end_path:out1:RootOutput                           3.1e-06     6.58633e-06    2.952e-05     5.35e-06     4.85369e-06      27     
end_path:out1:RootOutput(write)                    2.77008       3.39601       4.57061       3.19768       0.53372        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4601.99 MB
  Peak resident set size usage (VmHWM): 2616.18 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042421_0149_df-s04-d1_dw_0_20260203T145136_reco_stage0_20260203T232525_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042421_0149_df-s04-d1_dw_0_20260203T145136_reco_stage0_20260203T232525_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042421_0149_df-s04-d1_dw_0_20260203T145136_reco_stage0_20260203T232525_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770158867.0,
    "core.end_time": 1770161126.0,
    "core.events": [
      24215,
      24221,
      24227,
      24233,
      24239,
      24245,
      24251,
      24257,
      24263,
      24269,
      24275,
      24281,
      24287,
      24293,
      24299,
      24305,
      24311,
      24317,
      24323,
      24329,
      24335,
      24341,
      24347,
      24353,
      24359,
      24365,
      24371
    ],
    "core.event_count": 27,
    "core.first_event_number": 24215,
    "core.last_event_number": 24371,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42421
    ],
    "core.runs_subruns": [
      4242100001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042421_0149_df-s04-d1_dw_0_20260203T145136.hdf5"
    }
  ]
}
justIN time: 2026-02-04 05:58:11 UTC       justIN version: 01.06.00