justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303647.104@dunegpschedd01.fnal.gov

Jobsub ID303647.104@dunegpschedd01.fnal.gov
Workflow ID12717
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:46:24
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce514
Last heartbeat2026-02-03 23:34:57
From worker nodeHostnameb9p08p6650.cern.ch
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 22:47:10
Input filesvd-protodune:np02vd_raw_run042421_0120_df-s04-d1_dw_0_20260203T143555.hdf5
JobscriptExit code0
Real time45m (2720s)
CPU time43m (2606s = 95%)
Max RSS bytes3209461760 (3060 MiB)
Outputting started2026-02-03 23:32:31
Output files
Finished2026-02-03 23:34:57
Saved logsjustin-logs:303647.104-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

x1152c7d0
Caught exception in looking up space point ptr... 0x4910f7f0, 0x4910f660
Caught exception in looking up space point ptr... 0x4910f660, 0x1f6bfaf0
Caught exception in looking up space point ptr... 0xd1c6650, 0xd1c67e0
Caught exception in looking up space point ptr... 0xd1c67e0, 0xd1c6970
Caught exception in looking up space point ptr... 0x4f3306c0, 0x116dd4d0
Caught exception in looking up space point ptr... 0x4910efd0, 0x4f3306c0
Caught exception in looking up space point ptr... 0xd2374c0, 0x2ab4bf30
Caught exception in looking up space point ptr... 0x116c6d60, 0x116c45e0
Caught exception in looking up space point ptr... 0x116c45e0, 0xd2374c0
Caught exception in looking up space point ptr... 0x239879d0, 0x23987520
Caught exception in looking up space point ptr... 0x23987520, 0x23987840
Caught exception in looking up space point ptr... 0x2a592f00, 0x2a592dd0
Caught exception in looking up space point ptr... 0x2a592dd0, 0x2a592f00
Caught exception in looking up space point ptr... 0x2db5d370, 0x2db5f2e0
Caught exception in looking up space point ptr... 0x2db5f2e0, 0x2db626e0
Caught exception in looking up space point ptr... 0x2aea73b0, 0x2aea7590
Caught exception in looking up space point ptr... 0x2aea7590, 0x2aea6eb0
Caught exception in looking up space point ptr... 0x2aea6eb0, 0x2aea7090
Caught exception in looking up space point ptr... 0x2aea7090, 0x2aea73b0
Caught exception in looking up space point ptr... 0x52c3b720, 0x52c3b620
Caught exception in looking up space point ptr... 0x52c3b620, 0x52c3b820
Caught exception in looking up space point ptr... 0x4f0f1e0, 0x4f0f0b0
Caught exception in looking up space point ptr... 0x4f0f0b0, 0x4f0f1e0
Caught exception in looking up space point ptr... 0x5faf62a0, 0x31619d10
Caught exception in looking up space point ptr... 0x31619d10, 0x5faf62a0
Caught exception in looking up space point ptr... 0x5bf62ae0, 0x5bf624e0
Caught exception in looking up space point ptr... 0x5bf624e0, 0x5bf45c40
Caught exception in looking up space point ptr... 0x5c525910, 0x5c526b80
Caught exception in looking up space point ptr... 0x5c526b80, 0x5c526cb0
Caught exception in looking up space point ptr... 0x40a10840, 0x40a23890
Caught exception in looking up space point ptr... 0x40a23890, 0x40a10840
Caught exception in looking up space point ptr... 0x62c6a680, 0x3071c650
Caught exception in looking up space point ptr... 0x3071c650, 0x67a5a110
Caught exception in looking up space point ptr... 0x35ec5480, 0x58257610
Caught exception in looking up space point ptr... 0x58257610, 0x62c6a680
Caught exception in looking up space point ptr... 0x116b3d80, 0x2ab20540
Caught exception in looking up space point ptr... 0x2ab20540, 0x116b3d80
Caught exception in looking up space point ptr... 0x31616ca0, 0x31616b10
Caught exception in looking up space point ptr... 0x31616b10, 0x31616ca0
Caught exception in looking up space point ptr... 0x132b4f70, 0x132b4c50
Caught exception in looking up space point ptr... 0x132b4c50, 0x132b4ac0
Caught exception in looking up space point ptr... 0xd1ce410, 0x2fc876e0
Caught exception in looking up space point ptr... 0x2fc876e0, 0x2fc86d30
Caught exception in looking up space point ptr... 0x116ee2d0, 0x116ede20
Caught exception in looking up space point ptr... 0x116ede20, 0x116ee2d0
Caught exception in looking up space point ptr... 0x677648c0, 0x67764730
Caught exception in looking up space point ptr... 0x67764730, 0x677648c0
Caught exception in looking up space point ptr... 0xd1ba990, 0xd1ba800
Caught exception in looking up space point ptr... 0xd1ba800, 0xd1ba990
Caught exception in looking up space point ptr... 0xd1be5d0, 0x2fc95450
Caught exception in looking up space point ptr... 0x2fc95450, 0x2fc88910
Caught exception in looking up space point ptr... 0x2398c510, 0x3d510b90
Caught exception in looking up space point ptr... 0x3d510b90, 0x3d5159c0
Caught exception in looking up space point ptr... 0x33e37300, 0x2e269fc0
Caught exception in looking up space point ptr... 0x2e269fc0, 0x33e36990
Caught exception in looking up space point ptr... 0x45963570, 0x584f0ad0
Caught exception in looking up space point ptr... 0x584f0ad0, 0x45963570
Caught exception in looking up space point ptr... 0x45963570, 0x45963670
Caught exception in looking up space point ptr... 0x45963670, 0x45963570
Caught exception in looking up space point ptr... 0x2a8d7b30, 0x2a8d77c0
Caught exception in looking up space point ptr... 0x2a8d77c0, 0x2a8d79a0
Caught exception in looking up space point ptr... 0x2a708320, 0x2a7081f0
Caught exception in looking up space point ptr... 0x2a7081f0, 0x2a708470
Caught exception in looking up space point ptr... 0x30095a50, 0x30098a30
Caught exception in looking up space point ptr... 0x30098a30, 0x30095a50
++++>>>> total num hits: 83804, num free: 80774
04-Feb-2026 00:32:23 CET  Closed output file "np02vd_raw_run042421_0120_df-s04-d1_dw_0_20260203T143555_reco_stage0_20260203T233223_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         76.6931       98.2632       186.474       95.4317       18.4992        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       5.4071e-05    0.000103031   0.000232523   8.3021e-05    4.55873e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                4.45786       5.0522        5.54428       5.06884      0.265503        27     
produce:triggerrawdecoder:PDVDTriggerReader4      0.0123833     0.0176177     0.053662      0.0148888    0.00850752       27     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000488806   0.000981554   0.00627832    0.000592007    0.0012698       27     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.00172328    0.00241362    0.00344822    0.00228402     0.0004252       27     
produce:beamevent:BeamEvent                      7.4941e-05    0.000158712   0.00112333    0.000107081   0.000194697      27     
produce:pdvddaphne:DAPHNEReaderPDVD                2.45601       3.03144       3.4025        3.0376       0.209079        27     
produce:ophit:OpHitFinder                         0.0235443     0.0407477     0.0511029     0.0417713    0.00536848       27     
produce:wclsdatavd:WireCellToolkit                 57.6963       70.3855       80.2699       71.0134       4.92593        27     
produce:gaushit:GausHitFinder                     0.625892       1.06503       2.06631       1.03088      0.314434        27     
produce:nhitsfilter:NumberOfHitsFilter           0.000174722   0.000283197   0.000596257   0.000244953   9.53057e-05      27     
produce:reco3d:SpacePointSolver                    5.27888       10.4598       37.684        9.76587       6.02686        27     
produce:hitpdune:DisambigFromSpacePoints          0.0504358     0.121115      0.577755      0.0956778     0.0995409       27     
produce:cluster3d:Cluster3D                       0.892609       4.0072        59.1491       1.51955       10.848         27     
[art]:TriggerResults:TriggerResultInserter        2.14e-05     3.4764e-05    0.000107911    3.172e-05    1.63326e-05      27     
end_path:out1:RootOutput                          5.82e-06     9.47678e-06    3.354e-05     7.71e-06     6.31679e-06      27     
end_path:out1:RootOutput(write)                    2.93864       4.05606       8.30465       3.80667       1.04086        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5246.34 MB
  Peak resident set size usage (VmHWM): 3209.46 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042421_0120_df-s04-d1_dw_0_20260203T143555_reco_stage0_20260203T233223_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042421_0120_df-s04-d1_dw_0_20260203T143555_reco_stage0_20260203T233223_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042421_0120_df-s04-d1_dw_0_20260203T143555_reco_stage0_20260203T233223_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770158843.0,
    "core.end_time": 1770161544.0,
    "core.events": [
      19511,
      19517,
      19523,
      19529,
      19535,
      19541,
      19547,
      19553,
      19559,
      19565,
      19571,
      19577,
      19583,
      19589,
      19595,
      19601,
      19607,
      19613,
      19619,
      19625,
      19631,
      19637,
      19643,
      19649,
      19655,
      19661,
      19667
    ],
    "core.event_count": 27,
    "core.first_event_number": 19511,
    "core.last_event_number": 19667,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42421
    ],
    "core.runs_subruns": [
      4242100001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042421_0120_df-s04-d1_dw_0_20260203T143555.hdf5"
    }
  ]
}
justIN time: 2026-02-04 06:08:24 UTC       justIN version: 01.06.00