justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303652.164@dunegpschedd01.fnal.gov

Jobsub ID303652.164@dunegpschedd01.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:50:26
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce512
Last heartbeat2026-02-04 00:20:49
From worker nodeHostnameb9g24p4713.cern.ch
cpuinfoAMD EPYC 7302 16-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:33:10
Input filesvd-protodune:np02vd_raw_run042433_0278_df-s03-d1_dw_0_20260203T202746.hdf5
JobscriptExit code0
Real time45m (2716s)
CPU time43m (2598s = 95%)
Max RSS bytes3415789568 (3257 MiB)
Outputting started2026-02-04 00:18:28
Output files
Finished2026-02-04 00:20:49
Saved logsjustin-logs:303652.164-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

oint ptr... 0x6900ce30, 0x6900cff0
Caught exception in looking up space point ptr... 0x422f06d0, 0x422f0320
Caught exception in looking up space point ptr... 0x422f0320, 0x422f06d0
Caught exception in looking up space point ptr... 0x61ba5a20, 0x61ba58a0
Caught exception in looking up space point ptr... 0x61ba58a0, 0x61ba5a20
Caught exception in looking up space point ptr... 0x18455db0, 0x1845bba0
Caught exception in looking up space point ptr... 0x1845bba0, 0x18455db0
Caught exception in looking up space point ptr... 0x4899ae10, 0x4899aab0
Caught exception in looking up space point ptr... 0x4899b060, 0x4899ae10
Caught exception in looking up space point ptr... 0x575af9b0, 0x58e016e0
Caught exception in looking up space point ptr... 0x58e016e0, 0x575af9b0
Caught exception in looking up space point ptr... 0xe87bc20, 0xe8781c0
Caught exception in looking up space point ptr... 0xe8781c0, 0x644ca650
Caught exception in looking up space point ptr... 0x422dcd80, 0x422dd080
Caught exception in looking up space point ptr... 0x422dd080, 0x422dd1b0
Caught exception in looking up space point ptr... 0xcc96210, 0x1370f4b0
Caught exception in looking up space point ptr... 0xcc963a0, 0xcc96210
Caught exception in looking up space point ptr... 0x6fb25950, 0x6fb1f920
Caught exception in looking up space point ptr... 0x6fb1f920, 0x6ef00670
Caught exception in looking up space point ptr... 0x3cc62c70, 0x3cc62b70
Caught exception in looking up space point ptr... 0x3cc62b70, 0x3cc7a8f0
Caught exception in looking up space point ptr... 0x5606cdd0, 0x5606d0f0
Caught exception in looking up space point ptr... 0x5606d0f0, 0x2cd8e0e0
Caught exception in looking up space point ptr... 0x3c7cb050, 0x5b758db0
Caught exception in looking up space point ptr... 0x5b758db0, 0x3c7cb050
Caught exception in looking up space point ptr... 0x16ab3c50, 0x6afc0810
Caught exception in looking up space point ptr... 0x6afc0810, 0x3a8198f0
Caught exception in looking up space point ptr... 0x2f36da10, 0x2f3725c0
Caught exception in looking up space point ptr... 0x2f3725c0, 0x2f373b70
Caught exception in looking up space point ptr... 0x674fde00, 0x67504730
Caught exception in looking up space point ptr... 0x67504730, 0x67503750
Caught exception in looking up space point ptr... 0xcb33950, 0x58f4d4c0
Caught exception in looking up space point ptr... 0x58f4d4c0, 0xcb33950
Caught exception in looking up space point ptr... 0xe7799d0, 0xe7798d0
Caught exception in looking up space point ptr... 0xe7798d0, 0xe7799d0
Caught exception in looking up space point ptr... 0x55e01ed0, 0x55e01d40
Caught exception in looking up space point ptr... 0x55e01d40, 0x55e01ed0
Caught exception in looking up space point ptr... 0x4cb98600, 0x4cb98290
Caught exception in looking up space point ptr... 0x4cb98290, 0x4cb98600
Caught exception in looking up space point ptr... 0x5a045c70, 0x7a2bc7c0
Caught exception in looking up space point ptr... 0x7a2bc7c0, 0x4cf044e0
Caught exception in looking up space point ptr... 0x3a9e55d0, 0x3a9e57e0
Caught exception in looking up space point ptr... 0x71a5b370, 0x3a9e55d0
Caught exception in looking up space point ptr... 0x5a08fc70, 0x5a08c170
Caught exception in looking up space point ptr... 0x5a08c170, 0x5a091280
Caught exception in looking up space point ptr... 0x109eb420, 0x109ee490
Caught exception in looking up space point ptr... 0x109ee490, 0x109eb420
Caught exception in looking up space point ptr... 0x488697f0, 0x48869920
Caught exception in looking up space point ptr... 0x48869920, 0x48869b80
Caught exception in looking up space point ptr... 0x13743ae0, 0x137434a0
Caught exception in looking up space point ptr... 0x137434a0, 0x13743ae0
Caught exception in looking up space point ptr... 0x13743ae0, 0x13743630
Caught exception in looking up space point ptr... 0x13743630, 0x13743ae0
Caught exception in looking up space point ptr... 0x6b0cb7b0, 0x6b0cb460
Caught exception in looking up space point ptr... 0x6b0cb460, 0x6b0cb7b0
Caught exception in looking up space point ptr... 0x6eee34c0, 0x6af984a0
Caught exception in looking up space point ptr... 0x6af984a0, 0x6eee34c0
Caught exception in looking up space point ptr... 0x5b642720, 0x5b63f840
Caught exception in looking up space point ptr... 0x5b63f840, 0x5b642720
Caught exception in looking up space point ptr... 0x14dda540, 0x3880c120
Caught exception in looking up space point ptr... 0x3880c120, 0x14dd6f00
Caught exception in looking up space point ptr... 0x7e84ab30, 0x127be050
Caught exception in looking up space point ptr... 0x127be050, 0x7e84a3f0
Caught exception in looking up space point ptr... 0x7e84a3f0, 0x36581300
Caught exception in looking up space point ptr... 0x36581300, 0x7e84ab30
Caught exception in looking up space point ptr... 0x13a63ec0, 0x6b0a4890
Caught exception in looking up space point ptr... 0x6b0a4890, 0x13a689f0
++++>>>> total num hits: 106957, num free: 101376
04-Feb-2026 01:18:18 CET  Closed output file "np02vd_raw_run042433_0278_df-s03-d1_dw_0_20260203T202746_reco_stage0_20260204T001818_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         128.66        139.63        150.606       139.464       7.45048        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       5.4371e-05    9.10197e-05   0.000170164   8.4172e-05    3.23346e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                6.48755       8.32895       12.5947       7.64725       1.64804        19     
produce:triggerrawdecoder:PDVDTriggerReader4     0.00858236     0.0574434     0.289994      0.0244834      0.06768        19     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000407058    0.0015027     0.0115829    0.000547121   0.00276815       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.000332677   0.000506286   0.000919448   0.00048902    0.000141075      19     
produce:beamevent:BeamEvent                      5.2651e-05    0.000127649   0.000702923   9.7052e-05    0.000138392      19     
produce:pdvddaphne:DAPHNEReaderPDVD                2.08665       2.54059       3.20874       2.48072      0.297198        19     
produce:ophit:OpHitFinder                         0.0257396     0.0340603     0.0418348     0.0331627    0.00472306       19     
produce:wclsdatavd:WireCellToolkit                 97.3553       104.495       116.133       103.357       5.1115         19     
produce:gaushit:GausHitFinder                     0.773239       1.22557       2.03401       1.16317      0.316438        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000309136   0.000403166   0.000746354   0.000373647   9.60484e-05      19     
produce:reco3d:SpacePointSolver                    10.928        15.3165       23.6105       14.4322       3.05898        19     
produce:hitpdune:DisambigFromSpacePoints          0.113294      0.201438      0.317694      0.177645      0.0584517       19     
produce:cluster3d:Cluster3D                        1.52545       2.49677       5.12007       2.34834      0.828764        19     
[art]:TriggerResults:TriggerResultInserter        2.243e-05    3.08915e-05   9.1632e-05    2.6481e-05    1.48551e-05      19     
end_path:out1:RootOutput                          4.09e-06     8.52553e-06   2.7911e-05     7.741e-06    4.73178e-06      19     
end_path:out1:RootOutput(write)                    4.57052       4.90254       5.65832       4.8281       0.320448        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5362.04 MB
  Peak resident set size usage (VmHWM): 3415.79 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0278_df-s03-d1_dw_0_20260203T202746_reco_stage0_20260204T001818_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0278_df-s03-d1_dw_0_20260203T202746_reco_stage0_20260204T001818_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0278_df-s03-d1_dw_0_20260203T202746_reco_stage0_20260204T001818_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770161600.0,
    "core.end_time": 1770164300.0,
    "core.events": [
      31701,
      31707,
      31713,
      31719,
      31725,
      31731,
      31737,
      31743,
      31749,
      31755,
      31761,
      31767,
      31773,
      31779,
      31785,
      31791,
      31797,
      31803,
      31809
    ],
    "core.event_count": 19,
    "core.first_event_number": 31701,
    "core.last_event_number": 31809,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0278_df-s03-d1_dw_0_20260203T202746.hdf5"
    }
  ]
}
justIN time: 2026-02-04 07:37:17 UTC       justIN version: 01.06.00