justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303652.177@dunegpschedd01.fnal.gov

Jobsub ID303652.177@dunegpschedd01.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:50:26
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce514
Last heartbeat2026-02-04 00:30:05
From worker nodeHostnameb9p08p6650.cern.ch
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:34:33
Input filesvd-protodune:np02vd_raw_run042433_0519_df-s04-d1_dw_0_20260203T215923.hdf5
JobscriptExit code0
Real time53m (3186s)
CPU time50m (3038s = 95%)
Max RSS bytes3411038208 (3253 MiB)
Outputting started2026-02-04 00:27:40
Output files
Finished2026-02-04 00:30:05
Saved logsjustin-logs:303652.177-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

int ptr... 0xd74b910, 0xd74ba70
Caught exception in looking up space point ptr... 0x2981b0b0, 0x523ee990
Caught exception in looking up space point ptr... 0x523ee990, 0x13f9b570
Caught exception in looking up space point ptr... 0x7f6649d0, 0x7f6665a0
Caught exception in looking up space point ptr... 0x7f6665a0, 0x7f665b50
Caught exception in looking up space point ptr... 0x19819a90, 0x3164a6b0
Caught exception in looking up space point ptr... 0x3164a6b0, 0x19819a90
Caught exception in looking up space point ptr... 0x7fc86430, 0x7fc86300
Caught exception in looking up space point ptr... 0x7fc86300, 0x7fc85e40
Caught exception in looking up space point ptr... 0x8204fe70, 0x82050110
Caught exception in looking up space point ptr... 0x82050110, 0x8204fe70
Caught exception in looking up space point ptr... 0x1585ca80, 0x1585bdb0
Caught exception in looking up space point ptr... 0x1585bdb0, 0x15859600
Caught exception in looking up space point ptr... 0x15859600, 0x1585c760
Caught exception in looking up space point ptr... 0x1585c8f0, 0x1585ca80
Caught exception in looking up space point ptr... 0x66e2a2b0, 0x66e269a0
Caught exception in looking up space point ptr... 0x66e269a0, 0x66e256f0
Caught exception in looking up space point ptr... 0x54bf4bc0, 0x54bf4d10
Caught exception in looking up space point ptr... 0x54bf4d10, 0x54bf4bc0
Caught exception in looking up space point ptr... 0x51b69f60, 0x51b69ab0
Caught exception in looking up space point ptr... 0x51b69ab0, 0x51b657d0
Caught exception in looking up space point ptr... 0x7f0f28b0, 0x7f0f26e0
Caught exception in looking up space point ptr... 0x7f0f26e0, 0x7f0f2c50
Caught exception in looking up space point ptr... 0x76f68050, 0x76f60f10
Caught exception in looking up space point ptr... 0x76f60f10, 0x76f682b0
Caught exception in looking up space point ptr... 0x322ae0e0, 0x322adbe0
Caught exception in looking up space point ptr... 0x322adbe0, 0x322ae0e0
Caught exception in looking up space point ptr... 0xe8ff770, 0xe907ef0
Caught exception in looking up space point ptr... 0xe907ef0, 0xe8ff770
Caught exception in looking up space point ptr... 0x8205cbf0, 0x31f98dd0
Caught exception in looking up space point ptr... 0x31f98dd0, 0x8205cbf0
Caught exception in looking up space point ptr... 0x8205cbf0, 0x8205ccf0
Caught exception in looking up space point ptr... 0x8205ccf0, 0x8205cbf0
Caught exception in looking up space point ptr... 0x58a9c0f0, 0x58a9bfa0
Caught exception in looking up space point ptr... 0x58a9bfa0, 0x58a9be00
Caught exception in looking up space point ptr... 0x73667c90, 0x73667130
Caught exception in looking up space point ptr... 0x73667130, 0x73666b90
Caught exception in looking up space point ptr... 0x2eea4870, 0x2ee9fdf0
Caught exception in looking up space point ptr... 0x2eea4b90, 0x2eea4870
Caught exception in looking up space point ptr... 0xe901490, 0xe900f90
Caught exception in looking up space point ptr... 0xe900f90, 0xe901490
Caught exception in looking up space point ptr... 0x35f5e2e0, 0x35f59f80
Caught exception in looking up space point ptr... 0x35f59f80, 0x35f5e2e0
Caught exception in looking up space point ptr... 0x37c962c0, 0x37c96460
Caught exception in looking up space point ptr... 0x37c96460, 0x37c962c0
Caught exception in looking up space point ptr... 0x4c7c9ae0, 0x4c7ce4e0
Caught exception in looking up space point ptr... 0x4c7ce4e0, 0x4c7c9ae0
Caught exception in looking up space point ptr... 0x4c7c9ae0, 0x4c7ce170
Caught exception in looking up space point ptr... 0x4c7ce170, 0x4c7c9ae0
Caught exception in looking up space point ptr... 0x76f67260, 0x76f673f0
Caught exception in looking up space point ptr... 0x76f673f0, 0x76f67260
Caught exception in looking up space point ptr... 0x51ce0490, 0x51ce0c30
Caught exception in looking up space point ptr... 0x51ce0c30, 0x51ce0490
Caught exception in looking up space point ptr... 0x37e574a0, 0x37e576f0
Caught exception in looking up space point ptr... 0x37e576f0, 0x37e574a0
Caught exception in looking up space point ptr... 0x37e806a0, 0x37e82d10
Caught exception in looking up space point ptr... 0x37e82d10, 0x37e81bf0
Caught exception in looking up space point ptr... 0x187f33b0, 0x187f2f70
Caught exception in looking up space point ptr... 0x187f2f70, 0x187ea9c0
Caught exception in looking up space point ptr... 0x689b7800, 0x63b4be20
Caught exception in looking up space point ptr... 0x63b4be20, 0x689b7800
Caught exception in looking up space point ptr... 0x763a8a00, 0x763a8620
Caught exception in looking up space point ptr... 0x763a8620, 0x763a8a00
Caught exception in looking up space point ptr... 0x7928d520, 0x68a70ca0
Caught exception in looking up space point ptr... 0x68a70ca0, 0x7928d520
Caught exception in looking up space point ptr... 0x36974e70, 0x36974d20
Caught exception in looking up space point ptr... 0x36974d20, 0x36974e70
++++>>>> total num hits: 116345, num free: 110904
04-Feb-2026 01:27:30 CET  Closed output file "np02vd_raw_run042433_0519_df-s04-d1_dw_0_20260203T215923_reco_stage0_20260204T002730_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         143.94        163.997       184.622       166.125       12.1734        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       6.1061e-05    0.000122035   0.000362095   9.4681e-05    6.51214e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                7.74972       8.5837        10.9575       8.36825      0.812812        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.0138509     0.0381759     0.0805457     0.0352938     0.0203018       19     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000471885   0.000749715   0.00161842    0.000700598   0.000227322      19     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.000442476   0.000636391   0.00149961    0.000589157   0.000221108      19     
produce:beamevent:BeamEvent                      9.1301e-05    0.000191398   0.000935521   0.000128821   0.000186131      19     
produce:pdvddaphne:DAPHNEReaderPDVD                2.61616       3.08291       3.42709       3.08589      0.234938        19     
produce:ophit:OpHitFinder                         0.0267259      0.03903      0.0508665     0.040704     0.00567442       19     
produce:wclsdatavd:WireCellToolkit                 107.642       121.289       136.851       119.364       8.22292        19     
produce:gaushit:GausHitFinder                      0.90105        1.503        2.32458       1.52701      0.362457        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000239743   0.00038901    0.000684818   0.000353094   0.000102388      19     
produce:reco3d:SpacePointSolver                    9.4136        20.5472       32.6422       20.234        5.92531        19     
produce:hitpdune:DisambigFromSpacePoints          0.120394      0.213892      0.306287      0.213826      0.0567484       19     
produce:cluster3d:Cluster3D                        1.4217        2.83107       5.06025       2.5174       0.950644        19     
[art]:TriggerResults:TriggerResultInserter       2.6201e-05    3.89058e-05   0.000111582   3.6281e-05    1.8187e-05       19     
end_path:out1:RootOutput                          3.64e-06     8.79016e-06    3.06e-05      7.87e-06     5.69937e-06      19     
end_path:out1:RootOutput(write)                    4.27171       5.82602       7.04124       5.74173      0.847501        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5364.7 MB
  Peak resident set size usage (VmHWM): 3411.04 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0519_df-s04-d1_dw_0_20260203T215923_reco_stage0_20260204T002730_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0519_df-s04-d1_dw_0_20260203T215923_reco_stage0_20260204T002730_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0519_df-s04-d1_dw_0_20260203T215923_reco_stage0_20260204T002730_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770161685.0,
    "core.end_time": 1770164852.0,
    "core.events": [
      59177,
      59183,
      59189,
      59195,
      59201,
      59207,
      59213,
      59219,
      59225,
      59231,
      59237,
      59243,
      59249,
      59255,
      59261,
      59267,
      59273,
      59279,
      59285
    ],
    "core.event_count": 19,
    "core.first_event_number": 59177,
    "core.last_event_number": 59285,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0519_df-s04-d1_dw_0_20260203T215923.hdf5"
    }
  ]
}
justIN time: 2026-02-04 09:14:41 UTC       justIN version: 01.06.00