justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297919.134@dunegpschedd02.fnal.gov

Jobsub ID297919.134@dunegpschedd02.fnal.gov
Workflow ID12717
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:54:25
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce504
Last heartbeat2026-02-04 00:14:50
From worker nodeHostnameb9p07p8370.cern.ch
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:35:59
Input filesvd-protodune:np02vd_raw_run042421_0146_df-s03-d0_dw_0_20260203T145001.hdf5
JobscriptExit code0
Real time36m (2190s)
CPU time34m (2076s = 94%)
Max RSS bytes2631860224 (2509 MiB)
Outputting started2026-02-04 00:12:30
Output files
Finished2026-02-04 00:14:50
Saved logsjustin-logs:297919.134-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

700
Caught exception in looking up space point ptr... 0x15908700, 0x159089a0
Caught exception in looking up space point ptr... 0x368f5660, 0x368f5460
Caught exception in looking up space point ptr... 0x368f5460, 0x32b99b50
Caught exception in looking up space point ptr... 0x3dcf9810, 0x3dd010b0
Caught exception in looking up space point ptr... 0x572d7d20, 0x3dcf9810
Caught exception in looking up space point ptr... 0x2e5f34b0, 0x2e5ec930
Caught exception in looking up space point ptr... 0x2e5ec930, 0x2e5f3640
Caught exception in looking up space point ptr... 0xe5f3d20, 0x13d21990
Caught exception in looking up space point ptr... 0x13d21990, 0xe5f3d20
Caught exception in looking up space point ptr... 0x436be0f0, 0x436bdf50
Caught exception in looking up space point ptr... 0x436bdf50, 0x436be0f0
Caught exception in looking up space point ptr... 0x51ca7a60, 0x51ca7120
Caught exception in looking up space point ptr... 0x51ca75c0, 0x51ca7a60
Caught exception in looking up space point ptr... 0x35c29a00, 0x35c29900
Caught exception in looking up space point ptr... 0x35c29900, 0x35c29800
Caught exception in looking up space point ptr... 0x9bb5760, 0x16bcfea0
Caught exception in looking up space point ptr... 0x16bcfea0, 0x9bb5760
Caught exception in looking up space point ptr... 0x95e6110, 0x5b183620
Caught exception in looking up space point ptr... 0x5b183620, 0x95e6110
Caught exception in looking up space point ptr... 0x51f0a190, 0xfdfc820
Caught exception in looking up space point ptr... 0xfdfc820, 0x51f0a190
Caught exception in looking up space point ptr... 0x27a5c2c0, 0x27a574d0
Caught exception in looking up space point ptr... 0x27a574d0, 0x27a5c2c0
Caught exception in looking up space point ptr... 0x11ce8170, 0x11ce8490
Caught exception in looking up space point ptr... 0x11ce8490, 0x11ce8620
Caught exception in looking up space point ptr... 0x57c3f890, 0x364a6160
Caught exception in looking up space point ptr... 0x364a6160, 0x57c3f890
Caught exception in looking up space point ptr... 0x3d181f40, 0x32a25c20
Caught exception in looking up space point ptr... 0x32a25c20, 0x32a0cf40
Caught exception in looking up space point ptr... 0x251495d0, 0x2514a960
Caught exception in looking up space point ptr... 0x2514a960, 0x251495d0
Caught exception in looking up space point ptr... 0x251495d0, 0x251494c0
Caught exception in looking up space point ptr... 0x251494c0, 0x251495d0
Caught exception in looking up space point ptr... 0x13cbf3e0, 0x13cbf2e0
Caught exception in looking up space point ptr... 0x57192a30, 0x13cbf3e0
Caught exception in looking up space point ptr... 0x5580c260, 0x558123c0
Caught exception in looking up space point ptr... 0x558123c0, 0x55812610
Caught exception in looking up space point ptr... 0x5881dde0, 0x5881dbe0
Caught exception in looking up space point ptr... 0x5881dbe0, 0x48ad7840
Caught exception in looking up space point ptr... 0x2cda1450, 0x2cda1020
Caught exception in looking up space point ptr... 0x2cda1020, 0x2cda1450
Caught exception in looking up space point ptr... 0x2cda1450, 0x2cda15e0
Caught exception in looking up space point ptr... 0x2cda15e0, 0x2cda1450
Caught exception in looking up space point ptr... 0x4a568910, 0x4a56cd90
Caught exception in looking up space point ptr... 0x4a56cd90, 0x4a568910
Caught exception in looking up space point ptr... 0x2ffe74f0, 0x2ffe7360
Caught exception in looking up space point ptr... 0x2ffe7360, 0x2ffe74f0
Caught exception in looking up space point ptr... 0x2ffe74f0, 0x2ffe71d0
Caught exception in looking up space point ptr... 0x2ffe71d0, 0x2ffe74f0
Caught exception in looking up space point ptr... 0x36a8b200, 0x365244a0
Caught exception in looking up space point ptr... 0x36a89b80, 0x36a8b200
Caught exception in looking up space point ptr... 0x53125f30, 0x5311aca0
Caught exception in looking up space point ptr... 0x5311aca0, 0x5311aba0
Caught exception in looking up space point ptr... 0x572c6290, 0x572c6420
Caught exception in looking up space point ptr... 0x572c6420, 0x572c6740
Caught exception in looking up space point ptr... 0x12eb8cb0, 0x12eb8a10
Caught exception in looking up space point ptr... 0x12eb8a10, 0x12eb8e40
Caught exception in looking up space point ptr... 0xf971fa0, 0x4a5c5c70
Caught exception in looking up space point ptr... 0x4a5c5c70, 0xf971fa0
Caught exception in looking up space point ptr... 0x159031b0, 0x15902f90
Caught exception in looking up space point ptr... 0x15902f90, 0x159031b0
Caught exception in looking up space point ptr... 0x15906350, 0x15905d50
Caught exception in looking up space point ptr... 0x15905d50, 0x15906350
Caught exception in looking up space point ptr... 0x16adb8a0, 0x16adb5a0
Caught exception in looking up space point ptr... 0x16adb5a0, 0x16adb8a0
++++>>>> total num hits: 96244, num free: 92098
04-Feb-2026 01:12:23 CET  Closed output file "np02vd_raw_run042421_0146_df-s03-d0_dw_0_20260203T145001_reco_stage0_20260204T001223_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         60.9953       79.0653       96.571        78.5466       6.88196        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       4.1151e-05    8.00744e-05   0.000227764   7.1441e-05    3.67783e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                3.6206        5.12594       24.7502       4.41259       3.86367        27     
produce:triggerrawdecoder:PDVDTriggerReader4     0.00884881     0.0145701     0.0421695     0.0131079    0.00640963       27     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000364037   0.000501402   0.00111485    0.000464398   0.000139341      27     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.00135928    0.00199985    0.00499469    0.00173317    0.000808106      27     
produce:beamevent:BeamEvent                      5.0031e-05    0.00011916    0.000986718   8.3152e-05    0.000171665      27     
produce:pdvddaphne:DAPHNEReaderPDVD                1.88068       2.67885       3.22729       2.6794       0.382489        27     
produce:ophit:OpHitFinder                         0.0260282     0.0348276     0.0440679     0.0342319    0.00509327       27     
produce:wclsdatavd:WireCellToolkit                 48.4274       57.5215       67.0798       56.4491       3.83952        27     
produce:gaushit:GausHitFinder                     0.331153       0.85684       1.41095      0.835426      0.246137        27     
produce:nhitsfilter:NumberOfHitsFilter           0.000121323   0.000212563   0.000458868   0.000192144   6.9959e-05       27     
produce:reco3d:SpacePointSolver                    2.51767       7.75238       11.5095       7.42695       2.20498        27     
produce:hitpdune:DisambigFromSpacePoints          0.0256353     0.0767695     0.137185      0.0669204     0.0275251       27     
produce:cluster3d:Cluster3D                       0.292622       1.52111       3.44786       1.39775      0.659126        27     
[art]:TriggerResults:TriggerResultInserter        1.461e-05    3.13569e-05   8.7342e-05    2.5711e-05    1.73208e-05      27     
end_path:out1:RootOutput                           3.3e-06     7.46085e-06   2.5351e-05     6.82e-06     4.4303e-06       27     
end_path:out1:RootOutput(write)                    2.49128       3.46222       4.50263       3.41033      0.486437        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4600.49 MB
  Peak resident set size usage (VmHWM): 2631.86 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042421_0146_df-s03-d0_dw_0_20260203T145001_reco_stage0_20260204T001223_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042421_0146_df-s03-d0_dw_0_20260203T145001_reco_stage0_20260204T001223_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042421_0146_df-s03-d0_dw_0_20260203T145001_reco_stage0_20260204T001223_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770161769.0,
    "core.end_time": 1770163944.0,
    "core.events": [
      23732,
      23738,
      23744,
      23750,
      23756,
      23762,
      23768,
      23774,
      23780,
      23786,
      23792,
      23798,
      23804,
      23810,
      23816,
      23822,
      23828,
      23834,
      23840,
      23846,
      23852,
      23858,
      23864,
      23870,
      23876,
      23882,
      23888
    ],
    "core.event_count": 27,
    "core.first_event_number": 23732,
    "core.last_event_number": 23888,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42421
    ],
    "core.runs_subruns": [
      4242100001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042421_0146_df-s03-d0_dw_0_20260203T145001.hdf5"
    }
  ]
}
justIN time: 2026-02-04 07:44:21 UTC       justIN version: 01.06.00