justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297921.155@dunegpschedd02.fnal.gov

Jobsub ID297921.155@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:58:26
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce504
Last heartbeat2026-02-04 01:04:31
From worker nodeHostnameb9p06p4022.cern.ch
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:56:03
Input filesvd-protodune:np02vd_raw_run042433_0385_df-s04-d0_dw_0_20260203T210827.hdf5
JobscriptExit code0
Real time1h (3943s)
CPU time1h (3792s = 96%)
Max RSS bytes3382956032 (3226 MiB)
Outputting started2026-02-04 01:01:48
Output files
Finished2026-02-04 01:04:31
Saved logsjustin-logs:297921.155-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

nt ptr... 0x3a3db300, 0x3a3dc9d0
Caught exception in looking up space point ptr... 0x3a3dc9d0, 0x3a3db300
Caught exception in looking up space point ptr... 0x3a3db300, 0x3a3dccd0
Caught exception in looking up space point ptr... 0x3a3dccd0, 0x3a3db300
Caught exception in looking up space point ptr... 0x5dfe2380, 0x5dfe2540
Caught exception in looking up space point ptr... 0x5dfe2540, 0x5dfe2c60
Caught exception in looking up space point ptr... 0x5dfe2c60, 0x5dfe42f0
Caught exception in looking up space point ptr... 0x3a9e4430, 0x1366ec10
Caught exception in looking up space point ptr... 0x1366eb10, 0x3a9e4430
Caught exception in looking up space point ptr... 0x67e4a290, 0x67e469f0
Caught exception in looking up space point ptr... 0x67e469f0, 0x41bb1ee0
Caught exception in looking up space point ptr... 0x41bb1ee0, 0x5ed1ca00
Caught exception in looking up space point ptr... 0x5ed1ca00, 0x5e0697d0
Caught exception in looking up space point ptr... 0x4c211820, 0x3a050fc0
Caught exception in looking up space point ptr... 0x39360e50, 0x4c211820
Caught exception in looking up space point ptr... 0x3bacf0f0, 0x3baceff0
Caught exception in looking up space point ptr... 0x3baceff0, 0x3bab5b80
Caught exception in looking up space point ptr... 0x143f5a30, 0x73c601d0
Caught exception in looking up space point ptr... 0x73c601d0, 0x143f6300
Caught exception in looking up space point ptr... 0x1898aab0, 0x39dfac70
Caught exception in looking up space point ptr... 0x39dfac70, 0x1898aab0
Caught exception in looking up space point ptr... 0x3968d9c0, 0x74256fe0
Caught exception in looking up space point ptr... 0x3968d7e0, 0x3968d9c0
Caught exception in looking up space point ptr... 0x3dff7710, 0x3a29c330
Caught exception in looking up space point ptr... 0x3a29c330, 0x3dff7710
Caught exception in looking up space point ptr... 0x136a64d0, 0x4c0ed480
Caught exception in looking up space point ptr... 0x4c0ed480, 0x3dfe5720
Caught exception in looking up space point ptr... 0x4432c2e0, 0x330a78c0
Caught exception in looking up space point ptr... 0x3dace6b0, 0x4432c2e0
Caught exception in looking up space point ptr... 0x4146daf0, 0x4146d7f0
Caught exception in looking up space point ptr... 0x4146d7f0, 0x4146daf0
Caught exception in looking up space point ptr... 0x1236cef0, 0x1236cc90
Caught exception in looking up space point ptr... 0x12377140, 0x1236cef0
Caught exception in looking up space point ptr... 0x1234d110, 0x1234cd40
Caught exception in looking up space point ptr... 0x12354ce0, 0x1234d110
Caught exception in looking up space point ptr... 0x35872de0, 0x365289c0
Caught exception in looking up space point ptr... 0x365289c0, 0x35872de0
Caught exception in looking up space point ptr... 0x4be2bd80, 0x3b905300
Caught exception in looking up space point ptr... 0x3b905300, 0x3a791180
Caught exception in looking up space point ptr... 0x5c213420, 0x5c218420
Caught exception in looking up space point ptr... 0x5c218420, 0x5c212f60
Caught exception in looking up space point ptr... 0x60c898c0, 0x84edc80
Caught exception in looking up space point ptr... 0x84edc80, 0x73e6f210
Caught exception in looking up space point ptr... 0x74236ee0, 0x74236d50
Caught exception in looking up space point ptr... 0x74236d50, 0x74236ee0
Caught exception in looking up space point ptr... 0x772f6260, 0x3a436560
Caught exception in looking up space point ptr... 0x3a436560, 0x3a435f00
Caught exception in looking up space point ptr... 0x12d9f6e0, 0x3bad06f0
Caught exception in looking up space point ptr... 0x3bad06f0, 0x3bad05f0
Caught exception in looking up space point ptr... 0x73c61930, 0x73c63c60
Caught exception in looking up space point ptr... 0x73c63c60, 0x73c69450
Caught exception in looking up space point ptr... 0x4ff43da0, 0x59543340
Caught exception in looking up space point ptr... 0x59543340, 0x4ff43da0
Caught exception in looking up space point ptr... 0x18d53940, 0x18d53cb0
Caught exception in looking up space point ptr... 0x18d53cb0, 0x18d53e40
Caught exception in looking up space point ptr... 0x5c2158c0, 0x4d261180
Caught exception in looking up space point ptr... 0x4d261180, 0x5c21d730
Caught exception in looking up space point ptr... 0x670ff010, 0x65adcfe0
Caught exception in looking up space point ptr... 0x65adcfe0, 0x670ff5a0
Caught exception in looking up space point ptr... 0x786c7a0, 0x786c650
Caught exception in looking up space point ptr... 0x786c650, 0x786c7a0
Caught exception in looking up space point ptr... 0x12d9e8d0, 0x12d9e420
Caught exception in looking up space point ptr... 0x12d9e420, 0x12d9e8d0
Caught exception in looking up space point ptr... 0x143c43b0, 0x7829c40
Caught exception in looking up space point ptr... 0x7829c40, 0x143ee010
Caught exception in looking up space point ptr... 0x2eff1d30, 0x65624b00
Caught exception in looking up space point ptr... 0x5c4b27d0, 0x2eff1d30
++++>>>> total num hits: 56104, num free: 53753
04-Feb-2026 02:01:33 CET  Closed output file "np02vd_raw_run042433_0385_df-s04-d0_dw_0_20260203T210827_reco_stage0_20260204T010133_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         108.449       200.499       246.126       216.297       40.7361        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                        5.936e-05    0.000212359   0.000599469   0.000177983   0.000141219      19     
produce:tpcrawdecoder:PDVDTPCReader                5.58477       8.74135       10.0705       9.29913       1.32163        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.0114726     0.0303387     0.0575426     0.028452      0.0123907       19     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000392796   0.00157178    0.00307244    0.00132603    0.000965946      19     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.000378526   0.00150092    0.00361994    0.00135465    0.000934787      19     
produce:beamevent:BeamEvent                      5.1661e-05    0.000452389   0.00404676    0.000214193   0.000896079      19     
produce:pdvddaphne:DAPHNEReaderPDVD                2.17471       3.49208       5.08272       3.53238      0.745202        19     
produce:ophit:OpHitFinder                         0.026464      0.0443096     0.0751481      0.0437       0.0108742       19     
produce:wclsdatavd:WireCellToolkit                 84.9487       159.928       198.887       170.818       34.372         19     
produce:gaushit:GausHitFinder                     0.780343       1.78042       2.89709       1.92226      0.591312        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000200223   0.00131033    0.00398315    0.000573159   0.00119405       19     
produce:reco3d:SpacePointSolver                    9.03262       16.2255       22.6144       17.3631       3.85489        19     
produce:hitpdune:DisambigFromSpacePoints          0.0936832     0.266152       0.42004      0.260709      0.0975668       19     
produce:cluster3d:Cluster3D                        1.27067       3.58536       7.05876       3.26766       1.65066        19     
[art]:TriggerResults:TriggerResultInserter        1.865e-05    6.17236e-05   0.000147152   5.5011e-05    3.79882e-05      19     
end_path:out1:RootOutput                          4.74e-06     1.67539e-05   3.8591e-05    1.2261e-05    1.07212e-05      19     
end_path:out1:RootOutput(write)                    3.84635       6.32463       8.0712        6.51907       1.13721        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5333.66 MB
  Peak resident set size usage (VmHWM): 3382.96 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0385_df-s04-d0_dw_0_20260203T210827_reco_stage0_20260204T010133_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0385_df-s04-d0_dw_0_20260203T210827_reco_stage0_20260204T010133_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0385_df-s04-d0_dw_0_20260203T210827_reco_stage0_20260204T010133_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770162986.0,
    "core.end_time": 1770166895.0,
    "core.events": [
      43900,
      43906,
      43912,
      43918,
      43924,
      43930,
      43936,
      43942,
      43948,
      43954,
      43960,
      43966,
      43972,
      43978,
      43984,
      43990,
      43996,
      44002,
      44008
    ],
    "core.event_count": 19,
    "core.first_event_number": 43900,
    "core.last_event_number": 44008,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0385_df-s04-d0_dw_0_20260203T210827.hdf5"
    }
  ]
}
justIN time: 2026-02-04 06:06:07 UTC       justIN version: 01.06.00