justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303653.92@dunegpschedd01.fnal.gov

Jobsub ID303653.92@dunegpschedd01.fnal.gov
Workflow ID12717
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:52:24
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce508
Last heartbeat2026-02-04 00:12:11
From worker nodeHostnameb9p35p0022.cern.ch
cpuinfoAMD EPYC 9655P 96-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:41:08
Input filesvd-protodune:np02vd_raw_run042421_0047_df-s03-d1_dw_0_20260203T135626.hdf5
JobscriptExit code0
Real time28m (1732s)
CPU time27m (1645s = 94%)
Max RSS bytes2631852032 (2509 MiB)
Outputting started2026-02-04 00:10:01
Output files
Finished2026-02-04 00:12:11
Saved logsjustin-logs:303653.92-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ptr... 0x134946f0, 0x13494820
Caught exception in looking up space point ptr... 0x13494820, 0x13494950
Caught exception in looking up space point ptr... 0x13494950, 0x13494820
Caught exception in looking up space point ptr... 0x26828980, 0x347c3c40
Caught exception in looking up space point ptr... 0x347c3c40, 0x1386b1a0
Caught exception in looking up space point ptr... 0x15c799c0, 0x635a7710
Caught exception in looking up space point ptr... 0x635a7710, 0x4cee3c50
Caught exception in looking up space point ptr... 0x2be1bc30, 0x2be1e8f0
Caught exception in looking up space point ptr... 0x2be1e8f0, 0x2be1e190
Caught exception in looking up space point ptr... 0x357d3e10, 0x357d3af0
Caught exception in looking up space point ptr... 0x134945c0, 0x357d3e10
Caught exception in looking up space point ptr... 0x1376aba0, 0x1376a9b0
Caught exception in looking up space point ptr... 0x1376a9b0, 0x1376aba0
Caught exception in looking up space point ptr... 0x557b1980, 0x53eca7b0
Caught exception in looking up space point ptr... 0x53eca7b0, 0x557b1980
Caught exception in looking up space point ptr... 0x2db286c0, 0x121a1340
Caught exception in looking up space point ptr... 0x121a1340, 0x47c69250
Caught exception in looking up space point ptr... 0x45b353e0, 0x360dd000
Caught exception in looking up space point ptr... 0x360dd000, 0x58422220
Caught exception in looking up space point ptr... 0x360da140, 0x34ab2f20
Caught exception in looking up space point ptr... 0x34ab2f20, 0x34ab2d40
Caught exception in looking up space point ptr... 0xb35ac10, 0xb358990
Caught exception in looking up space point ptr... 0xb358990, 0xb34b0e0
Caught exception in looking up space point ptr... 0xb359000, 0xb35ac10
Caught exception in looking up space point ptr... 0xfbacc70, 0x4e9ca270
Caught exception in looking up space point ptr... 0x4e9ca270, 0xfbacc70
Caught exception in looking up space point ptr... 0x47dd2530, 0x47dc07e0
Caught exception in looking up space point ptr... 0x47dc07e0, 0x47dd2d10
Caught exception in looking up space point ptr... 0x3b48cc30, 0x3b48c990
Caught exception in looking up space point ptr... 0x3b48c990, 0x3b48cc30
Caught exception in looking up space point ptr... 0x469b9b90, 0x469b96e0
Caught exception in looking up space point ptr... 0x469b96e0, 0x469b9b90
Caught exception in looking up space point ptr... 0x16968860, 0x45f2f7c0
Caught exception in looking up space point ptr... 0x45f2f7c0, 0x13f80100
Caught exception in looking up space point ptr... 0x131a46d0, 0x131a5160
Caught exception in looking up space point ptr... 0x131a5160, 0x131a61b0
Caught exception in looking up space point ptr... 0x296286c0, 0x29619940
Caught exception in looking up space point ptr... 0x29619940, 0x29619870
Caught exception in looking up space point ptr... 0x29619870, 0x296252f0
Caught exception in looking up space point ptr... 0x296252f0, 0x296286c0
Caught exception in looking up space point ptr... 0x598a2800, 0x598a26d0
Caught exception in looking up space point ptr... 0x598a26d0, 0x598a2800
Caught exception in looking up space point ptr... 0x27f0d720, 0x2962be60
Caught exception in looking up space point ptr... 0x21cf8180, 0x27f0d720
Caught exception in looking up space point ptr... 0x15477340, 0x49ab9e30
Caught exception in looking up space point ptr... 0x49ab9e30, 0x49ab9c50
Caught exception in looking up space point ptr... 0x242e7fe0, 0x24301b60
Caught exception in looking up space point ptr... 0x24301b60, 0x28c56920
Caught exception in looking up space point ptr... 0x33a41060, 0x134a19f0
Caught exception in looking up space point ptr... 0x134a19f0, 0x33a41060
Caught exception in looking up space point ptr... 0x21d0d490, 0x464e5490
Caught exception in looking up space point ptr... 0x464e5490, 0x464e56b0
Caught exception in looking up space point ptr... 0x13f9a130, 0x16951ef0
Caught exception in looking up space point ptr... 0x16951ef0, 0x13f9a130
Caught exception in looking up space point ptr... 0x598ccb70, 0x2d4b5e60
Caught exception in looking up space point ptr... 0x2d4b5e60, 0x36c187c0
Caught exception in looking up space point ptr... 0x169668c0, 0x169665a0
Caught exception in looking up space point ptr... 0x169665a0, 0x169668c0
Caught exception in looking up space point ptr... 0x47dbf340, 0x47dbf700
Caught exception in looking up space point ptr... 0x47dbf700, 0x47dcb2e0
Caught exception in looking up space point ptr... 0x46077110, 0x46077870
Caught exception in looking up space point ptr... 0x46077870, 0x46077110
Caught exception in looking up space point ptr... 0x296122e0, 0x29612c90
Caught exception in looking up space point ptr... 0x29612c90, 0x296122e0
Caught exception in looking up space point ptr... 0x38b8eec0, 0x38b8fc20
Caught exception in looking up space point ptr... 0x38b8fc20, 0x38b8eec0
++++>>>> total num hits: 50345, num free: 48235
04-Feb-2026 01:09:55 CET  Closed output file "np02vd_raw_run042421_0047_df-s03-d1_dw_0_20260203T135626_reco_stage0_20260204T000955_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         53.2481       62.5221       74.3197       61.1471       5.10397        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                         5.4e-05     7.72981e-05   0.000248241   7.2951e-05    3.50602e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                3.64053       4.11873       4.52591       4.12005      0.242282        27     
produce:triggerrawdecoder:PDVDTriggerReader4     0.00763948     0.013028      0.0309441     0.0101452    0.00602492       27     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000318162   0.000561276   0.00105829    0.000459802   0.000227111      27     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.00110679     0.002014      0.0071857    0.00145156    0.00135725       27     
produce:beamevent:BeamEvent                      5.6741e-05    9.54764e-05   0.000631314    7.202e-05    0.000106369      27     
produce:pdvddaphne:DAPHNEReaderPDVD                1.46239       1.78279       2.26203       1.75962      0.188856        27     
produce:ophit:OpHitFinder                         0.0233554     0.0299191     0.0361769     0.0298394    0.00376473       27     
produce:wclsdatavd:WireCellToolkit                 38.4649       43.0664       53.4066       41.7177       3.4899         27     
produce:gaushit:GausHitFinder                     0.384667      0.650299      0.928656      0.654186      0.127154        27     
produce:nhitsfilter:NumberOfHitsFilter           0.00011596    0.000197013   0.000479062   0.000179511   8.15492e-05      27     
produce:reco3d:SpacePointSolver                    4.31395       7.93643       11.9364       7.43835       2.00181        27     
produce:hitpdune:DisambigFromSpacePoints          0.0412296     0.0712951     0.114762      0.0680716     0.0206614       27     
produce:cluster3d:Cluster3D                        0.50268       1.35738       2.14786       1.40279      0.428165        27     
[art]:TriggerResults:TriggerResultInserter        2.322e-05    3.05931e-05   7.4841e-05     2.726e-05    1.02111e-05      27     
end_path:out1:RootOutput                          4.28e-06     5.84856e-06    1.965e-05     5.04e-06     2.80097e-06      27     
end_path:out1:RootOutput(write)                    2.61345       3.46953       5.42172       3.46893       0.57625        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4671.39 MB
  Peak resident set size usage (VmHWM): 2631.85 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042421_0047_df-s03-d1_dw_0_20260203T135626_reco_stage0_20260204T000955_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042421_0047_df-s03-d1_dw_0_20260203T135626_reco_stage0_20260204T000955_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042421_0047_df-s03-d1_dw_0_20260203T135626_reco_stage0_20260204T000955_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770162074.0,
    "core.end_time": 1770163796.0,
    "core.events": [
      7665,
      7671,
      7677,
      7683,
      7689,
      7695,
      7701,
      7707,
      7713,
      7719,
      7725,
      7731,
      7737,
      7743,
      7749,
      7755,
      7761,
      7767,
      7773,
      7779,
      7785,
      7791,
      7797,
      7803,
      7809,
      7815,
      7821
    ],
    "core.event_count": 27,
    "core.first_event_number": 7665,
    "core.last_event_number": 7821,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42421
    ],
    "core.runs_subruns": [
      4242100001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042421_0047_df-s03-d1_dw_0_20260203T135626.hdf5"
    }
  ]
}
justIN time: 2026-02-04 09:13:32 UTC       justIN version: 01.06.00