justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303652.183@dunegpschedd01.fnal.gov

Jobsub ID303652.183@dunegpschedd01.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:50:26
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce515
Last heartbeat2026-02-04 00:14:27
From worker nodeHostnameb9p12p0201.cern.ch
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:34:58
Input filesvd-protodune:np02vd_raw_run042433_0625_df-s04-d0_dw_0_20260203T223940.hdf5
JobscriptExit code0
Real time37m (2237s)
CPU time35m (2147s = 95%)
Max RSS bytes3423121408 (3264 MiB)
Outputting started2026-02-04 00:12:16
Output files
Finished2026-02-04 00:14:27
Saved logsjustin-logs:303652.183-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ce point ptr... 0x68726220, 0x687260d0
Caught exception in looking up space point ptr... 0x4a65fb40, 0x2e54d8a0
Caught exception in looking up space point ptr... 0x2e54d8a0, 0x4a65fb40
Caught exception in looking up space point ptr... 0x293c28a0, 0x29319aa0
Caught exception in looking up space point ptr... 0x29319aa0, 0x293c2a40
Caught exception in looking up space point ptr... 0x6bbf5050, 0x28c06840
Caught exception in looking up space point ptr... 0x28c06840, 0x6bbf5050
Caught exception in looking up space point ptr... 0x8b10a10, 0x8b10880
Caught exception in looking up space point ptr... 0x8b10880, 0x8b10a10
Caught exception in looking up space point ptr... 0x2c63ac30, 0x2c63af50
Caught exception in looking up space point ptr... 0x2c63af50, 0x2c63aaa0
Caught exception in looking up space point ptr... 0x8b05590, 0x8b05400
Caught exception in looking up space point ptr... 0x8b05400, 0x8b05590
Caught exception in looking up space point ptr... 0x60785380, 0x32d3d1b0
Caught exception in looking up space point ptr... 0x32d3d1b0, 0x60785380
Caught exception in looking up space point ptr... 0x29eab1a0, 0x60912b60
Caught exception in looking up space point ptr... 0x60912b60, 0x29eab1a0
Caught exception in looking up space point ptr... 0x173a0720, 0x5197d270
Caught exception in looking up space point ptr... 0x5197d270, 0x5a75acf0
Caught exception in looking up space point ptr... 0x804c2ec0, 0x193455e0
Caught exception in looking up space point ptr... 0x193455e0, 0x804c2ec0
Caught exception in looking up space point ptr... 0x43d9ca10, 0x43d9c8c0
Caught exception in looking up space point ptr... 0x43d9c8c0, 0x43d9ca10
Caught exception in looking up space point ptr... 0x2c675690, 0x2c673ed0
Caught exception in looking up space point ptr... 0x2c673ed0, 0x2c675690
Caught exception in looking up space point ptr... 0x6bec8730, 0x14c5e630
Caught exception in looking up space point ptr... 0x14c5e630, 0x6bec8730
Caught exception in looking up space point ptr... 0x3f0302e0, 0x3f030150
Caught exception in looking up space point ptr... 0x3f030150, 0x3f0302e0
Caught exception in looking up space point ptr... 0x29ea99d0, 0xdb71000
Caught exception in looking up space point ptr... 0xdb71000, 0x29ea9730
Caught exception in looking up space point ptr... 0x37fd0e80, 0x37fd0b10
Caught exception in looking up space point ptr... 0x37fd0b10, 0x37fd0e80
Caught exception in looking up space point ptr... 0x37fd0e80, 0x37fd0cf0
Caught exception in looking up space point ptr... 0x37fd0cf0, 0x37fd0e80
Caught exception in looking up space point ptr... 0x8050bf90, 0x8050b0a0
Caught exception in looking up space point ptr... 0x8050b0a0, 0x8050bf90
Caught exception in looking up space point ptr... 0x6be1de40, 0x6be1dbb0
Caught exception in looking up space point ptr... 0x6be1e360, 0x6be1de40
Caught exception in looking up space point ptr... 0x37ffda40, 0x37ffddb0
Caught exception in looking up space point ptr... 0x37ffddb0, 0x37ffda40
Caught exception in looking up space point ptr... 0x37ffda40, 0x37ffe0d0
Caught exception in looking up space point ptr... 0x37ffe0d0, 0x37ffda40
Caught exception in looking up space point ptr... 0x800788c0, 0x80078770
Caught exception in looking up space point ptr... 0x80078770, 0x800788c0
Caught exception in looking up space point ptr... 0x8b07540, 0x8b07220
Caught exception in looking up space point ptr... 0x8b07220, 0x8b07540
Caught exception in looking up space point ptr... 0x583ea920, 0x5df63450
Caught exception in looking up space point ptr... 0x5df63450, 0x583ea920
Caught exception in looking up space point ptr... 0x583ea920, 0x68fc5bd0
Caught exception in looking up space point ptr... 0x68fc5bd0, 0x583ea920
Caught exception in looking up space point ptr... 0x10bf81f0, 0x10bf55e0
Caught exception in looking up space point ptr... 0x10bf55e0, 0x10bf5900
Caught exception in looking up space point ptr... 0x6be30320, 0x44585980
Caught exception in looking up space point ptr... 0x44585980, 0x6be30320
Caught exception in looking up space point ptr... 0x2c679150, 0x2c678240
Caught exception in looking up space point ptr... 0x2c678240, 0x2ea8e370
Caught exception in looking up space point ptr... 0x29218ff0, 0x29219250
Caught exception in looking up space point ptr... 0x29219250, 0x29218ff0
Caught exception in looking up space point ptr... 0x29218ff0, 0x29219120
Caught exception in looking up space point ptr... 0x29219120, 0x29218ff0
Caught exception in looking up space point ptr... 0x14e0b8e0, 0x57ef62d0
Caught exception in looking up space point ptr... 0x57ef62d0, 0x14e0b8e0
Caught exception in looking up space point ptr... 0x29219710, 0x292195e0
Caught exception in looking up space point ptr... 0x292195e0, 0x29219710
Caught exception in looking up space point ptr... 0x12cc7ba0, 0x12cc8050
Caught exception in looking up space point ptr... 0x12cc8050, 0x12cc7ba0
++++>>>> total num hits: 94916, num free: 91082
04-Feb-2026 01:12:09 CET  Closed output file "np02vd_raw_run042433_0625_df-s04-d0_dw_0_20260203T223940_reco_stage0_20260204T001209_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         107.416       114.812       128.007       113.967       5.63334        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                        4.463e-05    7.72893e-05   0.000196571   6.3931e-05    3.88813e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                5.56189       6.02831       7.11712       5.96964      0.372131        19     
produce:triggerrawdecoder:PDVDTriggerReader4     0.00949526     0.0187384     0.101937      0.0110164     0.0207555       19     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000371811   0.00047154    0.000704612   0.000448361   9.20956e-05      19     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.000370262   0.000449437   0.000724972   0.000415172   8.92916e-05      19     
produce:beamevent:BeamEvent                       5.174e-05    0.000127549   0.000879183    8.658e-05    0.000177997      19     
produce:pdvddaphne:DAPHNEReaderPDVD                1.86818       2.19672       2.62658       2.11281      0.216472        19     
produce:ophit:OpHitFinder                         0.022327      0.0299593     0.0356442     0.0308076    0.00374126       19     
produce:wclsdatavd:WireCellToolkit                 81.3682       83.7317       86.3635       83.3618       1.5882         19     
produce:gaushit:GausHitFinder                      0.81262       1.02136       1.1865        1.02537      0.115937        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000212881   0.00028776    0.000667022   0.000258841   9.76303e-05      19     
produce:reco3d:SpacePointSolver                    9.79914       15.4974       24.7846       13.7218       4.58959        19     
produce:hitpdune:DisambigFromSpacePoints          0.0878921     0.145508      0.189058       0.14353      0.0219987       19     
produce:cluster3d:Cluster3D                        1.09934       2.04397       3.0991        1.9572       0.472836        19     
[art]:TriggerResults:TriggerResultInserter        2.11e-05     3.44006e-05   0.000118631    2.998e-05    2.02302e-05      19     
end_path:out1:RootOutput                          3.55e-06     8.58105e-06    3.134e-05     7.66e-06     5.69829e-06      19     
end_path:out1:RootOutput(write)                    3.88492       4.07016       4.38665       4.04741      0.135711        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5432.77 MB
  Peak resident set size usage (VmHWM): 3423.12 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0625_df-s04-d0_dw_0_20260203T223940_reco_stage0_20260204T001209_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0625_df-s04-d0_dw_0_20260203T223940_reco_stage0_20260204T001209_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0625_df-s04-d0_dw_0_20260203T223940_reco_stage0_20260204T001209_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770161712.0,
    "core.end_time": 1770163930.0,
    "core.events": [
      71260,
      71266,
      71272,
      71278,
      71284,
      71290,
      71296,
      71302,
      71308,
      71314,
      71320,
      71326,
      71332,
      71338,
      71344,
      71350,
      71356,
      71362,
      71368
    ],
    "core.event_count": 19,
    "core.first_event_number": 71260,
    "core.last_event_number": 71368,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0625_df-s04-d0_dw_0_20260203T223940.hdf5"
    }
  ]
}
justIN time: 2026-02-04 07:42:30 UTC       justIN version: 01.06.00