justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303652.13@dunegpschedd01.fnal.gov

Jobsub ID303652.13@dunegpschedd01.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:50:26
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce503
Last heartbeat2026-02-04 00:08:58
From worker nodeHostnameb9p05p0564.cern.ch
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:25:52
Input filesvd-protodune:np02vd_raw_run042433_0099_df-s03-d1_dw_0_20260203T191945.hdf5
JobscriptExit code0
Real time40m (2442s)
CPU time39m (2348s = 96%)
Max RSS bytes3437555712 (3278 MiB)
Outputting started2026-02-04 00:06:35
Output files
Finished2026-02-04 00:08:58
Saved logsjustin-logs:303652.13-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

king up space point ptr... 0x45e67a80, 0x8e6cbb0
Caught exception in looking up space point ptr... 0x8e6cbb0, 0x45e67450
Caught exception in looking up space point ptr... 0x37d02a70, 0x37d02c00
Caught exception in looking up space point ptr... 0x37d02c00, 0x37d0b9d0
Caught exception in looking up space point ptr... 0x4e837b80, 0x4e8378b0
Caught exception in looking up space point ptr... 0x4e8378b0, 0x4e837b80
Caught exception in looking up space point ptr... 0x4e837b80, 0x4e836580
Caught exception in looking up space point ptr... 0x4e836580, 0x4e837b80
Caught exception in looking up space point ptr... 0x4e837b80, 0x4e8368a0
Caught exception in looking up space point ptr... 0x4e8368a0, 0x4e837b80
Caught exception in looking up space point ptr... 0x4e837b80, 0x4e837d10
Caught exception in looking up space point ptr... 0x4e837d10, 0x4e837b80
Caught exception in looking up space point ptr... 0x38a1daf0, 0x38a1dc80
Caught exception in looking up space point ptr... 0x42fd3a10, 0x38a1daf0
Caught exception in looking up space point ptr... 0xf97b8e0, 0xf97a9f0
Caught exception in looking up space point ptr... 0x6b5ba8f0, 0xf97b8e0
Caught exception in looking up space point ptr... 0x36857f60, 0x36858060
Caught exception in looking up space point ptr... 0x36858060, 0x6354a9e0
Caught exception in looking up space point ptr... 0x6857beb0, 0x6857c870
Caught exception in looking up space point ptr... 0x6857c870, 0x3653a1d0
Caught exception in looking up space point ptr... 0x53d459f0, 0x53d45b80
Caught exception in looking up space point ptr... 0x53d45b80, 0x53d52270
Caught exception in looking up space point ptr... 0x53d52270, 0x53d47f30
Caught exception in looking up space point ptr... 0x887eedc0, 0x887ee6c0
Caught exception in looking up space point ptr... 0x887ee6c0, 0x887eedc0
Caught exception in looking up space point ptr... 0x70a8cac0, 0x70a8e9e0
Caught exception in looking up space point ptr... 0x70a8e9e0, 0x70a8cac0
Caught exception in looking up space point ptr... 0x19d9bc40, 0x3bf547f0
Caught exception in looking up space point ptr... 0x3bf547f0, 0x19d9bc40
Caught exception in looking up space point ptr... 0x17a29320, 0x17a2b760
Caught exception in looking up space point ptr... 0x17a2b760, 0x17a29320
Caught exception in looking up space point ptr... 0x40fc9ed0, 0x43254500
Caught exception in looking up space point ptr... 0x43254500, 0x40fc9ed0
Caught exception in looking up space point ptr... 0x545858d0, 0x3f69c330
Caught exception in looking up space point ptr... 0x3f69c330, 0x545858d0
Caught exception in looking up space point ptr... 0x3f77acf0, 0x3f779180
Caught exception in looking up space point ptr... 0x3f779180, 0x3f77acf0
Caught exception in looking up space point ptr... 0xf963660, 0xf95cf70
Caught exception in looking up space point ptr... 0xf963c50, 0xf963660
Caught exception in looking up space point ptr... 0x37d00860, 0x50e7eff0
Caught exception in looking up space point ptr... 0x50e7eff0, 0x37d0dcd0
Caught exception in looking up space point ptr... 0x469c4750, 0x194724c0
Caught exception in looking up space point ptr... 0x194724c0, 0x469c4750
Caught exception in looking up space point ptr... 0xf4009a0, 0x4991d750
Caught exception in looking up space point ptr... 0x4991d750, 0x1a351470
Caught exception in looking up space point ptr... 0x96a82e0, 0x366cc950
Caught exception in looking up space point ptr... 0x366cc950, 0x96a82e0
Caught exception in looking up space point ptr... 0x647013e0, 0x64701250
Caught exception in looking up space point ptr... 0x64701250, 0x647013e0
Caught exception in looking up space point ptr... 0x432547d0, 0x5384ca00
Caught exception in looking up space point ptr... 0x5384ca00, 0x432547d0
Caught exception in looking up space point ptr... 0x432547d0, 0x43254960
Caught exception in looking up space point ptr... 0x43254960, 0x432547d0
Caught exception in looking up space point ptr... 0x432547d0, 0x5384c800
Caught exception in looking up space point ptr... 0x5384c800, 0x432547d0
Caught exception in looking up space point ptr... 0xf953140, 0xf952e90
Caught exception in looking up space point ptr... 0xf952e90, 0xf953010
Caught exception in looking up space point ptr... 0x91c1990, 0x13b55020
Caught exception in looking up space point ptr... 0x13b55020, 0x13b54e60
Caught exception in looking up space point ptr... 0xf9520e0, 0xf951fb0
Caught exception in looking up space point ptr... 0xf951fb0, 0xf9520e0
Caught exception in looking up space point ptr... 0x4c959ab0, 0x4c957d50
Caught exception in looking up space point ptr... 0x4c957d50, 0x4c9556c0
Caught exception in looking up space point ptr... 0x64027fe0, 0x7f0e7d80
Caught exception in looking up space point ptr... 0x7f0e7d80, 0x5ca33ba0
Caught exception in looking up space point ptr... 0xa6ca830, 0xd90dae0
Caught exception in looking up space point ptr... 0xd90dae0, 0xa6ca830
++++>>>> total num hits: 74545, num free: 71887
04-Feb-2026 01:06:27 CET  Closed output file "np02vd_raw_run042433_0099_df-s03-d1_dw_0_20260203T191945_reco_stage0_20260204T000627_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         116.264       125.694       149.311       122.281       8.2006         19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       4.8551e-05    0.000100221   0.000291863   7.4791e-05    6.16893e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                5.79586       6.86318       9.61503       6.6642        1.01692        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.0106164     0.0170062     0.0423959     0.0135822    0.00837952       19     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000403835   0.000837768   0.00559864    0.000558256   0.00112965       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.000363595   0.000543258   0.000835781   0.000522057   0.000120938      19     
produce:beamevent:BeamEvent                      6.1411e-05    0.000173472   0.000967732   0.000116762   0.000195682      19     
produce:pdvddaphne:DAPHNEReaderPDVD                2.05203       2.61986       3.2311        2.69092       0.35466        19     
produce:ophit:OpHitFinder                         0.0202032     0.034085      0.0420338     0.0352011    0.00536336       19     
produce:wclsdatavd:WireCellToolkit                 87.9101       94.1781       103.17        92.0755       4.36736        19     
produce:gaushit:GausHitFinder                     0.701769       1.20856       1.86954       1.07656      0.379841        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000192093   0.000290567   0.000467446   0.000272133   6.64122e-05      19     
produce:reco3d:SpacePointSolver                    7.90524       13.8338       23.2941       12.456        3.87725        19     
produce:hitpdune:DisambigFromSpacePoints          0.0838792     0.153102      0.227543      0.137504      0.0453763       19     
produce:cluster3d:Cluster3D                        1.02108       2.3382        4.31254       2.25235       0.92029        19     
[art]:TriggerResults:TriggerResultInserter        2.178e-05    3.16683e-05   0.000105641    2.629e-05    1.81379e-05      19     
end_path:out1:RootOutput                          3.48e-06     8.07689e-06    2.308e-05     7.78e-06     4.16162e-06      19     
end_path:out1:RootOutput(write)                    3.96553       4.4115        5.17598       4.29822      0.327802        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5448.41 MB
  Peak resident set size usage (VmHWM): 3437.56 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0099_df-s03-d1_dw_0_20260203T191945_reco_stage0_20260204T000627_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0099_df-s03-d1_dw_0_20260203T191945_reco_stage0_20260204T000627_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0099_df-s03-d1_dw_0_20260203T191945_reco_stage0_20260204T000627_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770161162.0,
    "core.end_time": 1770163588.0,
    "core.events": [
      11295,
      11301,
      11307,
      11313,
      11319,
      11325,
      11331,
      11337,
      11343,
      11349,
      11355,
      11361,
      11367,
      11373,
      11379,
      11385,
      11391,
      11397,
      11403
    ],
    "core.event_count": 19,
    "core.first_event_number": 11295,
    "core.last_event_number": 11403,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0099_df-s03-d1_dw_0_20260203T191945.hdf5"
    }
  ]
}
justIN time: 2026-02-04 07:44:00 UTC       justIN version: 01.06.00