justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303652.122@dunegpschedd01.fnal.gov

Jobsub ID303652.122@dunegpschedd01.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:50:26
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce506
Last heartbeat2026-02-04 00:17:38
From worker nodeHostnameb9p06p2822.cern.ch
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:28:54
Input filesvd-protodune:np02vd_raw_run042433_0305_df-s04-d0_dw_0_20260203T203802.hdf5
JobscriptExit code0
Real time46m (2773s)
CPU time44m (2658s = 95%)
Max RSS bytes3406442496 (3248 MiB)
Outputting started2026-02-04 00:15:09
Output files
Finished2026-02-04 00:17:38
Saved logsjustin-logs:303652.122-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

e point ptr... 0x5ae7cf80, 0x5ae944a0
Caught exception in looking up space point ptr... 0x5ae944a0, 0x5ae97ae0
Caught exception in looking up space point ptr... 0x18018660, 0x42ee4780
Caught exception in looking up space point ptr... 0x42ee4780, 0x18018660
Caught exception in looking up space point ptr... 0x451358c0, 0x45134650
Caught exception in looking up space point ptr... 0x45134650, 0x110a2300
Caught exception in looking up space point ptr... 0x5035f180, 0x142417b0
Caught exception in looking up space point ptr... 0x142417b0, 0x5035f180
Caught exception in looking up space point ptr... 0x515cdf40, 0x3a0e4390
Caught exception in looking up space point ptr... 0x515ce7c0, 0x515cdf40
Caught exception in looking up space point ptr... 0x3309bfd0, 0x330a5510
Caught exception in looking up space point ptr... 0x330a5510, 0x330a9560
Caught exception in looking up space point ptr... 0x330a5250, 0x330a5eb0
Caught exception in looking up space point ptr... 0x330a5eb0, 0x3309bfd0
Caught exception in looking up space point ptr... 0x3f102980, 0x3f1026e0
Caught exception in looking up space point ptr... 0x3f1026e0, 0x3f102830
Caught exception in looking up space point ptr... 0x29c75820, 0x29c75690
Caught exception in looking up space point ptr... 0x29c75690, 0x29c759b0
Caught exception in looking up space point ptr... 0x29c75cd0, 0x29c79370
Caught exception in looking up space point ptr... 0x29c79370, 0x29c75820
Caught exception in looking up space point ptr... 0x51945890, 0x51946a70
Caught exception in looking up space point ptr... 0x51946a70, 0x51945890
Caught exception in looking up space point ptr... 0x130d3960, 0x33a62b60
Caught exception in looking up space point ptr... 0x33a62b60, 0x130d3960
Caught exception in looking up space point ptr... 0x4e2ae830, 0x5015fb20
Caught exception in looking up space point ptr... 0x5015fb20, 0x367bad80
Caught exception in looking up space point ptr... 0x18959920, 0x189597d0
Caught exception in looking up space point ptr... 0x189597d0, 0x18959920
Caught exception in looking up space point ptr... 0x66a0b5a0, 0x669ebd20
Caught exception in looking up space point ptr... 0x669ebd20, 0x669ec5b0
Caught exception in looking up space point ptr... 0x669ec5b0, 0x66a09130
Caught exception in looking up space point ptr... 0x66a09130, 0x66a0b5a0
Caught exception in looking up space point ptr... 0x4a8b85d0, 0x4a8b8530
Caught exception in looking up space point ptr... 0x4a8b8530, 0x6fce94b0
Caught exception in looking up space point ptr... 0x16ee9a00, 0x16ee9b90
Caught exception in looking up space point ptr... 0x16ee9b90, 0x16ee9eb0
Caught exception in looking up space point ptr... 0xde2abb0, 0xde2aa60
Caught exception in looking up space point ptr... 0xde2aa60, 0xde2abb0
Caught exception in looking up space point ptr... 0x6f3c4490, 0x6f3c4000
Caught exception in looking up space point ptr... 0x6f3c4000, 0x6f3c4490
Caught exception in looking up space point ptr... 0x4a5869b0, 0x4a5868b0
Caught exception in looking up space point ptr... 0x4a5868b0, 0x3cf66570
Caught exception in looking up space point ptr... 0x66a05a40, 0x66a09850
Caught exception in looking up space point ptr... 0x66a09850, 0x66a05a40
Caught exception in looking up space point ptr... 0x100e9860, 0x50160c90
Caught exception in looking up space point ptr... 0x50160c90, 0x100e9860
Caught exception in looking up space point ptr... 0x2c200ec0, 0x2c2057f0
Caught exception in looking up space point ptr... 0x2c2057f0, 0x8a8be620
Caught exception in looking up space point ptr... 0x8f6e5c90, 0x3afbc3e0
Caught exception in looking up space point ptr... 0x3afbc3e0, 0x8f6e5c90
Caught exception in looking up space point ptr... 0x4bb99a40, 0x412b76a0
Caught exception in looking up space point ptr... 0x412b76a0, 0x9d04730
Caught exception in looking up space point ptr... 0x41233ce0, 0x685786d0
Caught exception in looking up space point ptr... 0x685786d0, 0x51893850
Caught exception in looking up space point ptr... 0x6f865300, 0x623de580
Caught exception in looking up space point ptr... 0x623de580, 0x19540d50
Caught exception in looking up space point ptr... 0xfc8b6f0, 0xfc91ca0
Caught exception in looking up space point ptr... 0xfc91ca0, 0xfc92ad0
Caught exception in looking up space point ptr... 0xfc92ad0, 0xfc929a0
Caught exception in looking up space point ptr... 0x2dc317b0, 0xde49c40
Caught exception in looking up space point ptr... 0xde49c40, 0x2dc317b0
Caught exception in looking up space point ptr... 0x540855e0, 0x1a4c8a80
Caught exception in looking up space point ptr... 0x1a4c8a80, 0x540855e0
Caught exception in looking up space point ptr... 0x38cd2550, 0x19e30890
Caught exception in looking up space point ptr... 0x19e30890, 0x38cd2550
Caught exception in looking up space point ptr... 0x6f3cb3b0, 0x6f3cb4b0
Caught exception in looking up space point ptr... 0x6f3cb4b0, 0x6f3c14e0
++++>>>> total num hits: 79116, num free: 75807
04-Feb-2026 01:14:58 CET  Closed output file "np02vd_raw_run042433_0305_df-s04-d0_dw_0_20260203T203802_reco_stage0_20260204T001458_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         124.994       142.548       170.895       139.736       10.6484        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       5.7871e-05    0.000111821   0.000196521   0.000109891   3.96931e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                 6.515        7.65304       12.5713       7.46714       1.22949        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.0109109     0.0150861     0.0269602     0.0137598    0.00413783       19     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000389772   0.00152769     0.0179172    0.000528632   0.00386875       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.000385521   0.000551421   0.00127407    0.000505993   0.00018746       19     
produce:beamevent:BeamEvent                       7.43e-05     0.000158326   0.000969644   0.000117921   0.000192114      19     
produce:pdvddaphne:DAPHNEReaderPDVD                2.10511       2.8031        3.27473       2.76015      0.330906        19     
produce:ophit:OpHitFinder                         0.0214515     0.0355385     0.0437401     0.0364125    0.00594324       19     
produce:wclsdatavd:WireCellToolkit                 89.8932       103.192        129.1        102.57        8.72327        19     
produce:gaushit:GausHitFinder                     0.860624       1.29306       2.25697       1.2374       0.364835        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000220711   0.000360683   0.000792753   0.000351641   0.000118078      19     
produce:reco3d:SpacePointSolver                    8.27847       19.4951       33.0734       18.9723       6.3543         19     
produce:hitpdune:DisambigFromSpacePoints           0.10001      0.200267      0.319122      0.194844      0.0575175       19     
produce:cluster3d:Cluster3D                        1.25579       2.71811       4.58742       2.8498       0.909371        19     
[art]:TriggerResults:TriggerResultInserter        2.993e-05    4.02096e-05    7.991e-05     3.672e-05    1.24879e-05      19     
end_path:out1:RootOutput                          4.05e-06     7.78111e-06    2.311e-05     7.29e-06     3.96595e-06      19     
end_path:out1:RootOutput(write)                    4.12093       5.10255       7.56208       4.94928      0.866974        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5373.24 MB
  Peak resident set size usage (VmHWM): 3406.44 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0305_df-s04-d0_dw_0_20260203T203802_reco_stage0_20260204T001458_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0305_df-s04-d0_dw_0_20260203T203802_reco_stage0_20260204T001458_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0305_df-s04-d0_dw_0_20260203T203802_reco_stage0_20260204T001458_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770161344.0,
    "core.end_time": 1770164100.0,
    "core.events": [
      34780,
      34786,
      34792,
      34798,
      34804,
      34810,
      34816,
      34822,
      34828,
      34834,
      34840,
      34846,
      34852,
      34858,
      34864,
      34870,
      34876,
      34882,
      34888
    ],
    "core.event_count": 19,
    "core.first_event_number": 34780,
    "core.last_event_number": 34888,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0305_df-s04-d0_dw_0_20260203T203802.hdf5"
    }
  ]
}
justIN time: 2026-02-04 06:00:25 UTC       justIN version: 01.06.00