justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303654.67@dunegpschedd01.fnal.gov

Jobsub ID303654.67@dunegpschedd01.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:52:26
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce503
Last heartbeat2026-02-04 00:43:35
From worker nodeHostnameb9p05p0564.cern.ch
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-04 00:01:00
Input filesvd-protodune:np02vd_raw_run042433_0569_df-s04-d1_dw_0_20260203T221823.hdf5
JobscriptExit code0
Real time40m (2417s)
CPU time38m (2331s = 96%)
Max RSS bytes3430666240 (3271 MiB)
Outputting started2026-02-04 00:41:19
Output files
Finished2026-02-04 00:43:35
Saved logsjustin-logs:303654.67-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

up space point ptr... 0x3855d620, 0x3855d7a0
Caught exception in looking up space point ptr... 0x3855d7a0, 0x3855d8d0
Caught exception in looking up space point ptr... 0x59116490, 0x59116a50
Caught exception in looking up space point ptr... 0x59116a50, 0x59116490
Caught exception in looking up space point ptr... 0x33c2f540, 0x33c2f3b0
Caught exception in looking up space point ptr... 0x33c2f3b0, 0x2e6ed360
Caught exception in looking up space point ptr... 0x632aa3a0, 0x632a8760
Caught exception in looking up space point ptr... 0x632a8760, 0x632aa3a0
Caught exception in looking up space point ptr... 0x4905e300, 0x5950bca0
Caught exception in looking up space point ptr... 0x3a812e50, 0x4905e300
Caught exception in looking up space point ptr... 0x3c3ba700, 0x5393ab50
Caught exception in looking up space point ptr... 0x5393ab50, 0x3c3ba700
Caught exception in looking up space point ptr... 0x6329f430, 0x6329f2a0
Caught exception in looking up space point ptr... 0x6329f2a0, 0x6329ef80
Caught exception in looking up space point ptr... 0x45f97c30, 0x678afe10
Caught exception in looking up space point ptr... 0x678afe10, 0x45f97c30
Caught exception in looking up space point ptr... 0x970f480, 0x970f330
Caught exception in looking up space point ptr... 0x970f330, 0x970f480
Caught exception in looking up space point ptr... 0x356867a0, 0x35684f00
Caught exception in looking up space point ptr... 0x35684f00, 0x356867a0
Caught exception in looking up space point ptr... 0x5e59da70, 0x5e59e8a0
Caught exception in looking up space point ptr... 0x5e59e8a0, 0x5e59da70
Caught exception in looking up space point ptr... 0x2f4fbd60, 0x13f004a0
Caught exception in looking up space point ptr... 0x13efe370, 0x2f4fbd60
Caught exception in looking up space point ptr... 0x4ba01200, 0x549c1c10
Caught exception in looking up space point ptr... 0x549c1c10, 0x6ef12260
Caught exception in looking up space point ptr... 0x6ef12260, 0x632c1d20
Caught exception in looking up space point ptr... 0x632c1d20, 0x35914ff0
Caught exception in looking up space point ptr... 0x35914ff0, 0x4ba01200
Caught exception in looking up space point ptr... 0x8d883f0, 0x2ed15730
Caught exception in looking up space point ptr... 0x2ed15730, 0x495da1c0
Caught exception in looking up space point ptr... 0x316ca4c0, 0x316d9520
Caught exception in looking up space point ptr... 0x316d9520, 0x316ca4c0
Caught exception in looking up space point ptr... 0x4eff34f0, 0x4efea5c0
Caught exception in looking up space point ptr... 0x4efea5c0, 0x4eff34f0
Caught exception in looking up space point ptr... 0x3bfb29a0, 0x3bfb2b80
Caught exception in looking up space point ptr... 0x3bfb2b80, 0x3bfb29a0
Caught exception in looking up space point ptr... 0x76f59160, 0x76f592b0
Caught exception in looking up space point ptr... 0x76f592b0, 0x76f59010
Caught exception in looking up space point ptr... 0x5939d590, 0x37e1e130
Caught exception in looking up space point ptr... 0x51e25fa0, 0x5939d590
Caught exception in looking up space point ptr... 0x3a8390a0, 0x38f34b80
Caught exception in looking up space point ptr... 0x38f34b80, 0x3a8390a0
Caught exception in looking up space point ptr... 0x13956dc0, 0x13947940
Caught exception in looking up space point ptr... 0x13947940, 0x13956dc0
Caught exception in looking up space point ptr... 0x2f51ea30, 0x2f51eee0
Caught exception in looking up space point ptr... 0x2f51eee0, 0x2f51ea30
Caught exception in looking up space point ptr... 0x79ebde10, 0x43fbeae0
Caught exception in looking up space point ptr... 0x43fbeae0, 0x79ebde10
Caught exception in looking up space point ptr... 0x632b5aa0, 0x632b5c30
Caught exception in looking up space point ptr... 0x632b5c30, 0x632b5f50
Caught exception in looking up space point ptr... 0x51df5b30, 0x51df50e0
Caught exception in looking up space point ptr... 0x51df50e0, 0x51df5b30
Caught exception in looking up space point ptr... 0x7a3dd850, 0x13d58440
Caught exception in looking up space point ptr... 0x13d58440, 0x7a3dd850
Caught exception in looking up space point ptr... 0x97113a0, 0x9713840
Caught exception in looking up space point ptr... 0x9713840, 0x9714490
Caught exception in looking up space point ptr... 0xa4e5e80, 0xa4e5460
Caught exception in looking up space point ptr... 0xa4e5460, 0xa4e52c0
Caught exception in looking up space point ptr... 0xdf8e5b0, 0x37e08b50
Caught exception in looking up space point ptr... 0x37e08b50, 0xdf8e5b0
Caught exception in looking up space point ptr... 0xdf8e5b0, 0x6302bc80
Caught exception in looking up space point ptr... 0x6302bc80, 0xdf8e5b0
Caught exception in looking up space point ptr... 0x4a501df0, 0x4a501a00
Caught exception in looking up space point ptr... 0x4a501a00, 0x4a501df0
Caught exception in looking up space point ptr... 0x884e670, 0x884ba60
Caught exception in looking up space point ptr... 0x884ba60, 0x884e670
++++>>>> total num hits: 66677, num free: 62846
04-Feb-2026 01:41:11 CET  Closed output file "np02vd_raw_run042433_0569_df-s04-d1_dw_0_20260203T221823_reco_stage0_20260204T004111_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         108.807       123.987       158.266       121.471       10.6541        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       4.1041e-05    7.80131e-05   0.000152742   7.4661e-05    2.80701e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                5.64941       6.36918       6.81971       6.34267      0.327125        19     
produce:triggerrawdecoder:PDVDTriggerReader4     0.00913689     0.0311127     0.221418      0.0128256     0.0472369       19     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000361144   0.000505864    0.001302     0.000440855   0.000211605      19     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.000354064   0.000476785   0.00138645    0.000415385   0.000221287      19     
produce:beamevent:BeamEvent                       5.204e-05    0.00011741    0.000914652    6.596e-05    0.000189555      19     
produce:pdvddaphne:DAPHNEReaderPDVD                2.07745       2.52149       3.22473       2.53472      0.270841        19     
produce:ophit:OpHitFinder                         0.0283518     0.0349088     0.0440361     0.0340238    0.00473101       19     
produce:wclsdatavd:WireCellToolkit                 87.2831       94.5166       105.03        94.9894       4.83633        19     
produce:gaushit:GausHitFinder                     0.645953       1.13031       1.59188       1.12691      0.241602        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000213533   0.00030694    0.000496316   0.000281913   8.36679e-05      19     
produce:reco3d:SpacePointSolver                    6.91812       12.004        20.606        11.6692       3.16999        19     
produce:hitpdune:DisambigFromSpacePoints          0.0707071     0.144435      0.335784      0.146811      0.054179        19     
produce:cluster3d:Cluster3D                       0.954227       2.82375       20.0101       1.85233       4.08391        19     
[art]:TriggerResults:TriggerResultInserter        2.124e-05    3.25481e-05   0.000132052    2.537e-05    2.39433e-05      19     
end_path:out1:RootOutput                          3.45e-06     6.85221e-06    4.102e-05     3.97e-06     8.23893e-06      19     
end_path:out1:RootOutput(write)                    3.85104       4.38559       6.05208       4.26152      0.455837        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5430.87 MB
  Peak resident set size usage (VmHWM): 3430.67 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0569_df-s04-d1_dw_0_20260203T221823_reco_stage0_20260204T004111_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0569_df-s04-d1_dw_0_20260203T221823_reco_stage0_20260204T004111_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0569_df-s04-d1_dw_0_20260203T221823_reco_stage0_20260204T004111_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770163273.0,
    "core.end_time": 1770165672.0,
    "core.events": [
      64877,
      64883,
      64889,
      64895,
      64901,
      64907,
      64913,
      64919,
      64925,
      64931,
      64937,
      64943,
      64949,
      64955,
      64961,
      64967,
      64973,
      64979,
      64985
    ],
    "core.event_count": 19,
    "core.first_event_number": 64877,
    "core.last_event_number": 64985,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0569_df-s04-d1_dw_0_20260203T221823.hdf5"
    }
  ]
}
justIN time: 2026-02-04 06:06:08 UTC       justIN version: 01.06.00