justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303652.5@dunegpschedd01.fnal.gov

Jobsub ID303652.5@dunegpschedd01.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:50:26
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce512
Last heartbeat2026-02-04 00:07:27
From worker nodeHostnameb9p10p5730.cern.ch
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:24:56
Input filesvd-protodune:np02vd_raw_run042433_0313_df-s03-d1_dw_0_20260203T204105.hdf5
JobscriptExit code0
Real time40m (2412s)
CPU time38m (2322s = 96%)
Max RSS bytes3424702464 (3266 MiB)
Outputting started2026-02-04 00:05:09
Output files
Finished2026-02-04 00:07:27
Saved logsjustin-logs:303652.5-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

.. 0x694a1d60, 0x694a15f0
Caught exception in looking up space point ptr... 0x694a15f0, 0x694a1d60
Caught exception in looking up space point ptr... 0x67adb830, 0x6fd721d0
Caught exception in looking up space point ptr... 0x6fd721d0, 0x5e0c6ef0
Caught exception in looking up space point ptr... 0x18b767d0, 0x18b76460
Caught exception in looking up space point ptr... 0x18b76460, 0x18b767d0
Caught exception in looking up space point ptr... 0x59c48c20, 0x6fd603e0
Caught exception in looking up space point ptr... 0x6fd603e0, 0x5ebab790
Caught exception in looking up space point ptr... 0x169c6520, 0x169c6720
Caught exception in looking up space point ptr... 0x169c6720, 0x169c6520
Caught exception in looking up space point ptr... 0x4cecf210, 0x41a81000
Caught exception in looking up space point ptr... 0x41a81000, 0x4cecf210
Caught exception in looking up space point ptr... 0x3bf7a8e0, 0x3bf79900
Caught exception in looking up space point ptr... 0x3bf79900, 0x3bf78250
Caught exception in looking up space point ptr... 0x3bf78250, 0x3bf77840
Caught exception in looking up space point ptr... 0x3bf77840, 0x3bf7a8e0
Caught exception in looking up space point ptr... 0x199d31d0, 0x199d3320
Caught exception in looking up space point ptr... 0x199d3320, 0x199d31d0
Caught exception in looking up space point ptr... 0x48f9f9d0, 0x48f9fad0
Caught exception in looking up space point ptr... 0x48f9fad0, 0x48f9b4a0
Caught exception in looking up space point ptr... 0x3d96a990, 0x3d96a0c0
Caught exception in looking up space point ptr... 0x3d96a0c0, 0x3d969f40
Caught exception in looking up space point ptr... 0x18b992e0, 0x18b8ee60
Caught exception in looking up space point ptr... 0x18b8f310, 0x18b992e0
Caught exception in looking up space point ptr... 0x3bf6d690, 0x3bf71bb0
Caught exception in looking up space point ptr... 0x3bf71bb0, 0x3bf764d0
Caught exception in looking up space point ptr... 0x77851890, 0x7787fdc0
Caught exception in looking up space point ptr... 0x129cb6d0, 0x778514a0
Caught exception in looking up space point ptr... 0x778514a0, 0x77851890
Caught exception in looking up space point ptr... 0x884caed0, 0x884ca8e0
Caught exception in looking up space point ptr... 0x884ca8e0, 0x884d10f0
Caught exception in looking up space point ptr... 0x67ad4560, 0x67ad1ac0
Caught exception in looking up space point ptr... 0x67ad1ac0, 0x67ae4570
Caught exception in looking up space point ptr... 0x194c8220, 0x194cad20
Caught exception in looking up space point ptr... 0x194cad20, 0x194ca5e0
Caught exception in looking up space point ptr... 0x81f6a440, 0x81f45cf0
Caught exception in looking up space point ptr... 0x81f674d0, 0x81f6a440
Caught exception in looking up space point ptr... 0x71936810, 0x719369a0
Caught exception in looking up space point ptr... 0x719369a0, 0x71936810
Caught exception in looking up space point ptr... 0x187fa300, 0x187fa490
Caught exception in looking up space point ptr... 0x422e0db0, 0x187fa300
Caught exception in looking up space point ptr... 0x6206b700, 0x6205c450
Caught exception in looking up space point ptr... 0x6205c450, 0x6206b700
Caught exception in looking up space point ptr... 0x6206b700, 0x6205c2d0
Caught exception in looking up space point ptr... 0x6205c2d0, 0x6206b700
Caught exception in looking up space point ptr... 0x59ec4e40, 0x3f046a20
Caught exception in looking up space point ptr... 0x3f046a20, 0x59ec4e40
Caught exception in looking up space point ptr... 0x1906fac0, 0x3d326290
Caught exception in looking up space point ptr... 0x3d326290, 0x1906e3c0
Caught exception in looking up space point ptr... 0x623bd2b0, 0x623bb110
Caught exception in looking up space point ptr... 0x623bb110, 0x623bd350
Caught exception in looking up space point ptr... 0x718e8830, 0x14436a00
Caught exception in looking up space point ptr... 0x14436a00, 0x718e8830
Caught exception in looking up space point ptr... 0x79900000, 0x798ffb60
Caught exception in looking up space point ptr... 0x340d7c40, 0x79900000
Caught exception in looking up space point ptr... 0x13f7e030, 0x13f7dee0
Caught exception in looking up space point ptr... 0x13f7dee0, 0x13f7dc70
Caught exception in looking up space point ptr... 0x6324c920, 0x6323eb20
Caught exception in looking up space point ptr... 0x6323eb20, 0x63243ea0
Caught exception in looking up space point ptr... 0x618e8730, 0x618e8a50
Caught exception in looking up space point ptr... 0x618e8a50, 0x618e8be0
Caught exception in looking up space point ptr... 0x186e6a20, 0x186e3030
Caught exception in looking up space point ptr... 0x186e3030, 0x186e6a20
Caught exception in looking up space point ptr... 0x186e6a20, 0x59d00370
Caught exception in looking up space point ptr... 0x59d00370, 0x186e6a20
Caught exception in looking up space point ptr... 0x186e6a20, 0x5ebbb9a0
Caught exception in looking up space point ptr... 0x5ebbb9a0, 0x186e6a20
++++>>>> total num hits: 89923, num free: 86658
04-Feb-2026 01:05:02 CET  Closed output file "np02vd_raw_run042433_0313_df-s03-d1_dw_0_20260203T204105_reco_stage0_20260204T000502_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         109.528       124.517       144.278       123.022       9.32407        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       4.1801e-05    7.21335e-05   0.000121531   7.3841e-05    2.24484e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                5.52507       6.76004       7.63005       6.73469      0.527775        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.0106183     0.0189096     0.0740713     0.0129855     0.0141195       19     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000372435   0.000553256    0.0014473    0.000485156   0.000240097      19     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.000374985   0.000502443   0.00158347    0.000438305   0.00025916       19     
produce:beamevent:BeamEvent                      5.0561e-05    0.000124148   0.000736388   8.3481e-05    0.000146761      19     
produce:pdvddaphne:DAPHNEReaderPDVD                1.95179       2.34643       2.87161       2.30016      0.283415        19     
produce:ophit:OpHitFinder                         0.0189452     0.0317016     0.0376246     0.033645     0.00533844       19     
produce:wclsdatavd:WireCellToolkit                 83.6198       92.7713       118.582       90.1553       9.09694        19     
produce:gaushit:GausHitFinder                     0.629472       1.0812        1.58442        1.05        0.218206        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000189402   0.000281774   0.000480277   0.000282363   6.12422e-05      19     
produce:reco3d:SpacePointSolver                    6.62655       15.0778       24.6022       14.226        4.57797        19     
produce:hitpdune:DisambigFromSpacePoints          0.0737974     0.142044      0.218779      0.134739      0.0307474       19     
produce:cluster3d:Cluster3D                       0.940006       1.96449       2.64463       1.93639      0.474751        19     
[art]:TriggerResults:TriggerResultInserter        1.861e-05    3.10208e-05   0.000109931    2.574e-05    1.9059e-05       19     
end_path:out1:RootOutput                          4.19e-06     8.41695e-06    3.269e-05     7.49e-06     5.98284e-06      19     
end_path:out1:RootOutput(write)                    3.72462       4.29538       5.79606       4.17989      0.464976        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5448.29 MB
  Peak resident set size usage (VmHWM): 3424.7 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0313_df-s03-d1_dw_0_20260203T204105_reco_stage0_20260204T000502_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0313_df-s03-d1_dw_0_20260203T204105_reco_stage0_20260204T000502_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0313_df-s03-d1_dw_0_20260203T204105_reco_stage0_20260204T000502_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770161105.0,
    "core.end_time": 1770163503.0,
    "core.events": [
      35691,
      35697,
      35703,
      35709,
      35715,
      35721,
      35727,
      35733,
      35739,
      35745,
      35751,
      35757,
      35763,
      35769,
      35775,
      35781,
      35787,
      35793,
      35799
    ],
    "core.event_count": 19,
    "core.first_event_number": 35691,
    "core.last_event_number": 35799,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0313_df-s03-d1_dw_0_20260203T204105.hdf5"
    }
  ]
}
justIN time: 2026-02-04 10:45:16 UTC       justIN version: 01.06.00