justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297923.114@dunegpschedd02.fnal.gov

Jobsub ID297923.114@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 23:00:26
SiteIT_CNAF
EntryCMSHTPC_T1_IT_CNAF_condor_ce02
Last heartbeat2026-02-04 01:57:48
From worker nodeHostnamewn-f01-15-03-09.cr.cnaf.infn.it
cpuinfoIntel(R) Xeon(R) CPU E5-2697 v4 @ 2.30GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-04 00:03:23
Input filesvd-protodune:np02vd_raw_run042433_0492_df-s03-d1_dw_0_20260203T214907.hdf5
JobscriptExit code0
Real time1h (6713s)
CPU time1h (6448s = 96%)
Max RSS bytes3607793664 (3440 MiB)
Outputting started2026-02-04 01:55:22
Output files
Finished2026-02-04 01:57:48
Saved logsjustin-logs:297923.114-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

oint ptr... 0x3c3ef390, 0x3c3fb720
Caught exception in looking up space point ptr... 0x63062ea0, 0x5be290b0
Caught exception in looking up space point ptr... 0x5be290b0, 0x630687a0
Caught exception in looking up space point ptr... 0x63651a10, 0x73840020
Caught exception in looking up space point ptr... 0x63651190, 0x63651a10
Caught exception in looking up space point ptr... 0x688ac250, 0x688d10f0
Caught exception in looking up space point ptr... 0x688d10f0, 0x688e3a20
Caught exception in looking up space point ptr... 0x6e023db0, 0x5ad10a90
Caught exception in looking up space point ptr... 0x5ad10a90, 0x6e023db0
Caught exception in looking up space point ptr... 0x44501940, 0x445016b0
Caught exception in looking up space point ptr... 0x445016b0, 0x44506030
Caught exception in looking up space point ptr... 0x64ed76b0, 0x56237620
Caught exception in looking up space point ptr... 0x56237620, 0x55a167b0
Caught exception in looking up space point ptr... 0x611ffe00, 0x1478cbe0
Caught exception in looking up space point ptr... 0x1478cbe0, 0x1478ca50
Caught exception in looking up space point ptr... 0x53d9f9b0, 0x69cf9ce0
Caught exception in looking up space point ptr... 0x69cf9ce0, 0x69cf9b50
Caught exception in looking up space point ptr... 0x3ce9b030, 0x4f5e3150
Caught exception in looking up space point ptr... 0x5f9acc30, 0x3ce9b030
Caught exception in looking up space point ptr... 0x8015e360, 0x8015e1e0
Caught exception in looking up space point ptr... 0x8015e1e0, 0x508ad390
Caught exception in looking up space point ptr... 0x508ad390, 0x508ad140
Caught exception in looking up space point ptr... 0x508ad140, 0x8015e6f0
Caught exception in looking up space point ptr... 0x7383a0b0, 0x738463e0
Caught exception in looking up space point ptr... 0x7383a540, 0x7383a0b0
Caught exception in looking up space point ptr... 0x756e3c90, 0x4e969380
Caught exception in looking up space point ptr... 0x4e969380, 0x4e959e30
Caught exception in looking up space point ptr... 0x177139a0, 0x17712ee0
Caught exception in looking up space point ptr... 0x17712ee0, 0x17713580
Caught exception in looking up space point ptr... 0x578abd60, 0x6898de10
Caught exception in looking up space point ptr... 0x6898de10, 0x6898dfa0
Caught exception in looking up space point ptr... 0x12f55660, 0x14bd4250
Caught exception in looking up space point ptr... 0x12f557f0, 0x12f55660
Caught exception in looking up space point ptr... 0x112e8230, 0x112e7cc0
Caught exception in looking up space point ptr... 0x112e7cc0, 0x112e7560
Caught exception in looking up space point ptr... 0x658d82e0, 0x9debe90
Caught exception in looking up space point ptr... 0x32c8dea0, 0x658d82e0
Caught exception in looking up space point ptr... 0x892af380, 0x122b27f0
Caught exception in looking up space point ptr... 0x892af510, 0x892af380
Caught exception in looking up space point ptr... 0x4bbf01f0, 0x64bfbfa0
Caught exception in looking up space point ptr... 0x4bc02940, 0x4bbf01f0
Caught exception in looking up space point ptr... 0x113fb440, 0x89364cc0
Caught exception in looking up space point ptr... 0x89364cc0, 0x6270d800
Caught exception in looking up space point ptr... 0x4bc315b0, 0x4bc1f4a0
Caught exception in looking up space point ptr... 0x4bc43b70, 0x4bc315b0
Caught exception in looking up space point ptr... 0x756dcf30, 0x4e9627b0
Caught exception in looking up space point ptr... 0x4e9627b0, 0x4e950b50
Caught exception in looking up space point ptr... 0x8017b5a0, 0x47b87420
Caught exception in looking up space point ptr... 0x5623bec0, 0x8017b5a0
Caught exception in looking up space point ptr... 0x5a9b96b0, 0x38944880
Caught exception in looking up space point ptr... 0x432180e0, 0x5a9b96b0
Caught exception in looking up space point ptr... 0x55adaef0, 0x404615a0
Caught exception in looking up space point ptr... 0x404615a0, 0x55adadf0
Caught exception in looking up space point ptr... 0x9aba730, 0x9aba9d0
Caught exception in looking up space point ptr... 0x9aba9d0, 0x9aba730
Caught exception in looking up space point ptr... 0x9aba730, 0x9aba880
Caught exception in looking up space point ptr... 0x9aba880, 0x9aba730
Caught exception in looking up space point ptr... 0x68892ad0, 0x68892c20
Caught exception in looking up space point ptr... 0x68892c20, 0x68892ad0
Caught exception in looking up space point ptr... 0x407c4380, 0x73fe3af0
Caught exception in looking up space point ptr... 0x18a60620, 0x407c4380
Caught exception in looking up space point ptr... 0x34821150, 0x3481f670
Caught exception in looking up space point ptr... 0x3481f670, 0x34821150
Caught exception in looking up space point ptr... 0x1a10e830, 0x8dcf860
Caught exception in looking up space point ptr... 0x8dcf860, 0x8dcf9a0
Caught exception in looking up space point ptr... 0x4c51a930, 0x3481c930
Caught exception in looking up space point ptr... 0x3481c930, 0x4c51a930
++++>>>> total num hits: 524577, num free: 516154
04-Feb-2026 02:55:04 CET  Closed output file "np02vd_raw_run042433_0492_df-s03-d1_dw_0_20260203T214907_reco_stage0_20260204T015504_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         221.919       347.088       463.329       348.167       58.9059        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       7.6879e-05    0.000145504   0.000288845   0.000117346   6.32144e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                13.8966       15.5329       17.6153       15.4773       1.00457        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.306943      0.317554      0.333483      0.318399     0.00642203       19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0103623     0.0105691     0.0108307     0.0105642    0.000101597      19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.010354      0.010579      0.0107964     0.010551     0.000110873      19     
produce:beamevent:BeamEvent                      0.000104819   0.000231733   0.00129597    0.000170599   0.000252861      19     
produce:pdvddaphne:DAPHNEReaderPDVD                5.19294       6.3916        7.65255       6.43965      0.489818        19     
produce:ophit:OpHitFinder                         0.040556      0.0529409     0.0663479     0.0545421    0.00712542       19     
produce:wclsdatavd:WireCellToolkit                 162.561       209.907       266.24        214.749       34.2342        19     
produce:gaushit:GausHitFinder                      2.17266       3.49782       5.14503       3.22814      0.880441        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000592065   0.000802371   0.00123754    0.000790145   0.000146024      19     
produce:reco3d:SpacePointSolver                    17.221        29.5508       40.2764       26.8706       6.69693        19     
produce:hitpdune:DisambigFromSpacePoints          0.281226      0.481394      0.832131      0.452955      0.138388        19     
produce:cluster3d:Cluster3D                        3.36019       72.0983       121.431       74.6411       38.0845        19     
[art]:TriggerResults:TriggerResultInserter       3.0829e-05    5.36947e-05   0.000116278   4.7782e-05    1.8998e-05       19     
end_path:out1:RootOutput                           6.3e-06     1.14722e-05   3.2747e-05     9.023e-06    5.87576e-06      19     
end_path:out1:RootOutput(write)                    5.91975       9.19357       12.9245       7.87061       2.18905        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5560.17 MB
  Peak resident set size usage (VmHWM): 3607.79 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0492_df-s03-d1_dw_0_20260203T214907_reco_stage0_20260204T015504_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0492_df-s03-d1_dw_0_20260203T214907_reco_stage0_20260204T015504_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0492_df-s03-d1_dw_0_20260203T214907_reco_stage0_20260204T015504_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770163420.0,
    "core.end_time": 1770170106.0,
    "core.events": [
      56097,
      56103,
      56109,
      56115,
      56121,
      56127,
      56133,
      56139,
      56145,
      56151,
      56157,
      56163,
      56169,
      56175,
      56181,
      56187,
      56193,
      56199,
      56205
    ],
    "core.event_count": 19,
    "core.first_event_number": 56097,
    "core.last_event_number": 56205,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0492_df-s03-d1_dw_0_20260203T214907.hdf5"
    }
  ]
}
justIN time: 2026-02-04 06:07:36 UTC       justIN version: 01.06.00