justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303654.57@dunegpschedd01.fnal.gov

Jobsub ID303654.57@dunegpschedd01.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:52:26
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce515
Last heartbeat2026-02-04 00:38:09
From worker nodeHostnameb9p12p0201.cern.ch
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:57:24
Input filesvd-protodune:np02vd_raw_run042433_0139_df-s03-d1_dw_0_20260203T193457.hdf5
JobscriptExit code0
Real time38m (2307s)
CPU time37m (2225s = 96%)
Max RSS bytes3412881408 (3254 MiB)
Outputting started2026-02-04 00:35:52
Output files
Finished2026-02-04 00:38:09
Saved logsjustin-logs:303654.57-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ce point ptr... 0x57f247d0, 0x36c126a0
Caught exception in looking up space point ptr... 0x36c126a0, 0x4fec2240
Caught exception in looking up space point ptr... 0x2bf95990, 0x2ec3c260
Caught exception in looking up space point ptr... 0x2ec3c260, 0x4bc79a40
Caught exception in looking up space point ptr... 0x852eb080, 0x30b34da0
Caught exception in looking up space point ptr... 0x30b34da0, 0x30b343e0
Caught exception in looking up space point ptr... 0x852f09c0, 0x852eb080
Caught exception in looking up space point ptr... 0x11271f60, 0x57c60cd0
Caught exception in looking up space point ptr... 0x57c60cd0, 0x11271f60
Caught exception in looking up space point ptr... 0x5f37e300, 0x5f37e080
Caught exception in looking up space point ptr... 0x5f37e080, 0x5f37e300
Caught exception in looking up space point ptr... 0x4f83f390, 0xe2bd8d0
Caught exception in looking up space point ptr... 0xe2bd8d0, 0xe2bd670
Caught exception in looking up space point ptr... 0x5997cde0, 0x388cdaf0
Caught exception in looking up space point ptr... 0x388cdaf0, 0x5997cde0
Caught exception in looking up space point ptr... 0x3c0cee30, 0x49970aa0
Caught exception in looking up space point ptr... 0x49970aa0, 0x49970910
Caught exception in looking up space point ptr... 0x49970910, 0x49970460
Caught exception in looking up space point ptr... 0x38ca2fd0, 0x3ac9eec0
Caught exception in looking up space point ptr... 0x3ac9eec0, 0x456796d0
Caught exception in looking up space point ptr... 0x456796d0, 0x3ac9ef60
Caught exception in looking up space point ptr... 0x37aab380, 0x115126b0
Caught exception in looking up space point ptr... 0x115126b0, 0x570b9c00
Caught exception in looking up space point ptr... 0x4136f220, 0x35b5e020
Caught exception in looking up space point ptr... 0x35b5e020, 0x4136f220
Caught exception in looking up space point ptr... 0x451e6150, 0x181372f0
Caught exception in looking up space point ptr... 0x181372f0, 0x451e6150
Caught exception in looking up space point ptr... 0x119de8d0, 0x119de740
Caught exception in looking up space point ptr... 0x119de740, 0x119de8d0
Caught exception in looking up space point ptr... 0x119de8d0, 0x8069830
Caught exception in looking up space point ptr... 0x8069830, 0x119de8d0
Caught exception in looking up space point ptr... 0x65fbccd0, 0x7ac844b0
Caught exception in looking up space point ptr... 0x7bdc5510, 0x65fbccd0
Caught exception in looking up space point ptr... 0x38b2d7f0, 0x38b299d0
Caught exception in looking up space point ptr... 0x38b299d0, 0x38b29840
Caught exception in looking up space point ptr... 0x6855fcb0, 0x2bf95520
Caught exception in looking up space point ptr... 0x177053a0, 0x6855fcb0
Caught exception in looking up space point ptr... 0x10cfa340, 0x10cfb390
Caught exception in looking up space point ptr... 0x10cfb390, 0x10cfa340
Caught exception in looking up space point ptr... 0x3d176b30, 0x38d63fe0
Caught exception in looking up space point ptr... 0x3e3c8360, 0x3d176b30
Caught exception in looking up space point ptr... 0x7286d080, 0x7286d3a0
Caught exception in looking up space point ptr... 0x7286d3a0, 0x7286d080
Caught exception in looking up space point ptr... 0x786f6d80, 0x786f2c40
Caught exception in looking up space point ptr... 0x786f2c40, 0x786ec080
Caught exception in looking up space point ptr... 0xf4fed80, 0x357712e0
Caught exception in looking up space point ptr... 0x35770dd0, 0xf4fed80
Caught exception in looking up space point ptr... 0x66ea0ca0, 0x374f08b0
Caught exception in looking up space point ptr... 0x374f08b0, 0x66ea0ca0
Caught exception in looking up space point ptr... 0x18ef2740, 0x5f868d70
Caught exception in looking up space point ptr... 0x786a73c0, 0x18ef2740
Caught exception in looking up space point ptr... 0x31c38eb0, 0x57d2ab10
Caught exception in looking up space point ptr... 0x57d2ab10, 0x31c38eb0
Caught exception in looking up space point ptr... 0x38dc9ee0, 0x3a372ac0
Caught exception in looking up space point ptr... 0x3a372ac0, 0x38dc9ee0
Caught exception in looking up space point ptr... 0xe2b2fb0, 0xe294cf0
Caught exception in looking up space point ptr... 0xe294cf0, 0xe2b2fb0
Caught exception in looking up space point ptr... 0xd2c6800, 0x7d37eb0
Caught exception in looking up space point ptr... 0x7d37eb0, 0x77bcc030
Caught exception in looking up space point ptr... 0x3a370780, 0x33606650
Caught exception in looking up space point ptr... 0x33606650, 0x799df2d0
Caught exception in looking up space point ptr... 0x38eb5360, 0x44c8baf0
Caught exception in looking up space point ptr... 0x44c8baf0, 0x44c8bc80
Caught exception in looking up space point ptr... 0x44c8bc80, 0x2b0f2a40
Caught exception in looking up space point ptr... 0x2b0f2a40, 0x38eb5360
Caught exception in looking up space point ptr... 0x511f29b0, 0x390c91c0
Caught exception in looking up space point ptr... 0x390c91c0, 0x37731fd0
++++>>>> total num hits: 78418, num free: 75882
04-Feb-2026 01:35:45 CET  Closed output file "np02vd_raw_run042433_0139_df-s03-d1_dw_0_20260203T193457_reco_stage0_20260204T003545_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         112.653       118.795       129.393       117.559       4.37723        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                        5.032e-05    0.000106414   0.000275291    8.29e-05     5.29023e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                5.15401       5.70985       6.13785       5.71749      0.255659        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.0095893     0.0130765     0.0205919     0.0115638    0.00348398       19     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000387561   0.000494848   0.000673012   0.000485312   8.37276e-05      19     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.000377312   0.000449063   0.000602142   0.000432751   5.86853e-05      19     
produce:beamevent:BeamEvent                       6.17e-05     0.000129273   0.000791633    9.351e-05    0.000157979      19     
produce:pdvddaphne:DAPHNEReaderPDVD                1.94185       2.1555        2.39741       2.14647      0.118257        19     
produce:ophit:OpHitFinder                         0.0213665     0.0304181     0.040986      0.0296609    0.00423552       19     
produce:wclsdatavd:WireCellToolkit                 80.7639       85.0919       87.7183       85.4303       1.75313        19     
produce:gaushit:GausHitFinder                     0.716647       1.05334       1.47602       1.05105      0.166949        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000254801   0.000322163   0.000494432   0.000305441   5.26334e-05      19     
produce:reco3d:SpacePointSolver                    13.0078       18.3904       24.5746       17.9485       3.06882        19     
produce:hitpdune:DisambigFromSpacePoints          0.0826451     0.145982      0.225581       0.14234      0.0301829       19     
produce:cluster3d:Cluster3D                        1.27308       2.06948       3.73066       1.85534      0.582457        19     
[art]:TriggerResults:TriggerResultInserter        2.15e-05     3.39896e-05   0.00011523     2.835e-05    2.08928e-05      19     
end_path:out1:RootOutput                          3.27e-06     8.41795e-06   3.4391e-05     7.21e-06     6.46598e-06      19     
end_path:out1:RootOutput(write)                    3.78168       4.10699       4.45689       4.10768      0.172947        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5385.27 MB
  Peak resident set size usage (VmHWM): 3412.88 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0139_df-s03-d1_dw_0_20260203T193457_reco_stage0_20260204T003545_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0139_df-s03-d1_dw_0_20260203T193457_reco_stage0_20260204T003545_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0139_df-s03-d1_dw_0_20260203T193457_reco_stage0_20260204T003545_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770163054.0,
    "core.end_time": 1770165346.0,
    "core.events": [
      15855,
      15861,
      15867,
      15873,
      15879,
      15885,
      15891,
      15897,
      15903,
      15909,
      15915,
      15921,
      15927,
      15933,
      15939,
      15945,
      15951,
      15957,
      15963
    ],
    "core.event_count": 19,
    "core.first_event_number": 15855,
    "core.last_event_number": 15963,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0139_df-s03-d1_dw_0_20260203T193457.hdf5"
    }
  ]
}
justIN time: 2026-02-04 07:37:42 UTC       justIN version: 01.06.00