justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303647.119@dunegpschedd01.fnal.gov

Jobsub ID303647.119@dunegpschedd01.fnal.gov
Workflow ID12717
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:46:24
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce512
Last heartbeat2026-02-03 23:32:34
From worker nodeHostnameb9g24p4713.cern.ch
cpuinfoAMD EPYC 7302 16-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 22:46:52
Input filesvd-protodune:np02vd_raw_run042421_0111_df-s04-d0_dw_0_20260203T143103.hdf5
JobscriptExit code0
Real time43m (2603s)
CPU time41m (2511s = 96%)
Max RSS bytes2687373312 (2562 MiB)
Outputting started2026-02-03 23:30:16
Output files
Finished2026-02-03 23:32:34
Saved logsjustin-logs:303647.119-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

16977320
Caught exception in looking up space point ptr... 0x2c0ae460, 0x2c0aee00
Caught exception in looking up space point ptr... 0x2c0aee00, 0x2c0afad0
Caught exception in looking up space point ptr... 0x100571f0, 0x60f5aa00
Caught exception in looking up space point ptr... 0x60f5aa00, 0x100571f0
Caught exception in looking up space point ptr... 0x17b16450, 0x17b16170
Caught exception in looking up space point ptr... 0x4033d710, 0x17b16450
Caught exception in looking up space point ptr... 0xb2ee530, 0xb2e01d0
Caught exception in looking up space point ptr... 0xb2e01d0, 0xb2df9d0
Caught exception in looking up space point ptr... 0x3da68440, 0x2bf4f7f0
Caught exception in looking up space point ptr... 0x2bf4f7f0, 0x3da68440
Caught exception in looking up space point ptr... 0x3da68440, 0x31032580
Caught exception in looking up space point ptr... 0x31032580, 0x3da68440
Caught exception in looking up space point ptr... 0x3da68440, 0x30c527f0
Caught exception in looking up space point ptr... 0x30c527f0, 0x3da68440
Caught exception in looking up space point ptr... 0x4ebf5450, 0x3cd0f3d0
Caught exception in looking up space point ptr... 0x3cd0f3d0, 0x3cd0f560
Caught exception in looking up space point ptr... 0x1973e1a0, 0xf2418e0
Caught exception in looking up space point ptr... 0xf2418e0, 0x1973e1a0
Caught exception in looking up space point ptr... 0x3debba90, 0x9519dd0
Caught exception in looking up space point ptr... 0x9519dd0, 0x9515640
Caught exception in looking up space point ptr... 0x5392bc40, 0x3debba90
Caught exception in looking up space point ptr... 0x587efd60, 0x62bc0bc0
Caught exception in looking up space point ptr... 0x62bc0bc0, 0x587efd60
Caught exception in looking up space point ptr... 0x16704f30, 0x16703330
Caught exception in looking up space point ptr... 0x16703330, 0x167031e0
Caught exception in looking up space point ptr... 0x167031e0, 0xd6e1aa0
Caught exception in looking up space point ptr... 0xd6e1aa0, 0x135be420
Caught exception in looking up space point ptr... 0x12f98bf0, 0x27fc2f10
Caught exception in looking up space point ptr... 0x27fc2f10, 0x12f98bf0
Caught exception in looking up space point ptr... 0x3a0222e0, 0x3a022150
Caught exception in looking up space point ptr... 0x3a022150, 0x3a0222e0
Caught exception in looking up space point ptr... 0xaed3660, 0x446a4b20
Caught exception in looking up space point ptr... 0x446a4b20, 0xaed3660
Caught exception in looking up space point ptr... 0x364886d0, 0x36488540
Caught exception in looking up space point ptr... 0x36488540, 0x36488090
Caught exception in looking up space point ptr... 0x3c0343a0, 0x3c032970
Caught exception in looking up space point ptr... 0x3c032970, 0x3c0343a0
Caught exception in looking up space point ptr... 0x1674e7e0, 0x1674ed40
Caught exception in looking up space point ptr... 0x1674ed40, 0x1674ebd0
Caught exception in looking up space point ptr... 0x1674ebd0, 0x46009a50
Caught exception in looking up space point ptr... 0x46009a50, 0x1674e7e0
Caught exception in looking up space point ptr... 0x3c7643e0, 0x3a0e43a0
Caught exception in looking up space point ptr... 0x3a0e43a0, 0x3c7643e0
Caught exception in looking up space point ptr... 0x3c7643e0, 0x17a38a30
Caught exception in looking up space point ptr... 0x17a38a30, 0x3c7643e0
Caught exception in looking up space point ptr... 0x1b12cc50, 0x2bf2f7e0
Caught exception in looking up space point ptr... 0x2bf2f7e0, 0x17a2c940
Caught exception in looking up space point ptr... 0x1463a750, 0x5120f6a0
Caught exception in looking up space point ptr... 0x5120f6a0, 0x5ff4c3a0
Caught exception in looking up space point ptr... 0x4bfcef00, 0x4031f390
Caught exception in looking up space point ptr... 0x4031f390, 0x1272ee00
Caught exception in looking up space point ptr... 0x3c23a830, 0x15c73830
Caught exception in looking up space point ptr... 0x15c73830, 0x31d61fd0
Caught exception in looking up space point ptr... 0x49d30000, 0x49d2fd60
Caught exception in looking up space point ptr... 0x49d2fd60, 0x3c7bf310
Caught exception in looking up space point ptr... 0x1c6c8f10, 0x1c6c9010
Caught exception in looking up space point ptr... 0x1c6c9010, 0x1c6c9210
Caught exception in looking up space point ptr... 0x53808d20, 0x538083f0
Caught exception in looking up space point ptr... 0x538083f0, 0x53808d20
Caught exception in looking up space point ptr... 0x3c6eff50, 0x28f412f0
Caught exception in looking up space point ptr... 0x28f412f0, 0x3c6eff50
Caught exception in looking up space point ptr... 0x3c6eff50, 0x61cf90a0
Caught exception in looking up space point ptr... 0x61cf90a0, 0x3c6eff50
Caught exception in looking up space point ptr... 0x2becd6e0, 0x94c7600
Caught exception in looking up space point ptr... 0x94c7600, 0x2becd6e0
++++>>>> total num hits: 39804, num free: 38032
04-Feb-2026 00:30:09 CET  Closed output file "np02vd_raw_run042421_0111_df-s04-d0_dw_0_20260203T143103_reco_stage0_20260203T233009_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         78.4786       93.8865       114.626       94.9655       9.58048        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       5.5121e-05    0.000102132   0.000288256   8.5291e-05    5.08818e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                3.88026       4.56395       7.28433       4.52991      0.583507        27     
produce:triggerrawdecoder:PDVDTriggerReader4     0.00924093     0.0173542     0.0490465     0.0120845    0.00971324       27     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000367288   0.000568324   0.00197596    0.000471149   0.000305224      27     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.00130774    0.00179879    0.00341205    0.00166524    0.000437581      27     
produce:beamevent:BeamEvent                      5.2381e-05    0.000120959   0.000872197   9.3312e-05    0.000152403      27     
produce:pdvddaphne:DAPHNEReaderPDVD                2.32036       2.7339        3.4171        2.72205      0.244125        27     
produce:ophit:OpHitFinder                         0.0236393     0.0349954     0.0557363     0.0339433    0.00642364       27     
produce:wclsdatavd:WireCellToolkit                 59.4907       68.1911       76.6279       68.8358       5.0136         27     
produce:gaushit:GausHitFinder                     0.495055      0.941252       1.47616      0.894159      0.278678        27     
produce:nhitsfilter:NumberOfHitsFilter           0.000173164   0.000301747   0.000463919   0.000289235   6.56554e-05      27     
produce:reco3d:SpacePointSolver                    4.32452       10.1071       16.6671       9.54492       3.0409         27     
produce:hitpdune:DisambigFromSpacePoints          0.0510885     0.112212       0.24031      0.101956      0.0468134       27     
produce:cluster3d:Cluster3D                       0.525162       3.05745       16.7981       2.01109       3.44353        27     
[art]:TriggerResults:TriggerResultInserter        2.179e-05    3.36034e-05   8.4852e-05    3.0491e-05    1.24757e-05      27     
end_path:out1:RootOutput                          3.47e-06     7.57122e-06    2.545e-05     7.85e-06     4.07967e-06      27     
end_path:out1:RootOutput(write)                    3.29225       4.10055       5.86908       4.01042       0.6202         27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4639.82 MB
  Peak resident set size usage (VmHWM): 2687.37 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042421_0111_df-s04-d0_dw_0_20260203T143103_reco_stage0_20260203T233009_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042421_0111_df-s04-d0_dw_0_20260203T143103_reco_stage0_20260203T233009_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042421_0111_df-s04-d0_dw_0_20260203T143103_reco_stage0_20260203T233009_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770158824.0,
    "core.end_time": 1770161411.0,
    "core.events": [
      18046,
      18052,
      18058,
      18064,
      18070,
      18076,
      18082,
      18088,
      18094,
      18100,
      18106,
      18112,
      18118,
      18124,
      18130,
      18136,
      18142,
      18148,
      18154,
      18160,
      18166,
      18172,
      18178,
      18184,
      18190,
      18196,
      18202
    ],
    "core.event_count": 27,
    "core.first_event_number": 18046,
    "core.last_event_number": 18202,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42421
    ],
    "core.runs_subruns": [
      4242100001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042421_0111_df-s04-d0_dw_0_20260203T143103.hdf5"
    }
  ]
}
justIN time: 2026-02-04 12:13:08 UTC       justIN version: 01.06.00