justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297917.183@dunegpschedd02.fnal.gov

Jobsub ID297917.183@dunegpschedd02.fnal.gov
Workflow ID12717
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:48:25
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce506
Last heartbeat2026-02-04 00:06:43
From worker nodeHostnameb9p06p2822.cern.ch
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:28:03
Input filesvd-protodune:np02vd_raw_run042421_0037_df-s04-d1_dw_0_20260203T135102.hdf5
JobscriptExit code0
Real time36m (2175s)
CPU time34m (2074s = 95%)
Max RSS bytes2620919808 (2499 MiB)
Outputting started2026-02-04 00:04:20
Output files
Finished2026-02-04 00:06:43
Saved logsjustin-logs:297917.183-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

int ptr... 0x30ce91a0, 0x3fb56110
Caught exception in looking up space point ptr... 0x2beb0f20, 0x2beb0cd0
Caught exception in looking up space point ptr... 0x6b30db30, 0x2beb0f20
Caught exception in looking up space point ptr... 0x560ba5b0, 0x57d5a730
Caught exception in looking up space point ptr... 0x560ba6b0, 0x560ba5b0
Caught exception in looking up space point ptr... 0x563c35f0, 0x29a22900
Caught exception in looking up space point ptr... 0x39579a40, 0x563c35f0
Caught exception in looking up space point ptr... 0x505fec70, 0x561a06a0
Caught exception in looking up space point ptr... 0x561a06a0, 0x3368c1d0
Caught exception in looking up space point ptr... 0x505fb7b0, 0x505fb8b0
Caught exception in looking up space point ptr... 0x505fb8b0, 0x12b2d420
Caught exception in looking up space point ptr... 0x5b556b20, 0x5b5569b0
Caught exception in looking up space point ptr... 0x5b5569b0, 0x5b556b20
Caught exception in looking up space point ptr... 0x5b556b20, 0x5b556270
Caught exception in looking up space point ptr... 0x5b556270, 0x5b556b20
Caught exception in looking up space point ptr... 0x29b524c0, 0x2db885f0
Caught exception in looking up space point ptr... 0x2db885f0, 0x2db87f80
Caught exception in looking up space point ptr... 0x3c1ac8e0, 0x3c1acc50
Caught exception in looking up space point ptr... 0x3c1acc50, 0x3c1acfc0
Caught exception in looking up space point ptr... 0x54f625b0, 0x54f62e10
Caught exception in looking up space point ptr... 0x54f62510, 0x54f625b0
Caught exception in looking up space point ptr... 0x36779500, 0x24693520
Caught exception in looking up space point ptr... 0x24693520, 0x340d33b0
Caught exception in looking up space point ptr... 0xb992130, 0x533dd0d0
Caught exception in looking up space point ptr... 0x533dd0d0, 0x4e434d20
Caught exception in looking up space point ptr... 0x4e434d20, 0x4d4ad5d0
Caught exception in looking up space point ptr... 0x15980e60, 0x2e1aec60
Caught exception in looking up space point ptr... 0x2e1aec60, 0x13c68e30
Caught exception in looking up space point ptr... 0x4d71cc90, 0x4d71cb00
Caught exception in looking up space point ptr... 0x4d71cb00, 0x150a3660
Caught exception in looking up space point ptr... 0x3c0d59e0, 0x5a24d330
Caught exception in looking up space point ptr... 0x5a24d330, 0x3c0d59e0
Caught exception in looking up space point ptr... 0x57c0b4f0, 0xb6d03f0
Caught exception in looking up space point ptr... 0xb6d03f0, 0x57c0b4f0
Caught exception in looking up space point ptr... 0x114b3e60, 0x114b4100
Caught exception in looking up space point ptr... 0x114b4100, 0x114b3e60
Caught exception in looking up space point ptr... 0x3632aea0, 0x3632b170
Caught exception in looking up space point ptr... 0x3632b170, 0x3632b040
Caught exception in looking up space point ptr... 0x336c6b80, 0x336c69f0
Caught exception in looking up space point ptr... 0x336c69f0, 0x336c6d10
Caught exception in looking up space point ptr... 0x23bb5b50, 0x14ef91e0
Caught exception in looking up space point ptr... 0x14ef91e0, 0x14efa6e0
Caught exception in looking up space point ptr... 0x29ee5070, 0x29ee4ee0
Caught exception in looking up space point ptr... 0x29ee4ee0, 0x29ee5200
Caught exception in looking up space point ptr... 0x3631a6a0, 0x3631c680
Caught exception in looking up space point ptr... 0x3631c680, 0x3631a6a0
Caught exception in looking up space point ptr... 0x2c10e6f0, 0x2c10e9f0
Caught exception in looking up space point ptr... 0x2c10e9f0, 0x2c10e6f0
Caught exception in looking up space point ptr... 0x15801e50, 0x15801f80
Caught exception in looking up space point ptr... 0x15801f80, 0x15801e50
Caught exception in looking up space point ptr... 0xf8a8a90, 0xf8a8900
Caught exception in looking up space point ptr... 0xf8a8900, 0xf8a8a90
Caught exception in looking up space point ptr... 0x242af040, 0x41de8890
Caught exception in looking up space point ptr... 0x41de8890, 0x4e433410
Caught exception in looking up space point ptr... 0x2ddbcba0, 0x2ddbc8a0
Caught exception in looking up space point ptr... 0x2ddbc8a0, 0x2ddbcba0
Caught exception in looking up space point ptr... 0xb24b280, 0xf868c30
Caught exception in looking up space point ptr... 0xf868c30, 0xb24b280
Caught exception in looking up space point ptr... 0x2be5ec90, 0x14bfd0b0
Caught exception in looking up space point ptr... 0x14bfd0b0, 0x256ab8b0
Caught exception in looking up space point ptr... 0x42e68c90, 0x40c1c630
Caught exception in looking up space point ptr... 0x40c1c630, 0x465a5d80
Caught exception in looking up space point ptr... 0x38acb490, 0x2e1d05f0
Caught exception in looking up space point ptr... 0x2e1d05f0, 0x10b5ce40
Caught exception in looking up space point ptr... 0x380797b0, 0x350629f0
Caught exception in looking up space point ptr... 0x350629f0, 0x40c094c0
++++>>>> total num hits: 60230, num free: 58341
04-Feb-2026 01:04:10 CET  Closed output file "np02vd_raw_run042421_0037_df-s04-d1_dw_0_20260203T135102_reco_stage0_20260204T000410_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         69.2566       77.8435       104.805       76.5154       7.2443         27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                        4.032e-05    8.23871e-05   0.000311201    7.386e-05    4.86582e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                3.54808       4.18337       4.96715       4.15606      0.338446        27     
produce:triggerrawdecoder:PDVDTriggerReader4      0.0098079     0.0118885     0.0221549     0.0115375    0.00227497       27     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000388202    0.0004809    0.000645833   0.000461232   6.16608e-05      27     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.001511     0.00173525    0.00222231    0.00166989    0.000183553      27     
produce:beamevent:BeamEvent                       5.979e-05    0.000101361   0.000688333    7.541e-05    0.000115878      27     
produce:pdvddaphne:DAPHNEReaderPDVD                2.12617       2.78444       3.42895       2.83217       0.31325        27     
produce:ophit:OpHitFinder                         0.0295089     0.0374664     0.0452639     0.037548     0.00385299       27     
produce:wclsdatavd:WireCellToolkit                 48.9261        56.88        76.4299       55.5039       6.09476        27     
produce:gaushit:GausHitFinder                     0.559291      0.858365       1.39042      0.844617      0.232401        27     
produce:nhitsfilter:NumberOfHitsFilter           0.000166771   0.000224713   0.000350802   0.000209611   4.43231e-05      27     
produce:reco3d:SpacePointSolver                    5.33398       8.29755       13.8512       8.28638       2.08596        27     
produce:hitpdune:DisambigFromSpacePoints          0.0434565     0.0744066     0.148258      0.0682687     0.0230436       27     
produce:cluster3d:Cluster3D                       0.749652       1.55819       3.79421       1.41773      0.733525        27     
[art]:TriggerResults:TriggerResultInserter        1.979e-05    3.15128e-05   0.00010647     2.836e-05    1.54484e-05      27     
end_path:out1:RootOutput                          3.15e-06     5.33559e-06    2.936e-05     3.96e-06     4.85925e-06      27     
end_path:out1:RootOutput(write)                    2.81965       3.1371        4.4434        3.01443      0.373795        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4647.88 MB
  Peak resident set size usage (VmHWM): 2620.92 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042421_0037_df-s04-d1_dw_0_20260203T135102_reco_stage0_20260204T000410_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042421_0037_df-s04-d1_dw_0_20260203T135102_reco_stage0_20260204T000410_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042421_0037_df-s04-d1_dw_0_20260203T135102_reco_stage0_20260204T000410_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770161300.0,
    "core.end_time": 1770163452.0,
    "core.events": [
      6047,
      6053,
      6059,
      6065,
      6071,
      6077,
      6083,
      6089,
      6095,
      6101,
      6107,
      6113,
      6119,
      6125,
      6131,
      6137,
      6143,
      6149,
      6155,
      6161,
      6167,
      6173,
      6179,
      6185,
      6191,
      6197,
      6203
    ],
    "core.event_count": 27,
    "core.first_event_number": 6047,
    "core.last_event_number": 6203,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42421
    ],
    "core.runs_subruns": [
      4242100001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042421_0037_df-s04-d1_dw_0_20260203T135102.hdf5"
    }
  ]
}
justIN time: 2026-02-04 06:00:26 UTC       justIN version: 01.06.00