justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303652.182@dunegpschedd01.fnal.gov

Jobsub ID303652.182@dunegpschedd01.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:50:26
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce509
Last heartbeat2026-02-04 00:16:10
From worker nodeHostnameb9p34p1022.cern.ch
cpuinfoAMD EPYC 9655 96-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:35:04
Input filesvd-protodune:np02vd_raw_run042433_0396_df-s04-d1_dw_0_20260203T211237.hdf5
JobscriptExit code0
Real time38m (2333s)
CPU time37m (2252s = 96%)
Max RSS bytes3405832192 (3248 MiB)
Outputting started2026-02-04 00:13:58
Output files
Finished2026-02-04 00:16:10
Saved logsjustin-logs:303652.182-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

space point ptr... 0xe689b30, 0x13044900
Caught exception in looking up space point ptr... 0x411f0ae0, 0x72d8d4f0
Caught exception in looking up space point ptr... 0x72d8d4f0, 0x592d2320
Caught exception in looking up space point ptr... 0x42169530, 0x42169690
Caught exception in looking up space point ptr... 0x42169690, 0x42169530
Caught exception in looking up space point ptr... 0x71843eb0, 0x71857bf0
Caught exception in looking up space point ptr... 0x71857bf0, 0x71857aa0
Caught exception in looking up space point ptr... 0x62a52e50, 0x3c37a1a0
Caught exception in looking up space point ptr... 0x3c37a1a0, 0x3a21bf80
Caught exception in looking up space point ptr... 0x28e739c0, 0x28e73870
Caught exception in looking up space point ptr... 0x28e73870, 0x28e739c0
Caught exception in looking up space point ptr... 0x335cbb80, 0x335cbd10
Caught exception in looking up space point ptr... 0x335cbd10, 0x335cbb80
Caught exception in looking up space point ptr... 0x11ad2080, 0x9ea7a00
Caught exception in looking up space point ptr... 0x9ea7a00, 0x3b41ba50
Caught exception in looking up space point ptr... 0x51f92b40, 0x51306ed0
Caught exception in looking up space point ptr... 0x51306ed0, 0x51f9adf0
Caught exception in looking up space point ptr... 0x60b2b9d0, 0x8e64630
Caught exception in looking up space point ptr... 0x8e64630, 0x2cf5fcf0
Caught exception in looking up space point ptr... 0x3aadfaf0, 0x7404cc00
Caught exception in looking up space point ptr... 0x7404cc00, 0x3aadfaf0
Caught exception in looking up space point ptr... 0x57c8d590, 0x57c8d730
Caught exception in looking up space point ptr... 0x57c8d730, 0x42136c50
Caught exception in looking up space point ptr... 0x42136c50, 0x421356f0
Caught exception in looking up space point ptr... 0x421356f0, 0x42135820
Caught exception in looking up space point ptr... 0x37b38dd0, 0x381f06b0
Caught exception in looking up space point ptr... 0x381f06b0, 0x37b38dd0
Caught exception in looking up space point ptr... 0x60b0caa0, 0x60b0c7f0
Caught exception in looking up space point ptr... 0x60b0c7f0, 0x60b0caa0
Caught exception in looking up space point ptr... 0x3a828550, 0x3a8287e0
Caught exception in looking up space point ptr... 0x3a828160, 0x3a828550
Caught exception in looking up space point ptr... 0xe69cba0, 0xe69d050
Caught exception in looking up space point ptr... 0xe69d050, 0xe69cba0
Caught exception in looking up space point ptr... 0x4d5fbca0, 0x4d5ffc10
Caught exception in looking up space point ptr... 0x4d5ffc10, 0x4d5d8ae0
Caught exception in looking up space point ptr... 0x2bead090, 0x2be97350
Caught exception in looking up space point ptr... 0x2be97350, 0x2bead090
Caught exception in looking up space point ptr... 0x2bead090, 0x2bea84c0
Caught exception in looking up space point ptr... 0x2bea84c0, 0x2bead090
Caught exception in looking up space point ptr... 0x387965c0, 0x37aabfd0
Caught exception in looking up space point ptr... 0x37aabfd0, 0x387965c0
Caught exception in looking up space point ptr... 0x12c99310, 0x12c99480
Caught exception in looking up space point ptr... 0x4038d2d0, 0x12c99310
Caught exception in looking up space point ptr... 0x39e13950, 0x37cf7170
Caught exception in looking up space point ptr... 0x37cf7170, 0x39e13950
Caught exception in looking up space point ptr... 0x51f68ce0, 0x51f68b90
Caught exception in looking up space point ptr... 0x51f68b90, 0x51f68ce0
Caught exception in looking up space point ptr... 0x4219e210, 0x4219df70
Caught exception in looking up space point ptr... 0x4219df70, 0x4219e210
Caught exception in looking up space point ptr... 0x7402a4d0, 0x74026330
Caught exception in looking up space point ptr... 0x74026330, 0x74026180
Caught exception in looking up space point ptr... 0x3572d270, 0x3572d400
Caught exception in looking up space point ptr... 0x3572d400, 0x3572d270
Caught exception in looking up space point ptr... 0x72c7e6a0, 0x5d329130
Caught exception in looking up space point ptr... 0x5d329130, 0x72c7e6a0
Caught exception in looking up space point ptr... 0x70bb7170, 0x16647ad0
Caught exception in looking up space point ptr... 0x16647ad0, 0x16647c60
Caught exception in looking up space point ptr... 0x38598420, 0x13485680
Caught exception in looking up space point ptr... 0x3add0ac0, 0x38598420
Caught exception in looking up space point ptr... 0x2be8c520, 0x2be8ec60
Caught exception in looking up space point ptr... 0x2be8ec60, 0x2be8c520
Caught exception in looking up space point ptr... 0xdef1ee0, 0xdef1bc0
Caught exception in looking up space point ptr... 0xdef1bc0, 0xdef1ee0
Caught exception in looking up space point ptr... 0x360b6da0, 0x360b5c20
Caught exception in looking up space point ptr... 0x360b5c20, 0x360b6da0
Caught exception in looking up space point ptr... 0xe68f060, 0xe68f380
Caught exception in looking up space point ptr... 0xe68f380, 0xe68f060
++++>>>> total num hits: 93579, num free: 89740
04-Feb-2026 01:13:51 CET  Closed output file "np02vd_raw_run042433_0396_df-s04-d1_dw_0_20260203T211237_reco_stage0_20260204T001351_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         106.098       119.861       142.274       120.712       8.99946        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       6.8031e-05    9.66984e-05   0.00024709    8.6209e-05    3.75484e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                5.43502       6.37689       6.91766       6.38218      0.316703        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.0119562     0.0235654     0.0745739     0.0192267     0.0148265       19     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000600538   0.000910894   0.00217049    0.000815861   0.000348778      19     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.000485787   0.000783729   0.00180864    0.000659887   0.000346205      19     
produce:beamevent:BeamEvent                      6.9955e-05    0.000145067   0.000724363   0.000118657   0.000138491      19     
produce:pdvddaphne:DAPHNEReaderPDVD                1.77128       2.08515       2.3655        2.10939      0.193957        19     
produce:ophit:OpHitFinder                         0.0282217     0.0345419     0.0418325     0.0343826    0.00412689       19     
produce:wclsdatavd:WireCellToolkit                 75.3965       82.4615       90.382        82.6253       4.24665        19     
produce:gaushit:GausHitFinder                     0.795415       1.22841       2.12379       1.19909      0.335381        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000217706   0.000301241   0.000588401   0.000270274   9.62936e-05      19     
produce:reco3d:SpacePointSolver                    12.5679       19.8462       28.3044       18.8975       3.80243        19     
produce:hitpdune:DisambigFromSpacePoints          0.114762      0.191845      0.317521      0.184711      0.0568946       19     
produce:cluster3d:Cluster3D                        1.21115       2.74399       6.97737       2.45198       1.34429        19     
[art]:TriggerResults:TriggerResultInserter       2.8022e-05    4.01142e-05   0.000108612   3.5032e-05    1.71912e-05      19     
end_path:out1:RootOutput                          6.47e-06     1.09176e-05   2.4297e-05     9.795e-06    4.71219e-06      19     
end_path:out1:RootOutput(write)                    3.9025        4.81439       6.28985       4.81483      0.578758        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5508.68 MB
  Peak resident set size usage (VmHWM): 3405.83 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0396_df-s04-d1_dw_0_20260203T211237_reco_stage0_20260204T001351_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0396_df-s04-d1_dw_0_20260203T211237_reco_stage0_20260204T001351_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0396_df-s04-d1_dw_0_20260203T211237_reco_stage0_20260204T001351_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770161712.0,
    "core.end_time": 1770164032.0,
    "core.events": [
      45155,
      45161,
      45167,
      45173,
      45179,
      45185,
      45191,
      45197,
      45203,
      45209,
      45215,
      45221,
      45227,
      45233,
      45239,
      45245,
      45251,
      45257,
      45263
    ],
    "core.event_count": 19,
    "core.first_event_number": 45155,
    "core.last_event_number": 45263,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0396_df-s04-d1_dw_0_20260203T211237.hdf5"
    }
  ]
}
justIN time: 2026-02-04 06:06:43 UTC       justIN version: 01.06.00