justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303652.15@dunegpschedd01.fnal.gov

Jobsub ID303652.15@dunegpschedd01.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:50:26
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce509
Last heartbeat2026-02-04 00:48:52
From worker nodeHostnameb9p04p3844.cern.ch
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:26:31
Input filesvd-protodune:np02vd_raw_run042433_0023_df-s03-d1_dw_0_20260203T185052.hdf5
JobscriptExit code0
Real time1h (4788s)
CPU time1h (4602s = 96%)
Max RSS bytes3402530816 (3244 MiB)
Outputting started2026-02-04 00:46:21
Output files
Finished2026-02-04 00:48:52
Saved logsjustin-logs:303652.15-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

looking up space point ptr... 0x409ce6f0, 0x409d4990
Caught exception in looking up space point ptr... 0x46d3e570, 0x5ef75550
Caught exception in looking up space point ptr... 0x5ef75550, 0x99ad150
Caught exception in looking up space point ptr... 0x88b3f150, 0x68496760
Caught exception in looking up space point ptr... 0x68496760, 0x47369480
Caught exception in looking up space point ptr... 0x38022d50, 0x398234e0
Caught exception in looking up space point ptr... 0x398234e0, 0x199f5ab0
Caught exception in looking up space point ptr... 0x46f27810, 0x46f27230
Caught exception in looking up space point ptr... 0x737a1c10, 0x46f27810
Caught exception in looking up space point ptr... 0x19cff750, 0xd0cd7b0
Caught exception in looking up space point ptr... 0xd0cd7b0, 0x5fed73e0
Caught exception in looking up space point ptr... 0x49e289c0, 0x69de8550
Caught exception in looking up space point ptr... 0x69de8550, 0x69de5b90
Caught exception in looking up space point ptr... 0x471752b0, 0x47174c20
Caught exception in looking up space point ptr... 0x47174c20, 0x471752b0
Caught exception in looking up space point ptr... 0x471752b0, 0x47174f90
Caught exception in looking up space point ptr... 0x47174f90, 0x471752b0
Caught exception in looking up space point ptr... 0x19c2aec0, 0x35d0cae0
Caught exception in looking up space point ptr... 0x35d0cae0, 0x19c2aec0
Caught exception in looking up space point ptr... 0x7966b340, 0x7966b090
Caught exception in looking up space point ptr... 0x7966b090, 0x7966b340
Caught exception in looking up space point ptr... 0x371e3b60, 0x5e95edd0
Caught exception in looking up space point ptr... 0x5e95edd0, 0x371e3b60
Caught exception in looking up space point ptr... 0x161a82b0, 0x161a8150
Caught exception in looking up space point ptr... 0x161a8150, 0x161a82b0
Caught exception in looking up space point ptr... 0x5cc7d250, 0x80be1290
Caught exception in looking up space point ptr... 0x80be1290, 0x5cc7d250
Caught exception in looking up space point ptr... 0x5cc7d250, 0x80be9560
Caught exception in looking up space point ptr... 0x80be9560, 0x5cc7d250
Caught exception in looking up space point ptr... 0x12c59a30, 0x12c5b6f0
Caught exception in looking up space point ptr... 0x12c5b6f0, 0x12c59a30
Caught exception in looking up space point ptr... 0x801bcf90, 0x801bce40
Caught exception in looking up space point ptr... 0x801bce40, 0x801bcf90
Caught exception in looking up space point ptr... 0x11d9fff0, 0x17719260
Caught exception in looking up space point ptr... 0x17719260, 0x11d9fff0
Caught exception in looking up space point ptr... 0xada40a0, 0xada41f0
Caught exception in looking up space point ptr... 0xada41f0, 0xada40a0
Caught exception in looking up space point ptr... 0x443e64a0, 0x443e6180
Caught exception in looking up space point ptr... 0x443e6180, 0x443e64a0
Caught exception in looking up space point ptr... 0x67b24910, 0x67b24780
Caught exception in looking up space point ptr... 0x67b24780, 0x67b24910
Caught exception in looking up space point ptr... 0x4714d340, 0x4714cfb0
Caught exception in looking up space point ptr... 0x4714cfb0, 0x4714d340
Caught exception in looking up space point ptr... 0x198be080, 0x198be210
Caught exception in looking up space point ptr... 0x198be210, 0x198be080
Caught exception in looking up space point ptr... 0x59e0d5b0, 0x31c7fae0
Caught exception in looking up space point ptr... 0x31c7fae0, 0x31c7fdb0
Caught exception in looking up space point ptr... 0x4715e330, 0x4715deb0
Caught exception in looking up space point ptr... 0x4715deb0, 0x4715e330
Caught exception in looking up space point ptr... 0x176efa20, 0x176ef380
Caught exception in looking up space point ptr... 0x176ef380, 0x176ef8c0
Caught exception in looking up space point ptr... 0x409d8af0, 0x409d87f0
Caught exception in looking up space point ptr... 0x409d87f0, 0x409d86f0
Caught exception in looking up space point ptr... 0x11dabdb0, 0x11da8f00
Caught exception in looking up space point ptr... 0x11da8f00, 0x11dabdb0
Caught exception in looking up space point ptr... 0x11e8f360, 0x40b187b0
Caught exception in looking up space point ptr... 0x40b187b0, 0x11e8f360
Caught exception in looking up space point ptr... 0x1970f8b0, 0x82538530
Caught exception in looking up space point ptr... 0x82538530, 0x771fced0
Caught exception in looking up space point ptr... 0x408d7ed0, 0x408d3be0
Caught exception in looking up space point ptr... 0x408d3be0, 0x408d6fa0
Caught exception in looking up space point ptr... 0x5cb5a7a0, 0x5cb5b120
Caught exception in looking up space point ptr... 0x5cb5b120, 0x5cb5a7a0
Caught exception in looking up space point ptr... 0x35c3ee60, 0x80152f50
Caught exception in looking up space point ptr... 0x80152f50, 0x80ba2050
Caught exception in looking up space point ptr... 0x46c86a00, 0x46c86880
Caught exception in looking up space point ptr... 0x46c86880, 0x46c86cb0
++++>>>> total num hits: 83967, num free: 80276
04-Feb-2026 01:46:11 CET  Closed output file "np02vd_raw_run042433_0023_df-s03-d1_dw_0_20260203T185052_reco_stage0_20260204T004611_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         178.491       247.736       279.636       257.21        28.2754        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       5.3771e-05    0.00011459    0.000288584   8.5692e-05    6.80768e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                6.31247       8.23301       9.83981       8.28618      0.950948        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.0110079     0.0152075     0.0260961     0.0140377    0.00409352       19     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000433906   0.000681272   0.00146738    0.000589129   0.000258869      19     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.000450507   0.000686686   0.00151904    0.00060881    0.000250785      19     
produce:beamevent:BeamEvent                      7.2472e-05    0.000201699   0.00123431    0.000126552   0.000253175      19     
produce:pdvddaphne:DAPHNEReaderPDVD                3.12011       4.51558       5.80497       4.41356      0.793531        19     
produce:ophit:OpHitFinder                         0.0343466     0.0712709     0.199851      0.0568704     0.0395738       19     
produce:wclsdatavd:WireCellToolkit                 132.342       187.626       215.461       189.24        22.877         19     
produce:gaushit:GausHitFinder                      1.37064       2.34933       3.51045       2.36817      0.591726        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000279965   0.000549389    0.0014651    0.000443857   0.000317808      19     
produce:reco3d:SpacePointSolver                    21.9076       31.7504       58.4627       28.0042       9.54333        19     
produce:hitpdune:DisambigFromSpacePoints          0.130274      0.306199       1.08351      0.249074      0.205317        19     
produce:cluster3d:Cluster3D                        2.05494       3.93576       8.50172       3.58277       1.37356        19     
[art]:TriggerResults:TriggerResultInserter       3.4791e-05    5.14947e-05   0.000108832   4.2361e-05    2.0178e-05       19     
end_path:out1:RootOutput                          6.91e-06     1.32887e-05   3.4591e-05     1.114e-05    7.00939e-06      19     
end_path:out1:RootOutput(write)                    6.57948       8.88806       10.4032       9.07441       1.06729        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5423.85 MB
  Peak resident set size usage (VmHWM): 3402.53 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0023_df-s03-d1_dw_0_20260203T185052_reco_stage0_20260204T004611_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0023_df-s03-d1_dw_0_20260203T185052_reco_stage0_20260204T004611_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0023_df-s03-d1_dw_0_20260203T185052_reco_stage0_20260204T004611_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770161204.0,
    "core.end_time": 1770165974.0,
    "core.events": [
      2631,
      2637,
      2643,
      2649,
      2655,
      2661,
      2667,
      2673,
      2679,
      2685,
      2691,
      2697,
      2703,
      2709,
      2715,
      2721,
      2727,
      2733,
      2739
    ],
    "core.event_count": 19,
    "core.first_event_number": 2631,
    "core.last_event_number": 2739,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0023_df-s03-d1_dw_0_20260203T185052.hdf5"
    }
  ]
}
justIN time: 2026-02-04 07:38:22 UTC       justIN version: 01.06.00