justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303654.73@dunegpschedd01.fnal.gov

Jobsub ID303654.73@dunegpschedd01.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:52:26
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce504
Last heartbeat2026-02-04 00:46:01
From worker nodeHostnameb9p07p8370.cern.ch
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-04 00:01:47
Input filesvd-protodune:np02vd_raw_run042433_0605_df-s02-d0_dw_0_20260203T223204.hdf5
JobscriptExit code0
Real time41m (2510s)
CPU time40m (2412s = 96%)
Max RSS bytes3407077376 (3249 MiB)
Outputting started2026-02-04 00:43:38
Output files
Finished2026-02-04 00:46:01
Saved logsjustin-logs:303654.73-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

p space point ptr... 0x620db320, 0xb03a600
Caught exception in looking up space point ptr... 0xb03a600, 0x620db320
Caught exception in looking up space point ptr... 0xd715af0, 0xd717390
Caught exception in looking up space point ptr... 0xd717390, 0xd715af0
Caught exception in looking up space point ptr... 0xd715af0, 0xd714d50
Caught exception in looking up space point ptr... 0xd714d50, 0xd715af0
Caught exception in looking up space point ptr... 0x62eff970, 0x747ae5a0
Caught exception in looking up space point ptr... 0x747ae5a0, 0x62eff970
Caught exception in looking up space point ptr... 0x570450e0, 0x43fa6d20
Caught exception in looking up space point ptr... 0x43fa6d20, 0x886a7ea0
Caught exception in looking up space point ptr... 0x36f13b60, 0x36f13c90
Caught exception in looking up space point ptr... 0x36f05f60, 0x36f13b60
Caught exception in looking up space point ptr... 0x15cb2f00, 0x15cb2d60
Caught exception in looking up space point ptr... 0x39be4cf0, 0x15cb2f00
Caught exception in looking up space point ptr... 0x82e06940, 0x82e06b10
Caught exception in looking up space point ptr... 0x82e06b10, 0x82e06940
Caught exception in looking up space point ptr... 0x3990b6d0, 0x8109ad00
Caught exception in looking up space point ptr... 0x8109ad00, 0x631b1130
Caught exception in looking up space point ptr... 0x631b1130, 0x540d10f0
Caught exception in looking up space point ptr... 0x36f2c130, 0x36f1f4f0
Caught exception in looking up space point ptr... 0x36f1f4f0, 0x36f25600
Caught exception in looking up space point ptr... 0x36f0db90, 0x36f0e870
Caught exception in looking up space point ptr... 0x36f0e870, 0x36f0db90
Caught exception in looking up space point ptr... 0x2d392a80, 0x2d376b30
Caught exception in looking up space point ptr... 0x2d376b30, 0x2d392a80
Caught exception in looking up space point ptr... 0x36f14210, 0x36f163f0
Caught exception in looking up space point ptr... 0x36f163f0, 0x36f15330
Caught exception in looking up space point ptr... 0x5199eb30, 0x519a4af0
Caught exception in looking up space point ptr... 0x519a4af0, 0x5199eb30
Caught exception in looking up space point ptr... 0x36f22130, 0x36f2d510
Caught exception in looking up space point ptr... 0x36f2d510, 0x36f35190
Caught exception in looking up space point ptr... 0x15cb5430, 0x15cb7700
Caught exception in looking up space point ptr... 0x15cb7700, 0x15cb5430
Caught exception in looking up space point ptr... 0x3145f870, 0x3145f530
Caught exception in looking up space point ptr... 0x3145f530, 0x3145f6d0
Caught exception in looking up space point ptr... 0x80fada60, 0x80fadb90
Caught exception in looking up space point ptr... 0x80fadb90, 0x80fada60
Caught exception in looking up space point ptr... 0x63e28510, 0x3ccde090
Caught exception in looking up space point ptr... 0x3ccde090, 0x63e28510
Caught exception in looking up space point ptr... 0x62283f80, 0x66d046f0
Caught exception in looking up space point ptr... 0x66d046f0, 0x34374380
Caught exception in looking up space point ptr... 0x34374380, 0x66d046f0
Caught exception in looking up space point ptr... 0x66d046f0, 0x343744d0
Caught exception in looking up space point ptr... 0x343744d0, 0x66d046f0
Caught exception in looking up space point ptr... 0x66d046f0, 0x6266c7c0
Caught exception in looking up space point ptr... 0x6266c7c0, 0x66d046f0
Caught exception in looking up space point ptr... 0x66d046f0, 0x6266c5f0
Caught exception in looking up space point ptr... 0x620ce640, 0x620ce7d0
Caught exception in looking up space point ptr... 0x620ce7d0, 0x620ce640
Caught exception in looking up space point ptr... 0xa3a5830, 0x146814e0
Caught exception in looking up space point ptr... 0x146814e0, 0x3f07d530
Caught exception in looking up space point ptr... 0x3b871320, 0x2d24d280
Caught exception in looking up space point ptr... 0x2d24d280, 0x3b871320
Caught exception in looking up space point ptr... 0x47dd2530, 0x556bcb00
Caught exception in looking up space point ptr... 0x556bcb00, 0x47dd2530
Caught exception in looking up space point ptr... 0xa393e70, 0xa393c90
Caught exception in looking up space point ptr... 0xa393c90, 0xa393e70
Caught exception in looking up space point ptr... 0x37f7da90, 0x37f7d720
Caught exception in looking up space point ptr... 0x37f7d720, 0x37f7da90
Caught exception in looking up space point ptr... 0xa462680, 0xdbcb190
Caught exception in looking up space point ptr... 0xdbcb190, 0xdbcb050
Caught exception in looking up space point ptr... 0x620ce320, 0x620ce4b0
Caught exception in looking up space point ptr... 0x620ce4b0, 0x620ce320
Caught exception in looking up space point ptr... 0x62f06540, 0x62f06280
Caught exception in looking up space point ptr... 0x62f06280, 0x62f063a0
Caught exception in looking up space point ptr... 0x2db2d9a0, 0x2db2d820
Caught exception in looking up space point ptr... 0x2db2d820, 0x2db2dc00
++++>>>> total num hits: 74075, num free: 71442
04-Feb-2026 01:43:29 CET  Closed output file "np02vd_raw_run042433_0605_df-s02-d0_dw_0_20260203T223204_reco_stage0_20260204T004329_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         109.694       129.193       145.989       129.966       10.4357        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       4.9191e-05    9.90904e-05   0.000301465   9.1992e-05    5.29965e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                6.28776       7.18182       8.15135       7.13351      0.489558        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.0105992     0.0331131     0.088813      0.0246259     0.0227502       19     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000441278   0.000592935   0.00123708     0.0005465    0.000184675      19     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.000403078   0.000493545   0.000744024   0.000470459   7.9572e-05       19     
produce:beamevent:BeamEvent                      5.7301e-05    0.000150955   0.00109935    0.000104242   0.000224107      19     
produce:pdvddaphne:DAPHNEReaderPDVD                2.1279        2.5549        2.99364       2.60039      0.276115        19     
produce:ophit:OpHitFinder                         0.0239517     0.0338523     0.0435398      0.03468     0.00512026       19     
produce:wclsdatavd:WireCellToolkit                 84.2808       94.9367       102.958       97.3479       6.27703        19     
produce:gaushit:GausHitFinder                     0.796573       1.19489       1.75758       1.15179      0.288115        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000230364   0.000338631   0.00057786    0.000307935   9.78021e-05      19     
produce:reco3d:SpacePointSolver                    9.38542       16.3711       27.1204       15.7078       4.55571        19     
produce:hitpdune:DisambigFromSpacePoints          0.0760144     0.153451      0.322583      0.149496      0.0507592       19     
produce:cluster3d:Cluster3D                        1.11722       2.02782       3.02652       2.03963      0.555885        19     
[art]:TriggerResults:TriggerResultInserter        2.66e-05     4.20565e-05   0.000145562   2.9891e-05    2.93402e-05      19     
end_path:out1:RootOutput                          3.711e-06    7.60584e-06    2.858e-05     7.05e-06     5.2395e-06       19     
end_path:out1:RootOutput(write)                    3.82985       4.67331       5.62372       4.64709      0.421074        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5398.82 MB
  Peak resident set size usage (VmHWM): 3407.08 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0605_df-s02-d0_dw_0_20260203T223204_reco_stage0_20260204T004329_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0605_df-s02-d0_dw_0_20260203T223204_reco_stage0_20260204T004329_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0605_df-s02-d0_dw_0_20260203T223204_reco_stage0_20260204T004329_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770163318.0,
    "core.end_time": 1770165810.0,
    "core.events": [
      68976,
      68982,
      68988,
      68994,
      69000,
      69006,
      69012,
      69018,
      69024,
      69030,
      69036,
      69042,
      69048,
      69054,
      69060,
      69066,
      69072,
      69078,
      69084
    ],
    "core.event_count": 19,
    "core.first_event_number": 68976,
    "core.last_event_number": 69084,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0605_df-s02-d0_dw_0_20260203T223204.hdf5"
    }
  ]
}
justIN time: 2026-02-04 09:13:31 UTC       justIN version: 01.06.00