justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297923.113@dunegpschedd02.fnal.gov

Jobsub ID297923.113@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 23:00:26
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce504
Last heartbeat2026-02-04 00:47:08
From worker nodeHostnameb9p12p0289.cern.ch
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-04 00:03:27
Input filesvd-protodune:np02vd_raw_run042433_0417_df-s02-d0_dw_0_20260203T212036.hdf5
JobscriptExit code0
Real time41m (2479s)
CPU time39m (2391s = 96%)
Max RSS bytes3415486464 (3257 MiB)
Outputting started2026-02-04 00:44:48
Output files
Finished2026-02-04 00:47:08
Saved logsjustin-logs:297923.113-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

. 0x28a155d0, 0x28a15480
Caught exception in looking up space point ptr... 0x66cbbc40, 0x66cb78a0
Caught exception in looking up space point ptr... 0x66cb78a0, 0x66cbc6c0
Caught exception in looking up space point ptr... 0x540b0660, 0x540b08c0
Caught exception in looking up space point ptr... 0x540b08c0, 0x540b0660
Caught exception in looking up space point ptr... 0x540b0660, 0x5407ed40
Caught exception in looking up space point ptr... 0x5407ed40, 0x540b0660
Caught exception in looking up space point ptr... 0x44484ff0, 0x37f58fb0
Caught exception in looking up space point ptr... 0x37f58fb0, 0x37f57170
Caught exception in looking up space point ptr... 0x37009970, 0x37009320
Caught exception in looking up space point ptr... 0x37009320, 0x37009970
Caught exception in looking up space point ptr... 0x7fcb3420, 0x7fcb2f70
Caught exception in looking up space point ptr... 0x7fcb2f70, 0x7fcb3420
Caught exception in looking up space point ptr... 0x443a5b00, 0x443b4400
Caught exception in looking up space point ptr... 0x443b4400, 0x443b50b0
Caught exception in looking up space point ptr... 0x2ede2b40, 0xdfc3690
Caught exception in looking up space point ptr... 0xdfc3690, 0x2ede2b40
Caught exception in looking up space point ptr... 0x2e49bf00, 0x429ae450
Caught exception in looking up space point ptr... 0x429ae450, 0x2e49bf00
Caught exception in looking up space point ptr... 0x411e3250, 0x4107f6b0
Caught exception in looking up space point ptr... 0x4107f6b0, 0x4112eb50
Caught exception in looking up space point ptr... 0x375bfd20, 0x375bf9b0
Caught exception in looking up space point ptr... 0x375bf9b0, 0x375bfb90
Caught exception in looking up space point ptr... 0x192a3520, 0x192a3670
Caught exception in looking up space point ptr... 0x192a3670, 0x192a3520
Caught exception in looking up space point ptr... 0x3beb9fe0, 0x3beca8f0
Caught exception in looking up space point ptr... 0x3beca8f0, 0x17881b80
Caught exception in looking up space point ptr... 0x3bec0d10, 0x17893e60
Caught exception in looking up space point ptr... 0x17893e60, 0x3bec0d10
Caught exception in looking up space point ptr... 0x3c362b20, 0x3c3628d0
Caught exception in looking up space point ptr... 0x3c3628d0, 0x3c3627d0
Caught exception in looking up space point ptr... 0x31536020, 0x314d0e60
Caught exception in looking up space point ptr... 0x314d0e60, 0x31536020
Caught exception in looking up space point ptr... 0x36939de0, 0x4a71d520
Caught exception in looking up space point ptr... 0x4a71d520, 0x36939de0
Caught exception in looking up space point ptr... 0x366d1820, 0x366d1b20
Caught exception in looking up space point ptr... 0x366d1b20, 0x366d1820
Caught exception in looking up space point ptr... 0x17887af0, 0x178877d0
Caught exception in looking up space point ptr... 0x3bfa0f80, 0x3bfa1120
Caught exception in looking up space point ptr... 0x3bfa1120, 0x3bfa12c0
Caught exception in looking up space point ptr... 0x3bfa12c0, 0x17887af0
Caught exception in looking up space point ptr... 0x37551e80, 0x37551f80
Caught exception in looking up space point ptr... 0x37551f80, 0x37552080
Caught exception in looking up space point ptr... 0x37491750, 0x37491050
Caught exception in looking up space point ptr... 0x37491050, 0x37490d30
Caught exception in looking up space point ptr... 0x53dce670, 0x44a72740
Caught exception in looking up space point ptr... 0x44a72740, 0x45084dd0
Caught exception in looking up space point ptr... 0x70ca71d0, 0x44102c60
Caught exception in looking up space point ptr... 0x44102c60, 0x70ca71d0
Caught exception in looking up space point ptr... 0x3ca31410, 0x315b77b0
Caught exception in looking up space point ptr... 0x315b77b0, 0x315b75f0
Caught exception in looking up space point ptr... 0x11d45d70, 0x11d45c70
Caught exception in looking up space point ptr... 0x11d45c70, 0x50a58610
Caught exception in looking up space point ptr... 0x429dfb70, 0x429e16e0
Caught exception in looking up space point ptr... 0x429e16e0, 0x429e0ca0
Caught exception in looking up space point ptr... 0x45e7ab40, 0x45e7c190
Caught exception in looking up space point ptr... 0x45e7c190, 0x45e7ab40
Caught exception in looking up space point ptr... 0x55e7c200, 0x55e7c0b0
Caught exception in looking up space point ptr... 0x55e7c0b0, 0x55e7c200
Caught exception in looking up space point ptr... 0x3bfba6a0, 0x3bfba550
Caught exception in looking up space point ptr... 0x3bfba550, 0x3bfbf6b0
Caught exception in looking up space point ptr... 0x28a23070, 0x28a22d50
Caught exception in looking up space point ptr... 0x28a22d50, 0x28a22ab0
Caught exception in looking up space point ptr... 0x2f5b7d50, 0x697c1210
Caught exception in looking up space point ptr... 0x697c1210, 0x2f5b7d50
Caught exception in looking up space point ptr... 0x4045c7d0, 0x316173d0
Caught exception in looking up space point ptr... 0x316173d0, 0x414bbdb0
++++>>>> total num hits: 105316, num free: 100145
04-Feb-2026 01:44:40 CET  Closed output file "np02vd_raw_run042433_0417_df-s02-d0_dw_0_20260203T212036_reco_stage0_20260204T004440_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         114.114       127.731       144.754       128.928       8.02881        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       4.6901e-05    0.000102038   0.000319041    7.627e-05    6.87224e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                5.65795       6.34402        7.081        6.38953      0.391396        19     
produce:triggerrawdecoder:PDVDTriggerReader4     0.00976806     0.0138673     0.025496      0.0119273    0.00448595       19     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000417151   0.000513039   0.000694382   0.000510232   7.52212e-05      19     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.00037658    0.000461395   0.000603652   0.000458712   5.54831e-05      19     
produce:beamevent:BeamEvent                      5.8031e-05    0.000125804   0.000753943    9.826e-05    0.000149791      19     
produce:pdvddaphne:DAPHNEReaderPDVD                2.08632       2.57004       3.10328       2.55881      0.313305        19     
produce:ophit:OpHitFinder                         0.0231909     0.0331186     0.0440119     0.0305437    0.00626686       19     
produce:wclsdatavd:WireCellToolkit                 84.0168       93.228        102.725       94.0316       4.46879        19     
produce:gaushit:GausHitFinder                     0.802876       1.17581       1.66883       1.20621      0.272221        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000249121   0.000354716   0.000848892   0.000329231   0.000126803      19     
produce:reco3d:SpacePointSolver                    10.8336       17.2504       24.628        18.0094       3.78688        19     
produce:hitpdune:DisambigFromSpacePoints          0.0937145     0.147819      0.207189      0.146628      0.0330601       19     
produce:cluster3d:Cluster3D                        1.33355       2.30865       5.13286       2.04998      0.842331        19     
[art]:TriggerResults:TriggerResultInserter        2.199e-05    3.14232e-05   0.000100391    2.591e-05    1.71716e-05      19     
end_path:out1:RootOutput                          3.55e-06     6.86737e-06    2.821e-05     5.43e-06     5.25495e-06      19     
end_path:out1:RootOutput(write)                    4.15388       4.62801       5.71905       4.48781      0.357663        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5394.93 MB
  Peak resident set size usage (VmHWM): 3415.49 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0417_df-s02-d0_dw_0_20260203T212036_reco_stage0_20260204T004440_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0417_df-s02-d0_dw_0_20260203T212036_reco_stage0_20260204T004440_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0417_df-s02-d0_dw_0_20260203T212036_reco_stage0_20260204T004440_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770163417.0,
    "core.end_time": 1770165881.0,
    "core.events": [
      47544,
      47550,
      47556,
      47562,
      47568,
      47574,
      47580,
      47586,
      47592,
      47598,
      47604,
      47610,
      47616,
      47622,
      47628,
      47634,
      47640,
      47646,
      47652
    ],
    "core.event_count": 19,
    "core.first_event_number": 47544,
    "core.last_event_number": 47652,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0417_df-s02-d0_dw_0_20260203T212036.hdf5"
    }
  ]
}
justIN time: 2026-02-04 10:35:51 UTC       justIN version: 01.06.00