justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297923.147@dunegpschedd02.fnal.gov

Jobsub ID297923.147@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 23:00:26
SiteUK_RAL-Tier1
EntryLIGO_UK_RAL_arc_ce02
Last heartbeat2026-02-04 01:15:47
From worker nodeHostnamedune002-1603661.0-lcg2443.gridpp.rl.ac.uk
cpuinfoAMD EPYC 7452 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit216000 (60 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-04 00:08:23
Input filesvd-protodune:np02vd_raw_run042433_0167_df-s04-d1_dw_0_20260203T194536.hdf5
JobscriptExit code0
Real time1h (3916s)
CPU time59m (3541s = 90%)
Max RSS bytes3345268736 (3190 MiB)
Outputting started2026-02-04 01:13:40
Output files
Finished2026-02-04 01:15:47
Saved logsjustin-logs:297923.147-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

.. 0x3717e420, 0x3717e520
Caught exception in looking up space point ptr... 0x83bc9690, 0x83bc94b0
Caught exception in looking up space point ptr... 0x83bc94b0, 0x83bc9690
Caught exception in looking up space point ptr... 0xa6743850, 0x79c1cc80
Caught exception in looking up space point ptr... 0x6620e280, 0xa6743850
Caught exception in looking up space point ptr... 0x57059a80, 0x57059990
Caught exception in looking up space point ptr... 0x57059990, 0x57059a80
Caught exception in looking up space point ptr... 0xa92c6440, 0x57057e00
Caught exception in looking up space point ptr... 0x57057e00, 0x570598a0
Caught exception in looking up space point ptr... 0xa92c62d0, 0x57057e00
Caught exception in looking up space point ptr... 0x57057e00, 0xa92c6370
Caught exception in looking up space point ptr... 0xa92c6370, 0x57057e00
Caught exception in looking up space point ptr... 0x57057e00, 0x57057b30
Caught exception in looking up space point ptr... 0x64a60f30, 0x9bd88bc0
Caught exception in looking up space point ptr... 0x9bd88bc0, 0x78f77a50
Caught exception in looking up space point ptr... 0x61bd7b80, 0x64a60f30
Caught exception in looking up space point ptr... 0x61c93c10, 0x76e5ae80
Caught exception in looking up space point ptr... 0x76e5ae80, 0x61c93c10
Caught exception in looking up space point ptr... 0x5705a600, 0x5705a3a0
Caught exception in looking up space point ptr... 0x5705a3a0, 0x5705a600
Caught exception in looking up space point ptr... 0x8361a000, 0x2f955dd0
Caught exception in looking up space point ptr... 0x2f955dd0, 0x8361a000
Caught exception in looking up space point ptr... 0x67dfabe0, 0x60adeaf0
Caught exception in looking up space point ptr... 0x60adeaf0, 0x67dfabe0
Caught exception in looking up space point ptr... 0x5d1bb7d0, 0x5d1cbef0
Caught exception in looking up space point ptr... 0x5d1cbef0, 0x5d1c6090
Caught exception in looking up space point ptr... 0x72594190, 0x72594330
Caught exception in looking up space point ptr... 0x72594330, 0x62c271c0
Caught exception in looking up space point ptr... 0x62c271c0, 0x54750be0
Caught exception in looking up space point ptr... 0x54750be0, 0x72594190
Caught exception in looking up space point ptr... 0x83e8ea90, 0x31fce940
Caught exception in looking up space point ptr... 0x31fce940, 0x83e8aec0
Caught exception in looking up space point ptr... 0x57060840, 0x570606b0
Caught exception in looking up space point ptr... 0x570606b0, 0xa92b9bb0
Caught exception in looking up space point ptr... 0x86f636d0, 0x86f63540
Caught exception in looking up space point ptr... 0x86f63240, 0x86f68ec0
Caught exception in looking up space point ptr... 0x86f68ec0, 0x86f71040
Caught exception in looking up space point ptr... 0x86f71040, 0x86f6ff20
Caught exception in looking up space point ptr... 0x86f6ff20, 0x86f636d0
Caught exception in looking up space point ptr... 0x5d1c4fa0, 0x5d1c6810
Caught exception in looking up space point ptr... 0x5d1c6810, 0x5d1c4fa0
Caught exception in looking up space point ptr... 0x5d1c4fa0, 0x5d1c66c0
Caught exception in looking up space point ptr... 0x5d1c66c0, 0x5d1c4fa0
Caught exception in looking up space point ptr... 0x82c6ac90, 0x9b124230
Caught exception in looking up space point ptr... 0x82c6d120, 0x82c6ac90
Caught exception in looking up space point ptr... 0x5cf4c890, 0x5cf4c9c0
Caught exception in looking up space point ptr... 0x735be700, 0x5cf4c890
Caught exception in looking up space point ptr... 0x8cfd3490, 0x8cfd5860
Caught exception in looking up space point ptr... 0x8cfd5860, 0x8cfd3300
Caught exception in looking up space point ptr... 0x441339c0, 0x44133b50
Caught exception in looking up space point ptr... 0x44133b50, 0x44133ce0
Caught exception in looking up space point ptr... 0x3df47880, 0x3df47950
Caught exception in looking up space point ptr... 0x3df47950, 0x3df4f6c0
Caught exception in looking up space point ptr... 0x31fca2c0, 0x83e87fc0
Caught exception in looking up space point ptr... 0x83e87fc0, 0x83e88330
Caught exception in looking up space point ptr... 0x380f9140, 0x380f8e20
Caught exception in looking up space point ptr... 0x380f8e20, 0x380f9140
Caught exception in looking up space point ptr... 0x5706cad0, 0x5706b500
Caught exception in looking up space point ptr... 0x5706b500, 0x5706cad0
Caught exception in looking up space point ptr... 0x410cc000, 0x410cbeb0
Caught exception in looking up space point ptr... 0x410cbeb0, 0x410cc000
Caught exception in looking up space point ptr... 0x78838290, 0x788383c0
Caught exception in looking up space point ptr... 0x788383c0, 0x78838290
Caught exception in looking up space point ptr... 0x380c5420, 0x380c93a0
Caught exception in looking up space point ptr... 0x380c93a0, 0x380c8730
Caught exception in looking up space point ptr... 0x94dbf780, 0x56be5b40
Caught exception in looking up space point ptr... 0x56be5b40, 0x4412f580
++++>>>> total num hits: 56937, num free: 54401
04-Feb-2026 01:13:28 UTC  Closed output file "np02vd_raw_run042433_0167_df-s04-d1_dw_0_20260203T194536_reco_stage0_20260204T011328_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         176.358       200.128       217.022       200.607       9.34642        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       0.00010042    0.000154455   0.000377083   0.00013216    6.39979e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                17.7411       20.9886       26.0722       20.4826       2.1946         19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.690996      0.802858      0.915793      0.807237      0.0628012       19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0230379     0.0262526     0.0343485     0.0250848    0.00336257       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.023074      0.0259653     0.0535517     0.0235307     0.0067714       19     
produce:beamevent:BeamEvent                      0.000112462   0.000444939   0.00451611    0.000203434   0.000961967      19     
produce:pdvddaphne:DAPHNEReaderPDVD                3.61996       4.08195       4.46927       4.12899      0.221096        19     
produce:ophit:OpHitFinder                         0.0374284     0.0466088     0.0668042     0.0447643    0.00618642       19     
produce:wclsdatavd:WireCellToolkit                 129.552       146.159       157.129       146.209       6.06325        19     
produce:gaushit:GausHitFinder                      0.78547       1.5596        2.33146       1.48464      0.360729        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000335263   0.000559169   0.000889761    0.0005132    0.000159658      19     
produce:reco3d:SpacePointSolver                    9.95081       16.8738       25.0455       17.6654       4.12341        19     
produce:hitpdune:DisambigFromSpacePoints          0.109766      0.240897       0.37926      0.246545      0.0708383       19     
produce:cluster3d:Cluster3D                        1.48632       2.73341       5.12525       2.5022       0.946859        19     
[art]:TriggerResults:TriggerResultInserter       2.7031e-05    4.72273e-05   0.000159993   3.5216e-05    3.29355e-05      19     
end_path:out1:RootOutput                          5.15e-06     2.82734e-05   0.000344922    9.708e-06    7.4879e-05       19     
end_path:out1:RootOutput(write)                    5.11245       6.5325        7.28233       6.68272      0.575445        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5478.4 MB
  Peak resident set size usage (VmHWM): 3345.27 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0167_df-s04-d1_dw_0_20260203T194536_reco_stage0_20260204T011328_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0167_df-s04-d1_dw_0_20260203T194536_reco_stage0_20260204T011328_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0167_df-s04-d1_dw_0_20260203T194536_reco_stage0_20260204T011328_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770163717.0,
    "core.end_time": 1770167611.0,
    "core.events": [
      19049,
      19055,
      19061,
      19067,
      19073,
      19079,
      19085,
      19091,
      19097,
      19103,
      19109,
      19115,
      19121,
      19127,
      19133,
      19139,
      19145,
      19151,
      19157
    ],
    "core.event_count": 19,
    "core.first_event_number": 19049,
    "core.last_event_number": 19157,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0167_df-s04-d1_dw_0_20260203T194536.hdf5"
    }
  ]
}
justIN time: 2026-02-04 04:33:29 UTC       justIN version: 01.06.00