justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297993.1@dunegpschedd02.fnal.gov

Jobsub ID297993.1@dunegpschedd02.fnal.gov
Workflow ID12718
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-04 01:20:32
SiteUK_QMUL
EntryDUNE_UK_London_QMUL_arcce02
Last heartbeat2026-02-04 03:09:55
From worker nodeHostnamecn053.htc.esc.qmul
cpuinfoIntel(R) Xeon(R) Gold 6248R CPU @ 3.00GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-04 01:21:21
Input filesvd-protodune:np02vd_raw_run042422_0096_df-s02-d1_dw_0_20260203T160641.hdf5
JobscriptExit code0
Real time1h (6379s)
CPU time49m (2945s = 46%)
Max RSS bytes2684141568 (2559 MiB)
Outputting started2026-02-04 03:07:41
Output files
Finished2026-02-04 03:09:55
Saved logsjustin-logs:297993.1-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

320
Caught exception in looking up space point ptr... 0xe1ffe30, 0x5de01e40
Caught exception in looking up space point ptr... 0x366eb530, 0x366eb630
Caught exception in looking up space point ptr... 0x366ecf70, 0x366eb530
Caught exception in looking up space point ptr... 0x330a6a20, 0x6a6bec0
Caught exception in looking up space point ptr... 0x6a6bff0, 0x330a6a20
Caught exception in looking up space point ptr... 0xd95a630, 0x33222510
Caught exception in looking up space point ptr... 0x331d6a70, 0xd95a630
Caught exception in looking up space point ptr... 0x47a0c380, 0x47a0c070
Caught exception in looking up space point ptr... 0x5949d2f0, 0x47a0c380
Caught exception in looking up space point ptr... 0xaaaa3e0, 0x23c60690
Caught exception in looking up space point ptr... 0x23c60690, 0x3285d470
Caught exception in looking up space point ptr... 0x3285d470, 0x41642bd0
Caught exception in looking up space point ptr... 0x41642bd0, 0xaaaa3e0
Caught exception in looking up space point ptr... 0x3c112930, 0x59230850
Caught exception in looking up space point ptr... 0x59230850, 0x11265180
Caught exception in looking up space point ptr... 0x379f4220, 0x379f4040
Caught exception in looking up space point ptr... 0x379f4040, 0x379f4220
Caught exception in looking up space point ptr... 0x28aa7740, 0x38294cf0
Caught exception in looking up space point ptr... 0x38294cf0, 0x28aa7740
Caught exception in looking up space point ptr... 0x28aa7740, 0x149985d0
Caught exception in looking up space point ptr... 0x149985d0, 0x28aa7740
Caught exception in looking up space point ptr... 0x28aa7740, 0x2557d9e0
Caught exception in looking up space point ptr... 0x2557d9e0, 0x28aa7740
Caught exception in looking up space point ptr... 0x28aa7740, 0x145eae50
Caught exception in looking up space point ptr... 0x145eae50, 0x28aa7740
Caught exception in looking up space point ptr... 0x2f0eced0, 0x37e9eb90
Caught exception in looking up space point ptr... 0x14b81fe0, 0x2f0eced0
Caught exception in looking up space point ptr... 0x30d6b750, 0x30d6b8b0
Caught exception in looking up space point ptr... 0x30d6b8b0, 0x30cd9ac0
Caught exception in looking up space point ptr... 0x2c29f7e0, 0x15889c10
Caught exception in looking up space point ptr... 0x15889c10, 0x2c29f7e0
Caught exception in looking up space point ptr... 0x2c29f7e0, 0x255970b0
Caught exception in looking up space point ptr... 0x255970b0, 0x2c29f7e0
Caught exception in looking up space point ptr... 0x3d60ae10, 0xef994a0
Caught exception in looking up space point ptr... 0x2592a8c0, 0x3d60ae10
Caught exception in looking up space point ptr... 0x370cc980, 0x370cc430
Caught exception in looking up space point ptr... 0x370cc430, 0x370cc980
Caught exception in looking up space point ptr... 0x26af6780, 0x38436f10
Caught exception in looking up space point ptr... 0x38436f10, 0x26af6780
Caught exception in looking up space point ptr... 0x26af6780, 0x38436df0
Caught exception in looking up space point ptr... 0x38436df0, 0x26af6780
Caught exception in looking up space point ptr... 0x26af6780, 0x26af6880
Caught exception in looking up space point ptr... 0x26af6880, 0x26af6780
Caught exception in looking up space point ptr... 0x365a16e0, 0x2f0f60b0
Caught exception in looking up space point ptr... 0x2a8cd690, 0x365a16e0
Caught exception in looking up space point ptr... 0x2381e150, 0x23820130
Caught exception in looking up space point ptr... 0x23820130, 0x2381e150
Caught exception in looking up space point ptr... 0x26bcdc90, 0x418ebe90
Caught exception in looking up space point ptr... 0x418ebe90, 0x26bcde50
Caught exception in looking up space point ptr... 0x26bcde50, 0x14ae8100
Caught exception in looking up space point ptr... 0x14ae8100, 0x26bcdc90
Caught exception in looking up space point ptr... 0x219a7c60, 0x219a76d0
Caught exception in looking up space point ptr... 0x219a76d0, 0x219a7860
Caught exception in looking up space point ptr... 0x5fb72f30, 0x60b5d920
Caught exception in looking up space point ptr... 0x60b5d920, 0x4ddd14e0
Caught exception in looking up space point ptr... 0x3d53dc50, 0x2e0f5ee0
Caught exception in looking up space point ptr... 0x3d53da00, 0x3d53dc50
Caught exception in looking up space point ptr... 0x6b2aba0, 0x23a192c0
Caught exception in looking up space point ptr... 0x23a192c0, 0x6b2aba0
Caught exception in looking up space point ptr... 0x5ba646f0, 0x28bdef90
Caught exception in looking up space point ptr... 0x28bdef90, 0x5ba646f0
Caught exception in looking up space point ptr... 0x3685d060, 0x3685cf40
Caught exception in looking up space point ptr... 0x3685cf40, 0x3685d060
Caught exception in looking up space point ptr... 0x2a893850, 0x20423970
Caught exception in looking up space point ptr... 0x20423970, 0x2667e340
++++>>>> total num hits: 41950, num free: 40407
04-Feb-2026 03:07:31 GMT  Closed output file "np02vd_raw_run042422_0096_df-s02-d1_dw_0_20260203T160641_reco_stage0_20260204T030731_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         186.839       232.293       278.186       234.081       23.7888        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       8.0938e-05    0.000109709   0.000241713   0.000106023   3.05588e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                86.0078       117.625       147.987       121.008       16.5447        27     
produce:triggerrawdecoder:PDVDTriggerReader4      0.556287      0.805024       1.05224      0.833234      0.159529        27     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0167429     0.0227218     0.0647828     0.0195207     0.0102426       27     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0817906     0.086104      0.114224      0.0830551    0.00797236       27     
produce:beamevent:BeamEvent                      0.000115168   0.000199442    0.0013272    0.000152198   0.000222066      27     
produce:pdvddaphne:DAPHNEReaderPDVD                8.08801       18.3471       21.7806       18.9874        2.626         27     
produce:ophit:OpHitFinder                         0.032404      0.0468551     0.0648369     0.0420304     0.0105877       27     
produce:wclsdatavd:WireCellToolkit                 68.0101       77.6333       101.163       72.1998       10.8683        27     
produce:gaushit:GausHitFinder                     0.765387       1.14821       2.07499       1.0788       0.328518        27     
produce:nhitsfilter:NumberOfHitsFilter           0.000352501   0.000498372   0.000756955   0.000483602   8.98285e-05      27     
produce:reco3d:SpacePointSolver                    5.7215        10.6852       15.8281       10.2741       2.7928         27     
produce:hitpdune:DisambigFromSpacePoints          0.0821114     0.137466      0.262516      0.121436      0.0473266       27     
produce:cluster3d:Cluster3D                       0.927949       2.00746       12.4714       1.34729       2.12778        27     
[art]:TriggerResults:TriggerResultInserter       2.3717e-05    3.17399e-05   8.2654e-05    2.9134e-05    1.09294e-05      27     
end_path:out1:RootOutput                          3.41e-06     7.85215e-06   2.3923e-05     7.899e-06    3.63902e-06      27     
end_path:out1:RootOutput(write)                    3.15939       3.71539       5.14818       3.51229      0.538027        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4518.57 MB
  Peak resident set size usage (VmHWM): 2684.14 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042422_0096_df-s02-d1_dw_0_20260203T160641_reco_stage0_20260204T030731_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042422_0096_df-s02-d1_dw_0_20260203T160641_reco_stage0_20260204T030731_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042422_0096_df-s02-d1_dw_0_20260203T160641_reco_stage0_20260204T030731_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770168095.0,
    "core.end_time": 1770174456.0,
    "core.events": [
      15565,
      15571,
      15577,
      15583,
      15589,
      15595,
      15601,
      15607,
      15613,
      15619,
      15625,
      15631,
      15637,
      15643,
      15649,
      15655,
      15661,
      15667,
      15673,
      15679,
      15685,
      15691,
      15697,
      15703,
      15709,
      15715,
      15721
    ],
    "core.event_count": 27,
    "core.first_event_number": 15565,
    "core.last_event_number": 15721,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42422
    ],
    "core.runs_subruns": [
      4242200001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042422_0096_df-s02-d1_dw_0_20260203T160641.hdf5"
    }
  ]
}
justIN time: 2026-02-04 04:28:58 UTC       justIN version: 01.06.00