justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297950.1@dunegpschedd02.fnal.gov

Jobsub ID297950.1@dunegpschedd02.fnal.gov
Workflow ID12718
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 23:50:26
SiteUK_QMUL
EntryDUNE_UK_London_QMUL_arcce03
Last heartbeat2026-02-04 02:17:15
From worker nodeHostnamecn118.htc.esc.qmul
cpuinfoIntel(R) Xeon(R) Gold 6248R CPU @ 3.00GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-04 00:14:26
Input filesvd-protodune:np02vd_raw_run042422_0044_df-s04-d0_dw_0_20260203T153834.hdf5
JobscriptExit code0
Real time1h (7143s)
CPU time1h (3707s = 51%)
Max RSS bytes2626846720 (2505 MiB)
Outputting started2026-02-04 02:13:34
Output files
Finished2026-02-04 02:17:15
Saved logsjustin-logs:297950.1-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

... 0x369a5da0, 0x369a53c0
Caught exception in looking up space point ptr... 0x369a53c0, 0x369a5da0
Caught exception in looking up space point ptr... 0x41c8cac0, 0x33bb2a80
Caught exception in looking up space point ptr... 0x33bb2a80, 0x1572a9d0
Caught exception in looking up space point ptr... 0x37ecc450, 0x37edaf70
Caught exception in looking up space point ptr... 0x37edaf70, 0x37ecc450
Caught exception in looking up space point ptr... 0x16755590, 0x167470b0
Caught exception in looking up space point ptr... 0x16746640, 0x16755590
Caught exception in looking up space point ptr... 0x47e7ce00, 0x47e77de0
Caught exception in looking up space point ptr... 0x47e77de0, 0x47e85920
Caught exception in looking up space point ptr... 0x59522af0, 0x59518ac0
Caught exception in looking up space point ptr... 0x59518ac0, 0x595189c0
Caught exception in looking up space point ptr... 0x3651ef20, 0x3651f240
Caught exception in looking up space point ptr... 0x3651f240, 0x3651f3d0
Caught exception in looking up space point ptr... 0x2e682590, 0x16fc9780
Caught exception in looking up space point ptr... 0x16fc9780, 0x2e682590
Caught exception in looking up space point ptr... 0x37eea8e0, 0x37eec2b0
Caught exception in looking up space point ptr... 0x37eec2b0, 0x37eea8e0
Caught exception in looking up space point ptr... 0x622728b0, 0x62272a40
Caught exception in looking up space point ptr... 0xe8baeb0, 0x622728b0
Caught exception in looking up space point ptr... 0x14bf2170, 0x14bf3f20
Caught exception in looking up space point ptr... 0x14bf3f20, 0x14bf2170
Caught exception in looking up space point ptr... 0x59523230, 0x59523370
Caught exception in looking up space point ptr... 0x59523370, 0x59523230
Caught exception in looking up space point ptr... 0x33d72ef0, 0x33d69b90
Caught exception in looking up space point ptr... 0x33d69b90, 0x33d69a00
Caught exception in looking up space point ptr... 0x112acc80, 0x50349860
Caught exception in looking up space point ptr... 0x50349860, 0x50349680
Caught exception in looking up space point ptr... 0x434cb590, 0x434cb6c0
Caught exception in looking up space point ptr... 0x434cb6c0, 0x434cb970
Caught exception in looking up space point ptr... 0x113b4bb0, 0x113b47f0
Caught exception in looking up space point ptr... 0x113b47f0, 0x113b4bb0
Caught exception in looking up space point ptr... 0xe8c5f40, 0xe8c6040
Caught exception in looking up space point ptr... 0x58f6c890, 0xe8c5f40
Caught exception in looking up space point ptr... 0x23dbdf00, 0x3dc01e70
Caught exception in looking up space point ptr... 0x3dc01e70, 0x3dc02050
Caught exception in looking up space point ptr... 0x16d22ec0, 0x16d23010
Caught exception in looking up space point ptr... 0x16d23010, 0x16d22ec0
Caught exception in looking up space point ptr... 0x4b661230, 0x4b6615a0
Caught exception in looking up space point ptr... 0x4b6615a0, 0x4b661410
Caught exception in looking up space point ptr... 0x2fdeb100, 0x4324f540
Caught exception in looking up space point ptr... 0x4324f540, 0x2fdeb100
Caught exception in looking up space point ptr... 0x61c892f0, 0x61c89680
Caught exception in looking up space point ptr... 0x61c89680, 0x61c8a600
Caught exception in looking up space point ptr... 0x3676bc70, 0x16d27fc0
Caught exception in looking up space point ptr... 0x16d27fc0, 0x3676bc70
Caught exception in looking up space point ptr... 0x594edb30, 0x594ee070
Caught exception in looking up space point ptr... 0x594ee070, 0x594ee1e0
Caught exception in looking up space point ptr... 0x594cccd0, 0x594ccc30
Caught exception in looking up space point ptr... 0x594ccc30, 0x594cccd0
Caught exception in looking up space point ptr... 0x66b91ba0, 0x66b90080
Caught exception in looking up space point ptr... 0x66b90080, 0x66b91ba0
Caught exception in looking up space point ptr... 0x13de2410, 0x13de2250
Caught exception in looking up space point ptr... 0x13de2250, 0x13de2780
Caught exception in looking up space point ptr... 0x36519d20, 0x36519b20
Caught exception in looking up space point ptr... 0x36519b20, 0x36519a20
Caught exception in looking up space point ptr... 0x47bd34b0, 0x47bd1ce0
Caught exception in looking up space point ptr... 0x47bd1ce0, 0x47bd34b0
Caught exception in looking up space point ptr... 0x355a3860, 0x2fded360
Caught exception in looking up space point ptr... 0x2fded360, 0x355a3860
Caught exception in looking up space point ptr... 0x66af2b70, 0x66af2800
Caught exception in looking up space point ptr... 0x66af2800, 0x66af2b70
Caught exception in looking up space point ptr... 0x16fcd550, 0x16fcd420
Caught exception in looking up space point ptr... 0x16fcd420, 0x16fcd550
Caught exception in looking up space point ptr... 0x16fce130, 0x16fce000
Caught exception in looking up space point ptr... 0x16fce000, 0x16fce130
++++>>>> total num hits: 79547, num free: 76281
04-Feb-2026 02:13:19 GMT  Closed output file "np02vd_raw_run042422_0044_df-s04-d0_dw_0_20260203T153834_reco_stage0_20260204T021319_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         206.638       260.332       322.131       261.787       30.211         27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       7.9323e-05    0.000103969   0.000335386   9.3959e-05    4.68984e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                82.8857       117.458       168.266       120.918       21.3402        27     
produce:triggerrawdecoder:PDVDTriggerReader4       0.5497       0.711198      0.987392      0.694557      0.119856        27     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0164782     0.0228691     0.060911      0.0191606    0.00953995       27     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0814868     0.0879214      0.1246       0.0829225    0.00990797       27     
produce:beamevent:BeamEvent                      0.000103432   0.000174787   0.00131468    0.000124782   0.000224269      27     
produce:pdvddaphne:DAPHNEReaderPDVD                13.0031       18.2447       23.5077       18.1164       2.59015        27     
produce:ophit:OpHitFinder                         0.0334359     0.0550549     0.0692539     0.0566042     0.0091496       27     
produce:wclsdatavd:WireCellToolkit                 73.9712       102.281       111.052       105.461       9.29583        27     
produce:gaushit:GausHitFinder                     0.785716       1.51211       2.74051       1.35604      0.508551        27     
produce:nhitsfilter:NumberOfHitsFilter           0.000327795   0.00055751    0.00131784    0.000522101   0.000198036      27     
produce:reco3d:SpacePointSolver                    5.03697       12.235        22.6206       10.9635       4.57036        27     
produce:hitpdune:DisambigFromSpacePoints          0.076075      0.195869      0.429342      0.173621      0.0924259       27     
produce:cluster3d:Cluster3D                        0.77238       2.54076       10.8619       2.11428       1.96555        27     
[art]:TriggerResults:TriggerResultInserter       2.2568e-05    6.48655e-05   0.000640945   3.9838e-05    0.000115411      27     
end_path:out1:RootOutput                          3.543e-06    8.24604e-06   4.3687e-05     6.345e-06    7.44544e-06      27     
end_path:out1:RootOutput(write)                    3.15407       4.95382       6.10164       4.93692      0.702898        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4466.52 MB
  Peak resident set size usage (VmHWM): 2626.85 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042422_0044_df-s04-d0_dw_0_20260203T153834_reco_stage0_20260204T021319_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042422_0044_df-s04-d0_dw_0_20260203T153834_reco_stage0_20260204T021319_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042422_0044_df-s04-d0_dw_0_20260203T153834_reco_stage0_20260204T021319_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770164078.0,
    "core.end_time": 1770171203.0,
    "core.events": [
      7138,
      7144,
      7150,
      7156,
      7162,
      7168,
      7174,
      7180,
      7186,
      7192,
      7198,
      7204,
      7210,
      7216,
      7222,
      7228,
      7234,
      7240,
      7246,
      7252,
      7258,
      7264,
      7270,
      7276,
      7282,
      7288,
      7294
    ],
    "core.event_count": 27,
    "core.first_event_number": 7138,
    "core.last_event_number": 7294,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42422
    ],
    "core.runs_subruns": [
      4242200001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042422_0044_df-s04-d0_dw_0_20260203T153834.hdf5"
    }
  ]
}
justIN time: 2026-02-04 04:31:48 UTC       justIN version: 01.06.00