justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303646.109@dunegpschedd01.fnal.gov

Jobsub ID303646.109@dunegpschedd01.fnal.gov
Workflow ID12718
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:44:24
SiteUK_Imperial
EntryDUNE_T2_UK_London_IC_ceprod01
Last heartbeat2026-02-04 00:05:12
From worker nodeHostnamewl06.grid.hep.ph.ic.ac.uk
cpuinfoAMD EPYC 7742 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 22:46:07
Input filesvd-protodune:np02vd_raw_run042422_0103_df-s03-d0_dw_0_20260203T161028.hdf5
JobscriptExit code0
Real time1h (4572s)
CPU time1h (3709s = 81%)
Max RSS bytes2620268544 (2498 MiB)
Outputting started2026-02-04 00:02:21
Output files
Finished2026-02-04 00:05:12
Saved logsjustin-logs:303646.109-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

9db0
Caught exception in looking up space point ptr... 0xfb19db0, 0xae2e920
Caught exception in looking up space point ptr... 0xae2e920, 0x33808180
Caught exception in looking up space point ptr... 0x33808180, 0x39bf8e30
Caught exception in looking up space point ptr... 0x39bf8e30, 0x46e49ec0
Caught exception in looking up space point ptr... 0xfb48580, 0xfb48a00
Caught exception in looking up space point ptr... 0xfb48a00, 0x3765b520
Caught exception in looking up space point ptr... 0x3765b520, 0x3765b430
Caught exception in looking up space point ptr... 0x15892ca0, 0x4d7f2b50
Caught exception in looking up space point ptr... 0x4d7f2b50, 0x60229fe0
Caught exception in looking up space point ptr... 0x60229fe0, 0x3d7b92e0
Caught exception in looking up space point ptr... 0x3139eab0, 0x14848b70
Caught exception in looking up space point ptr... 0x14848b70, 0x3d1c31c0
Caught exception in looking up space point ptr... 0x3d1c31c0, 0x3d1c3410
Caught exception in looking up space point ptr... 0x3d1c3410, 0x3139eab0
Caught exception in looking up space point ptr... 0x16145b40, 0x37ef0850
Caught exception in looking up space point ptr... 0x554c0990, 0x16145b40
Caught exception in looking up space point ptr... 0x44b27e80, 0x2be69030
Caught exception in looking up space point ptr... 0x2be69030, 0x44b27e80
Caught exception in looking up space point ptr... 0x44b27e80, 0x5cdcad40
Caught exception in looking up space point ptr... 0x5cdcad40, 0x44b27e80
Caught exception in looking up space point ptr... 0x15cd5160, 0x59bc9930
Caught exception in looking up space point ptr... 0x59bc9930, 0x9de1d70
Caught exception in looking up space point ptr... 0x48faf600, 0x48fb1350
Caught exception in looking up space point ptr... 0x48fb1350, 0x3765c7e0
Caught exception in looking up space point ptr... 0x3bbb3560, 0x39a73c30
Caught exception in looking up space point ptr... 0x39a73c30, 0x3bbb3560
Caught exception in looking up space point ptr... 0x4e3d2880, 0x4e3d2cd0
Caught exception in looking up space point ptr... 0x4e3d2cd0, 0x4e3d2b40
Caught exception in looking up space point ptr... 0x27566360, 0x2e7630b0
Caught exception in looking up space point ptr... 0x2e7630b0, 0x4d714060
Caught exception in looking up space point ptr... 0x24301860, 0x24302a80
Caught exception in looking up space point ptr... 0x24301b90, 0x24301a60
Caught exception in looking up space point ptr... 0x24301a60, 0x24301860
Caught exception in looking up space point ptr... 0x47d95370, 0x585246a0
Caught exception in looking up space point ptr... 0x585246a0, 0x47d95370
Caught exception in looking up space point ptr... 0x47d95370, 0x3876df00
Caught exception in looking up space point ptr... 0x3876df00, 0x47d95370
Caught exception in looking up space point ptr... 0x3b007210, 0x4fcb8170
Caught exception in looking up space point ptr... 0x4fcb8170, 0x520dcaf0
Caught exception in looking up space point ptr... 0x3b0dfd80, 0x3b0dde30
Caught exception in looking up space point ptr... 0x3b0dde30, 0x3b0de120
Caught exception in looking up space point ptr... 0x375f4aa0, 0x4bf77890
Caught exception in looking up space point ptr... 0x4bf77890, 0x4bf77190
Caught exception in looking up space point ptr... 0xd79c830, 0x39b219a0
Caught exception in looking up space point ptr... 0x39b219a0, 0x3242e190
Caught exception in looking up space point ptr... 0x4b863570, 0x532536e0
Caught exception in looking up space point ptr... 0x532536e0, 0x4b863570
Caught exception in looking up space point ptr... 0x4b863570, 0x2be461e0
Caught exception in looking up space point ptr... 0x2be461e0, 0x4b863570
Caught exception in looking up space point ptr... 0x311a78f0, 0x311a7750
Caught exception in looking up space point ptr... 0xf805650, 0x311a78f0
Caught exception in looking up space point ptr... 0x11ebb9c0, 0x1439d7a0
Caught exception in looking up space point ptr... 0x1439d7a0, 0x4712bbe0
Caught exception in looking up space point ptr... 0x3780c920, 0x4d516f30
Caught exception in looking up space point ptr... 0x4d516f30, 0x3780c920
Caught exception in looking up space point ptr... 0x33e033a0, 0x4d35db40
Caught exception in looking up space point ptr... 0x4d35db40, 0x33e033a0
Caught exception in looking up space point ptr... 0x375f6480, 0x375f52d0
Caught exception in looking up space point ptr... 0x375f52d0, 0x375f5aa0
Caught exception in looking up space point ptr... 0x44da4e60, 0x29905b00
Caught exception in looking up space point ptr... 0x29905b00, 0x44da4e60
Caught exception in looking up space point ptr... 0x24318c80, 0x24317da0
Caught exception in looking up space point ptr... 0x24317da0, 0x24318c80
Caught exception in looking up space point ptr... 0x24318c80, 0xb3fd3d0
Caught exception in looking up space point ptr... 0xb3fd3d0, 0x24318c80
++++>>>> total num hits: 26329, num free: 25305
04-Feb-2026 00:02:06 GMT  Closed output file "np02vd_raw_run042422_0103_df-s03-d0_dw_0_20260203T161028_reco_stage0_20260204T000206_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         117.012       161.596       188.056       163.439       15.764         27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       7.4082e-05    0.000123222   0.000302169   0.000102883   5.53848e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                12.6201       18.2685       34.5681       14.9921       6.12047        27     
produce:triggerrawdecoder:PDVDTriggerReader4      0.484835      0.591463       1.50603       0.53423       0.18855        27     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0163025     0.0299901     0.132325      0.017043      0.0294179       27     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0808157     0.0882896      0.1165       0.0828386    0.00891337       27     
produce:beamevent:BeamEvent                      8.7873e-05    0.000171458   0.00141817    0.000110094   0.000247256      27     
produce:pdvddaphne:DAPHNEReaderPDVD                3.99536       5.80451       15.1222       5.2989        2.03514        27     
produce:ophit:OpHitFinder                         0.0382056     0.0580898     0.0711656     0.0588749    0.00796261       27     
produce:wclsdatavd:WireCellToolkit                 79.9022       105.944       114.192       109.923       11.0126        27     
produce:gaushit:GausHitFinder                     0.614038       1.5942        2.38486       1.59683      0.455846        27     
produce:nhitsfilter:NumberOfHitsFilter           0.000234807   0.00048976    0.00101988    0.000476524   0.000180107      27     
produce:reco3d:SpacePointSolver                    6.07573       16.5491       27.9759       14.8268       5.82908        27     
produce:hitpdune:DisambigFromSpacePoints          0.0683509     0.172258       0.29453      0.162343      0.0565577       27     
produce:cluster3d:Cluster3D                        1.18498       3.8321        7.34006       3.23465       1.63317        27     
[art]:TriggerResults:TriggerResultInserter        2.028e-05    4.27228e-05   0.000202986   3.3351e-05    3.67903e-05      27     
end_path:out1:RootOutput                          4.03e-06     1.27263e-05   9.2173e-05     9.85e-06     1.64933e-05      27     
end_path:out1:RootOutput(write)                    5.44595       8.63456       10.6067       8.83137       1.08867        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4526.85 MB
  Peak resident set size usage (VmHWM): 2620.27 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042422_0103_df-s03-d0_dw_0_20260203T161028_reco_stage0_20260204T000206_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042422_0103_df-s03-d0_dw_0_20260203T161028_reco_stage0_20260204T000206_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042422_0103_df-s03-d0_dw_0_20260203T161028_reco_stage0_20260204T000206_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770158785.0,
    "core.end_time": 1770163329.0,
    "core.events": [
      16700,
      16706,
      16712,
      16718,
      16724,
      16730,
      16736,
      16742,
      16748,
      16754,
      16760,
      16766,
      16772,
      16778,
      16784,
      16790,
      16796,
      16802,
      16808,
      16814,
      16820,
      16826,
      16832,
      16838,
      16844,
      16850,
      16856
    ],
    "core.event_count": 27,
    "core.first_event_number": 16700,
    "core.last_event_number": 16856,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42422
    ],
    "core.runs_subruns": [
      4242200001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042422_0103_df-s03-d0_dw_0_20260203T161028.hdf5"
    }
  ]
}
justIN time: 2026-02-04 05:59:14 UTC       justIN version: 01.06.00