justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297923.117@dunegpschedd02.fnal.gov

Jobsub ID297923.117@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 23:00:26
SiteUK_QMUL
EntryDUNE_UK_London_QMUL_arcce03
Last heartbeat2026-02-04 01:38:50
From worker nodeHostnamecn541.htc.esc.qmul
cpuinfoIntel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-04 00:02:45
Input filesvd-protodune:np02vd_raw_run042433_0384_df-s04-d0_dw_0_20260203T210804.hdf5
JobscriptExit code0
Real time1h (5581s)
CPU time1h (5341s = 95%)
Max RSS bytes3556679680 (3391 MiB)
Outputting started2026-02-04 01:35:53
Output files
Finished2026-02-04 01:38:50
Saved logsjustin-logs:297923.117-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ce point ptr... 0x6642dcf0, 0x6642d980
Caught exception in looking up space point ptr... 0x54af9050, 0x769edcf0
Caught exception in looking up space point ptr... 0x769edcf0, 0x54af9050
Caught exception in looking up space point ptr... 0x28d457e0, 0x4c66f6d0
Caught exception in looking up space point ptr... 0x4c66f6d0, 0x28d457e0
Caught exception in looking up space point ptr... 0x55859070, 0x186e5b40
Caught exception in looking up space point ptr... 0x186e5b40, 0x10746a80
Caught exception in looking up space point ptr... 0x4ae1e3c0, 0x4ae1e500
Caught exception in looking up space point ptr... 0x4ae1e500, 0x4ae1e3c0
Caught exception in looking up space point ptr... 0x8469c120, 0x8469bc80
Caught exception in looking up space point ptr... 0x8469ba30, 0x8469c120
Caught exception in looking up space point ptr... 0x12355f30, 0x8b9c470
Caught exception in looking up space point ptr... 0x8b929d0, 0x12355f30
Caught exception in looking up space point ptr... 0x66aedc50, 0x4337c780
Caught exception in looking up space point ptr... 0x186c34d0, 0x66aedc50
Caught exception in looking up space point ptr... 0x36b63370, 0x36b52070
Caught exception in looking up space point ptr... 0x8b956b0, 0x36b63370
Caught exception in looking up space point ptr... 0x398cd230, 0x398ccd10
Caught exception in looking up space point ptr... 0x398cca30, 0x398cd230
Caught exception in looking up space point ptr... 0x846787f0, 0xe7ca040
Caught exception in looking up space point ptr... 0x186c81f0, 0x846787f0
Caught exception in looking up space point ptr... 0xe791350, 0x558524b0
Caught exception in looking up space point ptr... 0x55856fc0, 0xe791350
Caught exception in looking up space point ptr... 0xe78d680, 0x187173d0
Caught exception in looking up space point ptr... 0x65b98a80, 0xe78d680
Caught exception in looking up space point ptr... 0x4d2158a0, 0x37c5f0f0
Caught exception in looking up space point ptr... 0x8b93d00, 0x4d2158a0
Caught exception in looking up space point ptr... 0x398c9130, 0x398c8980
Caught exception in looking up space point ptr... 0x398c8c10, 0x398c9130
Caught exception in looking up space point ptr... 0x398c4cf0, 0x398c47d0
Caught exception in looking up space point ptr... 0x398c4540, 0x398c4cf0
Caught exception in looking up space point ptr... 0x36b511f0, 0x36b5fb00
Caught exception in looking up space point ptr... 0x36b82600, 0x36b511f0
Caught exception in looking up space point ptr... 0x36b6dd40, 0x4d215320
Caught exception in looking up space point ptr... 0x4d1daa00, 0x36b6dd40
Caught exception in looking up space point ptr... 0x362bfdf0, 0x362bf340
Caught exception in looking up space point ptr... 0x362bf340, 0x362c0700
Caught exception in looking up space point ptr... 0x8b95270, 0x8ba03e0
Caught exception in looking up space point ptr... 0x8ba0560, 0x8b95270
Caught exception in looking up space point ptr... 0xe7c8c70, 0x5580b840
Caught exception in looking up space point ptr... 0x8468ae20, 0xe7c8c70
Caught exception in looking up space point ptr... 0x398c3050, 0x398c2b30
Caught exception in looking up space point ptr... 0x398c28a0, 0x398c3050
Caught exception in looking up space point ptr... 0x36b51760, 0x36b51f10
Caught exception in looking up space point ptr... 0x12388df0, 0x36b51760
Caught exception in looking up space point ptr... 0x398cfd90, 0x398cf590
Caught exception in looking up space point ptr... 0x398cf870, 0x398cfd90
Caught exception in looking up space point ptr... 0x6a468440, 0x6a468870
Caught exception in looking up space point ptr... 0x6a468870, 0x6a468440
Caught exception in looking up space point ptr... 0x37be7750, 0x62236b10
Caught exception in looking up space point ptr... 0x4d1fdd60, 0x37be7750
Caught exception in looking up space point ptr... 0x5a1fe220, 0x5a1fdf40
Caught exception in looking up space point ptr... 0x5a1fdf40, 0x5a1fddf0
Caught exception in looking up space point ptr... 0x652a0d00, 0x28d69c30
Caught exception in looking up space point ptr... 0x28d69c30, 0x28d69b00
Caught exception in looking up space point ptr... 0x13bbd8a0, 0x13bbdb00
Caught exception in looking up space point ptr... 0x13bbdb00, 0x13bbd8a0
Caught exception in looking up space point ptr... 0x62797e00, 0x1127d550
Caught exception in looking up space point ptr... 0x1127d550, 0x62797e00
Caught exception in looking up space point ptr... 0x1727c660, 0x3b38be70
Caught exception in looking up space point ptr... 0x3b38be70, 0x1727c660
Caught exception in looking up space point ptr... 0x11615330, 0x7b3988d0
Caught exception in looking up space point ptr... 0x7b3988d0, 0x11615330
Caught exception in looking up space point ptr... 0x15d15260, 0x15d15060
Caught exception in looking up space point ptr... 0x15d15060, 0x15d14f60
Caught exception in looking up space point ptr... 0x157110f0, 0x157111f0
Caught exception in looking up space point ptr... 0x157111f0, 0x157110f0
++++>>>> total num hits: 478055, num free: 468056
04-Feb-2026 01:35:39 GMT  Closed output file "np02vd_raw_run042433_0384_df-s04-d0_dw_0_20260203T210804_reco_stage0_20260204T013539_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         199.709       288.176       348.905       287.899       39.7199        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       0.000102597   0.000167448   0.00030516    0.000132672   6.83125e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                16.4196       16.934        18.0256       16.8768      0.395614        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.490145      0.513301      0.583952      0.508599      0.0232683       19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0165967     0.0174516     0.0296783     0.0168119    0.00288501       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0166028     0.0168154     0.0174556     0.0167985    0.000193875      19     
produce:beamevent:BeamEvent                      0.000130447   0.000239754   0.00106879    0.000194232   0.00019889       19     
produce:pdvddaphne:DAPHNEReaderPDVD                5.53988       6.6403        7.90762       6.58476      0.736852        19     
produce:ophit:OpHitFinder                         0.0390083     0.0573121     0.0808138     0.0498181     0.0144926       19     
produce:wclsdatavd:WireCellToolkit                 148.838       176.376       228.768       171.779       21.9163        19     
produce:gaushit:GausHitFinder                      1.46839       3.08433       4.4491        2.98995      0.806278        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000521014   0.000706325   0.000933687   0.000731978   0.000142483      19     
produce:reco3d:SpacePointSolver                    12.7753       17.5825       26.1299       17.8314       4.09617        19     
produce:hitpdune:DisambigFromSpacePoints          0.185008      0.428038      0.765133      0.378345      0.146243        19     
produce:cluster3d:Cluster3D                        1.33914       58.8865       101.624       66.4405       31.4933        19     
[art]:TriggerResults:TriggerResultInserter       2.7163e-05    4.03983e-05   9.2817e-05    3.6572e-05    1.36106e-05      19     
end_path:out1:RootOutput                          5.099e-06    1.02632e-05   2.4111e-05     8.756e-06    5.1209e-06       19     
end_path:out1:RootOutput(write)                    5.51917       7.59015       13.1252       7.25815       1.87868        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5505.9 MB
  Peak resident set size usage (VmHWM): 3556.68 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0384_df-s04-d0_dw_0_20260203T210804_reco_stage0_20260204T013539_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0384_df-s04-d0_dw_0_20260203T210804_reco_stage0_20260204T013539_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0384_df-s04-d0_dw_0_20260203T210804_reco_stage0_20260204T013539_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770163379.0,
    "core.end_time": 1770168942.0,
    "core.events": [
      43786,
      43792,
      43798,
      43804,
      43810,
      43816,
      43822,
      43828,
      43834,
      43840,
      43846,
      43852,
      43858,
      43864,
      43870,
      43876,
      43882,
      43888,
      43894
    ],
    "core.event_count": 19,
    "core.first_event_number": 43786,
    "core.last_event_number": 43894,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0384_df-s04-d0_dw_0_20260203T210804.hdf5"
    }
  ]
}
justIN time: 2026-02-04 05:50:00 UTC       justIN version: 01.06.00