justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303668.2@dunegpschedd01.fnal.gov

Jobsub ID303668.2@dunegpschedd01.fnal.gov
Workflow ID12718
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 23:28:25
SiteUK_QMUL
EntryDUNE_UK_London_QMUL_arcce02
Last heartbeat2026-02-04 02:57:14
From worker nodeHostnamecn519.htc.esc.qmul
cpuinfoIntel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-04 00:18:01
Input filesvd-protodune:np02vd_raw_run042422_0109_df-s04-d0_dw_0_20260203T161343.hdf5
JobscriptExit code0
Real time2h (9354s)
CPU time1h (6135s = 65%)
Max RSS bytes2698792960 (2573 MiB)
Outputting started2026-02-04 02:54:06
Output files
Finished2026-02-04 02:57:14
Saved logsjustin-logs:303668.2-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

Caught exception in looking up space point ptr... 0x504ffef0, 0x47a62060
Caught exception in looking up space point ptr... 0x47a62060, 0x46f06c70
Caught exception in looking up space point ptr... 0x46f06c70, 0x5f3caaf0
Caught exception in looking up space point ptr... 0x5f3caaf0, 0x45ea0c60
Caught exception in looking up space point ptr... 0x116436a0, 0x36f43f60
Caught exception in looking up space point ptr... 0x36f43f60, 0x5506f050
Caught exception in looking up space point ptr... 0x5506f050, 0x30d7a870
Caught exception in looking up space point ptr... 0x30d7a870, 0x13013290
Caught exception in looking up space point ptr... 0x13013290, 0x2a0e2270
Caught exception in looking up space point ptr... 0x530b8cf0, 0x914b340
Caught exception in looking up space point ptr... 0x914b340, 0x530b8cf0
Caught exception in looking up space point ptr... 0x530b8cf0, 0x91253a0
Caught exception in looking up space point ptr... 0x91253a0, 0x530b8cf0
Caught exception in looking up space point ptr... 0x530b8cf0, 0x52d66820
Caught exception in looking up space point ptr... 0x52d66820, 0x530b8cf0
Caught exception in looking up space point ptr... 0x11077d10, 0x9f495d0
Caught exception in looking up space point ptr... 0x9f495d0, 0x11077d10
Caught exception in looking up space point ptr... 0x29cb2c60, 0x29cb5540
Caught exception in looking up space point ptr... 0x29cb5540, 0x29cb52b0
Caught exception in looking up space point ptr... 0x78e94a0, 0x1aaeb440
Caught exception in looking up space point ptr... 0x1aaeb440, 0x2dbe53b0
Caught exception in looking up space point ptr... 0x55867db0, 0x37ec97e0
Caught exception in looking up space point ptr... 0x37ec97e0, 0x40e4d160
Caught exception in looking up space point ptr... 0x40e4d160, 0x3d373e80
Caught exception in looking up space point ptr... 0x1ab98c00, 0x55867db0
Caught exception in looking up space point ptr... 0x2e7f9f20, 0x2e7f9c50
Caught exception in looking up space point ptr... 0x2e7f9c50, 0x2e7ece40
Caught exception in looking up space point ptr... 0x53f344a0, 0x33e0f8c0
Caught exception in looking up space point ptr... 0x33e0f8c0, 0x53f344a0
Caught exception in looking up space point ptr... 0x35e04900, 0x2e4ec720
Caught exception in looking up space point ptr... 0x2e4ec720, 0x5577fc10
Caught exception in looking up space point ptr... 0x112f3500, 0x11305ac0
Caught exception in looking up space point ptr... 0x11305ac0, 0x11305c50
Caught exception in looking up space point ptr... 0x19cd1a80, 0x53379400
Caught exception in looking up space point ptr... 0x19cdd390, 0x19cd1a80
Caught exception in looking up space point ptr... 0x541ad950, 0x29cbe130
Caught exception in looking up space point ptr... 0x29cbe130, 0x316eb290
Caught exception in looking up space point ptr... 0x2d8e6b10, 0x484f23f0
Caught exception in looking up space point ptr... 0x2d8d51d0, 0x2d8e6b10
Caught exception in looking up space point ptr... 0x29f7dfd0, 0x536e3be0
Caught exception in looking up space point ptr... 0x536e3be0, 0xaaf4950
Caught exception in looking up space point ptr... 0x47aa16f0, 0x47aa13d0
Caught exception in looking up space point ptr... 0x47aa13d0, 0x47aa10b0
Caught exception in looking up space point ptr... 0x363ac9f0, 0x363ba7d0
Caught exception in looking up space point ptr... 0x363ba640, 0x363ac9f0
Caught exception in looking up space point ptr... 0x648d06e0, 0x525a8250
Caught exception in looking up space point ptr... 0x525a8250, 0x51b05d60
Caught exception in looking up space point ptr... 0x317a52a0, 0x31792b00
Caught exception in looking up space point ptr... 0x19a1c690, 0x317a52a0
Caught exception in looking up space point ptr... 0x46ef33a0, 0x2e7ce3a0
Caught exception in looking up space point ptr... 0x2e7ce3a0, 0x3db11270
Caught exception in looking up space point ptr... 0x35d5bde0, 0x3f950a70
Caught exception in looking up space point ptr... 0x98208d0, 0x35d5bde0
Caught exception in looking up space point ptr... 0x4fd44310, 0x556e4280
Caught exception in looking up space point ptr... 0x4fd444a0, 0x4fd44310
Caught exception in looking up space point ptr... 0x612362e0, 0x61236150
Caught exception in looking up space point ptr... 0x61236150, 0x4705ead0
Caught exception in looking up space point ptr... 0x34645f60, 0x34633b30
Caught exception in looking up space point ptr... 0x160c3a50, 0x34645f60
Caught exception in looking up space point ptr... 0x4bc78420, 0x27c079f0
Caught exception in looking up space point ptr... 0x27c079f0, 0x4bc78290
Caught exception in looking up space point ptr... 0x493c9e10, 0x3a33fab0
Caught exception in looking up space point ptr... 0x493c9b80, 0x493c9e10
Caught exception in looking up space point ptr... 0x49c87a30, 0x5171b1f0
Caught exception in looking up space point ptr... 0x5171b1f0, 0x49c87a30
++++>>>> total num hits: 321645, num free: 316369
04-Feb-2026 02:53:42 GMT  Closed output file "np02vd_raw_run042422_0109_df-s04-d0_dw_0_20260203T161343_reco_stage0_20260204T025342_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                          233.2        342.003       429.676       347.623       47.8895        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       9.1931e-05    0.00015449    0.000374877   0.000129897   6.54554e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                80.004        110.855       152.72        104.449       18.4366        27     
produce:triggerrawdecoder:PDVDTriggerReader4      0.554405      0.707538      0.942285      0.694628      0.107672        27     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0165963     0.0250452     0.0877376     0.0198721     0.0147754       27     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0812552     0.0861455     0.107816      0.0825537    0.00742342       27     
produce:beamevent:BeamEvent                      0.000130533   0.000247579   0.00164567    0.000195713    0.0002772       27     
produce:pdvddaphne:DAPHNEReaderPDVD                14.0831       18.1775       22.1378       17.8349       1.89821        27     
produce:ophit:OpHitFinder                         0.0422447     0.0719279      0.10377      0.0769323     0.0153881       27     
produce:wclsdatavd:WireCellToolkit                 94.7888       136.973       159.012       147.019       21.8089        27     
produce:gaushit:GausHitFinder                      1.08994       2.45893       4.61166       2.28736       0.74113        27     
produce:nhitsfilter:NumberOfHitsFilter           0.000458136   0.000644555   0.00103612    0.000618608   0.000122785      27     
produce:reco3d:SpacePointSolver                    6.97524       11.3536       18.9141       10.3165       3.26488        27     
produce:hitpdune:DisambigFromSpacePoints          0.124744      0.405981       1.04349      0.400315      0.188881        27     
produce:cluster3d:Cluster3D                        1.36633       53.4536       93.104        57.2576       29.6343        27     
[art]:TriggerResults:TriggerResultInserter       2.9099e-05    5.04661e-05   0.000102782   4.5971e-05    1.76358e-05      27     
end_path:out1:RootOutput                          4.404e-06    1.17803e-05   2.2903e-05    1.0258e-05    5.20709e-06      27     
end_path:out1:RootOutput(write)                    5.3395        7.38872       9.65796       7.24988       1.41652        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4646.42 MB
  Peak resident set size usage (VmHWM): 2698.79 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042422_0109_df-s04-d0_dw_0_20260203T161343_reco_stage0_20260204T025342_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042422_0109_df-s04-d0_dw_0_20260203T161343_reco_stage0_20260204T025342_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042422_0109_df-s04-d0_dw_0_20260203T161343_reco_stage0_20260204T025342_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770164296.0,
    "core.end_time": 1770173629.0,
    "core.events": [
      17674,
      17680,
      17686,
      17692,
      17698,
      17704,
      17710,
      17716,
      17722,
      17728,
      17734,
      17740,
      17746,
      17752,
      17758,
      17764,
      17770,
      17776,
      17782,
      17788,
      17794,
      17800,
      17806,
      17812,
      17818,
      17824,
      17830
    ],
    "core.event_count": 27,
    "core.first_event_number": 17674,
    "core.last_event_number": 17830,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42422
    ],
    "core.runs_subruns": [
      4242200001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042422_0109_df-s04-d0_dw_0_20260203T161343.hdf5"
    }
  ]
}
justIN time: 2026-02-04 04:17:31 UTC       justIN version: 01.06.00