justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303650.177@dunegpschedd01.fnal.gov

Jobsub ID303650.177@dunegpschedd01.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:48:28
SiteUK_Imperial
EntryDUNE_T2_UK_London_IC_ceprod01
Last heartbeat2026-02-04 00:49:58
From worker nodeHostnamewk13.grid.hep.ph.ic.ac.uk
cpuinfoIntel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:16:14
Input filesvd-protodune:np02vd_raw_run042433_0349_df-s02-d1_dw_0_20260203T205445.hdf5
JobscriptExit code0
Real time1h (5458s)
CPU time1h (5083s = 93%)
Max RSS bytes3431874560 (3272 MiB)
Outputting started2026-02-04 00:47:16
Output files
Finished2026-02-04 00:49:58
Saved logsjustin-logs:303650.177-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

... 0x14e4af40, 0x14e463c0
Caught exception in looking up space point ptr... 0x1b509ba0, 0x1b4f8db0
Caught exception in looking up space point ptr... 0x1b4f8db0, 0x1b509ba0
Caught exception in looking up space point ptr... 0x704edf00, 0x47b47900
Caught exception in looking up space point ptr... 0x47b47900, 0x704edf00
Caught exception in looking up space point ptr... 0x63c15310, 0x63c14e10
Caught exception in looking up space point ptr... 0x63c14e10, 0x63c15310
Caught exception in looking up space point ptr... 0x4a3c9f50, 0x4a3d3a60
Caught exception in looking up space point ptr... 0x4a3d3a60, 0x4a3ca400
Caught exception in looking up space point ptr... 0x471d0e80, 0x61fbb470
Caught exception in looking up space point ptr... 0x61fbb470, 0x3127eb10
Caught exception in looking up space point ptr... 0x466585e0, 0x4ac42370
Caught exception in looking up space point ptr... 0x4ac42370, 0x4ac42270
Caught exception in looking up space point ptr... 0x452d4bc0, 0x378ad630
Caught exception in looking up space point ptr... 0x378ad630, 0x452d4bc0
Caught exception in looking up space point ptr... 0x49dcb4f0, 0x49dcb1f0
Caught exception in looking up space point ptr... 0x49dcb1f0, 0x49dcb4f0
Caught exception in looking up space point ptr... 0x3f4d0460, 0x57db50a0
Caught exception in looking up space point ptr... 0x17121e80, 0x3f4d0460
Caught exception in looking up space point ptr... 0x329c38a0, 0x4baa94c0
Caught exception in looking up space point ptr... 0x4baa94c0, 0x2e50ebc0
Caught exception in looking up space point ptr... 0x6a4ba640, 0x30c81980
Caught exception in looking up space point ptr... 0x30c81980, 0x6a4ba640
Caught exception in looking up space point ptr... 0x5a986c30, 0xb9b1c90
Caught exception in looking up space point ptr... 0xb9b1c90, 0x5a986c30
Caught exception in looking up space point ptr... 0x5a986c30, 0x17120d10
Caught exception in looking up space point ptr... 0x17120d10, 0x5a986c30
Caught exception in looking up space point ptr... 0x624f7590, 0x15365bb0
Caught exception in looking up space point ptr... 0x15365bb0, 0x624f7590
Caught exception in looking up space point ptr... 0x40c43e50, 0x40c43ef0
Caught exception in looking up space point ptr... 0x40c43ef0, 0x40c43ba0
Caught exception in looking up space point ptr... 0x617935b0, 0x61793290
Caught exception in looking up space point ptr... 0x61793290, 0x617935b0
Caught exception in looking up space point ptr... 0x39d3c530, 0x620c5cc0
Caught exception in looking up space point ptr... 0x620c5cc0, 0x39d3c530
Caught exception in looking up space point ptr... 0x49df4ed0, 0x49df4a10
Caught exception in looking up space point ptr... 0x49df4a10, 0x49df4ed0
Caught exception in looking up space point ptr... 0x18a19980, 0x5fb68e60
Caught exception in looking up space point ptr... 0x5fb68e60, 0x18a19980
Caught exception in looking up space point ptr... 0x2f4a7b70, 0x2f4a97b0
Caught exception in looking up space point ptr... 0x2f4a97b0, 0x2f4a7b70
Caught exception in looking up space point ptr... 0x7b6f9d10, 0x7b725150
Caught exception in looking up space point ptr... 0x7b725150, 0x7b6f9e10
Caught exception in looking up space point ptr... 0x1907c940, 0x1907c7b0
Caught exception in looking up space point ptr... 0x1907c7b0, 0x2a57dc10
Caught exception in looking up space point ptr... 0x113522e0, 0x431bead0
Caught exception in looking up space point ptr... 0x431bead0, 0x113522e0
Caught exception in looking up space point ptr... 0x30cc84f0, 0x30cc8640
Caught exception in looking up space point ptr... 0x30cc8640, 0x30cc8250
Caught exception in looking up space point ptr... 0x3b75a940, 0x3b75a7f0
Caught exception in looking up space point ptr... 0x3b75a7f0, 0x3b75a940
Caught exception in looking up space point ptr... 0x7152d950, 0x599845c0
Caught exception in looking up space point ptr... 0x599845c0, 0x7152d950
Caught exception in looking up space point ptr... 0x7152d950, 0x89a14dc0
Caught exception in looking up space point ptr... 0x89a14dc0, 0x7152d950
Caught exception in looking up space point ptr... 0x7152d950, 0x12047890
Caught exception in looking up space point ptr... 0x12047890, 0x7152d950
Caught exception in looking up space point ptr... 0x3b5cffe0, 0x3b5cb0e0
Caught exception in looking up space point ptr... 0x3b5cb0e0, 0x3b5cffe0
Caught exception in looking up space point ptr... 0x11b0a8f0, 0x29d2e660
Caught exception in looking up space point ptr... 0x29d2e660, 0x11b0a8f0
Caught exception in looking up space point ptr... 0x49dde8e0, 0x49dde640
Caught exception in looking up space point ptr... 0x49dde640, 0x49dde8e0
Caught exception in looking up space point ptr... 0x63c01700, 0x63c01320
Caught exception in looking up space point ptr... 0x63c01320, 0x63c01700
Caught exception in looking up space point ptr... 0x13282820, 0x1328f790
Caught exception in looking up space point ptr... 0x1328f790, 0x13282820
++++>>>> total num hits: 94957, num free: 91585
04-Feb-2026 00:47:00 GMT  Closed output file "np02vd_raw_run042433_0349_df-s02-d1_dw_0_20260203T205445_reco_stage0_20260204T004700_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         195.057       281.301       365.574       288.387       46.5862        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       9.3041e-05    0.000153713   0.000250048   0.000134227   5.06982e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                16.3406       17.3292       18.0954       17.2518      0.527158        19     
produce:triggerrawdecoder:PDVDTriggerReader4       0.53327      0.582194      0.663297      0.573679      0.0354486       19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0165612     0.0184474     0.0291337     0.0168553    0.00356098       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0165392     0.0167701     0.0174488     0.0167001    0.000237805      19     
produce:beamevent:BeamEvent                      0.000130222   0.000254952   0.00124274    0.000197609   0.000238542      19     
produce:pdvddaphne:DAPHNEReaderPDVD                5.87649       7.6527        9.1333        7.68403      0.865869        19     
produce:ophit:OpHitFinder                         0.0358593     0.0667029     0.0840695     0.0707493     0.014919        19     
produce:wclsdatavd:WireCellToolkit                 145.755       204.43        250.537       215.867       36.9299        19     
produce:gaushit:GausHitFinder                      1.47834       2.85097       4.70015       2.77417      0.835362        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000488333   0.000803155   0.00179889    0.000777267   0.000284724      19     
produce:reco3d:SpacePointSolver                    13.979        30.7685       61.3036       27.6504       12.725         19     
produce:hitpdune:DisambigFromSpacePoints          0.192599       0.45432      0.866778      0.455957      0.190453        19     
produce:cluster3d:Cluster3D                        2.18355       5.19076       9.48355       4.7917        2.14621        19     
[art]:TriggerResults:TriggerResultInserter       3.0922e-05    4.58426e-05   0.000143729   4.0141e-05    2.42818e-05      19     
end_path:out1:RootOutput                          8.272e-06    1.23865e-05   4.1651e-05    1.0881e-05    7.05472e-06      19     
end_path:out1:RootOutput(write)                    8.17712       11.886        19.9205       11.2802       2.91458        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5333.32 MB
  Peak resident set size usage (VmHWM): 3431.87 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0349_df-s02-d1_dw_0_20260203T205445_reco_stage0_20260204T004700_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0349_df-s02-d1_dw_0_20260203T205445_reco_stage0_20260204T004700_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0349_df-s02-d1_dw_0_20260203T205445_reco_stage0_20260204T004700_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770160588.0,
    "core.end_time": 1770166024.0,
    "core.events": [
      39793,
      39799,
      39805,
      39811,
      39817,
      39823,
      39829,
      39835,
      39841,
      39847,
      39853,
      39859,
      39865,
      39871,
      39877,
      39883,
      39889,
      39895,
      39901
    ],
    "core.event_count": 19,
    "core.first_event_number": 39793,
    "core.last_event_number": 39901,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0349_df-s02-d1_dw_0_20260203T205445.hdf5"
    }
  ]
}
justIN time: 2026-02-04 04:28:39 UTC       justIN version: 01.06.00