justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297919.92@dunegpschedd02.fnal.gov

Jobsub ID297919.92@dunegpschedd02.fnal.gov
Workflow ID12717
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:54:25
SiteUK_Imperial
EntryDUNE_T2_UK_London_IC_ceprod02
Last heartbeat2026-02-04 00:47:30
From worker nodeHostnamewk00.grid.hep.ph.ic.ac.uk
cpuinfoIntel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:30:40
Input filesvd-protodune:np02vd_raw_run042421_0064_df-s02-d0_dw_0_20260203T140538.hdf5
JobscriptExit code0
Real time1h (4453s)
CPU time1h (4015s = 90%)
Max RSS bytes2609250304 (2488 MiB)
Outputting started2026-02-04 00:44:56
Output files
Finished2026-02-04 00:47:30
Saved logsjustin-logs:297919.92-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

a0
Caught exception in looking up space point ptr... 0x247fc7a0, 0x247fca00
Caught exception in looking up space point ptr... 0xc56ded0, 0x2cdb9de0
Caught exception in looking up space point ptr... 0x2cdb9de0, 0x556a2f40
Caught exception in looking up space point ptr... 0x5605b3a0, 0x5605b0f0
Caught exception in looking up space point ptr... 0x5605b0f0, 0x5605af70
Caught exception in looking up space point ptr... 0x302f8500, 0x302f8690
Caught exception in looking up space point ptr... 0x302f8690, 0x302f8370
Caught exception in looking up space point ptr... 0x357e4f00, 0x244614c0
Caught exception in looking up space point ptr... 0x2ab3f9c0, 0x357e4f00
Caught exception in looking up space point ptr... 0x4ff8e720, 0xe333ae0
Caught exception in looking up space point ptr... 0x15957e80, 0x4ff8e720
Caught exception in looking up space point ptr... 0x60a18760, 0x4bb2cb40
Caught exception in looking up space point ptr... 0x4bb2cb40, 0xa0a0ef0
Caught exception in looking up space point ptr... 0x463b5980, 0x463b52a0
Caught exception in looking up space point ptr... 0x463b52a0, 0x463b5980
Caught exception in looking up space point ptr... 0x463b5980, 0x463b5610
Caught exception in looking up space point ptr... 0x463b5610, 0x463b5980
Caught exception in looking up space point ptr... 0x463b5980, 0x463b57f0
Caught exception in looking up space point ptr... 0x463b57f0, 0x463b5980
Caught exception in looking up space point ptr... 0x514aacc0, 0x514aab30
Caught exception in looking up space point ptr... 0x514aab30, 0x514aa9a0
Caught exception in looking up space point ptr... 0x3c8893a0, 0xfe27da0
Caught exception in looking up space point ptr... 0x1460f6e0, 0x3c8893a0
Caught exception in looking up space point ptr... 0x3c808070, 0x253e6240
Caught exception in looking up space point ptr... 0x253e6240, 0x4f746f00
Caught exception in looking up space point ptr... 0x4f746f00, 0x3c8087e0
Caught exception in looking up space point ptr... 0x109a9680, 0x109a94f0
Caught exception in looking up space point ptr... 0x109a94f0, 0x109a9130
Caught exception in looking up space point ptr... 0x109a9130, 0x109abcd0
Caught exception in looking up space point ptr... 0x109abcd0, 0x109abff0
Caught exception in looking up space point ptr... 0x109ac550, 0x109a9680
Caught exception in looking up space point ptr... 0x5bc05750, 0x5bc05360
Caught exception in looking up space point ptr... 0x5bc05360, 0x5bc05750
Caught exception in looking up space point ptr... 0x46cb1820, 0x4bb2d020
Caught exception in looking up space point ptr... 0x49ce23c0, 0x46cb1820
Caught exception in looking up space point ptr... 0x5156e180, 0x5156e4a0
Caught exception in looking up space point ptr... 0x5156e4a0, 0x5156e310
Caught exception in looking up space point ptr... 0xc56e5e0, 0x3b8760b0
Caught exception in looking up space point ptr... 0x3b8760b0, 0x54e0ef60
Caught exception in looking up space point ptr... 0x455cd7c0, 0x455cd5c0
Caught exception in looking up space point ptr... 0x455cd5c0, 0x455cd7c0
Caught exception in looking up space point ptr... 0x400b8a80, 0x400b8820
Caught exception in looking up space point ptr... 0x400b8820, 0x400b8a80
Caught exception in looking up space point ptr... 0x56055010, 0x56054f20
Caught exception in looking up space point ptr... 0x56054f20, 0x56053860
Caught exception in looking up space point ptr... 0x47282890, 0x11d58e50
Caught exception in looking up space point ptr... 0x11d58e50, 0x47282890
Caught exception in looking up space point ptr... 0x1596a4a0, 0x49275b40
Caught exception in looking up space point ptr... 0x49275b40, 0x1596a4a0
Caught exception in looking up space point ptr... 0x2dc52e80, 0x801ae10
Caught exception in looking up space point ptr... 0x801ae10, 0x2dc52e80
Caught exception in looking up space point ptr... 0x419fe430, 0x419fe110
Caught exception in looking up space point ptr... 0x419fe110, 0x146ed5d0
Caught exception in looking up space point ptr... 0x3ee51650, 0x3ee514c0
Caught exception in looking up space point ptr... 0x29258180, 0x3ee51650
Caught exception in looking up space point ptr... 0x43762c20, 0x43762530
Caught exception in looking up space point ptr... 0x43762530, 0x43762780
Caught exception in looking up space point ptr... 0xa42aa50, 0x3c3ce880
Caught exception in looking up space point ptr... 0x3c3ce880, 0xa42aa50
Caught exception in looking up space point ptr... 0x2762d920, 0x3dc3f080
Caught exception in looking up space point ptr... 0x3dc3f080, 0x2762d920
Caught exception in looking up space point ptr... 0x463b0470, 0x3d0a4fd0
Caught exception in looking up space point ptr... 0x3d0a4fd0, 0x463b0470
Caught exception in looking up space point ptr... 0x14b79130, 0x49ff9020
Caught exception in looking up space point ptr... 0x49ff9020, 0x14b79130
++++>>>> total num hits: 47741, num free: 45529
04-Feb-2026 00:44:40 GMT  Closed output file "np02vd_raw_run042421_0064_df-s02-d0_dw_0_20260203T140538_reco_stage0_20260204T004440_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         125.749       160.497       190.57        168.138       22.8454        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       6.3809e-05    0.000115293   0.000277514   9.6721e-05    5.4702e-05       27     
produce:tpcrawdecoder:PDVDTPCReader                12.3375       13.2309       14.4398       13.327        0.55127        27     
produce:triggerrawdecoder:PDVDTriggerReader4      0.483689      0.513492      0.713351      0.500615      0.0455479       27     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0162448     0.0183412     0.0662474     0.0164735    0.00939701       27     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0804097     0.0902396     0.148046      0.0816952     0.0137999       27     
produce:beamevent:BeamEvent                      9.5434e-05    0.000191865   0.00130711    0.000138572   0.000224242      27     
produce:pdvddaphne:DAPHNEReaderPDVD                5.64097       7.37456       10.8613       7.35901       1.03821        27     
produce:ophit:OpHitFinder                         0.0343347     0.0611285     0.0776793     0.0657913     0.0142522       27     
produce:wclsdatavd:WireCellToolkit                 82.0904       111.855       136.132       122.269       19.2254        27     
produce:gaushit:GausHitFinder                     0.958911       1.67541       2.56576       1.4936       0.545521        27     
produce:nhitsfilter:NumberOfHitsFilter           0.000281124   0.000485787   0.000842921   0.000475254   0.000117074      27     
produce:reco3d:SpacePointSolver                    5.48965       13.6504       23.7567       13.4371       3.70358        27     
produce:hitpdune:DisambigFromSpacePoints          0.0885809     0.189015      0.346889      0.153316      0.0743976       27     
produce:cluster3d:Cluster3D                       0.846571       3.27516       6.36323       2.87207       1.34371        27     
[art]:TriggerResults:TriggerResultInserter        2.515e-05    4.18768e-05   0.000120296    3.92e-05     1.82267e-05      27     
end_path:out1:RootOutput                          4.081e-06    1.12224e-05   3.1186e-05     9.46e-06     5.5784e-06       27     
end_path:out1:RootOutput(write)                    6.31481       8.53625       10.6269       8.80906       1.39331        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4509.66 MB
  Peak resident set size usage (VmHWM): 2609.25 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042421_0064_df-s02-d0_dw_0_20260203T140538_reco_stage0_20260204T004440_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042421_0064_df-s02-d0_dw_0_20260203T140538_reco_stage0_20260204T004440_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042421_0064_df-s02-d0_dw_0_20260203T140538_reco_stage0_20260204T004440_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770161455.0,
    "core.end_time": 1770165884.0,
    "core.events": [
      10428,
      10434,
      10440,
      10446,
      10452,
      10458,
      10464,
      10470,
      10476,
      10482,
      10488,
      10494,
      10500,
      10506,
      10512,
      10518,
      10524,
      10530,
      10536,
      10542,
      10548,
      10554,
      10560,
      10566,
      10572,
      10578,
      10584
    ],
    "core.event_count": 27,
    "core.first_event_number": 10428,
    "core.last_event_number": 10584,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42421
    ],
    "core.runs_subruns": [
      4242100001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042421_0064_df-s02-d0_dw_0_20260203T140538.hdf5"
    }
  ]
}
justIN time: 2026-02-04 04:28:23 UTC       justIN version: 01.06.00