justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303656.30@dunegpschedd01.fnal.gov

Jobsub ID303656.30@dunegpschedd01.fnal.gov
Workflow ID12717
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:56:25
SiteUK_Imperial
EntryDUNE_T2_UK_London_IC_ceprod03
Last heartbeat2026-02-04 01:46:46
From worker nodeHostnamewl02.grid.hep.ph.ic.ac.uk
cpuinfoAMD EPYC 7742 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-04 00:11:36
Input filesvd-protodune:np02vd_raw_run042421_0128_df-s04-d1_dw_0_20260203T144014.hdf5
JobscriptExit code0
Real time1h (5525s)
CPU time1h (4884s = 88%)
Max RSS bytes6284734464 (5993 MiB)
Outputting started2026-02-04 01:43:44
Output files
Finished2026-02-04 01:46:46
Saved logsjustin-logs:303656.30-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

aught exception in looking up space point ptr... 0x220b1ef0, 0x36f450f0
Caught exception in looking up space point ptr... 0x36f450f0, 0x54737fe0
Caught exception in looking up space point ptr... 0x5094a6e0, 0x66608570
Caught exception in looking up space point ptr... 0x66608570, 0x5094a6e0
Caught exception in looking up space point ptr... 0x164ef2b0, 0x10785fc0
Caught exception in looking up space point ptr... 0x10785fc0, 0x33150ea0
Caught exception in looking up space point ptr... 0x230d3120, 0x509cf6c0
Caught exception in looking up space point ptr... 0x509cf6c0, 0x653d8dc0
Caught exception in looking up space point ptr... 0x3fd8a9b0, 0x35240e90
Caught exception in looking up space point ptr... 0x35240e90, 0x4a2d5110
Caught exception in looking up space point ptr... 0x7116d4f0, 0x433ace20
Caught exception in looking up space point ptr... 0x433ace20, 0x7116d4f0
Caught exception in looking up space point ptr... 0x306babe0, 0x10fab2a0
Caught exception in looking up space point ptr... 0x67072150, 0x306babe0
Caught exception in looking up space point ptr... 0x52f63640, 0x31468f80
Caught exception in looking up space point ptr... 0x31468f80, 0x56b270d0
Caught exception in looking up space point ptr... 0x33ae3f80, 0x525bc520
Caught exception in looking up space point ptr... 0x67094ee0, 0x33ae3f80
Caught exception in looking up space point ptr... 0x4fe58210, 0xd050e60
Caught exception in looking up space point ptr... 0xd050e60, 0x76dbf3d0
Caught exception in looking up space point ptr... 0x29e55f70, 0x66ffcfc0
Caught exception in looking up space point ptr... 0x66ffcfc0, 0x29e55f70
Caught exception in looking up space point ptr... 0x29e55f70, 0x2a5374f0
Caught exception in looking up space point ptr... 0x2a5374f0, 0x29e55f70
Caught exception in looking up space point ptr... 0x22875550, 0x45768b30
Caught exception in looking up space point ptr... 0x45768b30, 0x22875550
Caught exception in looking up space point ptr... 0x53593560, 0x33b101b0
Caught exception in looking up space point ptr... 0x3dd15cb0, 0x53593560
Caught exception in looking up space point ptr... 0x57ea7dc0, 0x3dd14600
Caught exception in looking up space point ptr... 0x3dd14600, 0x4a291110
Caught exception in looking up space point ptr... 0x4a290f00, 0x57ea7dc0
Caught exception in looking up space point ptr... 0x3524a610, 0x3f5f0ba0
Caught exception in looking up space point ptr... 0x66ea36c0, 0x3524a610
Caught exception in looking up space point ptr... 0x2d327f60, 0x24ca6170
Caught exception in looking up space point ptr... 0x24ca6170, 0x2d327f60
Caught exception in looking up space point ptr... 0x512226c0, 0x653840f0
Caught exception in looking up space point ptr... 0x653840f0, 0x512226c0
Caught exception in looking up space point ptr... 0xd11a2750, 0x43508940
Caught exception in looking up space point ptr... 0x43508940, 0xd11a2750
Caught exception in looking up space point ptr... 0x66e9a100, 0x5836f800
Caught exception in looking up space point ptr... 0x5836f800, 0x4333e740
Caught exception in looking up space point ptr... 0x56b2fcf0, 0x56b2fb30
Caught exception in looking up space point ptr... 0x56b2fb30, 0x66d277e0
Caught exception in looking up space point ptr... 0x31308fd0, 0x293c6170
Caught exception in looking up space point ptr... 0x293c6170, 0x31308fd0
Caught exception in looking up space point ptr... 0x56228a30, 0x13fbc7f0
Caught exception in looking up space point ptr... 0x13fbc7f0, 0x56228a30
Caught exception in looking up space point ptr... 0x25efbda0, 0x25efbc10
Caught exception in looking up space point ptr... 0x25efbc10, 0x25efba80
Caught exception in looking up space point ptr... 0x3fa31f80, 0x664a6870
Caught exception in looking up space point ptr... 0x529a2ee0, 0x3fa31f80
Caught exception in looking up space point ptr... 0x7d020e30, 0x6710c820
Caught exception in looking up space point ptr... 0x6710c820, 0x7d020e30
Caught exception in looking up space point ptr... 0x66e1e8d0, 0x66f57870
Caught exception in looking up space point ptr... 0x66f57870, 0x3b56de00
Caught exception in looking up space point ptr... 0x670d8b90, 0x52b3b980
Caught exception in looking up space point ptr... 0x52b3b980, 0x670d8b90
Caught exception in looking up space point ptr... 0x4a2d7330, 0x33ae3e90
Caught exception in looking up space point ptr... 0x33ae3e90, 0x510a10b0
Caught exception in looking up space point ptr... 0x9830ee0, 0x9830cc0
Caught exception in looking up space point ptr... 0x9830cc0, 0x5152fe80
Caught exception in looking up space point ptr... 0x43af1060, 0x487c31d0
Caught exception in looking up space point ptr... 0x487c31d0, 0x640a4e30
Caught exception in looking up space point ptr... 0x670734e0, 0x6533a9e0
Caught exception in looking up space point ptr... 0x6533a9e0, 0x670734e0
++++>>>> total num hits: 39655, num free: 37645
04-Feb-2026 01:43:30 GMT  Closed output file "np02vd_raw_run042421_0128_df-s04-d1_dw_0_20260203T144014_reco_stage0_20260204T014330_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         104.732       200.944       785.547       139.55        150.726        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       6.8813e-05    0.00011002    0.000247278   0.000102723   3.66343e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                10.8129       11.5678       12.388        11.4699       0.47404        27     
produce:triggerrawdecoder:PDVDTriggerReader4      0.483505       0.50877      0.535195      0.510395      0.0130065       27     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0162797     0.016678      0.0169415     0.0167072    0.000207624      27     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.080243      0.0829379     0.0994437     0.0819273     0.0042951       27     
produce:beamevent:BeamEvent                      9.9344e-05    0.00018913    0.00149807    0.000123184   0.000261076      27     
produce:pdvddaphne:DAPHNEReaderPDVD                3.48363       4.42499       6.75721       4.34283      0.642832        27     
produce:ophit:OpHitFinder                         0.0353734     0.0500077     0.0660889     0.0503906    0.00767502       27     
produce:wclsdatavd:WireCellToolkit                 75.0582       94.6483       117.352       92.197        11.9604        27     
produce:gaushit:GausHitFinder                     0.644158       2.24318       6.74312       1.62986       1.67403        27     
produce:nhitsfilter:NumberOfHitsFilter           0.000218877   0.00054587    0.00153384    0.000440114   0.000297032      27     
produce:reco3d:SpacePointSolver                    5.78514       20.7548       91.5543        14.57        17.4648        27     
produce:hitpdune:DisambigFromSpacePoints          0.0730187     0.701096       5.15373      0.185219       1.22093        27     
produce:cluster3d:Cluster3D                        1.19686       56.7822       528.447       3.26074       123.145        27     
[art]:TriggerResults:TriggerResultInserter       2.1411e-05    3.47486e-05   0.000126124   3.0401e-05    1.97493e-05      27     
end_path:out1:RootOutput                          3.351e-06    7.38556e-06   3.9882e-05     5.36e-06     6.8076e-06       27     
end_path:out1:RootOutput(write)                    5.82635       9.12598       19.162        7.97977       3.28175        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8658.53 MB
  Peak resident set size usage (VmHWM): 6284.73 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042421_0128_df-s04-d1_dw_0_20260203T144014_reco_stage0_20260204T014330_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042421_0128_df-s04-d1_dw_0_20260203T144014_reco_stage0_20260204T014330_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042421_0128_df-s04-d1_dw_0_20260203T144014_reco_stage0_20260204T014330_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770163910.0,
    "core.end_time": 1770169413.0,
    "core.events": [
      20807,
      20813,
      20819,
      20825,
      20831,
      20837,
      20843,
      20849,
      20855,
      20861,
      20867,
      20873,
      20879,
      20885,
      20891,
      20897,
      20903,
      20909,
      20915,
      20921,
      20927,
      20933,
      20939,
      20945,
      20951,
      20957,
      20963
    ],
    "core.event_count": 27,
    "core.first_event_number": 20807,
    "core.last_event_number": 20963,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42421
    ],
    "core.runs_subruns": [
      4242100001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042421_0128_df-s04-d1_dw_0_20260203T144014.hdf5"
    }
  ]
}
justIN time: 2026-02-04 04:31:38 UTC       justIN version: 01.06.00