justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303652.79@dunegpschedd01.fnal.gov

Jobsub ID303652.79@dunegpschedd01.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:50:26
SiteUK_Imperial
EntryDUNE_T2_UK_London_IC_ceprod03
Last heartbeat2026-02-04 00:53:36
From worker nodeHostnamewm21.grid.hep.ph.ic.ac.uk
cpuinfoAMD EPYC 7742 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:28:54
Input filesvd-protodune:np02vd_raw_run042433_0562_df-s02-d0_dw_0_20260203T221543.hdf5
JobscriptExit code0
Real time1h (4904s)
CPU time1h (4381s = 89%)
Max RSS bytes3632934912 (3464 MiB)
Outputting started2026-02-04 00:50:42
Output files
Finished2026-02-04 00:53:36
Saved logsjustin-logs:303652.79-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

space point ptr... 0x62fa78a0, 0x62f97090
Caught exception in looking up space point ptr... 0x62fa7a30, 0x62fa78a0
Caught exception in looking up space point ptr... 0x62f97860, 0x62fa8070
Caught exception in looking up space point ptr... 0x62fa8070, 0x62fa8200
Caught exception in looking up space point ptr... 0xd232500, 0x18bba850
Caught exception in looking up space point ptr... 0x18bba850, 0x3b2a1720
Caught exception in looking up space point ptr... 0x3b2a1720, 0x3e9ecbd0
Caught exception in looking up space point ptr... 0x360a0cd0, 0x360a0b40
Caught exception in looking up space point ptr... 0x360a0b40, 0x360a09b0
Caught exception in looking up space point ptr... 0x3a5c9c60, 0x3a58a780
Caught exception in looking up space point ptr... 0x6648a580, 0x3a5c9c60
Caught exception in looking up space point ptr... 0x36157d70, 0x361494a0
Caught exception in looking up space point ptr... 0x36169cf0, 0x36157d70
Caught exception in looking up space point ptr... 0x6756af20, 0x6756b0b0
Caught exception in looking up space point ptr... 0xd289320, 0x6756af20
Caught exception in looking up space point ptr... 0x3fe01df0, 0x181feda0
Caught exception in looking up space point ptr... 0x3fe01f80, 0x3fe01df0
Caught exception in looking up space point ptr... 0x7a96e20, 0x358a1770
Caught exception in looking up space point ptr... 0x7a96fb0, 0x7a96e20
Caught exception in looking up space point ptr... 0x6e5f00a0, 0x3c9726d0
Caught exception in looking up space point ptr... 0x6e5f01e0, 0x6e5f00a0
Caught exception in looking up space point ptr... 0x139f11f0, 0x608f28d0
Caught exception in looking up space point ptr... 0x139f1380, 0x139f11f0
Caught exception in looking up space point ptr... 0x5ff65c00, 0x6d2ddf10
Caught exception in looking up space point ptr... 0x6d2ddf10, 0xd017480
Caught exception in looking up space point ptr... 0x6e2d0740, 0x6e2e2d00
Caught exception in looking up space point ptr... 0x6e2be180, 0x6e2d0740
Caught exception in looking up space point ptr... 0x746a14f0, 0xe802430
Caught exception in looking up space point ptr... 0xe802430, 0x74617680
Caught exception in looking up space point ptr... 0x6319e8d0, 0x63187dc0
Caught exception in looking up space point ptr... 0x63187dc0, 0x7469eff0
Caught exception in looking up space point ptr... 0xf984c10, 0xf9732d0
Caught exception in looking up space point ptr... 0xf992d10, 0xf984c10
Caught exception in looking up space point ptr... 0x85741bb0, 0x85733c40
Caught exception in looking up space point ptr... 0x85733c40, 0x85741a20
Caught exception in looking up space point ptr... 0x84d2f000, 0x84d1d080
Caught exception in looking up space point ptr... 0x163db6d0, 0x84d2f000
Caught exception in looking up space point ptr... 0x3fe62df0, 0x3fe509c0
Caught exception in looking up space point ptr... 0x3fe509c0, 0x3fe3e400
Caught exception in looking up space point ptr... 0x1842a160, 0x184181e0
Caught exception in looking up space point ptr... 0x1843bdc0, 0x1842a160
Caught exception in looking up space point ptr... 0x8632160, 0x8622bf0
Caught exception in looking up space point ptr... 0x863f0c0, 0x8632160
Caught exception in looking up space point ptr... 0x130914d0, 0x53205310
Caught exception in looking up space point ptr... 0x53205310, 0x19369220
Caught exception in looking up space point ptr... 0x38cb3610, 0x5951f680
Caught exception in looking up space point ptr... 0x5951f680, 0x6e1527e0
Caught exception in looking up space point ptr... 0x3bd68310, 0x619873d0
Caught exception in looking up space point ptr... 0x619873d0, 0x3bd68310
Caught exception in looking up space point ptr... 0x3bf331f0, 0xa792120
Caught exception in looking up space point ptr... 0xa792120, 0x5fc90860
Caught exception in looking up space point ptr... 0x4f5b3330, 0x5fff14f0
Caught exception in looking up space point ptr... 0x5fff14f0, 0x4f5b3330
Caught exception in looking up space point ptr... 0x4f5b3330, 0x3c8b0dd0
Caught exception in looking up space point ptr... 0x3c8b0dd0, 0x4f5b3330
Caught exception in looking up space point ptr... 0x5a367690, 0x86a7e530
Caught exception in looking up space point ptr... 0x533646f0, 0x5a367690
Caught exception in looking up space point ptr... 0x47e315c0, 0xa36db80
Caught exception in looking up space point ptr... 0xa36db80, 0x460e6ac0
Caught exception in looking up space point ptr... 0x42588210, 0x174f4890
Caught exception in looking up space point ptr... 0x174f4890, 0x40458240
Caught exception in looking up space point ptr... 0x39e7f5c0, 0x39e7f730
Caught exception in looking up space point ptr... 0x39e7f730, 0x77c04ef0
Caught exception in looking up space point ptr... 0x3cda48a0, 0x3cd5a720
Caught exception in looking up space point ptr... 0x3cd5a720, 0x3cda48a0
Caught exception in looking up space point ptr... 0x10dfd210, 0x6afcb080
Caught exception in looking up space point ptr... 0x6afcb080, 0x10dfd210
++++>>>> total num hits: 529301, num free: 519761
04-Feb-2026 00:50:29 GMT  Closed output file "np02vd_raw_run042433_0562_df-s02-d0_dw_0_20260203T221543_reco_stage0_20260204T005029_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         171.441       253.333       332.433       253.081       35.1138        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       6.5153e-05    0.000130276   0.000243819   0.000138731   5.07872e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                13.5751       16.8353       18.5021       16.8528       1.17838        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.479347      0.493017       0.52255      0.488886      0.0126742       19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0159849     0.0161488     0.016545      0.0161117    0.000142471      19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0159186     0.0161131     0.0164211     0.0160417    0.000157571      19     
produce:beamevent:BeamEvent                      6.4281e-05    0.000150441   0.000856875   0.000103334   0.000169445      19     
produce:pdvddaphne:DAPHNEReaderPDVD                3.32056       3.65237       5.11477       3.55446      0.428469        19     
produce:ophit:OpHitFinder                         0.0297449     0.039078      0.0532736     0.0387059    0.00576193       19     
produce:wclsdatavd:WireCellToolkit                 120.888       134.774       173.485       128.671       14.3026        19     
produce:gaushit:GausHitFinder                      1.3801        2.63267       3.75573       2.56471      0.570418        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000396478   0.000584965   0.000994475   0.000589631   0.000142828      19     
produce:reco3d:SpacePointSolver                    16.9542       25.887        36.9158       27.1763       6.01152        19     
produce:hitpdune:DisambigFromSpacePoints          0.221222      0.389109      0.627581      0.373241      0.105696        19     
produce:cluster3d:Cluster3D                        2.95274       58.684        85.918        65.3751       24.6975        19     
[art]:TriggerResults:TriggerResultInserter       2.4907e-05    3.49907e-05   9.8807e-05    3.0848e-05    1.56464e-05      19     
end_path:out1:RootOutput                          5.35e-06     9.17621e-06    2.614e-05     8.405e-06    4.18411e-06      19     
end_path:out1:RootOutput(write)                    7.92625       9.88293       13.2193       9.60106       1.41805        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5545.67 MB
  Peak resident set size usage (VmHWM): 3632.93 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0562_df-s02-d0_dw_0_20260203T221543_reco_stage0_20260204T005029_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0562_df-s02-d0_dw_0_20260203T221543_reco_stage0_20260204T005029_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0562_df-s02-d0_dw_0_20260203T221543_reco_stage0_20260204T005029_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770161344.0,
    "core.end_time": 1770166232.0,
    "core.events": [
      64074,
      64080,
      64086,
      64092,
      64098,
      64104,
      64110,
      64116,
      64122,
      64128,
      64134,
      64140,
      64146,
      64152,
      64158,
      64164,
      64170,
      64176,
      64182
    ],
    "core.event_count": 19,
    "core.first_event_number": 64074,
    "core.last_event_number": 64182,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0562_df-s02-d0_dw_0_20260203T221543.hdf5"
    }
  ]
}
justIN time: 2026-02-04 06:06:30 UTC       justIN version: 01.06.00