justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303646.122@dunegpschedd01.fnal.gov

Jobsub ID303646.122@dunegpschedd01.fnal.gov
Workflow ID12718
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:44:24
SiteUK_Imperial
EntryDUNE_T2_UK_London_IC_ceprod00
Last heartbeat2026-02-04 00:28:52
From worker nodeHostnamewl02.grid.hep.ph.ic.ac.uk
cpuinfoAMD EPYC 7742 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 22:45:58
Input filesvd-protodune:np02vd_raw_run042422_0091_df-s03-d0_dw_0_20260203T160359.hdf5
JobscriptExit code0
Real time1h (5973s)
CPU time1h (5194s = 86%)
Max RSS bytes2772082688 (2643 MiB)
Outputting started2026-02-04 00:25:37
Output files
Finished2026-02-04 00:28:52
Saved logsjustin-logs:303646.122-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

Caught exception in looking up space point ptr... 0xfdae160, 0x5fc26f40
Caught exception in looking up space point ptr... 0x5fc26f40, 0x22e0d850
Caught exception in looking up space point ptr... 0x5028b1d0, 0x502909b0
Caught exception in looking up space point ptr... 0x502909b0, 0x5028b1d0
Caught exception in looking up space point ptr... 0x5028b1d0, 0x41ede430
Caught exception in looking up space point ptr... 0x41ede430, 0x5028b1d0
Caught exception in looking up space point ptr... 0x406c0880, 0x39b99ca0
Caught exception in looking up space point ptr... 0x39b99ca0, 0x4e6ff5f0
Caught exception in looking up space point ptr... 0xaddc1f0, 0x423a2750
Caught exception in looking up space point ptr... 0x423a2750, 0x4e1a75c0
Caught exception in looking up space point ptr... 0x2b224710, 0x143530b0
Caught exception in looking up space point ptr... 0x143530b0, 0x4239c190
Caught exception in looking up space point ptr... 0x4239c190, 0x4e1af4f0
Caught exception in looking up space point ptr... 0x165ac380, 0x41468ac0
Caught exception in looking up space point ptr... 0x41468ac0, 0x423bfd60
Caught exception in looking up space point ptr... 0x423bfd60, 0x4f5d3b00
Caught exception in looking up space point ptr... 0x4f5d3b00, 0x165ac380
Caught exception in looking up space point ptr... 0x14a9f230, 0x4705fe60
Caught exception in looking up space point ptr... 0x133fe160, 0x14a9f230
Caught exception in looking up space point ptr... 0x3dcf2d10, 0x2165fab0
Caught exception in looking up space point ptr... 0x41f33e30, 0x3dcf2d10
Caught exception in looking up space point ptr... 0x1484e990, 0x6821de50
Caught exception in looking up space point ptr... 0x56ed1590, 0x1484e990
Caught exception in looking up space point ptr... 0x11496730, 0x38eea480
Caught exception in looking up space point ptr... 0x38eea480, 0x5758d060
Caught exception in looking up space point ptr... 0xba18f30, 0xba06650
Caught exception in looking up space point ptr... 0xba2b860, 0xba18f30
Caught exception in looking up space point ptr... 0xe9a35b0, 0x483db8f0
Caught exception in looking up space point ptr... 0xe990af0, 0xe9a35b0
Caught exception in looking up space point ptr... 0x30dba240, 0x30dc8b10
Caught exception in looking up space point ptr... 0x57775ad0, 0x30dba240
Caught exception in looking up space point ptr... 0x384f2c00, 0x384e3cf0
Caught exception in looking up space point ptr... 0x47321d40, 0x384f2c00
Caught exception in looking up space point ptr... 0x12b535c0, 0x12b40b00
Caught exception in looking up space point ptr... 0x552684e0, 0x12b535c0
Caught exception in looking up space point ptr... 0x4932a1b0, 0x2c33e570
Caught exception in looking up space point ptr... 0x4e925770, 0x4932a1b0
Caught exception in looking up space point ptr... 0x1094d900, 0x50c11450
Caught exception in looking up space point ptr... 0x59dea7e0, 0x1094d900
Caught exception in looking up space point ptr... 0x59dfc7c0, 0x3a3dc830
Caught exception in looking up space point ptr... 0x3a3dc830, 0x109341e0
Caught exception in looking up space point ptr... 0x3070eb90, 0x3071e8b0
Caught exception in looking up space point ptr... 0x57948180, 0x3070eb90
Caught exception in looking up space point ptr... 0x21d89910, 0x21d8ade0
Caught exception in looking up space point ptr... 0x21d8ade0, 0x21d88440
Caught exception in looking up space point ptr... 0x261d0a00, 0x261be2b0
Caught exception in looking up space point ptr... 0x121086e0, 0x261d0a00
Caught exception in looking up space point ptr... 0x270f8d20, 0x270e9000
Caught exception in looking up space point ptr... 0x270e9000, 0x270f8b90
Caught exception in looking up space point ptr... 0x2624db00, 0x2623b860
Caught exception in looking up space point ptr... 0x2623b860, 0x59cf7920
Caught exception in looking up space point ptr... 0x5522c0d0, 0x3d95b210
Caught exception in looking up space point ptr... 0x3d95b210, 0x5522bf40
Caught exception in looking up space point ptr... 0x23c0d970, 0x6823c460
Caught exception in looking up space point ptr... 0x6823c460, 0x68224a50
Caught exception in looking up space point ptr... 0x40f638f0, 0x3a3351f0
Caught exception in looking up space point ptr... 0x3a3351f0, 0x40f638f0
Caught exception in looking up space point ptr... 0x4e8534e0, 0x4e853800
Caught exception in looking up space point ptr... 0x4e853800, 0x4e8534e0
Caught exception in looking up space point ptr... 0x4e8534e0, 0x3135ab30
Caught exception in looking up space point ptr... 0x3135ab30, 0x4e8534e0
Caught exception in looking up space point ptr... 0x39eb8190, 0x26584790
Caught exception in looking up space point ptr... 0x26584790, 0x4e841ff0
Caught exception in looking up space point ptr... 0x4970c8e0, 0x154c8e30
Caught exception in looking up space point ptr... 0x154c8e30, 0x4970c8e0
++++>>>> total num hits: 324447, num free: 318199
04-Feb-2026 00:25:23 GMT  Closed output file "np02vd_raw_run042422_0091_df-s03-d0_dw_0_20260203T160359_reco_stage0_20260204T002523_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         128.745       215.786       276.304       217.146       46.2101        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       0.000101323   0.000179977   0.000490307   0.000128494   9.33153e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                12.5163       13.8344       16.4716       13.4779       1.02737        27     
produce:triggerrawdecoder:PDVDTriggerReader4      0.492195      0.520868      0.604527      0.512168      0.0252566       27     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.016325      0.0196179     0.0329557     0.0169088     0.0053042       27     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0806791     0.0842688     0.099188      0.0826582    0.00507872       27     
produce:beamevent:BeamEvent                      0.000114834   0.000211099    0.0015226    0.000151874   0.000260028      27     
produce:pdvddaphne:DAPHNEReaderPDVD                3.89152       5.06944       6.29134       5.17069       0.59143        27     
produce:ophit:OpHitFinder                         0.0333171     0.0494915     0.0595017     0.0502402    0.00738055       27     
produce:wclsdatavd:WireCellToolkit                 83.5354       112.474       129.732       117.421       14.7192        27     
produce:gaushit:GausHitFinder                     0.886882       2.09578       3.65509       2.05959      0.713172        27     
produce:nhitsfilter:NumberOfHitsFilter           0.000377492   0.000571244   0.000886418   0.000563338   0.000132262      27     
produce:reco3d:SpacePointSolver                    9.84319       18.7994       29.7693       17.9759       4.90971        27     
produce:hitpdune:DisambigFromSpacePoints           0.12759      0.332974      0.605807      0.341518      0.149263        27     
produce:cluster3d:Cluster3D                        1.9988        53.0244       91.5996       69.4447       33.8846        27     
[art]:TriggerResults:TriggerResultInserter       3.0801e-05    4.87183e-05   0.000129815   4.1051e-05    2.30989e-05      27     
end_path:out1:RootOutput                          5.13e-06     1.36948e-05   3.9861e-05     1.277e-05    6.0437e-06       27     
end_path:out1:RootOutput(write)                    6.38142       9.44326       12.3048       9.95484       1.61168        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4608.61 MB
  Peak resident set size usage (VmHWM): 2772.08 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042422_0091_df-s03-d0_dw_0_20260203T160359_reco_stage0_20260204T002523_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042422_0091_df-s03-d0_dw_0_20260203T160359_reco_stage0_20260204T002523_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042422_0091_df-s03-d0_dw_0_20260203T160359_reco_stage0_20260204T002523_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770158775.0,
    "core.end_time": 1770164725.0,
    "core.events": [
      14756,
      14762,
      14768,
      14774,
      14780,
      14786,
      14792,
      14798,
      14804,
      14810,
      14816,
      14822,
      14828,
      14834,
      14840,
      14846,
      14852,
      14858,
      14864,
      14870,
      14876,
      14882,
      14888,
      14894,
      14900,
      14906,
      14912
    ],
    "core.event_count": 27,
    "core.first_event_number": 14756,
    "core.last_event_number": 14912,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42422
    ],
    "core.runs_subruns": [
      4242200001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042422_0091_df-s03-d0_dw_0_20260203T160359.hdf5"
    }
  ]
}
justIN time: 2026-02-04 05:59:20 UTC       justIN version: 01.06.00