justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303653.181@dunegpschedd01.fnal.gov

Jobsub ID303653.181@dunegpschedd01.fnal.gov
Workflow ID12717
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:52:24
SiteUK_Oxford
EntryDUNE_UK_SGrid_Oxford_arc01
Last heartbeat2026-02-04 01:45:42
From worker nodeHostnamet2wn068.physics.ox.ac.uk
cpuinfoAMD EPYC 7763 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:49:14
Input filesvd-protodune:np02vd_raw_run042421_0062_df-s02-d1_dw_0_20260203T140434.hdf5
JobscriptExit code0
Real time1h (6768s)
CPU time46m (2764s = 40%)
Max RSS bytes2518056960 (2401 MiB)
Outputting started2026-02-04 01:42:04
Output files
Finished2026-02-04 01:45:42
Saved logsjustin-logs:303653.181-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

21e000
Caught exception in looking up space point ptr... 0x3d01e060, 0x3d01db60
Caught exception in looking up space point ptr... 0x3d01db60, 0x3d01e060
Caught exception in looking up space point ptr... 0x36bbc220, 0x1302fe40
Caught exception in looking up space point ptr... 0x1302fe40, 0x46293c10
Caught exception in looking up space point ptr... 0x311cc700, 0x52219680
Caught exception in looking up space point ptr... 0x52219680, 0x311cc700
Caught exception in looking up space point ptr... 0x3a021300, 0x3a0210d0
Caught exception in looking up space point ptr... 0x3a0210d0, 0xa95feb0
Caught exception in looking up space point ptr... 0xcd7ca80, 0x2d1eefd0
Caught exception in looking up space point ptr... 0x2d1eefd0, 0xcd7ca80
Caught exception in looking up space point ptr... 0x2d094bd0, 0x3d01b140
Caught exception in looking up space point ptr... 0x3d01b140, 0x2d094bd0
Caught exception in looking up space point ptr... 0x153fe780, 0x2c2cb890
Caught exception in looking up space point ptr... 0x2c2cb890, 0x153fe780
Caught exception in looking up space point ptr... 0x36597060, 0x51511eb0
Caught exception in looking up space point ptr... 0x51511eb0, 0x515120b0
Caught exception in looking up space point ptr... 0x27669270, 0x36b1b410
Caught exception in looking up space point ptr... 0x36b1b410, 0x47054090
Caught exception in looking up space point ptr... 0xca73b30, 0x3919e40
Caught exception in looking up space point ptr... 0x3919e40, 0xca73b30
Caught exception in looking up space point ptr... 0x2d85f670, 0x412c6240
Caught exception in looking up space point ptr... 0x412c6240, 0x2d85f670
Caught exception in looking up space point ptr... 0x156a5810, 0x3d296200
Caught exception in looking up space point ptr... 0x3d296200, 0x156a63e0
Caught exception in looking up space point ptr... 0x2d2eac90, 0x22909720
Caught exception in looking up space point ptr... 0x22909720, 0x2d2eac90
Caught exception in looking up space point ptr... 0x30ac3d50, 0x30ac3890
Caught exception in looking up space point ptr... 0x458eb6d0, 0x30ac3d50
Caught exception in looking up space point ptr... 0x3b1e9790, 0x33263790
Caught exception in looking up space point ptr... 0x33263790, 0x3b1e9790
Caught exception in looking up space point ptr... 0x2cbced20, 0x2cbce240
Caught exception in looking up space point ptr... 0x2cbce240, 0x2cbced20
Caught exception in looking up space point ptr... 0x3319b0f0, 0x3319ac40
Caught exception in looking up space point ptr... 0x3319ac40, 0x3319b0f0
Caught exception in looking up space point ptr... 0x44589a20, 0x44589920
Caught exception in looking up space point ptr... 0x44589920, 0x44589a20
Caught exception in looking up space point ptr... 0x3da7d1c0, 0x3da7d4b0
Caught exception in looking up space point ptr... 0x3da7d4b0, 0x3da7d360
Caught exception in looking up space point ptr... 0x3d01d830, 0x3d01d5e0
Caught exception in looking up space point ptr... 0x3d01d5e0, 0x3d01d830
Caught exception in looking up space point ptr... 0x65160b90, 0x11395e00
Caught exception in looking up space point ptr... 0x11395e00, 0x65160b90
Caught exception in looking up space point ptr... 0x6652c3a0, 0x5d7931d0
Caught exception in looking up space point ptr... 0x5d7931d0, 0x5d792600
Caught exception in looking up space point ptr... 0xdee1ea0, 0x3700abc0
Caught exception in looking up space point ptr... 0x3700abc0, 0x3114b6e0
Caught exception in looking up space point ptr... 0x3114b6e0, 0x33422d60
Caught exception in looking up space point ptr... 0x20fa27b0, 0x20fa2c60
Caught exception in looking up space point ptr... 0x20fa2c60, 0x20fa2940
Caught exception in looking up space point ptr... 0x5a39a310, 0x5a39b550
Caught exception in looking up space point ptr... 0x5a39b550, 0x5a39a310
Caught exception in looking up space point ptr... 0x2cb7c2d0, 0x52258600
Caught exception in looking up space point ptr... 0x52258600, 0x2cb7c2d0
Caught exception in looking up space point ptr... 0x22799510, 0x8574a60
Caught exception in looking up space point ptr... 0x8574a60, 0x22799510
Caught exception in looking up space point ptr... 0x15982af0, 0x36dd4b20
Caught exception in looking up space point ptr... 0x36dd4b20, 0x15982af0
Caught exception in looking up space point ptr... 0x20fa1fe0, 0x20fa2300
Caught exception in looking up space point ptr... 0x20fa2300, 0x20fa2170
Caught exception in looking up space point ptr... 0x83b4f60, 0x83b4b40
Caught exception in looking up space point ptr... 0x83b4b40, 0x2cb7b2c0
Caught exception in looking up space point ptr... 0x2c427a60, 0x29b1f4d0
Caught exception in looking up space point ptr... 0x33333e50, 0x2c427a60
Caught exception in looking up space point ptr... 0x60fb0fd0, 0x60fb0ea0
Caught exception in looking up space point ptr... 0x60fb0ea0, 0x60fb0fd0
++++>>>> total num hits: 45964, num free: 44260
04-Feb-2026 01:41:51 GMT  Closed output file "np02vd_raw_run042421_0062_df-s02-d1_dw_0_20260203T140434_reco_stage0_20260204T014151_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         193.611       245.943       337.355       237.129       34.2246        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       9.1741e-05    0.000161839   0.000341388   0.000139741   5.96966e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                99.8374       133.855       200.03        127.629       26.5109        27     
produce:triggerrawdecoder:PDVDTriggerReader4      0.656596      0.890596       2.25454      0.763925      0.327004        27     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0196671     0.0249194     0.0823769     0.0199999     0.0123341       27     
produce:ctbrawdecoder:PDHDCTBRawDecoder            0.09696      0.156311       1.50021      0.0981647     0.263969        27     
produce:beamevent:BeamEvent                      0.000104656   0.000216315   0.00162518    0.000163526   0.000277399      27     
produce:pdvddaphne:DAPHNEReaderPDVD                13.1789       16.9666       23.9156       16.1609       2.65872        27     
produce:ophit:OpHitFinder                         0.0357305     0.046015      0.0546456     0.0473591    0.00505387       27     
produce:wclsdatavd:WireCellToolkit                 66.0481       75.2244       86.7637       76.0875       5.71718        27     
produce:gaushit:GausHitFinder                     0.471157       1.19141       2.10529       1.18151      0.375509        27     
produce:nhitsfilter:NumberOfHitsFilter           0.000158436   0.000310567   0.000765651   0.000305801   0.000105748      27     
produce:reco3d:SpacePointSolver                    4.80388       10.9265       21.0759       9.93087       3.51503        27     
produce:hitpdune:DisambigFromSpacePoints          0.0486154     0.110656      0.188736       0.10858      0.0387553       27     
produce:cluster3d:Cluster3D                       0.731754       2.17629       5.6935        2.07882       1.00917        27     
[art]:TriggerResults:TriggerResultInserter       2.4465e-05     3.921e-05    0.000116538   3.5126e-05    1.76708e-05      27     
end_path:out1:RootOutput                          7.123e-06    1.2769e-05    3.9384e-05     1.076e-05    7.33128e-06      27     
end_path:out1:RootOutput(write)                    3.29231       4.33619       6.26609       4.25227      0.734106        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4637.87 MB
  Peak resident set size usage (VmHWM): 2518.06 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042421_0062_df-s02-d1_dw_0_20260203T140434_reco_stage0_20260204T014151_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042421_0062_df-s02-d1_dw_0_20260203T140434_reco_stage0_20260204T014151_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042421_0062_df-s02-d1_dw_0_20260203T140434_reco_stage0_20260204T014151_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770162588.0,
    "core.end_time": 1770169313.0,
    "core.events": [
      10099,
      10105,
      10111,
      10117,
      10123,
      10129,
      10135,
      10141,
      10147,
      10153,
      10159,
      10165,
      10171,
      10177,
      10183,
      10189,
      10195,
      10201,
      10207,
      10213,
      10219,
      10225,
      10231,
      10237,
      10243,
      10249,
      10255
    ],
    "core.event_count": 27,
    "core.first_event_number": 10099,
    "core.last_event_number": 10255,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42421
    ],
    "core.runs_subruns": [
      4242100001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042421_0062_df-s02-d1_dw_0_20260203T140434.hdf5"
    }
  ]
}
justIN time: 2026-02-04 06:06:42 UTC       justIN version: 01.06.00