justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297920.128@dunegpschedd02.fnal.gov

Jobsub ID297920.128@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:54:27
SiteUK_Oxford
EntryDUNE_UK_SGrid_Oxford_arc01
Last heartbeat2026-02-04 01:49:07
From worker nodeHostnamet2wn064.physics.ox.ac.uk
cpuinfoAMD EPYC 7763 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:42:25
Input filesvd-protodune:np02vd_raw_run042433_0419_df-s04-d0_dw_0_20260203T212123.hdf5
JobscriptExit code0
Real time2h (7413s)
CPU time59m (3577s = 48%)
Max RSS bytes3307536384 (3154 MiB)
Outputting started2026-02-04 01:45:59
Output files
Finished2026-02-04 01:49:07
Saved logsjustin-logs:297920.128-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ptr... 0x180d59c0, 0x180d5b50
Caught exception in looking up space point ptr... 0x180d5b50, 0x180d59c0
Caught exception in looking up space point ptr... 0x81afe200, 0x81afe070
Caught exception in looking up space point ptr... 0x81afe070, 0x81afe200
Caught exception in looking up space point ptr... 0x2d9dd2f0, 0x2d9d56d0
Caught exception in looking up space point ptr... 0x2d9d7f90, 0x2d9dd2f0
Caught exception in looking up space point ptr... 0x5d818650, 0x5d819970
Caught exception in looking up space point ptr... 0x5d819970, 0x5d81a420
Caught exception in looking up space point ptr... 0x46224ae0, 0x32e32d40
Caught exception in looking up space point ptr... 0x32e32d40, 0x4621fe30
Caught exception in looking up space point ptr... 0x80a87330, 0x80a871e0
Caught exception in looking up space point ptr... 0x80a871e0, 0x80a87330
Caught exception in looking up space point ptr... 0x517391e0, 0x1929cd80
Caught exception in looking up space point ptr... 0x1929cd80, 0x517391e0
Caught exception in looking up space point ptr... 0x45772340, 0x45772520
Caught exception in looking up space point ptr... 0x45772520, 0x45782850
Caught exception in looking up space point ptr... 0x45782850, 0x4577dee0
Caught exception in looking up space point ptr... 0x4577dc90, 0x45772340
Caught exception in looking up space point ptr... 0x805bf1e0, 0x805bf050
Caught exception in looking up space point ptr... 0x805bf050, 0x805bf1e0
Caught exception in looking up space point ptr... 0x6139f3a0, 0x6139f640
Caught exception in looking up space point ptr... 0x6139f640, 0x6139f3a0
Caught exception in looking up space point ptr... 0x3f910790, 0x3f910ae0
Caught exception in looking up space point ptr... 0x3f910ae0, 0x3f910790
Caught exception in looking up space point ptr... 0x518a27d0, 0x6f19e3e0
Caught exception in looking up space point ptr... 0x6f19e3e0, 0x518a27d0
Caught exception in looking up space point ptr... 0x6f1b90b0, 0x6f1b8f20
Caught exception in looking up space point ptr... 0x6f1b8f20, 0x6f1b90b0
Caught exception in looking up space point ptr... 0x129b0ac0, 0x129b0930
Caught exception in looking up space point ptr... 0x129b0930, 0x129b0c50
Caught exception in looking up space point ptr... 0xea35000, 0xeaab510
Caught exception in looking up space point ptr... 0xeaab510, 0x36abd360
Caught exception in looking up space point ptr... 0x36abd360, 0xeb54a50
Caught exception in looking up space point ptr... 0x191e0b60, 0x191da5b0
Caught exception in looking up space point ptr... 0x191da5b0, 0x191da4b0
Caught exception in looking up space point ptr... 0xea83ed0, 0xea841a0
Caught exception in looking up space point ptr... 0xea841a0, 0xea83ed0
Caught exception in looking up space point ptr... 0x7b2ba760, 0x7b2ba830
Caught exception in looking up space point ptr... 0x7b2ba830, 0x7b2ba760
Caught exception in looking up space point ptr... 0x6eec0290, 0x517470c0
Caught exception in looking up space point ptr... 0x517470c0, 0x6eec0290
Caught exception in looking up space point ptr... 0x17956640, 0x17964890
Caught exception in looking up space point ptr... 0x17964890, 0x17956640
Caught exception in looking up space point ptr... 0x2f15c180, 0x2f15c960
Caught exception in looking up space point ptr... 0x2f15c960, 0x2f15c180
Caught exception in looking up space point ptr... 0x17ceb0d0, 0x17ceb3d0
Caught exception in looking up space point ptr... 0x17ceb3d0, 0x80725e40
Caught exception in looking up space point ptr... 0x5abdf890, 0x3235be70
Caught exception in looking up space point ptr... 0x3235be70, 0x5abdf890
Caught exception in looking up space point ptr... 0x149b8690, 0x149b9f40
Caught exception in looking up space point ptr... 0x149b9f40, 0x149b8690
Caught exception in looking up space point ptr... 0x2d666a50, 0x325c4c70
Caught exception in looking up space point ptr... 0x325c4c70, 0x2d666a50
Caught exception in looking up space point ptr... 0x6119a3a0, 0x61189350
Caught exception in looking up space point ptr... 0x61189350, 0x6119a3a0
Caught exception in looking up space point ptr... 0x159baf70, 0x159bade0
Caught exception in looking up space point ptr... 0x159bade0, 0x159baf70
Caught exception in looking up space point ptr... 0x800e0770, 0x390dd9a0
Caught exception in looking up space point ptr... 0x390dd9a0, 0x390dd810
Caught exception in looking up space point ptr... 0x5b9cb160, 0x5b9cb5f0
Caught exception in looking up space point ptr... 0x5b9cb5f0, 0x5b9cb160
Caught exception in looking up space point ptr... 0x3bdb6f40, 0x3bdb6de0
Caught exception in looking up space point ptr... 0x3bdb6de0, 0x3bdb6f40
Caught exception in looking up space point ptr... 0x3b252c30, 0x3b252dc0
Caught exception in looking up space point ptr... 0x3b252dc0, 0x3b252c30
Caught exception in looking up space point ptr... 0x3ff69040, 0x3ff6ca50
Caught exception in looking up space point ptr... 0x3ff6ca50, 0x3ff69040
++++>>>> total num hits: 113184, num free: 107828
04-Feb-2026 01:45:37 GMT  Closed output file "np02vd_raw_run042433_0419_df-s04-d0_dw_0_20260203T212123_reco_stage0_20260204T014537_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         305.307       384.141       492.03        384.434       51.8122        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       0.000103664   0.000207882   0.000441586   0.000196838   8.40915e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                141.419       188.406       263.796       177.321       35.9598        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.778075       0.9729        1.25625      0.982623      0.140182        19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0196901     0.025534      0.0655618     0.0202566     0.0112842       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0193805     0.0235146     0.0466147     0.0199435    0.00804413       19     
produce:beamevent:BeamEvent                      0.000150802   0.000285874    0.0016674    0.000207768   0.000326505      19     
produce:pdvddaphne:DAPHNEReaderPDVD                12.4956       16.8899       22.2297       15.9556       2.98198        19     
produce:ophit:OpHitFinder                         0.0354468     0.0456266     0.0603759     0.0453169    0.00533872       19     
produce:wclsdatavd:WireCellToolkit                 113.711       138.996       161.557       136.099       14.3353        19     
produce:gaushit:GausHitFinder                     0.787792       1.9952        3.16199       2.00606      0.598397        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000275014   0.000605115   0.00163197    0.000579804   0.000295768      19     
produce:reco3d:SpacePointSolver                    10.9373       25.8314       40.4326       27.942        8.04548        19     
produce:hitpdune:DisambigFromSpacePoints          0.0948592     0.282021      0.436464      0.298546      0.0900968       19     
produce:cluster3d:Cluster3D                        1.39943       3.77298       6.1256        3.75203       1.2769         19     
[art]:TriggerResults:TriggerResultInserter       2.5778e-05    4.69102e-05   0.000179245   3.3963e-05    3.46492e-05      19     
end_path:out1:RootOutput                          5.49e-06     1.11171e-05   4.5835e-05     8.546e-06    8.78784e-06      19     
end_path:out1:RootOutput(write)                    4.74428       6.82633       8.11516       7.39295       1.05574        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5362.33 MB
  Peak resident set size usage (VmHWM): 3307.54 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0419_df-s04-d0_dw_0_20260203T212123_reco_stage0_20260204T014537_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0419_df-s04-d0_dw_0_20260203T212123_reco_stage0_20260204T014537_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0419_df-s04-d0_dw_0_20260203T212123_reco_stage0_20260204T014537_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770162161.0,
    "core.end_time": 1770169539.0,
    "core.events": [
      47776,
      47782,
      47788,
      47794,
      47800,
      47806,
      47812,
      47818,
      47824,
      47830,
      47836,
      47842,
      47848,
      47854,
      47860,
      47866,
      47872,
      47878,
      47884
    ],
    "core.event_count": 19,
    "core.first_event_number": 47776,
    "core.last_event_number": 47884,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0419_df-s04-d0_dw_0_20260203T212123.hdf5"
    }
  ]
}
justIN time: 2026-02-04 06:06:42 UTC       justIN version: 01.06.00