justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303656.95@dunegpschedd01.fnal.gov

Jobsub ID303656.95@dunegpschedd01.fnal.gov
Workflow ID12717
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:56:25
SiteUK_Oxford
EntryDUNE_UK_SGrid_Oxford_arc01
Last heartbeat2026-02-04 02:14:14
From worker nodeHostnamet2wn011.physics.ox.ac.uk
cpuinfoAMD EPYC 7713P 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-04 00:13:22
Input filesvd-protodune:np02vd_raw_run042421_0092_df-s04-d0_dw_0_20260203T142045.hdf5
JobscriptExit code0
Real time1h (7054s)
CPU time50m (3023s = 42%)
Max RSS bytes2516221952 (2399 MiB)
Outputting started2026-02-04 02:10:57
Output files
Finished2026-02-04 02:14:14
Saved logsjustin-logs:303656.95-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ught exception in looking up space point ptr... 0x4e8b7640, 0xea6a510
Caught exception in looking up space point ptr... 0xea6a510, 0x41d92c70
Caught exception in looking up space point ptr... 0x5d608a30, 0x14764b70
Caught exception in looking up space point ptr... 0x14764b70, 0x4e8871e0
Caught exception in looking up space point ptr... 0x3cb89430, 0x3cb8b1a0
Caught exception in looking up space point ptr... 0x3cb8b1a0, 0x3e04e1f0
Caught exception in looking up space point ptr... 0x3e04e1f0, 0x3cb8af00
Caught exception in looking up space point ptr... 0x3cb8af00, 0x39e8f700
Caught exception in looking up space point ptr... 0x39e8f700, 0x143d89e0
Caught exception in looking up space point ptr... 0x143d89e0, 0x3cb89430
Caught exception in looking up space point ptr... 0x46bb3e20, 0x4c9ac6b0
Caught exception in looking up space point ptr... 0x4c9ac6b0, 0x34e0af50
Caught exception in looking up space point ptr... 0x3db075a0, 0x4019a5b0
Caught exception in looking up space point ptr... 0x4019a5b0, 0x27f362a0
Caught exception in looking up space point ptr... 0x27f362a0, 0x999d990
Caught exception in looking up space point ptr... 0x289b4aa0, 0x39cd8450
Caught exception in looking up space point ptr... 0x1376abe0, 0x289b4aa0
Caught exception in looking up space point ptr... 0x43794dd0, 0x444c4440
Caught exception in looking up space point ptr... 0x444c4440, 0x45d21140
Caught exception in looking up space point ptr... 0x3546b290, 0x3546c8e0
Caught exception in looking up space point ptr... 0x31b16910, 0x3546b290
Caught exception in looking up space point ptr... 0x41b26130, 0x41b62cf0
Caught exception in looking up space point ptr... 0x624d07d0, 0x41b26130
Caught exception in looking up space point ptr... 0x3dd69110, 0x3dd68df0
Caught exception in looking up space point ptr... 0x3fbc2b60, 0x3dd69110
Caught exception in looking up space point ptr... 0x23744310, 0x324af370
Caught exception in looking up space point ptr... 0x27a71600, 0x23744310
Caught exception in looking up space point ptr... 0x3eeca9d0, 0x240fbff0
Caught exception in looking up space point ptr... 0x240fbff0, 0x40631fa0
Caught exception in looking up space point ptr... 0x40631fa0, 0x2e492b10
Caught exception in looking up space point ptr... 0x226d5b80, 0x4095ea40
Caught exception in looking up space point ptr... 0x4095ea40, 0x86d6810
Caught exception in looking up space point ptr... 0x29aea1f0, 0x3bc785c0
Caught exception in looking up space point ptr... 0x3bc785c0, 0x403cd010
Caught exception in looking up space point ptr... 0x3de004c0, 0x3889f8c0
Caught exception in looking up space point ptr... 0x46404080, 0x3de004c0
Caught exception in looking up space point ptr... 0x50284a30, 0x53b0ff80
Caught exception in looking up space point ptr... 0x53b0ff80, 0x29ae8050
Caught exception in looking up space point ptr... 0x39e90e30, 0x50284a30
Caught exception in looking up space point ptr... 0x47e977d0, 0x5fe1a6f0
Caught exception in looking up space point ptr... 0x5fe1a6f0, 0x2a069930
Caught exception in looking up space point ptr... 0x14ebb410, 0x31a456b0
Caught exception in looking up space point ptr... 0x31a456b0, 0x3e0dc320
Caught exception in looking up space point ptr... 0x30afae90, 0x4cb558e0
Caught exception in looking up space point ptr... 0x4cb558e0, 0x30afae90
Caught exception in looking up space point ptr... 0x53c3e230, 0x30263990
Caught exception in looking up space point ptr... 0x53c3df10, 0x53c3e230
Caught exception in looking up space point ptr... 0x40f9b6b0, 0x40f9b390
Caught exception in looking up space point ptr... 0x40f9b520, 0x40f9b6b0
Caught exception in looking up space point ptr... 0x5f1c7a90, 0x40e458b0
Caught exception in looking up space point ptr... 0x40e458b0, 0x5f1c7a90
Caught exception in looking up space point ptr... 0x39cde720, 0x462bfd70
Caught exception in looking up space point ptr... 0x462bfd70, 0x39cde720
Caught exception in looking up space point ptr... 0x242fa7d0, 0x242fa8d0
Caught exception in looking up space point ptr... 0x242fa8d0, 0x3dd69480
Caught exception in looking up space point ptr... 0x603ef6d0, 0x4c239c10
Caught exception in looking up space point ptr... 0x4c239c10, 0x603ef6d0
Caught exception in looking up space point ptr... 0x3b475280, 0x13e3e4a0
Caught exception in looking up space point ptr... 0x13e3e4a0, 0x3b475280
Caught exception in looking up space point ptr... 0x4c273200, 0x5a7149d0
Caught exception in looking up space point ptr... 0x5a7149d0, 0x2a4f9b20
Caught exception in looking up space point ptr... 0x47d56d00, 0x47d56970
Caught exception in looking up space point ptr... 0x47d56970, 0x47d56d00
Caught exception in looking up space point ptr... 0x361f0090, 0x5fc36c50
Caught exception in looking up space point ptr... 0x5fc36c50, 0x361f0090
++++>>>> total num hits: 45986, num free: 44468
04-Feb-2026 02:10:45 GMT  Closed output file "np02vd_raw_run042421_0092_df-s04-d0_dw_0_20260203T142045_reco_stage0_20260204T021045_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         205.908       257.236       329.262       257.106       28.4681        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       8.5494e-05    0.000104621   0.000274629   9.5233e-05    3.53245e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                97.5513       135.942       192.263       132.42        23.8702        27     
produce:triggerrawdecoder:PDVDTriggerReader4      0.602564      0.802123       1.20188      0.761462      0.155714        27     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0192938     0.0218825     0.038924      0.0196482    0.00430149       27     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0956155     0.103367       0.12344      0.0967444    0.00980245       27     
produce:beamevent:BeamEvent                      0.000101676   0.000164938   0.00136777    0.000117576   0.000236207      27     
produce:pdvddaphne:DAPHNEReaderPDVD                13.328        17.3782       24.7383       17.2847       2.52726        27     
produce:ophit:OpHitFinder                         0.0302879     0.0463603     0.0573573     0.0476538    0.00620011       27     
produce:wclsdatavd:WireCellToolkit                 68.4497       81.7718       96.3596       82.7147       8.46375        27     
produce:gaushit:GausHitFinder                     0.525263       1.18991       2.08203       1.19725      0.348151        27     
produce:nhitsfilter:NumberOfHitsFilter           0.000217699   0.000295671   0.000486587   0.000283005   6.20962e-05      27     
produce:reco3d:SpacePointSolver                    6.60335       13.0735       21.6479       12.647        3.98231        27     
produce:hitpdune:DisambigFromSpacePoints          0.0604809     0.122446      0.263217      0.107218      0.0502155       27     
produce:cluster3d:Cluster3D                       0.947232       2.21228       5.57398       1.93229      0.987466        27     
[art]:TriggerResults:TriggerResultInserter       1.9016e-05    3.14154e-05    0.0001092    2.6151e-05    1.64333e-05      27     
end_path:out1:RootOutput                          3.567e-06    5.73478e-06   2.7693e-05     4.388e-06    4.74134e-06      27     
end_path:out1:RootOutput(write)                    3.66629       4.54045       5.95814       4.21191      0.719522        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4562.73 MB
  Peak resident set size usage (VmHWM): 2516.22 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042421_0092_df-s04-d0_dw_0_20260203T142045_reco_stage0_20260204T021045_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042421_0092_df-s04-d0_dw_0_20260203T142045_reco_stage0_20260204T021045_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042421_0092_df-s04-d0_dw_0_20260203T142045_reco_stage0_20260204T021045_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770164019.0,
    "core.end_time": 1770171047.0,
    "core.events": [
      14962,
      14968,
      14974,
      14980,
      14986,
      14992,
      14998,
      15004,
      15010,
      15016,
      15022,
      15028,
      15034,
      15040,
      15046,
      15052,
      15058,
      15064,
      15070,
      15076,
      15082,
      15088,
      15094,
      15100,
      15106,
      15112,
      15118
    ],
    "core.event_count": 27,
    "core.first_event_number": 14962,
    "core.last_event_number": 15118,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42421
    ],
    "core.runs_subruns": [
      4242100001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042421_0092_df-s04-d0_dw_0_20260203T142045.hdf5"
    }
  ]
}
justIN time: 2026-02-04 06:06:45 UTC       justIN version: 01.06.00