justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297923.48@dunegpschedd02.fnal.gov

Jobsub ID297923.48@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 23:00:26
SiteUK_Oxford
EntryDUNE_UK_SGrid_Oxford_arc01
Last heartbeat2026-02-04 01:53:54
From worker nodeHostnamet2wn173.physics.ox.ac.uk
cpuinfoAMD EPYC 9655 96-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:59:28
Input filesvd-protodune:np02vd_raw_run042433_0244_df-s04-d1_dw_0_20260203T201451.hdf5
JobscriptExit code0
Real time1h (6691s)
CPU time37m (2252s = 33%)
Max RSS bytes3366301696 (3210 MiB)
Outputting started2026-02-04 01:51:00
Output files
Finished2026-02-04 01:53:54
Saved logsjustin-logs:297923.48-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

up space point ptr... 0x5a0c4480, 0x5a0d8b80
Caught exception in looking up space point ptr... 0x5a0d8b80, 0x5a0c4480
Caught exception in looking up space point ptr... 0x61613900, 0x8274a250
Caught exception in looking up space point ptr... 0x8274a250, 0x61613c90
Caught exception in looking up space point ptr... 0x61613c90, 0x6b042af0
Caught exception in looking up space point ptr... 0x6b042af0, 0x6b08e930
Caught exception in looking up space point ptr... 0x6b08e930, 0x61613900
Caught exception in looking up space point ptr... 0x5eae61d0, 0x5eae69e0
Caught exception in looking up space point ptr... 0x5eae69e0, 0x5eae61d0
Caught exception in looking up space point ptr... 0x5eae61d0, 0x5eae76e0
Caught exception in looking up space point ptr... 0x5eae76e0, 0x5eae61d0
Caught exception in looking up space point ptr... 0x5eae61d0, 0x5eae5eb0
Caught exception in looking up space point ptr... 0x5eae5eb0, 0x5eae61d0
Caught exception in looking up space point ptr... 0xd0b4950, 0xd0b8cd0
Caught exception in looking up space point ptr... 0xd0b8cd0, 0xd0b4950
Caught exception in looking up space point ptr... 0x5a5f4620, 0x5a5f4850
Caught exception in looking up space point ptr... 0x5a5f4850, 0x5a5ec870
Caught exception in looking up space point ptr... 0x40a1b400, 0x40a1b720
Caught exception in looking up space point ptr... 0x40a1b720, 0x40a1b400
Caught exception in looking up space point ptr... 0xd445d40, 0xd44a880
Caught exception in looking up space point ptr... 0xd44a880, 0xd44b720
Caught exception in looking up space point ptr... 0x6da1a270, 0x6da19530
Caught exception in looking up space point ptr... 0x6da19530, 0x6da1ebf0
Caught exception in looking up space point ptr... 0x11c38e40, 0x11c36340
Caught exception in looking up space point ptr... 0x11c36340, 0x11c38e40
Caught exception in looking up space point ptr... 0x8ca7320, 0x386e0010
Caught exception in looking up space point ptr... 0x386e0010, 0x8ca7320
Caught exception in looking up space point ptr... 0xcdfd200, 0x13dadd70
Caught exception in looking up space point ptr... 0x13dadd70, 0xcdfd200
Caught exception in looking up space point ptr... 0x107360d0, 0x19c99b60
Caught exception in looking up space point ptr... 0x19c99b60, 0x107360d0
Caught exception in looking up space point ptr... 0x14f12bf0, 0x14f12fb0
Caught exception in looking up space point ptr... 0x14f12fb0, 0x14f12bf0
Caught exception in looking up space point ptr... 0xd6014d0, 0x41bbfa30
Caught exception in looking up space point ptr... 0x41bbfa30, 0xd6014d0
Caught exception in looking up space point ptr... 0x3bf89ad0, 0x37b8fd10
Caught exception in looking up space point ptr... 0x37b8fd10, 0x3bf89ad0
Caught exception in looking up space point ptr... 0x339b6e40, 0x339bd3b0
Caught exception in looking up space point ptr... 0x339bd3b0, 0x339b6e40
Caught exception in looking up space point ptr... 0x11c56830, 0x11c5a090
Caught exception in looking up space point ptr... 0x11c5a090, 0x11c5a1c0
Caught exception in looking up space point ptr... 0xd601760, 0x41bbf6a0
Caught exception in looking up space point ptr... 0x41bbf6a0, 0x624f8b10
Caught exception in looking up space point ptr... 0x123b9720, 0x123b9590
Caught exception in looking up space point ptr... 0x123b9590, 0x123b9720
Caught exception in looking up space point ptr... 0x4c3d2000, 0x4c3d1e90
Caught exception in looking up space point ptr... 0x4c3d1e90, 0x4c3d1cf0
Caught exception in looking up space point ptr... 0x9475030, 0x794f7ae0
Caught exception in looking up space point ptr... 0x794f78e0, 0x9475030
Caught exception in looking up space point ptr... 0x823b18c0, 0x83d2070
Caught exception in looking up space point ptr... 0x83d2070, 0x6dbae870
Caught exception in looking up space point ptr... 0x6dbc07a0, 0x6dbc0480
Caught exception in looking up space point ptr... 0x6dbc0480, 0x6dbc02f0
Caught exception in looking up space point ptr... 0x741c45c0, 0x741c4b10
Caught exception in looking up space point ptr... 0x741c4b10, 0x741c4930
Caught exception in looking up space point ptr... 0x741c8a10, 0x741c86f0
Caught exception in looking up space point ptr... 0x741c86f0, 0x741c8a10
Caught exception in looking up space point ptr... 0x4e624c40, 0x4e627d90
Caught exception in looking up space point ptr... 0x4e627d90, 0x4e624c40
Caught exception in looking up space point ptr... 0x741c8240, 0x741c80b0
Caught exception in looking up space point ptr... 0x741c80b0, 0x741c83d0
Caught exception in looking up space point ptr... 0x741c9050, 0x741c8d30
Caught exception in looking up space point ptr... 0x741c8d30, 0x741c9050
Caught exception in looking up space point ptr... 0x741c7460, 0x741c7f50
Caught exception in looking up space point ptr... 0x741c7f50, 0x741c7300
Caught exception in looking up space point ptr... 0x384be7d0, 0x384beb40
Caught exception in looking up space point ptr... 0x384beb40, 0x384becd0
++++>>>> total num hits: 144931, num free: 138905
04-Feb-2026 01:50:42 GMT  Closed output file "np02vd_raw_run042433_0244_df-s04-d1_dw_0_20260203T201451_reco_stage0_20260204T015042_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         260.308       348.141       478.006       340.512       48.9621        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       7.6917e-05    0.000107941   0.000160705    9.209e-05    2.48636e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                144.323       218.945       328.552       215.634       44.5086        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.683059       1.01597       1.43451       1.0307       0.202422        19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0193798     0.036371      0.267507      0.0224936     0.0547367       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0192734      0.01956      0.0199687     0.0195714    0.000174819      19     
produce:beamevent:BeamEvent                      0.000100383   0.00020168    0.00119925    0.000146833   0.000236063      19     
produce:pdvddaphne:DAPHNEReaderPDVD                13.9808       18.3917       27.4686       17.8029       3.20808        19     
produce:ophit:OpHitFinder                         0.0299967     0.034246      0.0375452     0.0351343    0.00239087       19     
produce:wclsdatavd:WireCellToolkit                 72.7959       82.0483       92.6066       81.1752       4.94285        19     
produce:gaushit:GausHitFinder                     0.762481       1.18194       1.83727       1.00925      0.346131        19     
produce:nhitsfilter:NumberOfHitsFilter           0.00021702    0.000306614   0.00070439    0.000264442   0.000109653      19     
produce:reco3d:SpacePointSolver                    11.6736       19.3307       29.4969       17.7134       5.95417        19     
produce:hitpdune:DisambigFromSpacePoints          0.0724812     0.180991      0.318516      0.155488      0.0733384       19     
produce:cluster3d:Cluster3D                        0.96068       2.40772       4.39944       2.16669      0.902139        19     
[art]:TriggerResults:TriggerResultInserter       2.6089e-05    4.59514e-05    0.0001579    3.5594e-05    3.22074e-05      19     
end_path:out1:RootOutput                          5.118e-06    8.40374e-06   2.7072e-05     6.66e-06     4.7709e-06       19     
end_path:out1:RootOutput(write)                    3.67912       4.49037       5.46299       4.12803       0.63134        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5624 MB
  Peak resident set size usage (VmHWM): 3366.3 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0244_df-s04-d1_dw_0_20260203T201451_reco_stage0_20260204T015042_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0244_df-s04-d1_dw_0_20260203T201451_reco_stage0_20260204T015042_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0244_df-s04-d1_dw_0_20260203T201451_reco_stage0_20260204T015042_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770163176.0,
    "core.end_time": 1770169843.0,
    "core.events": [
      27827,
      27833,
      27839,
      27845,
      27851,
      27857,
      27863,
      27869,
      27875,
      27881,
      27887,
      27893,
      27899,
      27905,
      27911,
      27917,
      27923,
      27929,
      27935
    ],
    "core.event_count": 19,
    "core.first_event_number": 27827,
    "core.last_event_number": 27935,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0244_df-s04-d1_dw_0_20260203T201451.hdf5"
    }
  ]
}
justIN time: 2026-02-04 04:31:32 UTC       justIN version: 01.06.00