justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303659.23@dunegpschedd01.fnal.gov

Jobsub ID303659.23@dunegpschedd01.fnal.gov
Workflow ID12717
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:58:25
SiteUK_RAL-Tier1
EntryLIGO_UK_RAL_arc_ce03
Last heartbeat2026-02-04 01:25:52
From worker nodeHostnamedune001-1533736.0-lcg2487.gridpp.rl.ac.uk
cpuinfoAMD EPYC 7452 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit216000 (60 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-04 00:16:53
Input filesvd-protodune:np02vd_raw_run042421_0122_df-s04-d1_dw_0_20260203T143659.hdf5
JobscriptExit code0
Real time1h (4024s)
CPU time58m (3522s = 87%)
Max RSS bytes2567012352 (2448 MiB)
Outputting started2026-02-04 01:23:58
Output files
Finished2026-02-04 01:25:52
Saved logsjustin-logs:303659.23-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

exception in looking up space point ptr... 0x3bd530e0, 0x3bd53980
Caught exception in looking up space point ptr... 0x47f8ba50, 0x47f8b950
Caught exception in looking up space point ptr... 0x47f8b950, 0x4d265910
Caught exception in looking up space point ptr... 0x31fe7e00, 0x31fe7ba0
Caught exception in looking up space point ptr... 0x31fe7ba0, 0x31fe7cd0
Caught exception in looking up space point ptr... 0x31fe7cd0, 0x31fe6480
Caught exception in looking up space point ptr... 0x31fe6480, 0x31fe7f30
Caught exception in looking up space point ptr... 0x4797cea0, 0x479765c0
Caught exception in looking up space point ptr... 0x47971190, 0x4797cea0
Caught exception in looking up space point ptr... 0x7e4e8100, 0x357c9c20
Caught exception in looking up space point ptr... 0x357c9c20, 0x4ecf2b70
Caught exception in looking up space point ptr... 0x314480e0, 0x6c63b0b0
Caught exception in looking up space point ptr... 0x6c63b0b0, 0x314480e0
Caught exception in looking up space point ptr... 0x72deb750, 0x7c3dc920
Caught exception in looking up space point ptr... 0x7c3dc920, 0x7c3df7d0
Caught exception in looking up space point ptr... 0x34665b50, 0x35e7be80
Caught exception in looking up space point ptr... 0x35e7be80, 0x34665b50
Caught exception in looking up space point ptr... 0x34665b50, 0x373b7a60
Caught exception in looking up space point ptr... 0x373b7a60, 0x34665b50
Caught exception in looking up space point ptr... 0x34665b50, 0x7e8fac90
Caught exception in looking up space point ptr... 0x7e8fac90, 0x34665b50
Caught exception in looking up space point ptr... 0x4ee97140, 0x4ee93010
Caught exception in looking up space point ptr... 0x4ee93010, 0x4ee96f00
Caught exception in looking up space point ptr... 0x66c24a00, 0x66c258b0
Caught exception in looking up space point ptr... 0x66c258b0, 0x66c24a00
Caught exception in looking up space point ptr... 0x7ee37050, 0x7ee36ec0
Caught exception in looking up space point ptr... 0x7ee36ec0, 0x7ee36ba0
Caught exception in looking up space point ptr... 0x4b96d2e0, 0x4b96cfc0
Caught exception in looking up space point ptr... 0x4b96cfc0, 0x4b96ce30
Caught exception in looking up space point ptr... 0x692fe270, 0x692fdfb0
Caught exception in looking up space point ptr... 0x692fdfb0, 0x692fe270
Caught exception in looking up space point ptr... 0x61883bd0, 0x4e1058c0
Caught exception in looking up space point ptr... 0x4e1058c0, 0x61883d20
Caught exception in looking up space point ptr... 0x38e69d50, 0x81a322f0
Caught exception in looking up space point ptr... 0x81a322f0, 0x38e69d50
Caught exception in looking up space point ptr... 0x80c47d20, 0x80c48220
Caught exception in looking up space point ptr... 0x80c48220, 0x80c47d20
Caught exception in looking up space point ptr... 0x67686da0, 0x3a2945f0
Caught exception in looking up space point ptr... 0x3a2945f0, 0x67685200
Caught exception in looking up space point ptr... 0x4bdac970, 0x859e0590
Caught exception in looking up space point ptr... 0x859e0590, 0x4bdac970
Caught exception in looking up space point ptr... 0x59665ac0, 0x660e7480
Caught exception in looking up space point ptr... 0x660e7480, 0x59665ac0
Caught exception in looking up space point ptr... 0x68389ac0, 0x3745bc90
Caught exception in looking up space point ptr... 0x3745bc90, 0x68389ac0
Caught exception in looking up space point ptr... 0x55a4ae70, 0x5b90e030
Caught exception in looking up space point ptr... 0x5b90e030, 0x55a4ae70
Caught exception in looking up space point ptr... 0x875da4b0, 0x53b11400
Caught exception in looking up space point ptr... 0x53b11400, 0x875da4b0
Caught exception in looking up space point ptr... 0x7d0d32b0, 0x7ee12720
Caught exception in looking up space point ptr... 0x7ee12720, 0x33bd44f0
Caught exception in looking up space point ptr... 0x6faa4a70, 0x6faa4750
Caught exception in looking up space point ptr... 0x6faa4750, 0x6faa4a70
Caught exception in looking up space point ptr... 0x4ee87900, 0x4ee93ce0
Caught exception in looking up space point ptr... 0x4ee93ce0, 0x4ee87900
Caught exception in looking up space point ptr... 0x3c361360, 0x3c360d20
Caught exception in looking up space point ptr... 0x3c360d20, 0x3c360b90
Caught exception in looking up space point ptr... 0x3c360b90, 0x3c3611d0
Caught exception in looking up space point ptr... 0x3c3611d0, 0x3c360eb0
Caught exception in looking up space point ptr... 0x3c360eb0, 0x3c361040
Caught exception in looking up space point ptr... 0x3c361040, 0x3c361360
Caught exception in looking up space point ptr... 0x851152b0, 0x55fce2c0
Caught exception in looking up space point ptr... 0x55fce2c0, 0x851152b0
Caught exception in looking up space point ptr... 0x4cd02e60, 0x4cd02cd0
Caught exception in looking up space point ptr... 0x4cd02cd0, 0x4cd02e60
++++>>>> total num hits: 60149, num free: 57146
04-Feb-2026 01:23:47 UTC  Closed output file "np02vd_raw_run042421_0122_df-s04-d1_dw_0_20260203T143659_reco_stage0_20260204T012347_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         121.578       144.516       162.126       143.948       9.07738        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       9.1981e-05    0.000130002   0.000377015   0.000115932    5.16e-05        27     
produce:tpcrawdecoder:PDVDTPCReader                15.9245       18.8399       21.6047       19.6902       1.57849        27     
produce:triggerrawdecoder:PDVDTriggerReader4      0.684728      0.711629      0.742665      0.713085      0.0136097       27     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0231193     0.0236273     0.0308408     0.0232957    0.00143819       27     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.114096      0.115278      0.119019      0.115027     0.00112989       27     
produce:beamevent:BeamEvent                      0.000116781   0.000354818   0.00386714    0.000158032   0.000751452      27     
produce:pdvddaphne:DAPHNEReaderPDVD                4.35133       4.8845        5.30672       4.95896      0.266005        27     
produce:ophit:OpHitFinder                         0.0367648     0.0518422     0.0596067     0.0534269    0.00493937       27     
produce:wclsdatavd:WireCellToolkit                 77.883        94.1389       105.192       94.1721       5.64769        27     
produce:gaushit:GausHitFinder                     0.846588       1.50309       2.48182       1.44915      0.403856        27     
produce:nhitsfilter:NumberOfHitsFilter           0.000272613   0.00043242    0.00084075    0.000413245   0.00012326       27     
produce:reco3d:SpacePointSolver                    8.5316        15.7414       30.378        15.1863       5.38231        27     
produce:hitpdune:DisambigFromSpacePoints          0.083571      0.153132      0.270723      0.154239      0.0429748       27     
produce:cluster3d:Cluster3D                       0.981347       2.68483       7.21744       2.54043       1.1106         27     
[art]:TriggerResults:TriggerResultInserter        2.56e-05     4.0936e-05    0.000109181   3.7511e-05    1.50948e-05      27     
end_path:out1:RootOutput                          4.791e-06    9.40104e-06   3.1481e-05     9.06e-06     5.66043e-06      27     
end_path:out1:RootOutput(write)                    4.21816       5.62818       6.51859       5.98972      0.737458        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4751.51 MB
  Peak resident set size usage (VmHWM): 2567.01 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042421_0122_df-s04-d1_dw_0_20260203T143659_reco_stage0_20260204T012347_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042421_0122_df-s04-d1_dw_0_20260203T143659_reco_stage0_20260204T012347_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042421_0122_df-s04-d1_dw_0_20260203T143659_reco_stage0_20260204T012347_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770164229.0,
    "core.end_time": 1770168229.0,
    "core.events": [
      19835,
      19841,
      19847,
      19853,
      19859,
      19865,
      19871,
      19877,
      19883,
      19889,
      19895,
      19901,
      19907,
      19913,
      19919,
      19925,
      19931,
      19937,
      19943,
      19949,
      19955,
      19961,
      19967,
      19973,
      19979,
      19985,
      19991
    ],
    "core.event_count": 27,
    "core.first_event_number": 19835,
    "core.last_event_number": 19991,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42421
    ],
    "core.runs_subruns": [
      4242100001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042421_0122_df-s04-d1_dw_0_20260203T143659.hdf5"
    }
  ]
}
justIN time: 2026-02-04 04:31:40 UTC       justIN version: 01.06.00