justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297921.131@dunegpschedd02.fnal.gov

Jobsub ID297921.131@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:58:26
SiteUK_RAL-Tier1
EntryLIGO_UK_RAL_arc_ce03
Last heartbeat2026-02-04 00:50:13
From worker nodeHostnamedune001-1535462.0-lcg2721.gridpp.rl.ac.uk
cpuinfoAMD EPYC 9654 96-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit216000 (60 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:55:43
Input filesvd-protodune:np02vd_raw_run042433_0122_df-s03-d1_dw_0_20260203T192830.hdf5
JobscriptExit code0
Real time52m (3169s)
CPU time47m (2841s = 89%)
Max RSS bytes3381293056 (3224 MiB)
Outputting started2026-02-04 00:48:33
Output files
Finished2026-02-04 00:50:13
Saved logsjustin-logs:297921.131-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

0x69622000, 0x15e70060
Caught exception in looking up space point ptr... 0x15e70060, 0x69622000
Caught exception in looking up space point ptr... 0x854c5550, 0x854c5650
Caught exception in looking up space point ptr... 0x854c5650, 0x854c5550
Caught exception in looking up space point ptr... 0x854c5550, 0x854c7a70
Caught exception in looking up space point ptr... 0x854c7a70, 0x854c5550
Caught exception in looking up space point ptr... 0x5c013c90, 0x5c013d90
Caught exception in looking up space point ptr... 0x5c013d90, 0x5c023f40
Caught exception in looking up space point ptr... 0x3f670610, 0x2332e4a0
Caught exception in looking up space point ptr... 0x2332e4a0, 0x4add1c60
Caught exception in looking up space point ptr... 0x21821330, 0x12fd43a0
Caught exception in looking up space point ptr... 0x12fd43a0, 0x21821330
Caught exception in looking up space point ptr... 0x19589780, 0x16401d00
Caught exception in looking up space point ptr... 0x5b24a320, 0x19589780
Caught exception in looking up space point ptr... 0x20d34340, 0x20d31860
Caught exception in looking up space point ptr... 0x20d31860, 0x20d34340
Caught exception in looking up space point ptr... 0x4c519050, 0x4c518cc0
Caught exception in looking up space point ptr... 0x4c518cc0, 0x4c519050
Caught exception in looking up space point ptr... 0x358143c0, 0x3a64dab0
Caught exception in looking up space point ptr... 0x3a64dab0, 0x358143c0
Caught exception in looking up space point ptr... 0x358143c0, 0x87c48dc0
Caught exception in looking up space point ptr... 0x87c48dc0, 0x358143c0
Caught exception in looking up space point ptr... 0x358143c0, 0x608a6e20
Caught exception in looking up space point ptr... 0x608a6e20, 0x358143c0
Caught exception in looking up space point ptr... 0x680f18b0, 0x680f0e30
Caught exception in looking up space point ptr... 0x680f0e30, 0x680f0fd0
Caught exception in looking up space point ptr... 0x680f0fd0, 0x680eb660
Caught exception in looking up space point ptr... 0x5e658660, 0x1c231470
Caught exception in looking up space point ptr... 0x1c231470, 0x8fdb9760
Caught exception in looking up space point ptr... 0x4c519770, 0x4c5192b0
Caught exception in looking up space point ptr... 0x4c5192b0, 0x4c519770
Caught exception in looking up space point ptr... 0x1a2e2450, 0x191eb720
Caught exception in looking up space point ptr... 0x191eb720, 0x1a2f28e0
Caught exception in looking up space point ptr... 0x52117bc0, 0x125e91d0
Caught exception in looking up space point ptr... 0x125e91d0, 0x52117bc0
Caught exception in looking up space point ptr... 0x8e6239b0, 0x8e624650
Caught exception in looking up space point ptr... 0x8e624650, 0x8e6239b0
Caught exception in looking up space point ptr... 0x624b76c0, 0x774042c0
Caught exception in looking up space point ptr... 0x774042c0, 0x4ab85430
Caught exception in looking up space point ptr... 0x330fad90, 0x764aaeb0
Caught exception in looking up space point ptr... 0x764aaeb0, 0x330fad90
Caught exception in looking up space point ptr... 0x21dadea0, 0x21dad910
Caught exception in looking up space point ptr... 0x21dad910, 0x21dadea0
Caught exception in looking up space point ptr... 0x21dadea0, 0x21dadc00
Caught exception in looking up space point ptr... 0x21dadc00, 0x21dadea0
Caught exception in looking up space point ptr... 0x1ceca2b0, 0x1cebf7c0
Caught exception in looking up space point ptr... 0x1cebf7c0, 0x1ceca2b0
Caught exception in looking up space point ptr... 0x1c6d5f30, 0x1c6d60c0
Caught exception in looking up space point ptr... 0x1c6dcb30, 0x1c6d5f30
Caught exception in looking up space point ptr... 0x5dc740b0, 0x5dc76350
Caught exception in looking up space point ptr... 0x5dc76350, 0x5dc70de0
Caught exception in looking up space point ptr... 0x4b2a23c0, 0x538fa180
Caught exception in looking up space point ptr... 0x538fa180, 0x4b2a23c0
Caught exception in looking up space point ptr... 0x439012c0, 0x43902e90
Caught exception in looking up space point ptr... 0x43902e90, 0x1a30b5f0
Caught exception in looking up space point ptr... 0x1a2e3e30, 0x1a2e3b10
Caught exception in looking up space point ptr... 0x1a2e3b10, 0x1a2e3e30
Caught exception in looking up space point ptr... 0x191f8d90, 0x43933490
Caught exception in looking up space point ptr... 0x43933490, 0x191f8d90
Caught exception in looking up space point ptr... 0x439203f0, 0x43920260
Caught exception in looking up space point ptr... 0x43920260, 0x43920080
Caught exception in looking up space point ptr... 0x43ec1df0, 0x42061000
Caught exception in looking up space point ptr... 0x42061000, 0x43ec1df0
Caught exception in looking up space point ptr... 0x43ec1df0, 0x5b1173b0
Caught exception in looking up space point ptr... 0x5b1173b0, 0x43ec1df0
Caught exception in looking up space point ptr... 0x124069b0, 0x12406880
Caught exception in looking up space point ptr... 0x12406880, 0x32bb7640
++++>>>> total num hits: 118245, num free: 113346
04-Feb-2026 00:48:24 UTC  Closed output file "np02vd_raw_run042433_0122_df-s03-d1_dw_0_20260203T192830_reco_stage0_20260204T004824_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         149.904       162.713       180.769       162.514       6.59318        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       9.8308e-05    0.000169722   0.000862561   0.000108904   0.000169937      19     
produce:tpcrawdecoder:PDVDTPCReader                16.5792       17.5444       20.3291       17.4195      0.808546        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.722127       0.80718      0.898007      0.814896      0.0455448       19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0231907     0.0288828     0.061426      0.0236094     0.0112096       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0231095     0.0233852     0.023901      0.0233371    0.000189707      19     
produce:beamevent:BeamEvent                      0.000147783   0.000279512    0.0014664    0.000228916   0.000284699      19     
produce:pdvddaphne:DAPHNEReaderPDVD                3.04767       3.55868       3.92544       3.59951      0.270147        19     
produce:ophit:OpHitFinder                         0.0344948     0.0410309     0.044328      0.0411789    0.00262966       19     
produce:wclsdatavd:WireCellToolkit                 100.47        111.193       122.437       110.896       6.42574        19     
produce:gaushit:GausHitFinder                     0.953283       1.5647        2.26419       1.48735      0.398177        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000210708   0.000353241   0.000631153   0.000379512   0.000100134      19     
produce:reco3d:SpacePointSolver                    11.2028       19.1827       28.1321       19.0331       4.73403        19     
produce:hitpdune:DisambigFromSpacePoints          0.126638      0.214125      0.362933      0.201524      0.0708734       19     
produce:cluster3d:Cluster3D                        1.70614       3.12098       4.9202        2.87443      0.875127        19     
[art]:TriggerResults:TriggerResultInserter       2.6029e-05    4.05718e-05   0.000120542   3.6125e-05    1.9693e-05       19     
end_path:out1:RootOutput                          4.086e-06    9.11384e-06   3.3681e-05     8.863e-06    6.33354e-06      19     
end_path:out1:RootOutput(write)                    4.00476       5.37635       6.35408       5.44436      0.710632        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5541.32 MB
  Peak resident set size usage (VmHWM): 3381.29 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0122_df-s03-d1_dw_0_20260203T192830_reco_stage0_20260204T004824_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0122_df-s03-d1_dw_0_20260203T192830_reco_stage0_20260204T004824_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0122_df-s03-d1_dw_0_20260203T192830_reco_stage0_20260204T004824_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770162955.0,
    "core.end_time": 1770166106.0,
    "core.events": [
      13917,
      13923,
      13929,
      13935,
      13941,
      13947,
      13953,
      13959,
      13965,
      13971,
      13977,
      13983,
      13989,
      13995,
      14001,
      14007,
      14013,
      14019,
      14025
    ],
    "core.event_count": 19,
    "core.first_event_number": 13917,
    "core.last_event_number": 14025,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0122_df-s03-d1_dw_0_20260203T192830.hdf5"
    }
  ]
}
justIN time: 2026-02-04 07:43:07 UTC       justIN version: 01.06.00