justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297923.92@dunegpschedd02.fnal.gov

Jobsub ID297923.92@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 23:00:26
SiteUK_RAL-PPD
EntryCMSHTPC_T2_UK_SGrid_RALPP_hep206
Last heartbeat2026-02-04 01:20:47
From worker nodeHostnameheplnc153.pp.rl.ac.uk
cpuinfoAMD EPYC 7763 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-04 00:00:40
Input filesvd-protodune:np02vd_raw_run042433_0146_df-s04-d1_dw_0_20260203T193737.hdf5
JobscriptExit code0
Real time1h (4626s)
CPU time1h (4303s = 93%)
Max RSS bytes3533037568 (3369 MiB)
Outputting started2026-02-04 01:17:49
Output files
Finished2026-02-04 01:20:47
Saved logsjustin-logs:297923.92-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

in looking up space point ptr... 0x5594eaa0, 0x339a62a0
Caught exception in looking up space point ptr... 0x55860f00, 0x558890d0
Caught exception in looking up space point ptr... 0x558113e0, 0x55860f00
Caught exception in looking up space point ptr... 0xfd76430, 0xfd765c0
Caught exception in looking up space point ptr... 0x559d57f0, 0xfd76430
Caught exception in looking up space point ptr... 0x10973fa0, 0x71035140
Caught exception in looking up space point ptr... 0x7107deb0, 0x10973fa0
Caught exception in looking up space point ptr... 0x11810440, 0x1181e400
Caught exception in looking up space point ptr... 0x118101b0, 0x11810440
Caught exception in looking up space point ptr... 0x512aab50, 0x512b8ed0
Caught exception in looking up space point ptr... 0x512b8ed0, 0x3b37a3f0
Caught exception in looking up space point ptr... 0x468e4500, 0x89dad70
Caught exception in looking up space point ptr... 0x89dad70, 0x4aa225d0
Caught exception in looking up space point ptr... 0x3b306ac0, 0x39c13700
Caught exception in looking up space point ptr... 0x39c13700, 0x6b6b9420
Caught exception in looking up space point ptr... 0x4049bb60, 0x7621e1d0
Caught exception in looking up space point ptr... 0x7621e1d0, 0x3c5b4680
Caught exception in looking up space point ptr... 0x329a12b0, 0x134f46e0
Caught exception in looking up space point ptr... 0x134f46e0, 0x134f6bf0
Caught exception in looking up space point ptr... 0x156a56f0, 0x156ac330
Caught exception in looking up space point ptr... 0x53ed6a70, 0x156a56f0
Caught exception in looking up space point ptr... 0xeb2f8c0, 0xeb1d620
Caught exception in looking up space point ptr... 0xeb2fa50, 0xeb2f8c0
Caught exception in looking up space point ptr... 0x1624a7c0, 0x1623c9e0
Caught exception in looking up space point ptr... 0x1624a950, 0x1624a7c0
Caught exception in looking up space point ptr... 0x307f1c60, 0x1a1bc4c0
Caught exception in looking up space point ptr... 0x1a1bc4c0, 0x3d87e490
Caught exception in looking up space point ptr... 0x6127eea0, 0x80a0f9a0
Caught exception in looking up space point ptr... 0x80a0f9a0, 0x6127eea0
Caught exception in looking up space point ptr... 0xda7dcf0, 0xdae3960
Caught exception in looking up space point ptr... 0xdae3960, 0xdae3e00
Caught exception in looking up space point ptr... 0x7ab89110, 0x7527310
Caught exception in looking up space point ptr... 0x7527310, 0x610b6e60
Caught exception in looking up space point ptr... 0xb3c65c0, 0xb3c6750
Caught exception in looking up space point ptr... 0xd555880, 0xb3c65c0
Caught exception in looking up space point ptr... 0x494da1d0, 0x494d9ec0
Caught exception in looking up space point ptr... 0x494d9ec0, 0x494dafb0
Caught exception in looking up space point ptr... 0x595b1550, 0x7a6efab0
Caught exception in looking up space point ptr... 0x7a6efab0, 0x595b1550
Caught exception in looking up space point ptr... 0x8eec4a0, 0x5145f1a0
Caught exception in looking up space point ptr... 0x59eab330, 0x8eec4a0
Caught exception in looking up space point ptr... 0xa595360, 0x8064f460
Caught exception in looking up space point ptr... 0xa5954f0, 0xa595360
Caught exception in looking up space point ptr... 0x339b1260, 0x339b10d0
Caught exception in looking up space point ptr... 0x339b10d0, 0x123f91e0
Caught exception in looking up space point ptr... 0xe732190, 0xe731ce0
Caught exception in looking up space point ptr... 0xe7324b0, 0xe732190
Caught exception in looking up space point ptr... 0x156af820, 0x513b8790
Caught exception in looking up space point ptr... 0x513b8790, 0x60eaab70
Caught exception in looking up space point ptr... 0x5087ef50, 0x5086c990
Caught exception in looking up space point ptr... 0x5086c990, 0x5087f0e0
Caught exception in looking up space point ptr... 0x616f5020, 0x3b6e4500
Caught exception in looking up space point ptr... 0x3b6e4500, 0x7bb1ccd0
Caught exception in looking up space point ptr... 0x7bb68080, 0x7bb68260
Caught exception in looking up space point ptr... 0x7bb68260, 0x3180d4e0
Caught exception in looking up space point ptr... 0x34b2ad20, 0x6b7a0a80
Caught exception in looking up space point ptr... 0x6b7a0a80, 0x3bd7c8e0
Caught exception in looking up space point ptr... 0x34b2ad20, 0x6b7a0b80
Caught exception in looking up space point ptr... 0x6b7a0b80, 0x34b2ad20
Caught exception in looking up space point ptr... 0x352216f0, 0x610a6c50
Caught exception in looking up space point ptr... 0x610a6c50, 0x352216f0
Caught exception in looking up space point ptr... 0xe812a50, 0x89c9e20
Caught exception in looking up space point ptr... 0x89c9e20, 0xe812a50
Caught exception in looking up space point ptr... 0x3262b5b0, 0x3c4c73d0
Caught exception in looking up space point ptr... 0x3c4c73d0, 0x3262b5b0
Caught exception in looking up space point ptr... 0x34fee720, 0x6c159de0
Caught exception in looking up space point ptr... 0x6c159de0, 0xea75ac0
++++>>>> total num hits: 500650, num free: 490854
04-Feb-2026 01:17:39 GMT  Closed output file "np02vd_raw_run042433_0146_df-s04-d1_dw_0_20260203T193737_reco_stage0_20260204T011739_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         183.904       239.234       281.128       246.074       28.8179        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       8.8055e-05    0.000166389   0.000706423   0.000119554   0.000133072      19     
produce:tpcrawdecoder:PDVDTPCReader                12.8683       13.4289       14.6792       13.246       0.459389        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.529296      0.541727      0.561286      0.544369     0.00904949       19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0177001     0.0179758     0.0183013     0.0179581    0.000150204      19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0176877     0.0179787     0.0185992     0.0179109    0.000231707      19     
produce:beamevent:BeamEvent                      0.00011742    0.000221555   0.00109894    0.00016581    0.000210396      19     
produce:pdvddaphne:DAPHNEReaderPDVD                3.32938       3.70019       4.19386       3.67048      0.236474        19     
produce:ophit:OpHitFinder                         0.0286561     0.0455134     0.0528225     0.0470975    0.00596589       19     
produce:wclsdatavd:WireCellToolkit                 121.964       137.572       156.035       137.359       8.74753        19     
produce:gaushit:GausHitFinder                      1.01412       2.38105       3.31726       2.40549      0.560168        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000297948   0.000541872   0.00102698    0.000504635   0.000204124      19     
produce:reco3d:SpacePointSolver                    11.5149       17.4241       27.4856       17.2589       4.06558        19     
produce:hitpdune:DisambigFromSpacePoints          0.0999358     0.346127      0.506848      0.344504       0.11204        19     
produce:cluster3d:Cluster3D                        2.09821       55.7692       83.8441       68.051        28.3431        19     
[art]:TriggerResults:TriggerResultInserter       2.3144e-05    4.38853e-05   0.000190046   3.3653e-05    3.5692e-05       19     
end_path:out1:RootOutput                          4.288e-06    1.17921e-05   5.7307e-05     8.987e-06    1.11404e-05      19     
end_path:out1:RootOutput(write)                    6.00289       7.93478       9.85679       8.07149       1.31518        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5573.12 MB
  Peak resident set size usage (VmHWM): 3533.04 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0146_df-s04-d1_dw_0_20260203T193737_reco_stage0_20260204T011739_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0146_df-s04-d1_dw_0_20260203T193737_reco_stage0_20260204T011739_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0146_df-s04-d1_dw_0_20260203T193737_reco_stage0_20260204T011739_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770163254.0,
    "core.end_time": 1770167860.0,
    "core.events": [
      16655,
      16661,
      16667,
      16673,
      16679,
      16685,
      16691,
      16697,
      16703,
      16709,
      16715,
      16721,
      16727,
      16733,
      16739,
      16745,
      16751,
      16757,
      16763
    ],
    "core.event_count": 19,
    "core.first_event_number": 16655,
    "core.last_event_number": 16763,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0146_df-s04-d1_dw_0_20260203T193737.hdf5"
    }
  ]
}
justIN time: 2026-02-04 06:06:10 UTC       justIN version: 01.06.00