justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297923.53@dunegpschedd02.fnal.gov

Jobsub ID297923.53@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 23:00:26
SiteUK_RAL-PPD
EntryCMSHTPC_T2_UK_SGrid_RALPP_hep207
Last heartbeat2026-02-04 01:02:47
From worker nodeHostnameheplnc153.pp.rl.ac.uk
cpuinfoAMD EPYC 7763 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:58:45
Input filesvd-protodune:np02vd_raw_run042433_0151_df-s04-d1_dw_0_20260203T193931.hdf5
JobscriptExit code0
Real time1h (3694s)
CPU time56m (3390s = 91%)
Max RSS bytes3325374464 (3171 MiB)
Outputting started2026-02-04 01:00:21
Output files
Finished2026-02-04 01:02:47
Saved logsjustin-logs:297923.53-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

point ptr... 0x71b8dc70, 0x71b89510
Caught exception in looking up space point ptr... 0x17c99040, 0x71b8dc70
Caught exception in looking up space point ptr... 0x45b1bd10, 0x45b1b290
Caught exception in looking up space point ptr... 0x45b1b290, 0x45b1a030
Caught exception in looking up space point ptr... 0xa726bc0, 0xa72d660
Caught exception in looking up space point ptr... 0xa72d660, 0xa726bc0
Caught exception in looking up space point ptr... 0x39d906d0, 0x39d90d80
Caught exception in looking up space point ptr... 0x39d90d80, 0x39d90af0
Caught exception in looking up space point ptr... 0x39d90af0, 0x70d6aa00
Caught exception in looking up space point ptr... 0x60a28960, 0x60a28b40
Caught exception in looking up space point ptr... 0x60a28b40, 0x60a2e120
Caught exception in looking up space point ptr... 0x17468bb0, 0x5b36a720
Caught exception in looking up space point ptr... 0x5b36a720, 0x17468bb0
Caught exception in looking up space point ptr... 0xa716e80, 0xa71aca0
Caught exception in looking up space point ptr... 0xa71aca0, 0xa715580
Caught exception in looking up space point ptr... 0x6855aec0, 0x6855ac20
Caught exception in looking up space point ptr... 0x6855ac20, 0x6855aec0
Caught exception in looking up space point ptr... 0xd2ee820, 0x1359b650
Caught exception in looking up space point ptr... 0x1359b650, 0xd2ee820
Caught exception in looking up space point ptr... 0xcd8da20, 0x4ad82de0
Caught exception in looking up space point ptr... 0x4ad82de0, 0xcd8da20
Caught exception in looking up space point ptr... 0x3243b530, 0x3243b3a0
Caught exception in looking up space point ptr... 0x3243b3a0, 0x3243b530
Caught exception in looking up space point ptr... 0x12d86d70, 0x12d87300
Caught exception in looking up space point ptr... 0x12d87300, 0x12d86d70
Caught exception in looking up space point ptr... 0x51ca8170, 0x51ca7cc0
Caught exception in looking up space point ptr... 0x51ca7cc0, 0x51ca7fe0
Caught exception in looking up space point ptr... 0x129045e0, 0x12902fe0
Caught exception in looking up space point ptr... 0x12902fe0, 0x128ff930
Caught exception in looking up space point ptr... 0x17dfe1c0, 0x17e02da0
Caught exception in looking up space point ptr... 0x17e02da0, 0x17dfe1c0
Caught exception in looking up space point ptr... 0x683679d0, 0x68364220
Caught exception in looking up space point ptr... 0x68364220, 0x683679d0
Caught exception in looking up space point ptr... 0x34f73960, 0x6c2e4d60
Caught exception in looking up space point ptr... 0x6c2e4d60, 0x6c2e4bd0
Caught exception in looking up space point ptr... 0x5b392710, 0x5b3928a0
Caught exception in looking up space point ptr... 0x5b3928a0, 0x5b392710
Caught exception in looking up space point ptr... 0x29816530, 0x297eff30
Caught exception in looking up space point ptr... 0x297eff30, 0x29819310
Caught exception in looking up space point ptr... 0x5a6f8fb0, 0x5a6fe0d0
Caught exception in looking up space point ptr... 0x5a6fe0d0, 0x5a6f8fb0
Caught exception in looking up space point ptr... 0x631ae180, 0x3a1b5620
Caught exception in looking up space point ptr... 0x3a1b5620, 0x4b26c250
Caught exception in looking up space point ptr... 0x5a6f5900, 0x5a6f64a0
Caught exception in looking up space point ptr... 0x5a6f64a0, 0x5a6f5900
Caught exception in looking up space point ptr... 0x5adab580, 0x3d43dc80
Caught exception in looking up space point ptr... 0x5adab330, 0x5adab580
Caught exception in looking up space point ptr... 0x68364040, 0x68362da0
Caught exception in looking up space point ptr... 0x68362da0, 0x68364040
Caught exception in looking up space point ptr... 0x4aef0ef0, 0x4aef0c50
Caught exception in looking up space point ptr... 0x4aef0c50, 0x4aef0ef0
Caught exception in looking up space point ptr... 0x51cb1ca0, 0x298334d0
Caught exception in looking up space point ptr... 0x298334d0, 0x51cb1ca0
Caught exception in looking up space point ptr... 0x5a1c49e0, 0x70931a10
Caught exception in looking up space point ptr... 0x70931a10, 0x5a1c49e0
Caught exception in looking up space point ptr... 0x5b3a2d90, 0x5b3a2b00
Caught exception in looking up space point ptr... 0x709292f0, 0x5b3a2d90
Caught exception in looking up space point ptr... 0x566c5a00, 0x566c37c0
Caught exception in looking up space point ptr... 0x566c37c0, 0x566c5640
Caught exception in looking up space point ptr... 0x12936cd0, 0x12938a20
Caught exception in looking up space point ptr... 0x12938a20, 0x12936cd0
Caught exception in looking up space point ptr... 0x10278230, 0x102780e0
Caught exception in looking up space point ptr... 0x102780e0, 0x10278230
Caught exception in looking up space point ptr... 0x607afc70, 0x607af980
Caught exception in looking up space point ptr... 0x607af980, 0x607afc70
Caught exception in looking up space point ptr... 0x607b0330, 0x607b01a0
Caught exception in looking up space point ptr... 0x607b01a0, 0x607b0330
++++>>>> total num hits: 99057, num free: 94348
04-Feb-2026 01:00:12 GMT  Closed output file "np02vd_raw_run042433_0151_df-s04-d1_dw_0_20260203T193931_reco_stage0_20260204T010012_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         169.06        189.741       211.984       190.107       12.4468        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       9.5299e-05    0.000144654   0.000359854   0.000131216   6.40065e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                13.049        13.3837       13.8536       13.3714      0.201775        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.528987      0.544841      0.556614      0.550368     0.00988385       19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0178853     0.0189822     0.0338987     0.018108      0.0035232       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.017698      0.0187843     0.0303577     0.0180726    0.00274869       19     
produce:beamevent:BeamEvent                      0.000155902   0.000266597   0.00162893    0.000180417   0.000322625      19     
produce:pdvddaphne:DAPHNEReaderPDVD                3.27724       3.73853       4.15946       3.81734      0.229657        19     
produce:ophit:OpHitFinder                         0.0366317     0.0443997     0.0511944     0.0445999    0.00342986       19     
produce:wclsdatavd:WireCellToolkit                 118.677       140.59        158.685       138.432       11.6825        19     
produce:gaushit:GausHitFinder                      1.1925        1.76715       2.36439       1.8486       0.396141        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000316222   0.000494183   0.00112284    0.000456255   0.000185719      19     
produce:reco3d:SpacePointSolver                    13.3356       18.9791       25.456        19.1509       3.18831        19     
produce:hitpdune:DisambigFromSpacePoints          0.112861      0.251651      0.324734      0.272644      0.062143        19     
produce:cluster3d:Cluster3D                        2.16159       3.4232        4.66279       3.35866      0.770772        19     
[art]:TriggerResults:TriggerResultInserter       3.0537e-05    5.04123e-05   0.000157635    4.284e-05    2.73446e-05      19     
end_path:out1:RootOutput                          5.32e-06     1.23706e-05    4.254e-05     1.064e-05    7.62899e-06      19     
end_path:out1:RootOutput(write)                    5.7335        6.92504       7.67986       7.1079       0.669685        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5321.45 MB
  Peak resident set size usage (VmHWM): 3325.37 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0151_df-s04-d1_dw_0_20260203T193931_reco_stage0_20260204T010012_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0151_df-s04-d1_dw_0_20260203T193931_reco_stage0_20260204T010012_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0151_df-s04-d1_dw_0_20260203T193931_reco_stage0_20260204T010012_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770163140.0,
    "core.end_time": 1770166814.0,
    "core.events": [
      17225,
      17231,
      17237,
      17243,
      17249,
      17255,
      17261,
      17267,
      17273,
      17279,
      17285,
      17291,
      17297,
      17303,
      17309,
      17315,
      17321,
      17327,
      17333
    ],
    "core.event_count": 19,
    "core.first_event_number": 17225,
    "core.last_event_number": 17333,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0151_df-s04-d1_dw_0_20260203T193931.hdf5"
    }
  ]
}
justIN time: 2026-02-04 07:43:32 UTC       justIN version: 01.06.00