justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297920.5@dunegpschedd02.fnal.gov

Jobsub ID297920.5@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:54:27
SiteUK_Manchester
EntryUBoone_T2_UK_Manchester_ce01
Last heartbeat2026-02-04 02:58:33
From worker nodeHostnamewn1207250.tier2.hep.manchester.ac.uk
cpuinfoAMD EPYC 7702P 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:36:09
Input filesvd-protodune:np02vd_raw_run042433_0277_df-s04-d1_dw_0_20260203T202724.hdf5
JobscriptExit code0
Real time3h (11934s)
CPU time1h (6093s = 51%)
Max RSS bytes3567390720 (3402 MiB)
Outputting started2026-02-04 02:55:06
Output files
Finished2026-02-04 02:58:33
Saved logsjustin-logs:297920.5-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

up space point ptr... 0x54b04be0, 0x54b04980
Caught exception in looking up space point ptr... 0x1980a790, 0x2962cbd0
Caught exception in looking up space point ptr... 0x2962cbd0, 0x1980a790
Caught exception in looking up space point ptr... 0x31a80380, 0x31a76b40
Caught exception in looking up space point ptr... 0x31a76b40, 0x31a54fb0
Caught exception in looking up space point ptr... 0x4d5370c0, 0x4d536ee0
Caught exception in looking up space point ptr... 0x4d536ee0, 0x4d5370c0
Caught exception in looking up space point ptr... 0x19e47d40, 0x19e46f00
Caught exception in looking up space point ptr... 0x19e46f00, 0x19e43d40
Caught exception in looking up space point ptr... 0x61f55380, 0x77b86de0
Caught exception in looking up space point ptr... 0x77b86de0, 0x61f55380
Caught exception in looking up space point ptr... 0x5fbe6330, 0x5fbe4d80
Caught exception in looking up space point ptr... 0x5fbe4d80, 0x5fbefa80
Caught exception in looking up space point ptr... 0x5b13a530, 0x5b13a7d0
Caught exception in looking up space point ptr... 0x5b13a7d0, 0x5b13a530
Caught exception in looking up space point ptr... 0x38508760, 0x38508660
Caught exception in looking up space point ptr... 0x38508660, 0x38508760
Caught exception in looking up space point ptr... 0x31a4ae50, 0x31a4ac50
Caught exception in looking up space point ptr... 0x31a4ac50, 0x31a4ae50
Caught exception in looking up space point ptr... 0xde53780, 0xde4e2e0
Caught exception in looking up space point ptr... 0xde4e2e0, 0xde53780
Caught exception in looking up space point ptr... 0x4200c1a0, 0x4db951a0
Caught exception in looking up space point ptr... 0x4db951a0, 0x4200c1a0
Caught exception in looking up space point ptr... 0x4200c1a0, 0x357acd50
Caught exception in looking up space point ptr... 0x357acd50, 0x4200c1a0
Caught exception in looking up space point ptr... 0x3691d5c0, 0x3691d470
Caught exception in looking up space point ptr... 0x3691d470, 0x3691d710
Caught exception in looking up space point ptr... 0x48bb4370, 0x39095b20
Caught exception in looking up space point ptr... 0x39095b20, 0x48bb4370
Caught exception in looking up space point ptr... 0x48bb4370, 0x48bb4270
Caught exception in looking up space point ptr... 0x48bb4270, 0x48bb4370
Caught exception in looking up space point ptr... 0x84d41c90, 0x84d42140
Caught exception in looking up space point ptr... 0x84d42140, 0x84d41fb0
Caught exception in looking up space point ptr... 0x33b07980, 0x33b077a0
Caught exception in looking up space point ptr... 0x33b077a0, 0x33b075c0
Caught exception in looking up space point ptr... 0x3444dcb0, 0x3444dad0
Caught exception in looking up space point ptr... 0x3444dad0, 0x3444d760
Caught exception in looking up space point ptr... 0x59dceb60, 0x14581360
Caught exception in looking up space point ptr... 0x14581360, 0x59dceb60
Caught exception in looking up space point ptr... 0x3847e790, 0x3847eab0
Caught exception in looking up space point ptr... 0x3847eab0, 0x3847ec40
Caught exception in looking up space point ptr... 0x6353df20, 0x6353e120
Caught exception in looking up space point ptr... 0x6353e120, 0x6353df20
Caught exception in looking up space point ptr... 0x48bc5c60, 0x48bba8e0
Caught exception in looking up space point ptr... 0x48bc81b0, 0x48bc5c60
Caught exception in looking up space point ptr... 0x3656a370, 0x3656c690
Caught exception in looking up space point ptr... 0x3656c690, 0x3656a370
Caught exception in looking up space point ptr... 0x86107bd0, 0x8610e8a0
Caught exception in looking up space point ptr... 0x8610e8a0, 0x86107bd0
Caught exception in looking up space point ptr... 0x31a7eb70, 0x393b3750
Caught exception in looking up space point ptr... 0x393b3750, 0x31a7eb70
Caught exception in looking up space point ptr... 0x38b90e70, 0x38b91000
Caught exception in looking up space point ptr... 0x38b91000, 0x38b90ce0
Caught exception in looking up space point ptr... 0x19e3f2c0, 0xd72c770
Caught exception in looking up space point ptr... 0xd72c770, 0x19e3f2c0
Caught exception in looking up space point ptr... 0xfd0fc50, 0xfd0fb50
Caught exception in looking up space point ptr... 0xfd0fb50, 0xfd0fc50
Caught exception in looking up space point ptr... 0x41e28e30, 0x41e29010
Caught exception in looking up space point ptr... 0x41e29010, 0x41e28e30
Caught exception in looking up space point ptr... 0xe12b7c0, 0xe12bb30
Caught exception in looking up space point ptr... 0xe12bb30, 0xe12b630
Caught exception in looking up space point ptr... 0xe12b630, 0xe12b4a0
Caught exception in looking up space point ptr... 0xe12b4a0, 0xe12b7c0
Caught exception in looking up space point ptr... 0x29619850, 0x296196b0
Caught exception in looking up space point ptr... 0x296196b0, 0x29619850
Caught exception in looking up space point ptr... 0xd724b60, 0xd724900
Caught exception in looking up space point ptr... 0xd724900, 0xd724b60
++++>>>> total num hits: 109980, num free: 103648
04-Feb-2026 02:54:52 GMT  Closed output file "np02vd_raw_run042433_0277_df-s04-d1_dw_0_20260203T202724_reco_stage0_20260204T025452_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         461.025       618.907       808.013       626.468       98.339         19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       9.9568e-05    0.000128203   0.000329272   0.000111401   5.07727e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                209.251       286.07        407.805       273.075       55.7184        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.897832       1.35844       2.43002       1.30029      0.355928        19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0222912     0.0421869     0.269486      0.0230917     0.0554638       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0222966     0.0250989     0.0655171     0.0228788    0.00953221       19     
produce:beamevent:BeamEvent                      9.6111e-05    0.000231037   0.00137267    0.000136588   0.000297324      19     
produce:pdvddaphne:DAPHNEReaderPDVD                17.8609       24.0082       34.6005       23.9777       3.87367        19     
produce:ophit:OpHitFinder                         0.0487371     0.0560982     0.0601026     0.0565253     0.0032746       19     
produce:wclsdatavd:WireCellToolkit                 161.522       201.36        239.87        202.709       19.468         19     
produce:gaushit:GausHitFinder                      1.8558        3.18704       5.17888       3.15877      0.841206        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000388944   0.000673085   0.00138885    0.000631953   0.000196535      19     
produce:reco3d:SpacePointSolver                    13.6388       30.4967       50.6246       31.6051       8.77437        19     
produce:hitpdune:DisambigFromSpacePoints          0.250562      0.456961      0.642386      0.467031       0.10723        19     
produce:cluster3d:Cluster3D                        3.70199       61.0867       120.039       86.6596       47.5743        19     
[art]:TriggerResults:TriggerResultInserter       2.3855e-05    4.53449e-05   0.000141948   3.6359e-05    2.55039e-05      19     
end_path:out1:RootOutput                          3.867e-06    8.55089e-06   3.1439e-05     5.23e-06     7.58963e-06      19     
end_path:out1:RootOutput(write)                    7.46953       10.6975       12.2779       10.8937       1.28592        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5560.61 MB
  Peak resident set size usage (VmHWM): 3567.39 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0277_df-s04-d1_dw_0_20260203T202724_reco_stage0_20260204T025452_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0277_df-s04-d1_dw_0_20260203T202724_reco_stage0_20260204T025452_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0277_df-s04-d1_dw_0_20260203T202724_reco_stage0_20260204T025452_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770161786.0,
    "core.end_time": 1770173695.0,
    "core.events": [
      31589,
      31595,
      31601,
      31607,
      31613,
      31619,
      31625,
      31631,
      31637,
      31643,
      31649,
      31655,
      31661,
      31667,
      31673,
      31679,
      31685,
      31691,
      31697
    ],
    "core.event_count": 19,
    "core.first_event_number": 31589,
    "core.last_event_number": 31697,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0277_df-s04-d1_dw_0_20260203T202724.hdf5"
    }
  ]
}
justIN time: 2026-02-04 04:26:59 UTC       justIN version: 01.06.00