justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303653.45@dunegpschedd01.fnal.gov

Jobsub ID303653.45@dunegpschedd01.fnal.gov
Workflow ID12717
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:52:24
SiteUK_Manchester
EntryUBoone_T2_UK_Manchester_ce01
Last heartbeat2026-02-04 03:00:29
From worker nodeHostnamewn1209241.tier2.hep.manchester.ac.uk
cpuinfoIntel(R) Xeon(R) Gold 5118 CPU @ 2.30GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:40:59
Input filesvd-protodune:np02vd_raw_run042421_0076_df-s02-d1_dw_0_20260203T141207.hdf5
JobscriptExit code0
Real time3h (11766s)
CPU time1h (6803s = 57%)
Max RSS bytes2688315392 (2563 MiB)
Outputting started2026-02-04 02:57:08
Output files
Finished2026-02-04 03:00:29
Saved logsjustin-logs:303653.45-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

51e2e0
Caught exception in looking up space point ptr... 0x2e51eab0, 0x2e51e790
Caught exception in looking up space point ptr... 0xe341c40, 0x100e7970
Caught exception in looking up space point ptr... 0x22917d20, 0xe341c40
Caught exception in looking up space point ptr... 0x237adec0, 0x57222510
Caught exception in looking up space point ptr... 0x57222510, 0x29a92400
Caught exception in looking up space point ptr... 0x4a182ba0, 0x37c8fce0
Caught exception in looking up space point ptr... 0x4a182d30, 0x4a182ba0
Caught exception in looking up space point ptr... 0x29a81800, 0x58c45900
Caught exception in looking up space point ptr... 0x2e4dd660, 0x29a81800
Caught exception in looking up space point ptr... 0x2a29ca90, 0x2a29c450
Caught exception in looking up space point ptr... 0x2a29c450, 0x94cde10
Caught exception in looking up space point ptr... 0x46ed2600, 0x59fb6670
Caught exception in looking up space point ptr... 0x59fad1d0, 0x46ed2600
Caught exception in looking up space point ptr... 0x4bfa2a60, 0x4bfa2c80
Caught exception in looking up space point ptr... 0x362b61c0, 0x4bfa2a60
Caught exception in looking up space point ptr... 0x356ebe80, 0x4bcb97b0
Caught exception in looking up space point ptr... 0x356ec010, 0x356ebe80
Caught exception in looking up space point ptr... 0xdd4abe0, 0xdd5d1a0
Caught exception in looking up space point ptr... 0xdd3aa10, 0xdd4abe0
Caught exception in looking up space point ptr... 0x355fd500, 0x355eb260
Caught exception in looking up space point ptr... 0x355eb260, 0x355fd370
Caught exception in looking up space point ptr... 0x5361b410, 0x5361b5a0
Caught exception in looking up space point ptr... 0x5361b5a0, 0x5361b410
Caught exception in looking up space point ptr... 0x5361b410, 0xebf60d0
Caught exception in looking up space point ptr... 0xebf60d0, 0x5361b410
Caught exception in looking up space point ptr... 0x245a4c70, 0x245a5120
Caught exception in looking up space point ptr... 0x245a5120, 0x245a4c70
Caught exception in looking up space point ptr... 0x245a4c70, 0x245a4ae0
Caught exception in looking up space point ptr... 0x245a4ae0, 0x245a4c70
Caught exception in looking up space point ptr... 0x245a4c70, 0x245a4950
Caught exception in looking up space point ptr... 0x245a4950, 0x245a4c70
Caught exception in looking up space point ptr... 0x36908270, 0x28474420
Caught exception in looking up space point ptr... 0xda0ba70, 0x36908270
Caught exception in looking up space point ptr... 0x62dafd20, 0x62db0040
Caught exception in looking up space point ptr... 0x62db0040, 0x62db0360
Caught exception in looking up space point ptr... 0x5237adc0, 0x2586a610
Caught exception in looking up space point ptr... 0x48e34e90, 0x5237adc0
Caught exception in looking up space point ptr... 0x62bb06d0, 0x62bb0810
Caught exception in looking up space point ptr... 0x62bb0810, 0x62bb06d0
Caught exception in looking up space point ptr... 0x62bb06d0, 0x62bb0d10
Caught exception in looking up space point ptr... 0x62bb0d10, 0x62bb06d0
Caught exception in looking up space point ptr... 0x62bb06d0, 0x62bb0950
Caught exception in looking up space point ptr... 0x62bb0950, 0x62bb06d0
Caught exception in looking up space point ptr... 0x351b59b0, 0x351b5ab0
Caught exception in looking up space point ptr... 0x351b5ab0, 0x351b57e0
Caught exception in looking up space point ptr... 0x62dc91f0, 0x25405860
Caught exception in looking up space point ptr... 0x25405860, 0x62dca440
Caught exception in looking up space point ptr... 0x621e26d0, 0xccb4730
Caught exception in looking up space point ptr... 0xccb4730, 0x52acd2c0
Caught exception in looking up space point ptr... 0x4c0fd300, 0x4c0fd550
Caught exception in looking up space point ptr... 0x4c0fd550, 0x49c416d0
Caught exception in looking up space point ptr... 0x284c27a0, 0x28466430
Caught exception in looking up space point ptr... 0x28466430, 0x9f6f580
Caught exception in looking up space point ptr... 0x4a808d00, 0x4a809000
Caught exception in looking up space point ptr... 0x4a809000, 0x4a808d00
Caught exception in looking up space point ptr... 0x21860dd0, 0x218610f0
Caught exception in looking up space point ptr... 0x218610f0, 0x21860dd0
Caught exception in looking up space point ptr... 0x4c7b6520, 0x245b7360
Caught exception in looking up space point ptr... 0x245b7360, 0x4c7b6520
Caught exception in looking up space point ptr... 0x49a89d80, 0x7ffa9f0
Caught exception in looking up space point ptr... 0x7ffa9f0, 0x5d17b8f0
Caught exception in looking up space point ptr... 0x62e07da0, 0x62e06b60
Caught exception in looking up space point ptr... 0x62e06b60, 0x56c97e90
Caught exception in looking up space point ptr... 0x35247410, 0x51cd4220
Caught exception in looking up space point ptr... 0x51cd4220, 0x35247410
++++>>>> total num hits: 347152, num free: 341003
04-Feb-2026 02:56:54 GMT  Closed output file "np02vd_raw_run042421_0076_df-s02-d1_dw_0_20260203T141207_reco_stage0_20260204T025654_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         331.418       430.347       508.847       430.159       45.5509        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       0.000108833   0.000228009   0.000557656   0.000216644   0.000103856      27     
produce:tpcrawdecoder:PDVDTPCReader                109.729       163.816       229.287       156.833        28.4          27     
produce:triggerrawdecoder:PDVDTriggerReader4      0.694629       1.02972       3.48514       0.92812      0.503617        27     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0224361     0.0299164     0.0605678     0.0266818     0.0103493       27     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.109604      0.115619      0.153018      0.110784      0.0106224       27     
produce:beamevent:BeamEvent                      0.000133677    0.0002847    0.00160558    0.000234016   0.000263197      27     
produce:pdvddaphne:DAPHNEReaderPDVD                17.4517       25.3878       58.6557       24.3524       7.46327        27     
produce:ophit:OpHitFinder                         0.0436498     0.0771349     0.116232      0.0769496     0.0116378       27     
produce:wclsdatavd:WireCellToolkit                 131.624       150.24        171.425       152.951       9.49464        27     
produce:gaushit:GausHitFinder                      1.38032       2.71101       4.11532       2.71154      0.656101        27     
produce:nhitsfilter:NumberOfHitsFilter           0.000521836   0.000710112   0.00106084    0.000687951   0.000140939      27     
produce:reco3d:SpacePointSolver                    8.44836       14.9392       23.2164       14.2102       4.07671        27     
produce:hitpdune:DisambigFromSpacePoints          0.158605      0.489241      0.989207      0.482269      0.190305        27     
produce:cluster3d:Cluster3D                        1.82497       63.1207       95.4418       80.1176       35.8935        27     
[art]:TriggerResults:TriggerResultInserter        4.654e-05    7.34329e-05   0.000139457   6.9129e-05    2.06119e-05      27     
end_path:out1:RootOutput                         1.0355e-05    1.63429e-05   3.2274e-05     1.386e-05    6.53242e-06      27     
end_path:out1:RootOutput(write)                    6.36235       8.33643       9.91751       8.68056       1.1784         27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4657.68 MB
  Peak resident set size usage (VmHWM): 2688.32 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042421_0076_df-s02-d1_dw_0_20260203T141207_reco_stage0_20260204T025654_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042421_0076_df-s02-d1_dw_0_20260203T141207_reco_stage0_20260204T025654_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042421_0076_df-s02-d1_dw_0_20260203T141207_reco_stage0_20260204T025654_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770162075.0,
    "core.end_time": 1770173818.0,
    "core.events": [
      12367,
      12373,
      12379,
      12385,
      12391,
      12397,
      12403,
      12409,
      12415,
      12421,
      12427,
      12433,
      12439,
      12445,
      12451,
      12457,
      12463,
      12469,
      12475,
      12481,
      12487,
      12493,
      12499,
      12505,
      12511,
      12517,
      12523
    ],
    "core.event_count": 27,
    "core.first_event_number": 12367,
    "core.last_event_number": 12523,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42421
    ],
    "core.runs_subruns": [
      4242100001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042421_0076_df-s02-d1_dw_0_20260203T141207.hdf5"
    }
  ]
}
justIN time: 2026-02-04 04:17:54 UTC       justIN version: 01.06.00