justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303728.21@dunegpschedd01.fnal.gov

Jobsub ID303728.21@dunegpschedd01.fnal.gov
Workflow ID12717
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-04 01:20:30
SiteUK_Manchester
EntryUBoone_T2_UK_Manchester_ce02
Last heartbeat2026-02-04 04:24:25
From worker nodeHostnamewn1907250.tier2.hep.manchester.ac.uk
cpuinfoIntel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-04 01:21:07
Input filesvd-protodune:np02vd_raw_run042421_0158_df-s02-d0_dw_0_20260203T145627.hdf5
JobscriptExit code0
Real time3h (10817s)
CPU time1h (5988s = 55%)
Max RSS bytes4425191424 (4220 MiB)
Outputting started2026-02-04 04:21:27
Output files
Finished2026-02-04 04:24:25
Saved logsjustin-logs:303728.21-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ht exception in looking up space point ptr... 0x2a772950, 0x2a771760
Caught exception in looking up space point ptr... 0x38573040, 0x38572f40
Caught exception in looking up space point ptr... 0x38572f40, 0x4e3bc110
Caught exception in looking up space point ptr... 0x2b500660, 0x2b5003b0
Caught exception in looking up space point ptr... 0x2b5003b0, 0x2b500660
Caught exception in looking up space point ptr... 0x3cd136c0, 0x380da0d0
Caught exception in looking up space point ptr... 0x380da0d0, 0x380d9fa0
Caught exception in looking up space point ptr... 0x38bf2790, 0x303ffcc0
Caught exception in looking up space point ptr... 0x38bf2890, 0x38bf2790
Caught exception in looking up space point ptr... 0x36048710, 0x2f7643a0
Caught exception in looking up space point ptr... 0x2f7643a0, 0x50545380
Caught exception in looking up space point ptr... 0x45ea4a70, 0x3b3525b0
Caught exception in looking up space point ptr... 0x3b3525b0, 0x305e7c60
Caught exception in looking up space point ptr... 0x374d7600, 0x374d7420
Caught exception in looking up space point ptr... 0x374d7420, 0x374d7970
Caught exception in looking up space point ptr... 0x5d54ae90, 0x5d54ac30
Caught exception in looking up space point ptr... 0x5d54ac30, 0x5d54ae90
Caught exception in looking up space point ptr... 0x24dfe290, 0x503d6e10
Caught exception in looking up space point ptr... 0x503d6e10, 0x24dfe290
Caught exception in looking up space point ptr... 0x6a422a10, 0x34c02dd0
Caught exception in looking up space point ptr... 0x34c02dd0, 0x6a422a10
Caught exception in looking up space point ptr... 0x2b4f4300, 0x2b4f41d0
Caught exception in looking up space point ptr... 0x2b4f41d0, 0x2b4f40a0
Caught exception in looking up space point ptr... 0x5112afa0, 0x5112ae10
Caught exception in looking up space point ptr... 0x5112ae10, 0x5112afa0
Caught exception in looking up space point ptr... 0x5112afa0, 0x511297f0
Caught exception in looking up space point ptr... 0x511297f0, 0x5112afa0
Caught exception in looking up space point ptr... 0x2b08c8c0, 0x2b08c5a0
Caught exception in looking up space point ptr... 0x2b08c5a0, 0x2b08c8c0
Caught exception in looking up space point ptr... 0x2b095d00, 0x2b095a50
Caught exception in looking up space point ptr... 0x2b095a50, 0x2b095d00
Caught exception in looking up space point ptr... 0x2b5f5be0, 0x305d7100
Caught exception in looking up space point ptr... 0x305d7100, 0x2b5f5be0
Caught exception in looking up space point ptr... 0x5959eb00, 0x5959e650
Caught exception in looking up space point ptr... 0x5959e650, 0x5959eb00
Caught exception in looking up space point ptr... 0x30489a80, 0x4dc50c00
Caught exception in looking up space point ptr... 0x4dc50c00, 0x4dc50960
Caught exception in looking up space point ptr... 0x37c63bc0, 0x2b522ea0
Caught exception in looking up space point ptr... 0x2b522ea0, 0x4c634a70
Caught exception in looking up space point ptr... 0x29de37b0, 0x29de3620
Caught exception in looking up space point ptr... 0x29de3620, 0x29de3260
Caught exception in looking up space point ptr... 0x4bfd12b0, 0x461e9f50
Caught exception in looking up space point ptr... 0x2a75aea0, 0x4bfd12b0
Caught exception in looking up space point ptr... 0x386e4010, 0x64f74300
Caught exception in looking up space point ptr... 0x64f74300, 0x386e4010
Caught exception in looking up space point ptr... 0x632b6760, 0x632b68b0
Caught exception in looking up space point ptr... 0x632b68b0, 0x632b65c0
Caught exception in looking up space point ptr... 0x3c2971c0, 0x4dc78da0
Caught exception in looking up space point ptr... 0x4dc78da0, 0x3c2971c0
Caught exception in looking up space point ptr... 0x4dc50dc0, 0x4dc51220
Caught exception in looking up space point ptr... 0x4dc51220, 0x4dc50dc0
Caught exception in looking up space point ptr... 0x4d96a540, 0x4d969af0
Caught exception in looking up space point ptr... 0x4d969af0, 0x4d968a90
Caught exception in looking up space point ptr... 0x5959dc00, 0x5959d890
Caught exception in looking up space point ptr... 0x5959d890, 0x5959dc00
Caught exception in looking up space point ptr... 0x5112e620, 0x5112e170
Caught exception in looking up space point ptr... 0x5112e170, 0x5112e620
Caught exception in looking up space point ptr... 0x3bf21a90, 0x502aab60
Caught exception in looking up space point ptr... 0x502aab60, 0x37ac52a0
Caught exception in looking up space point ptr... 0x85e5960, 0x5d54b0f0
Caught exception in looking up space point ptr... 0x5d54b0f0, 0x85e5960
Caught exception in looking up space point ptr... 0x29dddc20, 0x29ddfd70
Caught exception in looking up space point ptr... 0x29ddfd70, 0x29de3c50
Caught exception in looking up space point ptr... 0x363bd990, 0x38eefb30
Caught exception in looking up space point ptr... 0x38eefb30, 0x363bd990
++++>>>> total num hits: 85974, num free: 82412
04-Feb-2026 04:21:12 GMT  Closed output file "np02vd_raw_run042421_0158_df-s02-d0_dw_0_20260203T145627_reco_stage0_20260204T042112_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         288.341       394.933       692.379       362.587       101.015        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       0.000108856   0.000167632   0.000274537   0.000138461   5.66024e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                104.482       161.972       214.108       160.181       25.6816        27     
produce:triggerrawdecoder:PDVDTriggerReader4      0.779754       1.0071        1.93445      0.975869       0.23939        27     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0220717     0.0265981     0.0349812     0.0253523    0.00410418       27     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.109436       0.13028       0.38627      0.111116      0.0515913       27     
produce:beamevent:BeamEvent                      0.000138185   0.000248196   0.00122311    0.000210639   0.000196869      27     
produce:pdvddaphne:DAPHNEReaderPDVD                17.7115       24.267        30.367        24.2198       3.45803        27     
produce:ophit:OpHitFinder                         0.0446588     0.0739599     0.109249      0.0746095     0.014403        27     
produce:wclsdatavd:WireCellToolkit                 128.361       139.078       154.571       139.359       6.32155        27     
produce:gaushit:GausHitFinder                      1.04887       2.47218       8.19874       2.02406       1.52591        27     
produce:nhitsfilter:NumberOfHitsFilter           0.000337145   0.000654788    0.0014715    0.000535447   0.000273613      27     
produce:reco3d:SpacePointSolver                    9.23714       18.8354       43.7782       14.3228       9.87242        27     
produce:hitpdune:DisambigFromSpacePoints          0.144269      0.545424       2.58619       0.24058      0.685741        27     
produce:cluster3d:Cluster3D                        1.30231       38.5695       315.81        3.21106       86.7125        27     
[art]:TriggerResults:TriggerResultInserter       3.3154e-05    5.55564e-05   0.000161424   4.7444e-05    2.49391e-05      27     
end_path:out1:RootOutput                          6.53e-06     1.21491e-05    5.314e-05    1.0461e-05    8.41614e-06      27     
end_path:out1:RootOutput(write)                    5.30782       7.87134       17.7724       6.86036       3.24503        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6354.82 MB
  Peak resident set size usage (VmHWM): 4425.19 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042421_0158_df-s02-d0_dw_0_20260203T145627_reco_stage0_20260204T042112_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042421_0158_df-s02-d0_dw_0_20260203T145627_reco_stage0_20260204T042112_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042421_0158_df-s02-d0_dw_0_20260203T145627_reco_stage0_20260204T042112_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770168083.0,
    "core.end_time": 1770178877.0,
    "core.events": [
      25674,
      25680,
      25686,
      25692,
      25698,
      25704,
      25710,
      25716,
      25722,
      25728,
      25734,
      25740,
      25746,
      25752,
      25758,
      25764,
      25770,
      25776,
      25782,
      25788,
      25794,
      25800,
      25806,
      25812,
      25818,
      25824,
      25830
    ],
    "core.event_count": 27,
    "core.first_event_number": 25674,
    "core.last_event_number": 25830,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42421
    ],
    "core.runs_subruns": [
      4242100001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042421_0158_df-s02-d0_dw_0_20260203T145627.hdf5"
    }
  ]
}
justIN time: 2026-02-04 06:07:31 UTC       justIN version: 01.06.00