justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303730.14@dunegpschedd01.fnal.gov

Jobsub ID303730.14@dunegpschedd01.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-04 01:20:32
SiteUK_Manchester
EntryUBoone_T2_UK_Manchester_ce02
Last heartbeat2026-02-04 04:09:13
From worker nodeHostnamewn1907251.tier2.hep.manchester.ac.uk
cpuinfoIntel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-04 01:21:20
Input filesvd-protodune:np02vd_raw_run042433_0351_df-s02-d0_dw_0_20260203T205530.hdf5
JobscriptExit code0
Real time2h (9902s)
CPU time1h (5249s = 53%)
Max RSS bytes3407339520 (3249 MiB)
Outputting started2026-02-04 04:06:24
Output files
Finished2026-02-04 04:09:13
Saved logsjustin-logs:303730.14-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

r... 0x4d81fa90, 0x86305750
Caught exception in looking up space point ptr... 0x1030d3f0, 0x373d36a0
Caught exception in looking up space point ptr... 0x373d36a0, 0x1030d3f0
Caught exception in looking up space point ptr... 0x5bd71100, 0x3b2a1f00
Caught exception in looking up space point ptr... 0x3b2a1f00, 0x5bd71100
Caught exception in looking up space point ptr... 0x5ed65b70, 0x39f0ebd0
Caught exception in looking up space point ptr... 0x39f0ebd0, 0x32315f20
Caught exception in looking up space point ptr... 0x3b519f80, 0x4d6cb420
Caught exception in looking up space point ptr... 0x4d6cb420, 0x7b26bb0
Caught exception in looking up space point ptr... 0x37007950, 0x51fdf990
Caught exception in looking up space point ptr... 0x5ed64360, 0x37007950
Caught exception in looking up space point ptr... 0x10cb2620, 0x10cb26f0
Caught exception in looking up space point ptr... 0x10cb26f0, 0x6811a860
Caught exception in looking up space point ptr... 0x6811a860, 0x61c77100
Caught exception in looking up space point ptr... 0x61c77100, 0x10cb2620
Caught exception in looking up space point ptr... 0x3adfb7a0, 0x3a77b760
Caught exception in looking up space point ptr... 0x3a77b760, 0x3adfb7a0
Caught exception in looking up space point ptr... 0x766812d0, 0x7667f1d0
Caught exception in looking up space point ptr... 0x7667f1d0, 0x76681440
Caught exception in looking up space point ptr... 0x59807870, 0x59807770
Caught exception in looking up space point ptr... 0x59807770, 0x59807870
Caught exception in looking up space point ptr... 0x4971acc0, 0x3b359f60
Caught exception in looking up space point ptr... 0x3b359f60, 0x4971acc0
Caught exception in looking up space point ptr... 0x19b352e0, 0x19b2aed0
Caught exception in looking up space point ptr... 0x19b2aed0, 0x19b2a910
Caught exception in looking up space point ptr... 0x68143f60, 0x6e942ef0
Caught exception in looking up space point ptr... 0x6e942ef0, 0x68143f60
Caught exception in looking up space point ptr... 0x3a4ce8a0, 0x1925cbf0
Caught exception in looking up space point ptr... 0x1925cbf0, 0x3a4ce8a0
Caught exception in looking up space point ptr... 0x771ad070, 0x771acf50
Caught exception in looking up space point ptr... 0x771acf50, 0x6ab04c70
Caught exception in looking up space point ptr... 0x3ac8bf60, 0x307d32d0
Caught exception in looking up space point ptr... 0x307d32d0, 0x3ac8bf60
Caught exception in looking up space point ptr... 0x862f9330, 0x862ef0c0
Caught exception in looking up space point ptr... 0x862ef0c0, 0x862fd820
Caught exception in looking up space point ptr... 0x4c29e050, 0x4c29dec0
Caught exception in looking up space point ptr... 0x4c29dec0, 0x4c29dd30
Caught exception in looking up space point ptr... 0x45ae92d0, 0x45ae9460
Caught exception in looking up space point ptr... 0x45ae9460, 0x45ae9780
Caught exception in looking up space point ptr... 0x1082a020, 0x10829e90
Caught exception in looking up space point ptr... 0x10829e90, 0x1082a020
Caught exception in looking up space point ptr... 0x579440e0, 0x57948520
Caught exception in looking up space point ptr... 0x57948520, 0x579440e0
Caught exception in looking up space point ptr... 0x19246780, 0x6e996640
Caught exception in looking up space point ptr... 0x6e996640, 0x19246780
Caught exception in looking up space point ptr... 0x7a1a0060, 0xfd9ad80
Caught exception in looking up space point ptr... 0xfd9ad80, 0x7a1a0160
Caught exception in looking up space point ptr... 0x1082a7f0, 0x1082a660
Caught exception in looking up space point ptr... 0x1082a660, 0x1082a7f0
Caught exception in looking up space point ptr... 0x5b1c4860, 0x1801ba50
Caught exception in looking up space point ptr... 0x1801ba50, 0x61a4e950
Caught exception in looking up space point ptr... 0x59ab38f0, 0x65b909c0
Caught exception in looking up space point ptr... 0x65b909c0, 0x59ab38f0
Caught exception in looking up space point ptr... 0x499c6820, 0x3f03f320
Caught exception in looking up space point ptr... 0x3f03f320, 0x499c6820
Caught exception in looking up space point ptr... 0x499c6820, 0x3b5662d0
Caught exception in looking up space point ptr... 0x3b5662d0, 0x499c6820
Caught exception in looking up space point ptr... 0x3a7dcb80, 0x383a31d0
Caught exception in looking up space point ptr... 0x383a31d0, 0x3a7dcb80
Caught exception in looking up space point ptr... 0x45c59620, 0x3b609830
Caught exception in looking up space point ptr... 0x3b609830, 0x45b847c0
Caught exception in looking up space point ptr... 0x3abb28f0, 0x54c2c250
Caught exception in looking up space point ptr... 0x54c2c250, 0x3abb28f0
Caught exception in looking up space point ptr... 0x3abb28f0, 0x17fe8140
Caught exception in looking up space point ptr... 0x17fe8140, 0x3abb28f0
Caught exception in looking up space point ptr... 0x801a01b0, 0x55233500
Caught exception in looking up space point ptr... 0x55233500, 0x801a01b0
++++>>>> total num hits: 64700, num free: 62396
04-Feb-2026 04:06:07 GMT  Closed output file "np02vd_raw_run042433_0351_df-s02-d0_dw_0_20260203T205530_reco_stage0_20260204T040607_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         457.453       513.499       581.359       511.256       37.9925        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       0.000100744   0.000240168   0.000902757   0.000198041   0.000184131      19     
produce:tpcrawdecoder:PDVDTPCReader                167.422       230.823       302.716       232.984       35.2275        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.716451       1.09555       1.4134        1.12989      0.201138        19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0221346     0.0252493     0.0514542     0.0233683    0.00638994       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0220462     0.0224875     0.0254142     0.0223202    0.000711521      19     
produce:beamevent:BeamEvent                      0.000183607   0.000314918   0.00124372    0.000248266   0.000224743      19     
produce:pdvddaphne:DAPHNEReaderPDVD                14.5761       22.1947       27.8615       22.6596       3.71607        19     
produce:ophit:OpHitFinder                         0.0409166     0.0669585     0.0962761     0.069931      0.0154258       19     
produce:wclsdatavd:WireCellToolkit                  179.6        214.669       233.813       215.879       11.2681        19     
produce:gaushit:GausHitFinder                      1.77066       2.68586       3.90038       2.66115      0.550306        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000579041   0.000785324   0.00118078    0.000725058   0.000168886      19     
produce:reco3d:SpacePointSolver                    15.0167       27.0811       39.5155       27.1023       6.83712        19     
produce:hitpdune:DisambigFromSpacePoints           0.24675      0.445665      0.637104      0.449906      0.100464        19     
produce:cluster3d:Cluster3D                        2.83239       4.8248        7.99626       4.39428       1.58776        19     
[art]:TriggerResults:TriggerResultInserter       2.9838e-05    6.14437e-05   0.000141614   5.8773e-05    2.58439e-05      19     
end_path:out1:RootOutput                          7.681e-06    1.71346e-05   4.2982e-05    1.3963e-05    9.38941e-06      19     
end_path:out1:RootOutput(write)                    7.54847       9.48709       12.956        9.08349       1.28305        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5339.32 MB
  Peak resident set size usage (VmHWM): 3407.34 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0351_df-s02-d0_dw_0_20260203T205530_reco_stage0_20260204T040607_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0351_df-s02-d0_dw_0_20260203T205530_reco_stage0_20260204T040607_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0351_df-s02-d0_dw_0_20260203T205530_reco_stage0_20260204T040607_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770168098.0,
    "core.end_time": 1770177974.0,
    "core.events": [
      40020,
      40026,
      40032,
      40038,
      40044,
      40050,
      40056,
      40062,
      40068,
      40074,
      40080,
      40086,
      40092,
      40098,
      40104,
      40110,
      40116,
      40122,
      40128
    ],
    "core.event_count": 19,
    "core.first_event_number": 40020,
    "core.last_event_number": 40128,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0351_df-s02-d0_dw_0_20260203T205530.hdf5"
    }
  ]
}
justIN time: 2026-02-04 04:29:50 UTC       justIN version: 01.06.00