justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297920.169@dunegpschedd02.fnal.gov

Jobsub ID297920.169@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:54:27
SiteUK_Manchester
EntryUBoone_T2_UK_Manchester_ce02
Last heartbeat2026-02-04 03:03:16
From worker nodeHostnamewn1906250.tier2.hep.manchester.ac.uk
cpuinfoIntel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:43:53
Input filesvd-protodune:np02vd_raw_run042433_0266_df-s04-d0_dw_0_20260203T202314.hdf5
JobscriptExit code0
Real time3h (11805s)
CPU time1h (6044s = 51%)
Max RSS bytes3355983872 (3200 MiB)
Outputting started2026-02-04 03:00:40
Output files
Finished2026-02-04 03:03:16
Saved logsjustin-logs:297920.169-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

... 0x16992da0, 0x7ed6d400
Caught exception in looking up space point ptr... 0x5949d2c0, 0x5190cae0
Caught exception in looking up space point ptr... 0x5190cae0, 0x47aea330
Caught exception in looking up space point ptr... 0x6377b860, 0x57680430
Caught exception in looking up space point ptr... 0x57680430, 0x78cf14a0
Caught exception in looking up space point ptr... 0x7179fd50, 0x637f4a30
Caught exception in looking up space point ptr... 0x717a1d40, 0x7179fd50
Caught exception in looking up space point ptr... 0x31822350, 0x5ad77740
Caught exception in looking up space point ptr... 0x5f4c2750, 0x31822350
Caught exception in looking up space point ptr... 0x19c1a0f0, 0x5a2862d0
Caught exception in looking up space point ptr... 0x5a2862d0, 0x19c1aec0
Caught exception in looking up space point ptr... 0x11a96ef0, 0x17395a70
Caught exception in looking up space point ptr... 0x17395a70, 0x11a9bcc0
Caught exception in looking up space point ptr... 0x6f373990, 0x3ad8fed0
Caught exception in looking up space point ptr... 0x78cf92c0, 0x6f373990
Caught exception in looking up space point ptr... 0x774efb40, 0x51799450
Caught exception in looking up space point ptr... 0x58366bd0, 0x774efb40
Caught exception in looking up space point ptr... 0x315de1e0, 0x315dfda0
Caught exception in looking up space point ptr... 0x315dfda0, 0x315dd5c0
Caught exception in looking up space point ptr... 0x315dd5c0, 0x315e0000
Caught exception in looking up space point ptr... 0x315e0000, 0x315db4d0
Caught exception in looking up space point ptr... 0x4b4f3ee0, 0x4b4f3f80
Caught exception in looking up space point ptr... 0x4b4f3f80, 0x4b4f3ee0
Caught exception in looking up space point ptr... 0x635e1250, 0x5a50cf20
Caught exception in looking up space point ptr... 0x5a50cf20, 0x5a50ccd0
Caught exception in looking up space point ptr... 0x3f188ea0, 0xdd08d10
Caught exception in looking up space point ptr... 0xdd08d10, 0x3f188ea0
Caught exception in looking up space point ptr... 0x4e9b58a0, 0x4e9b15c0
Caught exception in looking up space point ptr... 0x5a284130, 0x4e9b58a0
Caught exception in looking up space point ptr... 0x3a498f60, 0x3a364120
Caught exception in looking up space point ptr... 0x3a365180, 0x3a498f60
Caught exception in looking up space point ptr... 0x78e44500, 0x6ed3efd0
Caught exception in looking up space point ptr... 0x6ed3efd0, 0x78e92a50
Caught exception in looking up space point ptr... 0x5a530f40, 0x5a5309c0
Caught exception in looking up space point ptr... 0x5a5309c0, 0x5a53b1c0
Caught exception in looking up space point ptr... 0x45846550, 0x4584cca0
Caught exception in looking up space point ptr... 0x4584cca0, 0x45846550
Caught exception in looking up space point ptr... 0x4da44190, 0x40b69b70
Caught exception in looking up space point ptr... 0x40b69b70, 0x4da44190
Caught exception in looking up space point ptr... 0x116aadf0, 0x4c90dac0
Caught exception in looking up space point ptr... 0x4c90dac0, 0x116aadf0
Caught exception in looking up space point ptr... 0x869e9a20, 0x4e5648b0
Caught exception in looking up space point ptr... 0x4e5648b0, 0x869e9890
Caught exception in looking up space point ptr... 0x3cb5fd60, 0x3cb5fbf0
Caught exception in looking up space point ptr... 0x3cb5fbf0, 0x3cb5fd60
Caught exception in looking up space point ptr... 0x109f1a00, 0x6e4f4180
Caught exception in looking up space point ptr... 0x6e4f4180, 0x109f1a00
Caught exception in looking up space point ptr... 0x541cd820, 0x541cda00
Caught exception in looking up space point ptr... 0x541cda00, 0x541cd820
Caught exception in looking up space point ptr... 0x133fec90, 0x4ec6f630
Caught exception in looking up space point ptr... 0x4ec6f630, 0x133fec90
Caught exception in looking up space point ptr... 0x33a031c0, 0x63b205a0
Caught exception in looking up space point ptr... 0x63b205a0, 0x31fc2100
Caught exception in looking up space point ptr... 0x594986e0, 0x5949a0d0
Caught exception in looking up space point ptr... 0x59499c10, 0x594986e0
Caught exception in looking up space point ptr... 0x365f4bd0, 0x7be4b940
Caught exception in looking up space point ptr... 0x7be4b940, 0x3c9924e0
Caught exception in looking up space point ptr... 0x376d8610, 0x3969e960
Caught exception in looking up space point ptr... 0x3969e960, 0x5a391870
Caught exception in looking up space point ptr... 0x459267c0, 0x459265e0
Caught exception in looking up space point ptr... 0x459265e0, 0x459269a0
Caught exception in looking up space point ptr... 0x792bd650, 0x792bdbd0
Caught exception in looking up space point ptr... 0x792bdbd0, 0x792bd650
Caught exception in looking up space point ptr... 0x5f039550, 0x35580a60
Caught exception in looking up space point ptr... 0x35580a60, 0x5f039550
Caught exception in looking up space point ptr... 0x792bc660, 0x40bbd440
Caught exception in looking up space point ptr... 0x40bbd440, 0x792bc660
++++>>>> total num hits: 87223, num free: 84446
04-Feb-2026 03:00:19 GMT  Closed output file "np02vd_raw_run042433_0266_df-s04-d0_dw_0_20260203T202314_reco_stage0_20260204T030019_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         506.432       612.883       732.588       612.466       56.1578        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       0.000114157   0.000196873   0.000327478   0.000171375   6.72268e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                198.743       280.854       405.249       279.59        50.6778        19     
produce:triggerrawdecoder:PDVDTriggerReader4       0.87647       1.26803       1.73383       1.19279      0.250153        19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0217634     0.0438096      0.31076      0.0259721     0.0633777       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0221157     0.0255277     0.0530203     0.0225129    0.00810221       19     
produce:beamevent:BeamEvent                      0.000157169   0.000307119   0.00148689    0.000243069   0.000283741      19     
produce:pdvddaphne:DAPHNEReaderPDVD                22.5576       27.5676       38.1073       26.4798       3.9986         19     
produce:ophit:OpHitFinder                         0.0451829     0.0680426     0.0893845     0.0725853     0.0112682       19     
produce:wclsdatavd:WireCellToolkit                 204.298       250.259       290.038       251.414       22.9201        19     
produce:gaushit:GausHitFinder                      1.79242       2.58856       3.75047       2.49408      0.502833        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000639072   0.000857737   0.00110276    0.000836798   0.00015806       19     
produce:reco3d:SpacePointSolver                    19.6092       32.0775       54.9112       30.7762       8.06075        19     
produce:hitpdune:DisambigFromSpacePoints          0.279264      0.416009      0.676259       0.39367      0.106381        19     
produce:cluster3d:Cluster3D                        3.0631        4.87016       7.47368       4.51622       1.38333        19     
[art]:TriggerResults:TriggerResultInserter       3.7671e-05    5.50086e-05   9.6189e-05    4.8182e-05    1.57232e-05      19     
end_path:out1:RootOutput                          6.877e-06    1.34765e-05   2.9229e-05    1.1731e-05    5.22441e-06      19     
end_path:out1:RootOutput(write)                    7.70797       12.752        22.3617       10.591        4.65123        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5345.34 MB
  Peak resident set size usage (VmHWM): 3355.98 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0266_df-s04-d0_dw_0_20260203T202314_reco_stage0_20260204T030019_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0266_df-s04-d0_dw_0_20260203T202314_reco_stage0_20260204T030019_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0266_df-s04-d0_dw_0_20260203T202314_reco_stage0_20260204T030019_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770162247.0,
    "core.end_time": 1770174033.0,
    "core.events": [
      30334,
      30340,
      30346,
      30352,
      30358,
      30364,
      30370,
      30376,
      30382,
      30388,
      30394,
      30400,
      30406,
      30412,
      30418,
      30424,
      30430,
      30436,
      30442
    ],
    "core.event_count": 19,
    "core.first_event_number": 30334,
    "core.last_event_number": 30442,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0266_df-s04-d0_dw_0_20260203T202314.hdf5"
    }
  ]
}
justIN time: 2026-02-04 04:32:53 UTC       justIN version: 01.06.00