justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297921.135@dunegpschedd02.fnal.gov

Jobsub ID297921.135@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:58:26
SiteUK_RAL-Tier1
EntryLIGO_UK_RAL_arc_ce05
Last heartbeat2026-02-04 02:09:50
From worker nodeHostnamedune001-1845822.0-lcg2351.gridpp.rl.ac.uk
cpuinfoIntel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit216000 (60 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:56:13
Input filesvd-protodune:np02vd_raw_run042433_0619_df-s04-d1_dw_0_20260203T223724.hdf5
JobscriptExit code0
Real time2h (7902s)
CPU time1h (4252s = 53%)
Max RSS bytes3355996160 (3200 MiB)
Outputting started2026-02-04 02:07:56
Output files
Finished2026-02-04 02:09:50
Saved logsjustin-logs:297921.135-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

.. 0x5b9da7d0, 0x5b9da960
Caught exception in looking up space point ptr... 0x578c5940, 0x366d3310
Caught exception in looking up space point ptr... 0x366d3310, 0x578c5940
Caught exception in looking up space point ptr... 0x5c007fb0, 0x94869560
Caught exception in looking up space point ptr... 0x94869560, 0x5c007fb0
Caught exception in looking up space point ptr... 0x9e278a50, 0x598625c0
Caught exception in looking up space point ptr... 0x598625c0, 0x35cf86d0
Caught exception in looking up space point ptr... 0x66bd2ff0, 0x5788b790
Caught exception in looking up space point ptr... 0x5788b790, 0x7ab5fbf0
Caught exception in looking up space point ptr... 0x73fc7050, 0x73fca010
Caught exception in looking up space point ptr... 0x73fca010, 0x5f9502f0
Caught exception in looking up space point ptr... 0x5f9502f0, 0x6038f4b0
Caught exception in looking up space point ptr... 0x6038f4b0, 0x73fc7050
Caught exception in looking up space point ptr... 0x566306e0, 0x5384bab0
Caught exception in looking up space point ptr... 0x5384bab0, 0x53834da0
Caught exception in looking up space point ptr... 0x50a3ee40, 0x5ec2c9c0
Caught exception in looking up space point ptr... 0x5ec2c9c0, 0x50a3ee40
Caught exception in looking up space point ptr... 0x601b82a0, 0x601ba670
Caught exception in looking up space point ptr... 0x601ba670, 0x601b82a0
Caught exception in looking up space point ptr... 0x5072fe40, 0x59941bf0
Caught exception in looking up space point ptr... 0x59941bf0, 0x5072fe40
Caught exception in looking up space point ptr... 0x91ef7610, 0x91ef77a0
Caught exception in looking up space point ptr... 0x91ef77a0, 0x91ef7610
Caught exception in looking up space point ptr... 0x91ef7610, 0x5cd9adf0
Caught exception in looking up space point ptr... 0x5cd9adf0, 0x91ef7610
Caught exception in looking up space point ptr... 0x58371060, 0x583773a0
Caught exception in looking up space point ptr... 0x583773a0, 0x58371060
Caught exception in looking up space point ptr... 0x5ced7340, 0x2ff50a20
Caught exception in looking up space point ptr... 0x2ff50a20, 0x5ced7340
Caught exception in looking up space point ptr... 0x61541f60, 0x620a9aa0
Caught exception in looking up space point ptr... 0x620a9aa0, 0x61541f60
Caught exception in looking up space point ptr... 0x53380140, 0x53380d10
Caught exception in looking up space point ptr... 0x53380d10, 0x53380360
Caught exception in looking up space point ptr... 0x69ef9b80, 0x69efc170
Caught exception in looking up space point ptr... 0x69efc170, 0x69ef9b80
Caught exception in looking up space point ptr... 0x3330c770, 0x3330c5e0
Caught exception in looking up space point ptr... 0x3330c5e0, 0x3330c770
Caught exception in looking up space point ptr... 0x3330c770, 0x3330c450
Caught exception in looking up space point ptr... 0x3330c450, 0x3330c770
Caught exception in looking up space point ptr... 0x2e319750, 0xa74b9d10
Caught exception in looking up space point ptr... 0xa74b9d10, 0x2e319750
Caught exception in looking up space point ptr... 0xa32ce590, 0xa32ce950
Caught exception in looking up space point ptr... 0xa32ce950, 0xa32ce770
Caught exception in looking up space point ptr... 0x81fc0eb0, 0x81fc0cf0
Caught exception in looking up space point ptr... 0x81fc0cf0, 0x81fc0b50
Caught exception in looking up space point ptr... 0x82123a80, 0x82123da0
Caught exception in looking up space point ptr... 0x82123da0, 0x82123a80
Caught exception in looking up space point ptr... 0x854b7b20, 0x854b7c40
Caught exception in looking up space point ptr... 0x854b7c40, 0x854d52e0
Caught exception in looking up space point ptr... 0x59100390, 0x794c5b40
Caught exception in looking up space point ptr... 0x794c5b40, 0x60917450
Caught exception in looking up space point ptr... 0x766bbbb0, 0x766bba20
Caught exception in looking up space point ptr... 0x766bba20, 0x766bbbb0
Caught exception in looking up space point ptr... 0x71ef96b0, 0x71ef9c00
Caught exception in looking up space point ptr... 0x71ef9c00, 0x71ef9890
Caught exception in looking up space point ptr... 0x56c21780, 0x56c21910
Caught exception in looking up space point ptr... 0x56c21910, 0x56c21780
Caught exception in looking up space point ptr... 0x69fc5040, 0x5c0a4ca0
Caught exception in looking up space point ptr... 0x5c0a4ca0, 0x69fc5040
Caught exception in looking up space point ptr... 0x73602270, 0x58b33eb0
Caught exception in looking up space point ptr... 0x58b33eb0, 0x73602270
Caught exception in looking up space point ptr... 0x53d63190, 0x53d60400
Caught exception in looking up space point ptr... 0x53d60400, 0x53d63190
Caught exception in looking up space point ptr... 0x5bea9a10, 0x4b0b4a40
Caught exception in looking up space point ptr... 0x4b0b4a40, 0x53bab0c0
Caught exception in looking up space point ptr... 0x534b39d0, 0x534b3cf0
Caught exception in looking up space point ptr... 0x534b3cf0, 0x534b3e80
++++>>>> total num hits: 108501, num free: 103202
04-Feb-2026 02:07:41 UTC  Closed output file "np02vd_raw_run042433_0619_df-s04-d1_dw_0_20260203T223724_reco_stage0_20260204T020741_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         335.605       410.847        524.8        410.988       44.154         19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       8.4409e-05    0.00016015    0.000288169   0.000127838   7.07488e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                117.939       185.036       274.904       180.888       35.0562        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.640825      0.851315       1.0152       0.881778      0.115949        19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0178796     0.0231135     0.0359566     0.0203044    0.00599576       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0179091     0.0199669     0.0521589     0.0180872    0.00759709       19     
produce:beamevent:BeamEvent                      0.000180423   0.000305353   0.00134937    0.000239572   0.000251577      19     
produce:pdvddaphne:DAPHNEReaderPDVD                14.7856       19.1419       27.5847       18.6615       3.08736        19     
produce:ophit:OpHitFinder                         0.039164      0.0647021     0.110022      0.0691839     0.0178497       19     
produce:wclsdatavd:WireCellToolkit                 151.637       174.965       205.049       175.611       14.1302        19     
produce:gaushit:GausHitFinder                      1.13035       1.97293       3.39465       1.87892       0.48963        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000472261   0.000679567   0.00136357    0.000595308   0.00022304       19     
produce:reco3d:SpacePointSolver                    14.2684       18.1865       24.498        17.3384       3.2385         19     
produce:hitpdune:DisambigFromSpacePoints          0.204124       0.3192       0.601409       0.28658      0.0921689       19     
produce:cluster3d:Cluster3D                        1.56648       3.30078       6.16551       2.99399       1.1231         19     
[art]:TriggerResults:TriggerResultInserter       3.6328e-05    7.04315e-05   0.000355702   4.5849e-05    7.11215e-05      19     
end_path:out1:RootOutput                          4.385e-06    6.70989e-05   0.00107516     9.674e-06    0.000237711      19     
end_path:out1:RootOutput(write)                    5.51646       6.90403       9.4786        6.56556       1.25889        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5517.96 MB
  Peak resident set size usage (VmHWM): 3356 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0619_df-s04-d1_dw_0_20260203T223724_reco_stage0_20260204T020741_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0619_df-s04-d1_dw_0_20260203T223724_reco_stage0_20260204T020741_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0619_df-s04-d1_dw_0_20260203T223724_reco_stage0_20260204T020741_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770162983.0,
    "core.end_time": 1770170864.0,
    "core.events": [
      70577,
      70583,
      70589,
      70595,
      70601,
      70607,
      70613,
      70619,
      70625,
      70631,
      70637,
      70643,
      70649,
      70655,
      70661,
      70667,
      70673,
      70679,
      70685
    ],
    "core.event_count": 19,
    "core.first_event_number": 70577,
    "core.last_event_number": 70685,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0619_df-s04-d1_dw_0_20260203T223724.hdf5"
    }
  ]
}
justIN time: 2026-02-04 07:44:25 UTC       justIN version: 01.06.00