justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303678.0@dunegpschedd01.fnal.gov

Jobsub ID303678.0@dunegpschedd01.fnal.gov
Workflow ID12718
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 23:48:27
SiteUK_RAL-Tier1
EntryLIGO_UK_RAL_arc_ce01
Last heartbeat2026-02-04 01:27:36
From worker nodeHostnamedune001-1527755.0-lcg2379.gridpp.rl.ac.uk
cpuinfoIntel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit216000 (60 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-04 00:18:12
Input filesvd-protodune:np02vd_raw_run042422_0050_df-s02-d1_dw_0_20260203T154148.hdf5
JobscriptExit code0
Real time1h (4037s)
CPU time1h (3714s = 91%)
Max RSS bytes2561662976 (2442 MiB)
Outputting started2026-02-04 01:25:31
Output files
Finished2026-02-04 01:27:36
Saved logsjustin-logs:303678.0-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

0x3e8d7c10, 0x3e8d7a30
Caught exception in looking up space point ptr... 0x3e8d7a30, 0x3e8d7c10
Caught exception in looking up space point ptr... 0x4e884b80, 0x4e884f70
Caught exception in looking up space point ptr... 0x4e884f70, 0x4e8850c0
Caught exception in looking up space point ptr... 0x60944740, 0x609445f0
Caught exception in looking up space point ptr... 0x609445f0, 0x60944740
Caught exception in looking up space point ptr... 0x35594830, 0x16dc81e0
Caught exception in looking up space point ptr... 0x16dc81e0, 0x16dc8370
Caught exception in looking up space point ptr... 0x4cf40060, 0x5dc92e00
Caught exception in looking up space point ptr... 0x2e2cd430, 0x4cf40060
Caught exception in looking up space point ptr... 0x3cffdec0, 0x442267f0
Caught exception in looking up space point ptr... 0x442267f0, 0x166bfa00
Caught exception in looking up space point ptr... 0x4d887b80, 0x4d886500
Caught exception in looking up space point ptr... 0x4d886500, 0x4d8863b0
Caught exception in looking up space point ptr... 0x3cd3c680, 0x3cd3c430
Caught exception in looking up space point ptr... 0x3cd3c430, 0x3cd3c680
Caught exception in looking up space point ptr... 0x17ccf580, 0x1d5b7160
Caught exception in looking up space point ptr... 0x1d5b7160, 0x2985ca10
Caught exception in looking up space point ptr... 0x5df7a6b0, 0x2dae5c80
Caught exception in looking up space point ptr... 0x2dae5c80, 0x5df7a6b0
Caught exception in looking up space point ptr... 0x4f4b17a0, 0x4f4b1610
Caught exception in looking up space point ptr... 0x4e2fcf90, 0x4f4b17a0
Caught exception in looking up space point ptr... 0x58be1570, 0x58be1270
Caught exception in looking up space point ptr... 0x58bfba70, 0x58be1570
Caught exception in looking up space point ptr... 0x43f5cc90, 0x43f5cf20
Caught exception in looking up space point ptr... 0x43f5cf20, 0x43f5e8e0
Caught exception in looking up space point ptr... 0x4a508990, 0x1739b4d0
Caught exception in looking up space point ptr... 0x1739b4d0, 0x4a508990
Caught exception in looking up space point ptr... 0x5418a550, 0x54185d30
Caught exception in looking up space point ptr... 0x54185d30, 0x54185ba0
Caught exception in looking up space point ptr... 0x297180b0, 0x29717930
Caught exception in looking up space point ptr... 0x29717930, 0x297180b0
Caught exception in looking up space point ptr... 0x2942a1d0, 0x2e18ea60
Caught exception in looking up space point ptr... 0x2e18ea60, 0x3e27d610
Caught exception in looking up space point ptr... 0x1cc25720, 0x394c6590
Caught exception in looking up space point ptr... 0x394c6590, 0x1cc25720
Caught exception in looking up space point ptr... 0x2e2cc760, 0x54935cd0
Caught exception in looking up space point ptr... 0x54935cd0, 0x2e2cc760
Caught exception in looking up space point ptr... 0x53d770c0, 0x53d76f70
Caught exception in looking up space point ptr... 0x53d76f70, 0x53d770c0
Caught exception in looking up space point ptr... 0x1bb888a0, 0x1bb8fc70
Caught exception in looking up space point ptr... 0x1bb8fc70, 0x1bb8ffe0
Caught exception in looking up space point ptr... 0x16c22a60, 0x3d5342e0
Caught exception in looking up space point ptr... 0x3d5342e0, 0x16c22a60
Caught exception in looking up space point ptr... 0x3e525b50, 0x4cf289c0
Caught exception in looking up space point ptr... 0x4cf289c0, 0x3e525b50
Caught exception in looking up space point ptr... 0x641dd330, 0x328e79e0
Caught exception in looking up space point ptr... 0x328e79e0, 0x641dd330
Caught exception in looking up space point ptr... 0x4aa39bf0, 0x1ca8ccf0
Caught exception in looking up space point ptr... 0x1ca8ccf0, 0x4aa39bf0
Caught exception in looking up space point ptr... 0x3ea21930, 0x3e2c62d0
Caught exception in looking up space point ptr... 0x3e2c62d0, 0x3ea21930
Caught exception in looking up space point ptr... 0x5e28e2c0, 0x5e28e1c0
Caught exception in looking up space point ptr... 0x5e28e1c0, 0x2974e6d0
Caught exception in looking up space point ptr... 0x5dc92910, 0x5dc93190
Caught exception in looking up space point ptr... 0x5dc93190, 0x5dc9b230
Caught exception in looking up space point ptr... 0x31ad2c60, 0x31ad4500
Caught exception in looking up space point ptr... 0x31ad4500, 0x31ad2df0
Caught exception in looking up space point ptr... 0x499d8d00, 0x2e67bea0
Caught exception in looking up space point ptr... 0x2e67bea0, 0x499d8d00
Caught exception in looking up space point ptr... 0x53d75490, 0x53d75340
Caught exception in looking up space point ptr... 0x53d75340, 0x53d75050
Caught exception in looking up space point ptr... 0x5b64d6d0, 0x5b64d2e0
Caught exception in looking up space point ptr... 0x5b64d2e0, 0x5b64d430
Caught exception in looking up space point ptr... 0x2aec3720, 0x2aec35f0
Caught exception in looking up space point ptr... 0x2aec35f0, 0x2aec3720
++++>>>> total num hits: 64831, num free: 62323
04-Feb-2026 01:25:19 UTC  Closed output file "np02vd_raw_run042422_0050_df-s02-d1_dw_0_20260203T154148_reco_stage0_20260204T012519_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         101.288       145.996       197.347       148.244       25.9477        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       6.8742e-05    0.000127664   0.000285257   0.000118493   4.83701e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                12.0049       13.9827       20.0091       14.0112       1.4669         27     
produce:triggerrawdecoder:PDVDTriggerReader4      0.530184      0.638154       2.71351      0.559476       0.40749        27     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0179777     0.019906      0.0356978     0.0181939     0.0041844       27     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0877904     0.0890526     0.091664      0.0889117    0.00067167       27     
produce:beamevent:BeamEvent                      8.3612e-05    0.000251297   0.00112574    0.000211059   0.000188098      27     
produce:pdvddaphne:DAPHNEReaderPDVD                4.91315       6.75706       8.10671       7.23311      0.989807        27     
produce:ophit:OpHitFinder                         0.0359176     0.0559862     0.0872761     0.0615527     0.0157485       27     
produce:wclsdatavd:WireCellToolkit                 71.1467       102.353       129.264       105.723       18.6366        27     
produce:gaushit:GausHitFinder                     0.585127       1.60267       3.45856       1.51303      0.576726        27     
produce:nhitsfilter:NumberOfHitsFilter           0.000235083   0.000436724   0.000813295   0.000394977   0.000123981      27     
produce:reco3d:SpacePointSolver                    3.56525       12.5614       23.6759       11.4926       4.71217        27     
produce:hitpdune:DisambigFromSpacePoints          0.0560515     0.183111      0.609967      0.153185      0.104432        27     
produce:cluster3d:Cluster3D                       0.592107       2.64528       13.8001       2.0853        2.33745        27     
[art]:TriggerResults:TriggerResultInserter       2.1887e-05    4.15034e-05   0.000125997   3.6948e-05     2.162e-05       27     
end_path:out1:RootOutput                          3.708e-06    1.89452e-05   0.000268283    8.261e-06    4.91264e-05      27     
end_path:out1:RootOutput(write)                    3.3185        5.07009       6.81164       5.06472       1.07911        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4932.48 MB
  Peak resident set size usage (VmHWM): 2561.66 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042422_0050_df-s02-d1_dw_0_20260203T154148_reco_stage0_20260204T012519_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042422_0050_df-s02-d1_dw_0_20260203T154148_reco_stage0_20260204T012519_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042422_0050_df-s02-d1_dw_0_20260203T154148_reco_stage0_20260204T012519_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770164302.0,
    "core.end_time": 1770168321.0,
    "core.events": [
      8107,
      8113,
      8119,
      8125,
      8131,
      8137,
      8143,
      8149,
      8155,
      8161,
      8167,
      8173,
      8179,
      8185,
      8191,
      8197,
      8203,
      8209,
      8215,
      8221,
      8227,
      8233,
      8239,
      8245,
      8251,
      8257,
      8263
    ],
    "core.event_count": 27,
    "core.first_event_number": 8107,
    "core.last_event_number": 8263,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42422
    ],
    "core.runs_subruns": [
      4242200001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042422_0050_df-s02-d1_dw_0_20260203T154148.hdf5"
    }
  ]
}
justIN time: 2026-02-04 04:31:39 UTC       justIN version: 01.06.00