justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303678.1@dunegpschedd01.fnal.gov

Jobsub ID303678.1@dunegpschedd01.fnal.gov
Workflow ID12718
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 23:48:27
SiteUK_RAL-Tier1
EntryLIGO_UK_RAL_arc_ce04
Last heartbeat2026-02-04 02:39:18
From worker nodeHostnamedune001-1520896.0-lcg2379.gridpp.rl.ac.uk
cpuinfoIntel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit216000 (60 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-04 00:18:11
Input filesvd-protodune:np02vd_raw_run042422_0125_df-s04-d1_dw_0_20260203T162223.hdf5
JobscriptExit code0
Real time2h (8358s)
CPU time1h (4137s = 49%)
Max RSS bytes2583048192 (2463 MiB)
Outputting started2026-02-04 02:37:30
Output files
Finished2026-02-04 02:39:18
Saved logsjustin-logs:303678.1-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

exception in looking up space point ptr... 0x57135fb0, 0x57135d60
Caught exception in looking up space point ptr... 0x7e7de260, 0x6ea37d80
Caught exception in looking up space point ptr... 0x6ea37d80, 0x7e7de260
Caught exception in looking up space point ptr... 0x55f74720, 0x55f74bd0
Caught exception in looking up space point ptr... 0x55f74bd0, 0x55f74720
Caught exception in looking up space point ptr... 0x4d683350, 0x6c936630
Caught exception in looking up space point ptr... 0x6c936630, 0x4a4e8180
Caught exception in looking up space point ptr... 0x214a64b0, 0x571205e0
Caught exception in looking up space point ptr... 0x571205e0, 0x214a64b0
Caught exception in looking up space point ptr... 0x78e13150, 0x6c8eead0
Caught exception in looking up space point ptr... 0x6c8eead0, 0x78e13150
Caught exception in looking up space point ptr... 0x66b18df0, 0x66b18f20
Caught exception in looking up space point ptr... 0x66b18f20, 0x66b18df0
Caught exception in looking up space point ptr... 0x74590400, 0x6edf1700
Caught exception in looking up space point ptr... 0x6edf1700, 0x7458ff20
Caught exception in looking up space point ptr... 0x687612a0, 0x2f6fd290
Caught exception in looking up space point ptr... 0x2f6fd290, 0x65fb5d00
Caught exception in looking up space point ptr... 0x449779a0, 0x6be75cb0
Caught exception in looking up space point ptr... 0x6be75cb0, 0x6f561770
Caught exception in looking up space point ptr... 0x41f83f90, 0x41f83d30
Caught exception in looking up space point ptr... 0x41f83d30, 0x41f83c00
Caught exception in looking up space point ptr... 0x617518d0, 0x61753550
Caught exception in looking up space point ptr... 0x61753550, 0x61750bc0
Caught exception in looking up space point ptr... 0x41f28670, 0x5eff9b00
Caught exception in looking up space point ptr... 0x5eff9b00, 0x41f28670
Caught exception in looking up space point ptr... 0x56a17510, 0x5dcaa9c0
Caught exception in looking up space point ptr... 0x5dcaa9c0, 0x4d2535d0
Caught exception in looking up space point ptr... 0x4985d4e0, 0x445e93e0
Caught exception in looking up space point ptr... 0x445e93e0, 0x760730f0
Caught exception in looking up space point ptr... 0x54f144a0, 0x42311210
Caught exception in looking up space point ptr... 0x42311210, 0x54f144a0
Caught exception in looking up space point ptr... 0x6f46c710, 0x6ac09ba0
Caught exception in looking up space point ptr... 0x6ac09ba0, 0x6ac09d50
Caught exception in looking up space point ptr... 0x43b895a0, 0x43b89910
Caught exception in looking up space point ptr... 0x43b89910, 0x43b895a0
Caught exception in looking up space point ptr... 0x4a8cc450, 0x4a8cc2c0
Caught exception in looking up space point ptr... 0x4a8cc2c0, 0x4a8cc130
Caught exception in looking up space point ptr... 0x5b8007e0, 0x76c96550
Caught exception in looking up space point ptr... 0x76c96550, 0x76c96460
Caught exception in looking up space point ptr... 0x69c3d330, 0x76f94be0
Caught exception in looking up space point ptr... 0x76f94be0, 0x69c3d330
Caught exception in looking up space point ptr... 0x55b86330, 0x55b863d0
Caught exception in looking up space point ptr... 0x55b863d0, 0x481449a0
Caught exception in looking up space point ptr... 0x43dce6e0, 0x43dce500
Caught exception in looking up space point ptr... 0x43dce500, 0x43dce360
Caught exception in looking up space point ptr... 0x2f392420, 0x2f3926c0
Caught exception in looking up space point ptr... 0x2f3926c0, 0x2f3925e0
Caught exception in looking up space point ptr... 0x452fec70, 0x43b299c0
Caught exception in looking up space point ptr... 0x43b299c0, 0x452fec70
Caught exception in looking up space point ptr... 0x6c7c9270, 0x4e4c52e0
Caught exception in looking up space point ptr... 0x4e4c52e0, 0x6c7c9270
Caught exception in looking up space point ptr... 0x5dd598f0, 0x2483b2e0
Caught exception in looking up space point ptr... 0x2483b2e0, 0x5dd598f0
Caught exception in looking up space point ptr... 0x6b96f430, 0x481d8710
Caught exception in looking up space point ptr... 0x481d8710, 0x4a50e490
Caught exception in looking up space point ptr... 0x41983cf0, 0x22d38580
Caught exception in looking up space point ptr... 0x22d38580, 0x41983cf0
Caught exception in looking up space point ptr... 0x41983cf0, 0x5dd589e0
Caught exception in looking up space point ptr... 0x5dd589e0, 0x41983cf0
Caught exception in looking up space point ptr... 0x743d2150, 0x2f188980
Caught exception in looking up space point ptr... 0x2f188980, 0x743d2250
Caught exception in looking up space point ptr... 0x3fe52f30, 0x43dd05b0
Caught exception in looking up space point ptr... 0x43dd05b0, 0x43dd0740
Caught exception in looking up space point ptr... 0x6a44ea10, 0x75f9b2b0
Caught exception in looking up space point ptr... 0x75f9b2b0, 0x6a44ea10
++++>>>> total num hits: 45249, num free: 43253
04-Feb-2026 02:37:19 UTC  Closed output file "np02vd_raw_run042422_0125_df-s04-d1_dw_0_20260203T162223_reco_stage0_20260204T023719_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         234.134       306.151       398.89        304.056       30.7019        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                        6.646e-05    0.000104718   0.000206303   9.7827e-05    2.71259e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                96.6995       144.511       210.248       143.242       26.5942        27     
produce:triggerrawdecoder:PDVDTriggerReader4      0.606419      0.969848       3.9843       0.838916      0.613588        27     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0180224     0.0226992     0.074928      0.0182756     0.0112371       27     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0884616     0.0934212     0.131495      0.0896014    0.00935135       27     
produce:beamevent:BeamEvent                      0.000116203   0.000237552   0.00155354    0.00018172    0.000261333      27     
produce:pdvddaphne:DAPHNEReaderPDVD                16.4315       21.5184       27.4713       22.1387       2.85982        27     
produce:ophit:OpHitFinder                         0.0377397     0.0654848     0.0909243     0.0691358     0.0151238       27     
produce:wclsdatavd:WireCellToolkit                 76.6121       110.703       137.282       114.819       14.6253        27     
produce:gaushit:GausHitFinder                      0.84016       1.55824       2.5827        1.43944      0.446785        27     
produce:nhitsfilter:NumberOfHitsFilter           0.000302739   0.000491321   0.000717812   0.000471899   0.000107116      27     
produce:reco3d:SpacePointSolver                    5.39462       14.7078       40.8268       14.8492       6.65544        27     
produce:hitpdune:DisambigFromSpacePoints          0.0982061     0.215829       0.86095      0.191375      0.134791        27     
produce:cluster3d:Cluster3D                       0.900357       6.27638       105.823       2.40045       19.5432        27     
[art]:TriggerResults:TriggerResultInserter       2.4674e-05    4.50682e-05   9.5755e-05    4.2046e-05    1.39966e-05      27     
end_path:out1:RootOutput                          3.673e-06    8.39056e-06   3.5288e-05     6.183e-06    6.30694e-06      27     
end_path:out1:RootOutput(write)                    3.71676       5.46941       6.65509       5.67538      0.874961        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5021.22 MB
  Peak resident set size usage (VmHWM): 2583.05 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042422_0125_df-s04-d1_dw_0_20260203T162223_reco_stage0_20260204T023719_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042422_0125_df-s04-d1_dw_0_20260203T162223_reco_stage0_20260204T023719_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042422_0125_df-s04-d1_dw_0_20260203T162223_reco_stage0_20260204T023719_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770164301.0,
    "core.end_time": 1770172641.0,
    "core.events": [
      20273,
      20279,
      20285,
      20291,
      20297,
      20303,
      20309,
      20315,
      20321,
      20327,
      20333,
      20339,
      20345,
      20351,
      20357,
      20363,
      20369,
      20375,
      20381,
      20387,
      20393,
      20399,
      20405,
      20411,
      20417,
      20423,
      20429
    ],
    "core.event_count": 27,
    "core.first_event_number": 20273,
    "core.last_event_number": 20429,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42422
    ],
    "core.runs_subruns": [
      4242200001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042422_0125_df-s04-d1_dw_0_20260203T162223.hdf5"
    }
  ]
}
justIN time: 2026-02-04 04:31:55 UTC       justIN version: 01.06.00