justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297923.116@dunegpschedd02.fnal.gov

Jobsub ID297923.116@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 23:00:26
SiteUK_Imperial
EntryDUNE_T2_UK_London_IC_ceprod03
Last heartbeat2026-02-04 01:14:45
From worker nodeHostnamewl01.grid.hep.ph.ic.ac.uk
cpuinfoAMD EPYC 7742 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-04 00:03:30
Input filesvd-protodune:np02vd_raw_run042433_0600_df-s03-d0_dw_0_20260203T223009.hdf5
JobscriptExit code0
Real time1h (4115s)
CPU time59m (3584s = 87%)
Max RSS bytes3379810304 (3223 MiB)
Outputting started2026-02-04 01:12:07
Output files
Finished2026-02-04 01:14:45
Saved logsjustin-logs:297923.116-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

point ptr... 0x2b2e7070, 0x2b2f3f30
Caught exception in looking up space point ptr... 0x2b2f3f30, 0x2b2ede10
Caught exception in looking up space point ptr... 0x2b2ede10, 0x2b2f3f30
Caught exception in looking up space point ptr... 0x38f53130, 0x2b13cb90
Caught exception in looking up space point ptr... 0x2b13cb90, 0x38c69b00
Caught exception in looking up space point ptr... 0x40595690, 0x5112a2a0
Caught exception in looking up space point ptr... 0x5112a2a0, 0x3ba5cfa0
Caught exception in looking up space point ptr... 0x5f51f220, 0x686db0e0
Caught exception in looking up space point ptr... 0x686db0e0, 0x5f51f220
Caught exception in looking up space point ptr... 0x15a49140, 0x584e7280
Caught exception in looking up space point ptr... 0x584e7280, 0x584bdf90
Caught exception in looking up space point ptr... 0x6255a820, 0x39877d60
Caught exception in looking up space point ptr... 0x39877d60, 0x6255a820
Caught exception in looking up space point ptr... 0x6255a820, 0xf167500
Caught exception in looking up space point ptr... 0xf167500, 0x6255a820
Caught exception in looking up space point ptr... 0x59a76250, 0x59a760c0
Caught exception in looking up space point ptr... 0x59a95800, 0x59a76250
Caught exception in looking up space point ptr... 0x39e11bd0, 0x39e2be80
Caught exception in looking up space point ptr... 0x39e2be80, 0x39e0dc70
Caught exception in looking up space point ptr... 0x6139bbb0, 0x6139ac40
Caught exception in looking up space point ptr... 0x6139ac40, 0x39296100
Caught exception in looking up space point ptr... 0x3f8bd6c0, 0x1155f270
Caught exception in looking up space point ptr... 0x1155f270, 0x3f8bcd30
Caught exception in looking up space point ptr... 0x3f8bcd30, 0x3f8bce80
Caught exception in looking up space point ptr... 0x3f8bce80, 0x3f8bd6c0
Caught exception in looking up space point ptr... 0x43d8d550, 0x3aeb57d0
Caught exception in looking up space point ptr... 0x3aeb57d0, 0x38fe86b0
Caught exception in looking up space point ptr... 0xf16ac20, 0xf16af40
Caught exception in looking up space point ptr... 0xf16af40, 0xf16b0d0
Caught exception in looking up space point ptr... 0x40048b40, 0x400489b0
Caught exception in looking up space point ptr... 0x400489b0, 0x3363b8f0
Caught exception in looking up space point ptr... 0x63788440, 0x4abb59c0
Caught exception in looking up space point ptr... 0x4abb59c0, 0x307408b0
Caught exception in looking up space point ptr... 0x73ae2160, 0x5fde39a0
Caught exception in looking up space point ptr... 0x5fde39a0, 0x5fde3570
Caught exception in looking up space point ptr... 0xf29a690, 0xf29a390
Caught exception in looking up space point ptr... 0xf29a390, 0xf29a690
Caught exception in looking up space point ptr... 0x4dceacd0, 0x4dceab40
Caught exception in looking up space point ptr... 0x4dceab40, 0x4dceacd0
Caught exception in looking up space point ptr... 0x3360d050, 0x107ce250
Caught exception in looking up space point ptr... 0x107ce250, 0x3360d050
Caught exception in looking up space point ptr... 0x4734eef0, 0x4735e940
Caught exception in looking up space point ptr... 0x4735e940, 0x4734eef0
Caught exception in looking up space point ptr... 0x6e706560, 0x6e706880
Caught exception in looking up space point ptr... 0x6e706880, 0x6e706560
Caught exception in looking up space point ptr... 0x170217a0, 0x59de2f00
Caught exception in looking up space point ptr... 0x59de2f00, 0x170217a0
Caught exception in looking up space point ptr... 0x3794dab0, 0x3794d860
Caught exception in looking up space point ptr... 0x3794d860, 0x3794dab0
Caught exception in looking up space point ptr... 0x39e34300, 0x39e33d50
Caught exception in looking up space point ptr... 0x39e33d50, 0x39e32420
Caught exception in looking up space point ptr... 0x3ae2eff0, 0x3ae2eaa0
Caught exception in looking up space point ptr... 0x3ae2eaa0, 0x3ae2eff0
Caught exception in looking up space point ptr... 0x1a65d860, 0x2b2d37f0
Caught exception in looking up space point ptr... 0x2b2d37f0, 0x1a65d860
Caught exception in looking up space point ptr... 0x107c5090, 0x107c4f20
Caught exception in looking up space point ptr... 0x107c4f20, 0x107c5090
Caught exception in looking up space point ptr... 0x61b48e30, 0x38f416c0
Caught exception in looking up space point ptr... 0x38f416c0, 0x62d73c90
Caught exception in looking up space point ptr... 0x59a9d360, 0x4e440340
Caught exception in looking up space point ptr... 0x4e440340, 0x59a9d360
Caught exception in looking up space point ptr... 0x535df580, 0x535df3a0
Caught exception in looking up space point ptr... 0x535df3a0, 0x535df710
Caught exception in looking up space point ptr... 0x51383900, 0x92acc50
Caught exception in looking up space point ptr... 0x92acc50, 0x51383900
Caught exception in looking up space point ptr... 0x535dbcd0, 0x535db450
Caught exception in looking up space point ptr... 0x535db450, 0x535dbcd0
++++>>>> total num hits: 96205, num free: 92875
04-Feb-2026 01:11:55 GMT  Closed output file "np02vd_raw_run042433_0600_df-s03-d0_dw_0_20260203T223009_reco_stage0_20260204T011155_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         185.259       211.341       256.867       202.019       23.8398        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       8.3313e-05    0.000130771   0.000273039   0.000110853   4.77781e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                14.0319       16.6894       20.5102       16.635        2.01699        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.483946       0.49736      0.517449      0.499635      0.0100541       19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.016161      0.0164894     0.0172414     0.0164335    0.000249089      19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0161126     0.0164036     0.0169207     0.0163005    0.000259857      19     
produce:beamevent:BeamEvent                      8.1092e-05    0.00022548    0.00140999    0.000134054   0.000307619      19     
produce:pdvddaphne:DAPHNEReaderPDVD                3.38056       4.17339       5.32403        3.973       0.520485        19     
produce:ophit:OpHitFinder                         0.0354974     0.0477665     0.0568085     0.0466846    0.00517433       19     
produce:wclsdatavd:WireCellToolkit                 128.756       144.289       183.861       135.646       16.2125        19     
produce:gaushit:GausHitFinder                      1.39679       1.72746       2.83013       1.57917      0.341166        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000423803   0.000601096   0.00118711    0.000564888   0.000179593      19     
produce:reco3d:SpacePointSolver                    20.8768       29.207        40.7727       29.1219       6.12809        19     
produce:hitpdune:DisambigFromSpacePoints          0.190733      0.292381      0.432705      0.273575      0.0691305       19     
produce:cluster3d:Cluster3D                        3.1274        4.77086       7.42933       4.41526       1.10954        19     
[art]:TriggerResults:TriggerResultInserter       2.7931e-05    4.01559e-05   0.000114934   3.6201e-05    1.84787e-05      19     
end_path:out1:RootOutput                          7.571e-06    3.75224e-05   0.000244678    9.36e-06     7.20472e-05      19     
end_path:out1:RootOutput(write)                    8.05877       9.57087       11.6106       9.1651        1.13102        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5320.13 MB
  Peak resident set size usage (VmHWM): 3379.81 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0600_df-s03-d0_dw_0_20260203T223009_reco_stage0_20260204T011155_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0600_df-s03-d0_dw_0_20260203T223009_reco_stage0_20260204T011155_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0600_df-s03-d0_dw_0_20260203T223009_reco_stage0_20260204T011155_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770163424.0,
    "core.end_time": 1770167517.0,
    "core.events": [
      68408,
      68414,
      68420,
      68426,
      68432,
      68438,
      68444,
      68450,
      68456,
      68462,
      68468,
      68474,
      68480,
      68486,
      68492,
      68498,
      68504,
      68510,
      68516
    ],
    "core.event_count": 19,
    "core.first_event_number": 68408,
    "core.last_event_number": 68516,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0600_df-s03-d0_dw_0_20260203T223009.hdf5"
    }
  ]
}
justIN time: 2026-02-04 04:29:12 UTC       justIN version: 01.06.00