justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303650.176@dunegpschedd01.fnal.gov

Jobsub ID303650.176@dunegpschedd01.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:48:28
SiteUK_Imperial
EntryDUNE_T2_UK_London_IC_ceprod01
Last heartbeat2026-02-04 01:13:56
From worker nodeHostnamebf1770077716.novalocal
cpuinfoAMD EPYC-Genoa Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:16:55
Input filesvd-protodune:np02vd_raw_run042433_0160_df-s03-d1_dw_0_20260203T194257.hdf5
JobscriptExit code0
Real time1h (6773s)
CPU time52m (3160s = 46%)
Max RSS bytes3608543232 (3441 MiB)
Outputting started2026-02-04 01:09:50
Output files
Finished2026-02-04 01:13:56
Saved logsjustin-logs:303650.176-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

int ptr... 0x5dfc4740, 0x5dfb2ae0
Caught exception in looking up space point ptr... 0x503adcc0, 0x503ada70
Caught exception in looking up space point ptr... 0x503ada70, 0x503b7c50
Caught exception in looking up space point ptr... 0x8b8ae330, 0x16ee7280
Caught exception in looking up space point ptr... 0x16ee7280, 0x5f4e5380
Caught exception in looking up space point ptr... 0x81f35a60, 0x4af60bb0
Caught exception in looking up space point ptr... 0x4af60bb0, 0x81f40a20
Caught exception in looking up space point ptr... 0x154f8b70, 0x154e6a60
Caught exception in looking up space point ptr... 0x154f8d00, 0x154f8b70
Caught exception in looking up space point ptr... 0x154fb280, 0x154e9170
Caught exception in looking up space point ptr... 0x154fb410, 0x154fb280
Caught exception in looking up space point ptr... 0x31f778c0, 0xdba0f00
Caught exception in looking up space point ptr... 0xdba0f00, 0x31f9bca0
Caught exception in looking up space point ptr... 0x17b18ca0, 0x11853fc0
Caught exception in looking up space point ptr... 0x11853fc0, 0x11853e30
Caught exception in looking up space point ptr... 0x2a888080, 0x35cfb940
Caught exception in looking up space point ptr... 0x35cfb940, 0x2a887f10
Caught exception in looking up space point ptr... 0x5253cfa0, 0x48ac1120
Caught exception in looking up space point ptr... 0x59d35890, 0x5253cfa0
Caught exception in looking up space point ptr... 0x324c8930, 0x2b39bfa0
Caught exception in looking up space point ptr... 0x2b39bfa0, 0x324c8930
Caught exception in looking up space point ptr... 0x3249f8d0, 0x61ece870
Caught exception in looking up space point ptr... 0x3248f890, 0x3249f8d0
Caught exception in looking up space point ptr... 0x42dbf260, 0x11539100
Caught exception in looking up space point ptr... 0x42dbf3f0, 0x42dbf260
Caught exception in looking up space point ptr... 0xdba32a0, 0x3a208db0
Caught exception in looking up space point ptr... 0x3a208db0, 0x293bf800
Caught exception in looking up space point ptr... 0x54e61e90, 0x624b53a0
Caught exception in looking up space point ptr... 0x624b53a0, 0x624b5530
Caught exception in looking up space point ptr... 0x47652720, 0x47652590
Caught exception in looking up space point ptr... 0x47652590, 0x1650a940
Caught exception in looking up space point ptr... 0x545b19d0, 0x5459f8c0
Caught exception in looking up space point ptr... 0x545c37c0, 0x545b19d0
Caught exception in looking up space point ptr... 0x85d38070, 0x61319930
Caught exception in looking up space point ptr... 0x61319930, 0x85d38070
Caught exception in looking up space point ptr... 0x93e31eb0, 0x59f9fdd0
Caught exception in looking up space point ptr... 0x59f9fdd0, 0x93e31eb0
Caught exception in looking up space point ptr... 0x93e31eb0, 0x660e2780
Caught exception in looking up space point ptr... 0x660e2780, 0x93e31eb0
Caught exception in looking up space point ptr... 0x96210e0, 0x9620dc0
Caught exception in looking up space point ptr... 0x9620dc0, 0x9620a40
Caught exception in looking up space point ptr... 0x9620a40, 0x9620be0
Caught exception in looking up space point ptr... 0x9620be0, 0x96210e0
Caught exception in looking up space point ptr... 0x29630080, 0x476fe900
Caught exception in looking up space point ptr... 0x476fe900, 0x5155fd10
Caught exception in looking up space point ptr... 0x18726ae0, 0x18726d40
Caught exception in looking up space point ptr... 0x18726d40, 0x18726ae0
Caught exception in looking up space point ptr... 0x6628bb10, 0x3e5990d0
Caught exception in looking up space point ptr... 0x3e5990d0, 0x6628bb10
Caught exception in looking up space point ptr... 0x473c7340, 0x663cda30
Caught exception in looking up space point ptr... 0x663cda30, 0x294a4d60
Caught exception in looking up space point ptr... 0x5f7ad5f0, 0x842a4950
Caught exception in looking up space point ptr... 0x842a4950, 0x842a4a50
Caught exception in looking up space point ptr... 0x3a6da890, 0x3a6dab40
Caught exception in looking up space point ptr... 0x3a6dab40, 0x3a6da890
Caught exception in looking up space point ptr... 0x3a6da890, 0x3a6da9b0
Caught exception in looking up space point ptr... 0x3a6da9b0, 0x3a6da890
Caught exception in looking up space point ptr... 0x592be0b0, 0x2e3bdc70
Caught exception in looking up space point ptr... 0x2e3bdc70, 0x592be0b0
Caught exception in looking up space point ptr... 0x47304080, 0x47303a60
Caught exception in looking up space point ptr... 0x47303a60, 0x47304080
Caught exception in looking up space point ptr... 0x36e3cb20, 0x36e3cc20
Caught exception in looking up space point ptr... 0x36e3cc20, 0x36e3cb20
Caught exception in looking up space point ptr... 0x2aac0290, 0x348ccbd0
Caught exception in looking up space point ptr... 0x348ccbd0, 0x2aac0290
Caught exception in looking up space point ptr... 0x348d1ec0, 0x348b9e50
Caught exception in looking up space point ptr... 0x348b9e50, 0x348d1ec0
++++>>>> total num hits: 527008, num free: 518076
04-Feb-2026 01:09:39 GMT  Closed output file "np02vd_raw_run042433_0160_df-s03-d1_dw_0_20260203T194257_reco_stage0_20260204T010939_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         270.129       352.834       459.02        339.226       54.0052        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                        6.863e-05    0.000118583   0.000290549    9.108e-05    6.15705e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                128.115       185.377       267.298       176.275       41.3401        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.618544      0.924954       1.59616       0.87065      0.236901        19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0171481     0.0259507     0.113896       0.01944      0.0209695       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0172471     0.0200916     0.0493726     0.0185169    0.00694537       19     
produce:beamevent:BeamEvent                       6.746e-05    0.000204887   0.00101008    0.00012523    0.000222185      19     
produce:pdvddaphne:DAPHNEReaderPDVD                7.39067       14.9895       22.3178       14.3723       3.48648        19     
produce:ophit:OpHitFinder                         0.029697      0.0374175     0.0581581     0.0346159    0.00706056       19     
produce:wclsdatavd:WireCellToolkit                 82.4375       89.5324       99.8685       89.2006       4.01512        19     
produce:gaushit:GausHitFinder                      1.02403       1.65771       2.25146       1.73038      0.274467        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000211629   0.000329475   0.000807887   0.000296489   0.000129017      19     
produce:reco3d:SpacePointSolver                    12.0653       16.9961       27.5885       17.0187       3.93007        19     
produce:hitpdune:DisambigFromSpacePoints          0.177706      0.242971      0.332034      0.236436      0.0436811       19     
produce:cluster3d:Cluster3D                        2.50513       37.6915       62.551        42.4926       18.6956        19     
[art]:TriggerResults:TriggerResultInserter        1.954e-05    3.67608e-05   0.000143199    2.872e-05    2.81618e-05      19     
end_path:out1:RootOutput                          3.65e-06     3.50786e-05   0.000558907    4.91e-06     0.000123507      19     
end_path:out1:RootOutput(write)                    4.43572       5.28539       5.87397        5.345       0.406504        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5528.09 MB
  Peak resident set size usage (VmHWM): 3608.54 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0160_df-s03-d1_dw_0_20260203T194257_reco_stage0_20260204T010939_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0160_df-s03-d1_dw_0_20260203T194257_reco_stage0_20260204T010939_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0160_df-s03-d1_dw_0_20260203T194257_reco_stage0_20260204T010939_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770160627.0,
    "core.end_time": 1770167380.0,
    "core.events": [
      18249,
      18255,
      18261,
      18267,
      18273,
      18279,
      18285,
      18291,
      18297,
      18303,
      18309,
      18315,
      18321,
      18327,
      18333,
      18339,
      18345,
      18351,
      18357
    ],
    "core.event_count": 19,
    "core.first_event_number": 18249,
    "core.last_event_number": 18357,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0160_df-s03-d1_dw_0_20260203T194257.hdf5"
    }
  ]
}
justIN time: 2026-02-04 10:34:22 UTC       justIN version: 01.06.00