justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297923.110@dunegpschedd02.fnal.gov

Jobsub ID297923.110@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 23:00:26
SiteUK_Imperial
EntryDUNE_T2_UK_London_IC_ceprod03
Last heartbeat2026-02-04 01:45:27
From worker nodeHostnamewl01.grid.hep.ph.ic.ac.uk
cpuinfoAMD EPYC 7742 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-04 00:03:07
Input filesvd-protodune:np02vd_raw_run042433_0597_df-s03-d0_dw_0_20260203T222902.hdf5
JobscriptExit code0
Real time1h (5964s)
CPU time1h (5214s = 87%)
Max RSS bytes3591753728 (3425 MiB)
Outputting started2026-02-04 01:42:35
Output files
Finished2026-02-04 01:45:27
Saved logsjustin-logs:297923.110-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

oint ptr... 0x6ffb68e0, 0x6ffb6a70
Caught exception in looking up space point ptr... 0x787f3340, 0x4213a3b0
Caught exception in looking up space point ptr... 0x4213a3b0, 0x787f3340
Caught exception in looking up space point ptr... 0x7b1d3670, 0x713a6910
Caught exception in looking up space point ptr... 0x7b1d34e0, 0x7b1d3670
Caught exception in looking up space point ptr... 0x92d90d0, 0x5cf25c80
Caught exception in looking up space point ptr... 0x92d92f0, 0x92d90d0
Caught exception in looking up space point ptr... 0xdc87f90, 0x5cf281d0
Caught exception in looking up space point ptr... 0xdc880a0, 0xdc87f90
Caught exception in looking up space point ptr... 0x59c91b10, 0x2e2d70c0
Caught exception in looking up space point ptr... 0x2e2d70c0, 0x2e2d6fc0
Caught exception in looking up space point ptr... 0x6c5e5080, 0xdbc8ef0
Caught exception in looking up space point ptr... 0xdbc8ef0, 0xdbc0d30
Caught exception in looking up space point ptr... 0x654a2e60, 0x65487130
Caught exception in looking up space point ptr... 0x654a2ff0, 0x654a2e60
Caught exception in looking up space point ptr... 0xd0983e0, 0x353abc20
Caught exception in looking up space point ptr... 0x6c42e850, 0xd0983e0
Caught exception in looking up space point ptr... 0x168b1260, 0x17b76070
Caught exception in looking up space point ptr... 0x17b76070, 0x168affe0
Caught exception in looking up space point ptr... 0x31cc2cd0, 0x81e6ed40
Caught exception in looking up space point ptr... 0x81e6ed40, 0x31ca8ba0
Caught exception in looking up space point ptr... 0x3a7ec400, 0x31cc2cd0
Caught exception in looking up space point ptr... 0x2d457d60, 0x5c41cc40
Caught exception in looking up space point ptr... 0x32dfbed0, 0x2d457d60
Caught exception in looking up space point ptr... 0x11b31cb0, 0x32ddc4e0
Caught exception in looking up space point ptr... 0x32dcf0c0, 0x11b31cb0
Caught exception in looking up space point ptr... 0x8d86bf0, 0x751741d0
Caught exception in looking up space point ptr... 0x751741d0, 0x75162ed0
Caught exception in looking up space point ptr... 0x14b81ec0, 0x14b6f590
Caught exception in looking up space point ptr... 0x14b6f590, 0x14b5ccb0
Caught exception in looking up space point ptr... 0x19d8edb0, 0x186c8100
Caught exception in looking up space point ptr... 0x186c8100, 0x19d8edb0
Caught exception in looking up space point ptr... 0x14a6df10, 0x32bdf560
Caught exception in looking up space point ptr... 0x32bdf560, 0x14a6dd80
Caught exception in looking up space point ptr... 0x5e900e70, 0x571bfc10
Caught exception in looking up space point ptr... 0x571bfc10, 0x571c0460
Caught exception in looking up space point ptr... 0x571c0460, 0x5e8de450
Caught exception in looking up space point ptr... 0x5e8b5350, 0x103643d0
Caught exception in looking up space point ptr... 0x10364620, 0x5e8b5350
Caught exception in looking up space point ptr... 0x421c0280, 0x634026e0
Caught exception in looking up space point ptr... 0x634026e0, 0x421c0280
Caught exception in looking up space point ptr... 0x654e5160, 0x3213f2c0
Caught exception in looking up space point ptr... 0x3213f2c0, 0x654e5160
Caught exception in looking up space point ptr... 0x6f30be60, 0x3313be90
Caught exception in looking up space point ptr... 0x3313be90, 0x6f30be60
Caught exception in looking up space point ptr... 0x633f5100, 0x633f5290
Caught exception in looking up space point ptr... 0x633f5290, 0x633f4f70
Caught exception in looking up space point ptr... 0x633f4f70, 0x633f4c50
Caught exception in looking up space point ptr... 0x633f4c50, 0x633f4de0
Caught exception in looking up space point ptr... 0x3bbb2790, 0x3bbb1a80
Caught exception in looking up space point ptr... 0x3bbb1a80, 0x3bbb3240
Caught exception in looking up space point ptr... 0x30dae620, 0x30dae7b0
Caught exception in looking up space point ptr... 0x30dae7b0, 0x30dae620
Caught exception in looking up space point ptr... 0x30dae620, 0x3d64b5e0
Caught exception in looking up space point ptr... 0x3d64b5e0, 0x30dae620
Caught exception in looking up space point ptr... 0x12a5e840, 0x3d1d02e0
Caught exception in looking up space point ptr... 0x3d1d02e0, 0x3b363560
Caught exception in looking up space point ptr... 0x11b97920, 0x11b97030
Caught exception in looking up space point ptr... 0x11b97030, 0x54b55c10
Caught exception in looking up space point ptr... 0x5a793090, 0x32bfd890
Caught exception in looking up space point ptr... 0x32bfd890, 0x5a793090
Caught exception in looking up space point ptr... 0x17b37da0, 0x17b35900
Caught exception in looking up space point ptr... 0x17b35900, 0x17b36c10
Caught exception in looking up space point ptr... 0x74859180, 0x466e6e40
Caught exception in looking up space point ptr... 0x466e6e40, 0x74859180
Caught exception in looking up space point ptr... 0x6062f080, 0x5803dab0
Caught exception in looking up space point ptr... 0x5803dab0, 0x58e467c0
++++>>>> total num hits: 558338, num free: 545917
04-Feb-2026 01:42:21 GMT  Closed output file "np02vd_raw_run042433_0597_df-s03-d0_dw_0_20260203T222902_reco_stage0_20260204T014221_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         191.024       307.711       401.559        297.6        48.2216        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       6.3742e-05    0.000152905   0.000283579   0.000125714   6.15881e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                15.4961       21.0289       54.4214       18.1854       8.51203        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.509657      0.604507      0.721634      0.595448      0.0570317       19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0163152     0.0274294     0.0615055     0.0199043     0.0139823       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0161762     0.0168673     0.0179789     0.0168699    0.000437876      19     
produce:beamevent:BeamEvent                      9.2163e-05    0.000255434   0.00134926    0.000160285   0.00031526       19     
produce:pdvddaphne:DAPHNEReaderPDVD                3.5269        4.6565        5.86548       4.7732       0.662017        19     
produce:ophit:OpHitFinder                         0.0359895     0.0480742     0.0542659     0.0502794    0.00523019       19     
produce:wclsdatavd:WireCellToolkit                 133.688       167.997       194.903       180.863       24.2599        19     
produce:gaushit:GausHitFinder                      1.23702       3.07476       4.38062       3.07044      0.867499        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000429564   0.000715158   0.00109943    0.000702152   0.00018046       19     
produce:reco3d:SpacePointSolver                    16.8089       30.0209       57.1047       27.9505       10.1182        19     
produce:hitpdune:DisambigFromSpacePoints          0.202138      0.459068      0.744016      0.501434      0.141677        19     
produce:cluster3d:Cluster3D                        2.8196        67.9813       106.352       77.0205       33.5122        19     
[art]:TriggerResults:TriggerResultInserter       3.3981e-05    4.91991e-05   0.000118743   4.2801e-05    2.02799e-05      19     
end_path:out1:RootOutput                          9.811e-06    1.87016e-05   0.000102143    1.189e-05    2.04676e-05      19     
end_path:out1:RootOutput(write)                    7.8097        11.7515       14.412        11.2748       1.91775        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5532.94 MB
  Peak resident set size usage (VmHWM): 3591.75 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0597_df-s03-d0_dw_0_20260203T222902_reco_stage0_20260204T014221_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0597_df-s03-d0_dw_0_20260203T222902_reco_stage0_20260204T014221_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0597_df-s03-d0_dw_0_20260203T222902_reco_stage0_20260204T014221_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770163403.0,
    "core.end_time": 1770169343.0,
    "core.events": [
      68066,
      68072,
      68078,
      68084,
      68090,
      68096,
      68102,
      68108,
      68114,
      68120,
      68126,
      68132,
      68138,
      68144,
      68150,
      68156,
      68162,
      68168,
      68174
    ],
    "core.event_count": 19,
    "core.first_event_number": 68066,
    "core.last_event_number": 68174,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0597_df-s03-d0_dw_0_20260203T222902.hdf5"
    }
  ]
}
justIN time: 2026-02-04 09:54:57 UTC       justIN version: 01.06.00