justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303650.178@dunegpschedd01.fnal.gov

Jobsub ID303650.178@dunegpschedd01.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:48:28
SiteUK_Imperial
EntryDUNE_T2_UK_London_IC_ceprod00
Last heartbeat2026-02-04 01:19:09
From worker nodeHostnamewh01.grid.hep.ph.ic.ac.uk
cpuinfoIntel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:16:31
Input filesvd-protodune:np02vd_raw_run042433_0458_df-s03-d0_dw_0_20260203T213611.hdf5
JobscriptExit code0
Real time1h (7149s)
CPU time1h (6161s = 86%)
Max RSS bytes3612983296 (3445 MiB)
Outputting started2026-02-04 01:15:52
Output files
Finished2026-02-04 01:19:09
Saved logsjustin-logs:303650.178-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

0x18cfc170, 0x40f10200
Caught exception in looking up space point ptr... 0x40f10200, 0x18cf6c40
Caught exception in looking up space point ptr... 0x169cd220, 0x76a0d510
Caught exception in looking up space point ptr... 0x76a0d510, 0x169cd220
Caught exception in looking up space point ptr... 0x15cf9000, 0x15cf9100
Caught exception in looking up space point ptr... 0x15cf9100, 0x58d38c60
Caught exception in looking up space point ptr... 0x7c99fdc0, 0x7c99fc30
Caught exception in looking up space point ptr... 0x7c99fc30, 0x7c99fdc0
Caught exception in looking up space point ptr... 0x3d2e2410, 0x3d2e2540
Caught exception in looking up space point ptr... 0x3d2e2670, 0x3d2e2410
Caught exception in looking up space point ptr... 0x815fc2d0, 0x815fc0f0
Caught exception in looking up space point ptr... 0x815fc0f0, 0x815fbf10
Caught exception in looking up space point ptr... 0x646081e0, 0x3d865590
Caught exception in looking up space point ptr... 0x3d865ad0, 0x646081e0
Caught exception in looking up space point ptr... 0x418b4a10, 0x62afe750
Caught exception in looking up space point ptr... 0x3c240780, 0x418b4a10
Caught exception in looking up space point ptr... 0x63e797e0, 0x63e798e0
Caught exception in looking up space point ptr... 0x63e798e0, 0x63e797e0
Caught exception in looking up space point ptr... 0x3708b2b0, 0x3fe8cfc0
Caught exception in looking up space point ptr... 0x3fe8cfc0, 0x3708b2b0
Caught exception in looking up space point ptr... 0x315699b0, 0x31566160
Caught exception in looking up space point ptr... 0x31566160, 0x315699b0
Caught exception in looking up space point ptr... 0x645d6df0, 0x645ce050
Caught exception in looking up space point ptr... 0x645ce050, 0x645d5a70
Caught exception in looking up space point ptr... 0x34e5c790, 0x3d9143a0
Caught exception in looking up space point ptr... 0x3d9143a0, 0x3d914140
Caught exception in looking up space point ptr... 0x9d62680, 0x51faebb0
Caught exception in looking up space point ptr... 0x51faebb0, 0x326c7d40
Caught exception in looking up space point ptr... 0x326c7d40, 0x2ac5db40
Caught exception in looking up space point ptr... 0x46c057c0, 0x3ceb7260
Caught exception in looking up space point ptr... 0x3ceb7260, 0x6c528980
Caught exception in looking up space point ptr... 0x59d70480, 0x59d702f0
Caught exception in looking up space point ptr... 0x59d702f0, 0x59d70610
Caught exception in looking up space point ptr... 0x645b7620, 0x645b74f0
Caught exception in looking up space point ptr... 0x645b74f0, 0x645b7620
Caught exception in looking up space point ptr... 0x3d812420, 0x3d812100
Caught exception in looking up space point ptr... 0x3d812100, 0x3d811f70
Caught exception in looking up space point ptr... 0x3d811f70, 0x3d816e10
Caught exception in looking up space point ptr... 0x3d816e10, 0x3d812420
Caught exception in looking up space point ptr... 0x76912fa0, 0x76912aa0
Caught exception in looking up space point ptr... 0x76912aa0, 0x76912fa0
Caught exception in looking up space point ptr... 0x76912fa0, 0x76912c30
Caught exception in looking up space point ptr... 0x76912c30, 0x76912fa0
Caught exception in looking up space point ptr... 0x383a65f0, 0x5dc621a0
Caught exception in looking up space point ptr... 0x5dc621a0, 0x5dc622e0
Caught exception in looking up space point ptr... 0x36fef030, 0x526fe720
Caught exception in looking up space point ptr... 0x526fe720, 0x7687ae10
Caught exception in looking up space point ptr... 0x7687ae10, 0x526fe720
Caught exception in looking up space point ptr... 0x526fe720, 0x76879280
Caught exception in looking up space point ptr... 0x36014db0, 0x36014c60
Caught exception in looking up space point ptr... 0x36014c60, 0x36014db0
Caught exception in looking up space point ptr... 0x34e65fd0, 0x45d14960
Caught exception in looking up space point ptr... 0x45d14960, 0x34e65fd0
Caught exception in looking up space point ptr... 0x42081350, 0x420744a0
Caught exception in looking up space point ptr... 0x420744a0, 0x42081350
Caught exception in looking up space point ptr... 0x3d0ed9d0, 0x8166ed10
Caught exception in looking up space point ptr... 0x8166ed10, 0x3d0ed610
Caught exception in looking up space point ptr... 0x36a44740, 0x36a44290
Caught exception in looking up space point ptr... 0x36a44290, 0x36a44420
Caught exception in looking up space point ptr... 0x3e3aed50, 0x3e3ad4d0
Caught exception in looking up space point ptr... 0x3e3ad4d0, 0x3e3aed50
Caught exception in looking up space point ptr... 0x49d549e0, 0x49d54ae0
Caught exception in looking up space point ptr... 0x49d54ae0, 0x49d54be0
Caught exception in looking up space point ptr... 0x2fc9a150, 0x2fc9a270
Caught exception in looking up space point ptr... 0x2fc9a270, 0x3d2fa0f0
Caught exception in looking up space point ptr... 0x16d0aa60, 0x16d051f0
Caught exception in looking up space point ptr... 0x16d051f0, 0x16d0aa60
++++>>>> total num hits: 112814, num free: 108986
04-Feb-2026 01:15:29 GMT  Closed output file "np02vd_raw_run042433_0458_df-s03-d0_dw_0_20260203T213611_reco_stage0_20260204T011529_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         237.06        370.152       438.601       390.543       59.2534        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       6.2681e-05    0.000118288   0.000216564   0.000104623   4.15153e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                43.8319       45.3888       50.1322       45.1406       1.43722        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.515089       0.57902      0.992159      0.547446      0.103454        19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0164049     0.0224164     0.0650392     0.0188539     0.0108341       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0164449      0.0166       0.0170232     0.0165888    0.000115995      19     
produce:beamevent:BeamEvent                      8.6016e-05    0.00019805    0.00126533    0.000138457   0.000252767      19     
produce:pdvddaphne:DAPHNEReaderPDVD                6.63443       7.79331       8.70726       7.84615      0.583095        19     
produce:ophit:OpHitFinder                         0.0382228     0.0469635     0.0538746     0.0489313    0.00512049       19     
produce:wclsdatavd:WireCellToolkit                 147.861       201.209       220.413       210.706       19.0579        19     
produce:gaushit:GausHitFinder                      1.87476       4.0442        5.78356       4.25847       1.19825        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000471122   0.000715001   0.00100204    0.000731559   0.000140289      19     
produce:reco3d:SpacePointSolver                    12.6063       28.3771       45.3884       26.0058       9.00115        19     
produce:hitpdune:DisambigFromSpacePoints          0.245311      0.535739       1.15592      0.588164      0.215426        19     
produce:cluster3d:Cluster3D                        3.64589       68.1193       120.001       87.6105       48.481         19     
[art]:TriggerResults:TriggerResultInserter       3.1795e-05    4.53439e-05   8.6645e-05    4.4752e-05    1.1407e-05       19     
end_path:out1:RootOutput                          9.361e-06    2.22104e-05   0.000209792   1.1757e-05    4.43386e-05      19     
end_path:out1:RootOutput(write)                    7.96389        13.99        19.2863       14.9193       2.96231        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5519.28 MB
  Peak resident set size usage (VmHWM): 3612.98 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0458_df-s03-d0_dw_0_20260203T213611_reco_stage0_20260204T011529_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0458_df-s03-d0_dw_0_20260203T213611_reco_stage0_20260204T011529_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0458_df-s03-d0_dw_0_20260203T213611_reco_stage0_20260204T011529_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770160607.0,
    "core.end_time": 1770167732.0,
    "core.events": [
      52220,
      52226,
      52232,
      52238,
      52244,
      52250,
      52256,
      52262,
      52268,
      52274,
      52280,
      52286,
      52292,
      52298,
      52304,
      52310,
      52316,
      52322,
      52328
    ],
    "core.event_count": 19,
    "core.first_event_number": 52220,
    "core.last_event_number": 52328,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0458_df-s03-d0_dw_0_20260203T213611.hdf5"
    }
  ]
}
justIN time: 2026-02-04 06:06:48 UTC       justIN version: 01.06.00