justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303652.87@dunegpschedd01.fnal.gov

Jobsub ID303652.87@dunegpschedd01.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:50:26
SiteUK_Imperial
EntryDUNE_T2_UK_London_IC_ceprod03
Last heartbeat2026-02-04 00:31:57
From worker nodeHostnamewm21.grid.hep.ph.ic.ac.uk
cpuinfoAMD EPYC 7742 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:28:50
Input filesvd-protodune:np02vd_raw_run042433_0533_df-s04-d1_dw_0_20260203T220441.hdf5
JobscriptExit code0
Real time1h (3641s)
CPU time53m (3211s = 88%)
Max RSS bytes3427647488 (3268 MiB)
Outputting started2026-02-04 00:29:32
Output files
Finished2026-02-04 00:31:57
Saved logsjustin-logs:303652.87-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

nt ptr... 0x319139f0, 0x4dbb0d10
Caught exception in looking up space point ptr... 0x77dd6e30, 0x77ddc4e0
Caught exception in looking up space point ptr... 0x77ddc4e0, 0x77ddbc90
Caught exception in looking up space point ptr... 0x61830d90, 0x61822980
Caught exception in looking up space point ptr... 0x61822980, 0x61822b30
Caught exception in looking up space point ptr... 0x828226a0, 0x82822d80
Caught exception in looking up space point ptr... 0x82822d80, 0x828226a0
Caught exception in looking up space point ptr... 0x3b392c20, 0xa4b3170
Caught exception in looking up space point ptr... 0xa4b3170, 0x3b392c20
Caught exception in looking up space point ptr... 0x587f4950, 0x587f4bf0
Caught exception in looking up space point ptr... 0x587f4bf0, 0x587f4950
Caught exception in looking up space point ptr... 0x11ce8e20, 0x12e3c480
Caught exception in looking up space point ptr... 0x12e3c480, 0x11ce8e20
Caught exception in looking up space point ptr... 0x11ce8e20, 0x12e07a80
Caught exception in looking up space point ptr... 0x12e07a80, 0x11ce8e20
Caught exception in looking up space point ptr... 0x6ce884f0, 0x6ce88410
Caught exception in looking up space point ptr... 0x6ce88410, 0x6ce884f0
Caught exception in looking up space point ptr... 0x6ce884f0, 0x6ce88330
Caught exception in looking up space point ptr... 0x6ce88330, 0x6ce884f0
Caught exception in looking up space point ptr... 0x2ecc15c0, 0x5ea6f870
Caught exception in looking up space point ptr... 0x5ea6f870, 0x2ecc15c0
Caught exception in looking up space point ptr... 0x3073b7c0, 0x63f6b000
Caught exception in looking up space point ptr... 0x63f6b000, 0x3073b7c0
Caught exception in looking up space point ptr... 0x3073b7c0, 0x379cfc50
Caught exception in looking up space point ptr... 0x379cfc50, 0x3073b7c0
Caught exception in looking up space point ptr... 0x6b35aaa0, 0x6b35a910
Caught exception in looking up space point ptr... 0x6b35a910, 0x6b358fb0
Caught exception in looking up space point ptr... 0x54249620, 0x54249c60
Caught exception in looking up space point ptr... 0x54249c60, 0x54249ad0
Caught exception in looking up space point ptr... 0x6f04b2f0, 0x6f04b0a0
Caught exception in looking up space point ptr... 0x6f04b0a0, 0x6f04b2f0
Caught exception in looking up space point ptr... 0x149372f0, 0x1493a430
Caught exception in looking up space point ptr... 0x1493a430, 0x1493a570
Caught exception in looking up space point ptr... 0x40226d10, 0x5e987940
Caught exception in looking up space point ptr... 0x5e987940, 0x40226d10
Caught exception in looking up space point ptr... 0x40226d10, 0x12601200
Caught exception in looking up space point ptr... 0x12601200, 0x40226d10
Caught exception in looking up space point ptr... 0x8281db90, 0xdb62d30
Caught exception in looking up space point ptr... 0xdb62d30, 0x8281e120
Caught exception in looking up space point ptr... 0x640720c0, 0x72c14920
Caught exception in looking up space point ptr... 0x72c14920, 0x640720c0
Caught exception in looking up space point ptr... 0xf8aa640, 0xf8a8ae0
Caught exception in looking up space point ptr... 0xf8a8ae0, 0xf8aa640
Caught exception in looking up space point ptr... 0x58813b60, 0x58813ee0
Caught exception in looking up space point ptr... 0x58813ee0, 0x58813b60
Caught exception in looking up space point ptr... 0x61c402c0, 0x61c3a8f0
Caught exception in looking up space point ptr... 0x61c3a8f0, 0x61c3ec50
Caught exception in looking up space point ptr... 0x53985f70, 0x61701f00
Caught exception in looking up space point ptr... 0x61701f00, 0x53985f70
Caught exception in looking up space point ptr... 0x3a8e7fd0, 0x58332db0
Caught exception in looking up space point ptr... 0x58332db0, 0x3a8e7fd0
Caught exception in looking up space point ptr... 0x152a02c0, 0x4c6506e0
Caught exception in looking up space point ptr... 0x4c6506e0, 0x152a02c0
Caught exception in looking up space point ptr... 0x14460290, 0x14460530
Caught exception in looking up space point ptr... 0x14460530, 0x144603e0
Caught exception in looking up space point ptr... 0x4c3ff630, 0x46bf97e0
Caught exception in looking up space point ptr... 0x46bf97e0, 0x77c42b50
Caught exception in looking up space point ptr... 0x1978ba90, 0x19776150
Caught exception in looking up space point ptr... 0x19776150, 0x1978ba90
Caught exception in looking up space point ptr... 0x33fdcfa0, 0x6ce88d10
Caught exception in looking up space point ptr... 0x6ce88d10, 0x33fdd080
Caught exception in looking up space point ptr... 0x31127470, 0x54294cb0
Caught exception in looking up space point ptr... 0x54294cb0, 0x31127470
Caught exception in looking up space point ptr... 0x6f043080, 0x6f052080
Caught exception in looking up space point ptr... 0x6f052080, 0x6f051e20
Caught exception in looking up space point ptr... 0x19334570, 0x19335d50
Caught exception in looking up space point ptr... 0x19335d50, 0x19335c70
++++>>>> total num hits: 96451, num free: 92618
04-Feb-2026 00:29:21 GMT  Closed output file "np02vd_raw_run042433_0533_df-s04-d1_dw_0_20260203T220441_reco_stage0_20260204T002921_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         173.13        186.578       207.34        185.583       9.28102        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       6.6375e-05    0.000100224   0.000188476   8.8457e-05    2.98949e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                12.575        14.8222       18.2325       14.6533       1.19741        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.480063      0.486841      0.499594      0.482554     0.00720028       19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.015942      0.0170939     0.0327228     0.0162563     0.0036875       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0159487     0.0161535     0.0166116     0.0161087    0.000171773      19     
produce:beamevent:BeamEvent                      5.7558e-05    0.000129334   0.000857847   8.2205e-05    0.000173084      19     
produce:pdvddaphne:DAPHNEReaderPDVD                3.23763       3.50707       3.89526       3.49767      0.184282        19     
produce:ophit:OpHitFinder                         0.0266676     0.0375132     0.044025      0.0383013    0.00480471       19     
produce:wclsdatavd:WireCellToolkit                 117.056       126.707       150.533       125.125       7.57229        19     
produce:gaushit:GausHitFinder                      1.16476       1.62949       2.86831       1.54391      0.366216        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000422396   0.000511996   0.000902652   0.000462602   0.000112379      19     
produce:reco3d:SpacePointSolver                    21.462        27.3001       36.8632       26.3673       4.17018        19     
produce:hitpdune:DisambigFromSpacePoints           0.18975      0.239543      0.367007      0.214536      0.0467606       19     
produce:cluster3d:Cluster3D                        2.51977       3.74456       6.75046       3.59458      0.993826        19     
[art]:TriggerResults:TriggerResultInserter        2.116e-05    3.25814e-05   0.000108234   2.5328e-05    2.11071e-05      19     
end_path:out1:RootOutput                          3.817e-06    1.25493e-05   9.2675e-05     8.015e-06    1.89666e-05      19     
end_path:out1:RootOutput(write)                    6.75013       8.03742       11.3627       7.90191       1.00374        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5343.25 MB
  Peak resident set size usage (VmHWM): 3427.65 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0533_df-s04-d1_dw_0_20260203T220441_reco_stage0_20260204T002921_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0533_df-s04-d1_dw_0_20260203T220441_reco_stage0_20260204T002921_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0533_df-s04-d1_dw_0_20260203T220441_reco_stage0_20260204T002921_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770161343.0,
    "core.end_time": 1770164964.0,
    "core.events": [
      60773,
      60779,
      60785,
      60791,
      60797,
      60803,
      60809,
      60815,
      60821,
      60827,
      60833,
      60839,
      60845,
      60851,
      60857,
      60863,
      60869,
      60875,
      60881
    ],
    "core.event_count": 19,
    "core.first_event_number": 60773,
    "core.last_event_number": 60881,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0533_df-s04-d1_dw_0_20260203T220441.hdf5"
    }
  ]
}
justIN time: 2026-02-04 04:31:57 UTC       justIN version: 01.06.00