justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297920.122@dunegpschedd02.fnal.gov

Jobsub ID297920.122@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:54:27
SiteUK_Oxford
EntryDUNE_UK_SGrid_Oxford_arc01
Last heartbeat2026-02-04 02:14:42
From worker nodeHostnamet2wn064.physics.ox.ac.uk
cpuinfoAMD EPYC 7763 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:43:01
Input filesvd-protodune:np02vd_raw_run042433_0059_df-s02-d1_dw_0_20260203T190431.hdf5
JobscriptExit code0
Real time2h (8858s)
CPU time1h (4555s = 51%)
Max RSS bytes3463667712 (3303 MiB)
Outputting started2026-02-04 02:10:41
Output files
Finished2026-02-04 02:14:42
Saved logsjustin-logs:297920.122-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

n in looking up space point ptr... 0x348050b0, 0x60fafb30
Caught exception in looking up space point ptr... 0x1a3057c0, 0x1a311de0
Caught exception in looking up space point ptr... 0x1a314220, 0x1a3057c0
Caught exception in looking up space point ptr... 0x7625b3d0, 0x3c33ec30
Caught exception in looking up space point ptr... 0x623f87e0, 0x7625b3d0
Caught exception in looking up space point ptr... 0x7df63b30, 0x3bd08700
Caught exception in looking up space point ptr... 0x3bd08700, 0x609ca1a0
Caught exception in looking up space point ptr... 0x393fe8e0, 0x586624c0
Caught exception in looking up space point ptr... 0x586624c0, 0x1632bd10
Caught exception in looking up space point ptr... 0x3477c010, 0x3477b840
Caught exception in looking up space point ptr... 0x3477b840, 0x3477aee0
Caught exception in looking up space point ptr... 0x3477aee0, 0x3477b520
Caught exception in looking up space point ptr... 0x3477b520, 0x3477c010
Caught exception in looking up space point ptr... 0xb6330e0, 0x1577d070
Caught exception in looking up space point ptr... 0x1577d070, 0xb038130
Caught exception in looking up space point ptr... 0x706a5070, 0x70695990
Caught exception in looking up space point ptr... 0x2dcf15e0, 0x706a5070
Caught exception in looking up space point ptr... 0xf8148d0, 0xf814c50
Caught exception in looking up space point ptr... 0xf814710, 0xf8148d0
Caught exception in looking up space point ptr... 0x7e0fffc0, 0x1250a4e0
Caught exception in looking up space point ptr... 0x1250a4e0, 0x7e100210
Caught exception in looking up space point ptr... 0x12157dc0, 0x12146930
Caught exception in looking up space point ptr... 0x12135180, 0x12157dc0
Caught exception in looking up space point ptr... 0x53518900, 0x162f5830
Caught exception in looking up space point ptr... 0x162f5830, 0x535196e0
Caught exception in looking up space point ptr... 0x33c1c1e0, 0x33c0a260
Caught exception in looking up space point ptr... 0x33c0a0d0, 0x33c1c1e0
Caught exception in looking up space point ptr... 0x10d98920, 0x10d986d0
Caught exception in looking up space point ptr... 0x10d986d0, 0x10d994b0
Caught exception in looking up space point ptr... 0xf5082e0, 0x700fe950
Caught exception in looking up space point ptr... 0x700fe950, 0x5e64daa0
Caught exception in looking up space point ptr... 0x4af79ce0, 0x4af79b50
Caught exception in looking up space point ptr... 0x4af799c0, 0x4af79ce0
Caught exception in looking up space point ptr... 0x16adee70, 0x160d5130
Caught exception in looking up space point ptr... 0x160d5130, 0x633b5fe0
Caught exception in looking up space point ptr... 0x6f99ee50, 0x6f98de70
Caught exception in looking up space point ptr... 0x6f98de70, 0x2aeccb20
Caught exception in looking up space point ptr... 0x3306a950, 0x406eda50
Caught exception in looking up space point ptr... 0x11fdb0b0, 0x3306a950
Caught exception in looking up space point ptr... 0x5f2db8d0, 0x5ead7d40
Caught exception in looking up space point ptr... 0x5ead7d40, 0x5ead7ed0
Caught exception in looking up space point ptr... 0x121c4a90, 0x3bf9f020
Caught exception in looking up space point ptr... 0x121c4f40, 0x121c4a90
Caught exception in looking up space point ptr... 0x2fcc00e0, 0xf8fec30
Caught exception in looking up space point ptr... 0x2fcc0720, 0x2fcc00e0
Caught exception in looking up space point ptr... 0x3c3e2720, 0x3c3e2270
Caught exception in looking up space point ptr... 0x3c3e2a40, 0x3c3e2720
Caught exception in looking up space point ptr... 0x60335740, 0x5b334c90
Caught exception in looking up space point ptr... 0x15683e90, 0x60335740
Caught exception in looking up space point ptr... 0x165ce870, 0x1a31f830
Caught exception in looking up space point ptr... 0xb9e6fa0, 0x165ce870
Caught exception in looking up space point ptr... 0x2f8980c0, 0x2f885790
Caught exception in looking up space point ptr... 0x2f885790, 0x2f872eb0
Caught exception in looking up space point ptr... 0x4c247da0, 0x4c259d20
Caught exception in looking up space point ptr... 0x4c259d20, 0x5a84d7e0
Caught exception in looking up space point ptr... 0x1353ff50, 0x61cd8d80
Caught exception in looking up space point ptr... 0x8ddc520, 0x1353ff50
Caught exception in looking up space point ptr... 0x5fd75fa0, 0x5e9bfae0
Caught exception in looking up space point ptr... 0x52544730, 0x5fd75fa0
Caught exception in looking up space point ptr... 0x51fd44e0, 0x11d30000
Caught exception in looking up space point ptr... 0x11d30000, 0x608849e0
Caught exception in looking up space point ptr... 0x5fc8d710, 0x709d94f0
Caught exception in looking up space point ptr... 0x709d94f0, 0x5b8c2a00
Caught exception in looking up space point ptr... 0xf8138b0, 0xf813cc0
Caught exception in looking up space point ptr... 0xf813cc0, 0xf813e50
Caught exception in looking up space point ptr... 0x39ef3560, 0x387a3980
Caught exception in looking up space point ptr... 0x387a3980, 0x39ef3560
++++>>>> total num hits: 513777, num free: 504870
04-Feb-2026 02:10:17 GMT  Closed output file "np02vd_raw_run042433_0059_df-s02-d1_dw_0_20260203T190431_reco_stage0_20260204T021017_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         386.254       460.167       575.907       455.875       55.5619        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       0.000107922   0.000244216   0.00157232    0.000167594   0.000317871      19     
produce:tpcrawdecoder:PDVDTPCReader                144.246       210.092       351.33        195.955       48.5475        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.644503       1.07366       1.73178       1.02941      0.317154        19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0194405     0.0220291     0.0439632     0.019945     0.00551059       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0195087     0.019848      0.0205141     0.0198363    0.000240319      19     
produce:beamevent:BeamEvent                      0.000134351   0.000272586   0.00146889    0.000208059   0.000283256      19     
produce:pdvddaphne:DAPHNEReaderPDVD                12.3708       18.0932       24.0541       18.4307       3.20772        19     
produce:ophit:OpHitFinder                         0.0313339     0.0444793     0.0535816     0.0455667    0.00734638       19     
produce:wclsdatavd:WireCellToolkit                  116.5        141.588       162.701       141.467       13.5535        19     
produce:gaushit:GausHitFinder                      1.21122       2.78843       3.72919       2.81662      0.590013        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000435154   0.000569005   0.00104954    0.000535551   0.000152063      19     
produce:reco3d:SpacePointSolver                    14.3468       22.9733       35.9655       20.9538       6.62528        19     
produce:hitpdune:DisambigFromSpacePoints          0.120117      0.398611      0.615824      0.383842      0.123423        19     
produce:cluster3d:Cluster3D                        1.82035       54.9216       83.8854       69.4367       31.264         19     
[art]:TriggerResults:TriggerResultInserter       3.4465e-05    5.94991e-05   0.000170288   5.2387e-05    2.9029e-05       19     
end_path:out1:RootOutput                          6.642e-06    1.36704e-05   3.8963e-05    1.1091e-05    8.22141e-06      19     
end_path:out1:RootOutput(write)                    5.49961       8.09227       9.94046       8.00861       1.34803        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5576.96 MB
  Peak resident set size usage (VmHWM): 3463.67 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0059_df-s02-d1_dw_0_20260203T190431_reco_stage0_20260204T021017_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0059_df-s02-d1_dw_0_20260203T190431_reco_stage0_20260204T021017_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0059_df-s02-d1_dw_0_20260203T190431_reco_stage0_20260204T021017_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770162198.0,
    "core.end_time": 1770171019.0,
    "core.events": [
      6733,
      6739,
      6745,
      6751,
      6757,
      6763,
      6769,
      6775,
      6781,
      6787,
      6793,
      6799,
      6805,
      6811,
      6817,
      6823,
      6829,
      6835,
      6841
    ],
    "core.event_count": 19,
    "core.first_event_number": 6733,
    "core.last_event_number": 6841,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0059_df-s02-d1_dw_0_20260203T190431.hdf5"
    }
  ]
}
justIN time: 2026-02-04 06:06:34 UTC       justIN version: 01.06.00