justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303654.2@dunegpschedd01.fnal.gov

Jobsub ID303654.2@dunegpschedd01.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:52:26
SiteUK_Oxford
EntryDUNE_UK_SGrid_Oxford_arc01
Last heartbeat2026-02-04 02:11:06
From worker nodeHostnamet2wn068.physics.ox.ac.uk
cpuinfoAMD EPYC 7763 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:48:54
Input filesvd-protodune:np02vd_raw_run042433_0007_df-s04-d0_dw_0_20260203T184446.hdf5
JobscriptExit code0
Real time2h (8340s)
CPU time56m (3405s = 40%)
Max RSS bytes3190341632 (3042 MiB)
Outputting started2026-02-04 02:07:56
Output files
Finished2026-02-04 02:11:06
Saved logsjustin-logs:303654.2-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ught exception in looking up space point ptr... 0x1ae08c70, 0x6f6146f0
Caught exception in looking up space point ptr... 0x59995f50, 0x5998e510
Caught exception in looking up space point ptr... 0x599960a0, 0x59995f50
Caught exception in looking up space point ptr... 0x3defc2e0, 0x3def3850
Caught exception in looking up space point ptr... 0x3def3850, 0x3dee8870
Caught exception in looking up space point ptr... 0x39195010, 0x39194ef0
Caught exception in looking up space point ptr... 0x39194ef0, 0x391944a0
Caught exception in looking up space point ptr... 0x59a5efc0, 0x59a5f120
Caught exception in looking up space point ptr... 0x59a5f120, 0x59a5f660
Caught exception in looking up space point ptr... 0x67558920, 0x6755d2c0
Caught exception in looking up space point ptr... 0x3f26ace0, 0x67558920
Caught exception in looking up space point ptr... 0x42e20480, 0x42e207a0
Caught exception in looking up space point ptr... 0x42e207a0, 0x42e20480
Caught exception in looking up space point ptr... 0x7c2e2420, 0x7c2e1c60
Caught exception in looking up space point ptr... 0x7c2e1c60, 0x7c2e1f40
Caught exception in looking up space point ptr... 0x36ca3de0, 0x36ca2a10
Caught exception in looking up space point ptr... 0x36ca5d00, 0x36ca3de0
Caught exception in looking up space point ptr... 0x5b11aaf0, 0x4edb0070
Caught exception in looking up space point ptr... 0x4edb0070, 0x5b10e7f0
Caught exception in looking up space point ptr... 0x627864a0, 0x627853a0
Caught exception in looking up space point ptr... 0x627853a0, 0x627864a0
Caught exception in looking up space point ptr... 0x5aec3960, 0x5aec3af0
Caught exception in looking up space point ptr... 0x5aec3af0, 0x5aec3960
Caught exception in looking up space point ptr... 0x5b245aa0, 0x5b245780
Caught exception in looking up space point ptr... 0x5b245780, 0x5b2455f0
Caught exception in looking up space point ptr... 0x62887dd0, 0x6288f110
Caught exception in looking up space point ptr... 0x6288f110, 0x62887dd0
Caught exception in looking up space point ptr... 0x14964d80, 0x1496f3b0
Caught exception in looking up space point ptr... 0x1496f3b0, 0x1496f4d0
Caught exception in looking up space point ptr... 0x6039f2c0, 0x4edae150
Caught exception in looking up space point ptr... 0x4edae150, 0x6039f2c0
Caught exception in looking up space point ptr... 0x43eac1c0, 0x43eac310
Caught exception in looking up space point ptr... 0x43eac310, 0x43eac1c0
Caught exception in looking up space point ptr... 0xed50c40, 0x7bf3bae0
Caught exception in looking up space point ptr... 0x7bf3bae0, 0xed50c40
Caught exception in looking up space point ptr... 0xed50c40, 0xed50dd0
Caught exception in looking up space point ptr... 0xed50dd0, 0xed50c40
Caught exception in looking up space point ptr... 0x79f7a4e0, 0x79f79c60
Caught exception in looking up space point ptr... 0x79f79c60, 0x79f7a4e0
Caught exception in looking up space point ptr... 0x6341e930, 0x6341ec50
Caught exception in looking up space point ptr... 0x6341ec50, 0x6341ede0
Caught exception in looking up space point ptr... 0x5b249270, 0x5b248fd0
Caught exception in looking up space point ptr... 0x5b248fd0, 0x5b249270
Caught exception in looking up space point ptr... 0x5b20c770, 0x3dd3bcd0
Caught exception in looking up space point ptr... 0x3dd3bcd0, 0x5b20c770
Caught exception in looking up space point ptr... 0x43dc2ab0, 0x43dbf1e0
Caught exception in looking up space point ptr... 0x43dbf1e0, 0x43dc2ab0
Caught exception in looking up space point ptr... 0x816074a0, 0x816072e0
Caught exception in looking up space point ptr... 0x816072e0, 0x816074a0
Caught exception in looking up space point ptr... 0x598e1f60, 0x7c2e5f00
Caught exception in looking up space point ptr... 0x7c2e5f00, 0x598e1f60
Caught exception in looking up space point ptr... 0x19db8100, 0x19db9350
Caught exception in looking up space point ptr... 0x19db9350, 0x19db8100
Caught exception in looking up space point ptr... 0x7c4eea80, 0x7c4ee950
Caught exception in looking up space point ptr... 0x7c4ee950, 0x7c4eea80
Caught exception in looking up space point ptr... 0x8163d880, 0x8163e510
Caught exception in looking up space point ptr... 0x8163b040, 0x8163d880
Caught exception in looking up space point ptr... 0x478100a0, 0x73dd2b60
Caught exception in looking up space point ptr... 0x73dd2b60, 0x47810930
Caught exception in looking up space point ptr... 0x1499ed30, 0x6038cf70
Caught exception in looking up space point ptr... 0x6038cf70, 0x3f286ae0
Caught exception in looking up space point ptr... 0x3deee860, 0x3def1cd0
Caught exception in looking up space point ptr... 0x3def1cd0, 0x3dee73c0
Caught exception in looking up space point ptr... 0x29d99fc0, 0x5aefaad0
Caught exception in looking up space point ptr... 0x5aefaad0, 0x29d99fc0
Caught exception in looking up space point ptr... 0x3dbe8660, 0x3dbe8980
Caught exception in looking up space point ptr... 0x3dbe8980, 0x3dbe8660
++++>>>> total num hits: 104506, num free: 100033
04-Feb-2026 02:07:32 GMT  Closed output file "np02vd_raw_run042433_0007_df-s04-d0_dw_0_20260203T184446_reco_stage0_20260204T020732_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         339.733       430.542       521.469       445.119       53.4748        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       0.000111158   0.000181246   0.00029976    0.000199011   5.31939e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                172.434       240.617       326.442       245.475       43.7436        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.762814       1.11871       1.78476       1.11543      0.254139        19     
produce:timingrawdecoder:PDHDTimingRawDecoder      0.01959      0.0237085     0.037994      0.0200164     0.0058161       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0195751     0.0201302     0.0250694     0.0198081    0.00118699       19     
produce:beamevent:BeamEvent                      0.000141725   0.000256348   0.00136686    0.000198752   0.000263333      19     
produce:pdvddaphne:DAPHNEReaderPDVD                13.0233       19.9998       28.1359       21.2792       4.05053        19     
produce:ophit:OpHitFinder                         0.0408967     0.0471705     0.0592372     0.0463278    0.00412422       19     
produce:wclsdatavd:WireCellToolkit                 117.923       137.16        158.216       138.847       9.74032        19     
produce:gaushit:GausHitFinder                      1.0247        1.79983       2.41091       1.87845      0.349275        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000364882   0.000571412   0.00245173    0.000448017   0.000452031      19     
produce:reco3d:SpacePointSolver                    12.2493        18.86        30.2512       17.8185       4.85322        19     
produce:hitpdune:DisambigFromSpacePoints          0.143885      0.259094      0.522674      0.262282      0.0861014       19     
produce:cluster3d:Cluster3D                        1.80965       3.96961       16.2205       3.33263       3.00981        19     
[art]:TriggerResults:TriggerResultInserter       3.1338e-05    4.46903e-05   0.000126346   4.1738e-05     2.012e-05       19     
end_path:out1:RootOutput                          6.102e-06    1.23426e-05   4.5766e-05    1.1551e-05    8.26789e-06      19     
end_path:out1:RootOutput(write)                    4.95307       6.60316       8.43307       6.85795       0.77991        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5406.93 MB
  Peak resident set size usage (VmHWM): 3190.34 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0007_df-s04-d0_dw_0_20260203T184446_reco_stage0_20260204T020732_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0007_df-s04-d0_dw_0_20260203T184446_reco_stage0_20260204T020732_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0007_df-s04-d0_dw_0_20260203T184446_reco_stage0_20260204T020732_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770162587.0,
    "core.end_time": 1770170854.0,
    "core.events": [
      808,
      814,
      820,
      826,
      832,
      838,
      844,
      850,
      856,
      862,
      868,
      874,
      880,
      886,
      892,
      898,
      904,
      910,
      916
    ],
    "core.event_count": 19,
    "core.first_event_number": 808,
    "core.last_event_number": 916,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0007_df-s04-d0_dw_0_20260203T184446.hdf5"
    }
  ]
}
justIN time: 2026-02-04 08:47:25 UTC       justIN version: 01.06.00