justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297921.37@dunegpschedd02.fnal.gov

Jobsub ID297921.37@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:58:26
SiteUK_Imperial
EntryDUNE_T2_UK_London_IC_ceprod03
Last heartbeat2026-02-04 01:11:51
From worker nodeHostnamewm10.grid.hep.ph.ic.ac.uk
cpuinfoAMD EPYC 7742 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:48:07
Input filesvd-protodune:np02vd_raw_run042433_0539_df-s03-d1_dw_0_20260203T220658.hdf5
JobscriptExit code0
Real time1h (4873s)
CPU time1h (4295s = 88%)
Max RSS bytes3387559936 (3230 MiB)
Outputting started2026-02-04 01:09:21
Output files
Finished2026-02-04 01:11:51
Saved logsjustin-logs:297921.37-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ptr... 0x4702fd10, 0x158c8fe0
Caught exception in looking up space point ptr... 0x158c8fe0, 0x56aacd40
Caught exception in looking up space point ptr... 0x3b75f370, 0x3b75f640
Caught exception in looking up space point ptr... 0x3b75f640, 0x3b75f370
Caught exception in looking up space point ptr... 0x6248beb0, 0x2dc6c4f0
Caught exception in looking up space point ptr... 0x6881ba40, 0x6248beb0
Caught exception in looking up space point ptr... 0x6c78b4d0, 0x6c78b3a0
Caught exception in looking up space point ptr... 0x6c78b3a0, 0x6c78b4d0
Caught exception in looking up space point ptr... 0x6c78b4d0, 0x6c78b0f0
Caught exception in looking up space point ptr... 0x6c78b0f0, 0x6c78b4d0
Caught exception in looking up space point ptr... 0x55d1d240, 0x55d1b790
Caught exception in looking up space point ptr... 0x55d1b790, 0x55d1d240
Caught exception in looking up space point ptr... 0x55d1d240, 0x55d1bc90
Caught exception in looking up space point ptr... 0x55d1bc90, 0x55d1d240
Caught exception in looking up space point ptr... 0x747ab600, 0x747ab410
Caught exception in looking up space point ptr... 0x11bf8390, 0x747ab600
Caught exception in looking up space point ptr... 0xb63a3c0, 0xb63a660
Caught exception in looking up space point ptr... 0xb63a660, 0xb63a270
Caught exception in looking up space point ptr... 0x4d3ca0f0, 0x4d3ca410
Caught exception in looking up space point ptr... 0x4d3ca410, 0x4d3ca5a0
Caught exception in looking up space point ptr... 0x506e91c0, 0x506e8ea0
Caught exception in looking up space point ptr... 0x506e8ea0, 0x506e91c0
Caught exception in looking up space point ptr... 0x506e91c0, 0x506e9030
Caught exception in looking up space point ptr... 0x506e9030, 0x506e91c0
Caught exception in looking up space point ptr... 0x2ab61d20, 0x2ab619b0
Caught exception in looking up space point ptr... 0x2ab619b0, 0x2ab61d20
Caught exception in looking up space point ptr... 0x2ab61d20, 0x374b43b0
Caught exception in looking up space point ptr... 0x374b43b0, 0x2ab61d20
Caught exception in looking up space point ptr... 0x2ab61d20, 0x2ab62490
Caught exception in looking up space point ptr... 0x2ab62490, 0x2ab61d20
Caught exception in looking up space point ptr... 0x684d52e0, 0x684d5470
Caught exception in looking up space point ptr... 0x684d5470, 0x684d5600
Caught exception in looking up space point ptr... 0x32b29e00, 0x32b29f00
Caught exception in looking up space point ptr... 0x32b29f00, 0x32b29e00
Caught exception in looking up space point ptr... 0x52e56df0, 0x52e56fc0
Caught exception in looking up space point ptr... 0x52e56fc0, 0x52e56df0
Caught exception in looking up space point ptr... 0x69678390, 0x69674150
Caught exception in looking up space point ptr... 0x69674150, 0x69678390
Caught exception in looking up space point ptr... 0x32e67500, 0x34601d80
Caught exception in looking up space point ptr... 0x34601d80, 0x56524d10
Caught exception in looking up space point ptr... 0x56524d10, 0x8dec270
Caught exception in looking up space point ptr... 0x8dec270, 0x47d79fe0
Caught exception in looking up space point ptr... 0x47d79fe0, 0x32e67500
Caught exception in looking up space point ptr... 0x534d9040, 0x54334d90
Caught exception in looking up space point ptr... 0x54334d90, 0x534d9040
Caught exception in looking up space point ptr... 0x725803d0, 0x72580240
Caught exception in looking up space point ptr... 0x72580240, 0x725803d0
Caught exception in looking up space point ptr... 0x33036b40, 0x614e88d0
Caught exception in looking up space point ptr... 0x614e88d0, 0x19eeb540
Caught exception in looking up space point ptr... 0x639312c0, 0x63930d80
Caught exception in looking up space point ptr... 0x63930d80, 0x639312c0
Caught exception in looking up space point ptr... 0x41ac5090, 0x32d99ef0
Caught exception in looking up space point ptr... 0x32d99ef0, 0x148dc990
Caught exception in looking up space point ptr... 0x74826fa0, 0x6ca63680
Caught exception in looking up space point ptr... 0x6ca63680, 0x74826fa0
Caught exception in looking up space point ptr... 0x374b4690, 0x374a4560
Caught exception in looking up space point ptr... 0x374a4560, 0x374b4690
Caught exception in looking up space point ptr... 0x5f7e0990, 0x5f7e0630
Caught exception in looking up space point ptr... 0x5f7e0630, 0x5f7e0990
Caught exception in looking up space point ptr... 0x728ebf80, 0x728ebe30
Caught exception in looking up space point ptr... 0x728ebe30, 0x728eb710
Caught exception in looking up space point ptr... 0x55d200f0, 0x55d1ffa0
Caught exception in looking up space point ptr... 0x55d1ffa0, 0x55d200f0
Caught exception in looking up space point ptr... 0x55d200f0, 0x55d1e3a0
Caught exception in looking up space point ptr... 0x55d1e3a0, 0x55d200f0
Caught exception in looking up space point ptr... 0x2d84d240, 0x419c19a0
Caught exception in looking up space point ptr... 0x419c19a0, 0x512fded0
++++>>>> total num hits: 100368, num free: 96307
04-Feb-2026 01:09:12 GMT  Closed output file "np02vd_raw_run042433_0539_df-s03-d1_dw_0_20260203T220658_reco_stage0_20260204T010912_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         228.006       250.224       265.34        251.339       9.71448        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       9.3576e-05    0.000135546   0.000227159   0.000105078   4.65489e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                15.6679       18.0495       19.2158       18.4331      0.976808        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.487293      0.500138      0.550801      0.493338      0.0170056       19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0164696     0.0170533     0.0246743      0.01664     0.00179954       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0164516     0.0166452     0.017002      0.0165774    0.000134607      19     
produce:beamevent:BeamEvent                      0.000100359   0.000189237   0.00127074    0.000123774   0.000255832      19     
produce:pdvddaphne:DAPHNEReaderPDVD                3.87358       4.54266       4.88526       4.56994      0.239191        19     
produce:ophit:OpHitFinder                         0.0311831     0.0499173     0.0550578     0.0514019    0.00518752       19     
produce:wclsdatavd:WireCellToolkit                 175.587       182.196       190.548       182.825       3.38893        19     
produce:gaushit:GausHitFinder                      1.08559       2.17342       2.74237       2.17409      0.403231        19     
produce:nhitsfilter:NumberOfHitsFilter           0.00036059    0.000582642   0.00111429    0.000556098   0.000170061      19     
produce:reco3d:SpacePointSolver                    12.4797       27.3289       37.625        28.7076       6.78982        19     
produce:hitpdune:DisambigFromSpacePoints          0.156898       0.32148      0.398906       0.32623      0.0569277       19     
produce:cluster3d:Cluster3D                        2.41019       4.47436       6.20097       4.41736       1.11156        19     
[art]:TriggerResults:TriggerResultInserter        3.174e-05    4.37104e-05   0.000108144   3.6779e-05    2.12505e-05      19     
end_path:out1:RootOutput                          7.975e-06    1.09966e-05   2.8534e-05     9.588e-06    4.58025e-06      19     
end_path:out1:RootOutput(write)                    9.09277       10.5106       11.5224       10.5521      0.617038        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5299.07 MB
  Peak resident set size usage (VmHWM): 3387.56 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0539_df-s03-d1_dw_0_20260203T220658_reco_stage0_20260204T010912_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0539_df-s03-d1_dw_0_20260203T220658_reco_stage0_20260204T010912_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0539_df-s03-d1_dw_0_20260203T220658_reco_stage0_20260204T010912_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770162503.0,
    "core.end_time": 1770167353.0,
    "core.events": [
      61455,
      61461,
      61467,
      61473,
      61479,
      61485,
      61491,
      61497,
      61503,
      61509,
      61515,
      61521,
      61527,
      61533,
      61539,
      61545,
      61551,
      61557,
      61563
    ],
    "core.event_count": 19,
    "core.first_event_number": 61455,
    "core.last_event_number": 61563,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0539_df-s03-d1_dw_0_20260203T220658.hdf5"
    }
  ]
}
justIN time: 2026-02-04 04:28:18 UTC       justIN version: 01.06.00