justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303654.62@dunegpschedd01.fnal.gov

Jobsub ID303654.62@dunegpschedd01.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:52:26
SiteUK_QMUL
EntryDUNE_UK_London_QMUL_arcce03
Last heartbeat2026-02-04 02:17:50
From worker nodeHostnamecn023.htc.esc.qmul
cpuinfoIntel(R) Xeon(R) Gold 6248R CPU @ 3.00GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-04 00:01:03
Input filesvd-protodune:np02vd_raw_run042433_0161_df-s04-d1_dw_0_20260203T194320.hdf5
JobscriptExit code0
Real time2h (8046s)
CPU time1h (4760s = 59%)
Max RSS bytes3496542208 (3334 MiB)
Outputting started2026-02-04 02:15:11
Output files
Finished2026-02-04 02:17:50
Saved logsjustin-logs:303654.62-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

point ptr... 0x3f8fa4a0, 0x3f8fa7c0
Caught exception in looking up space point ptr... 0x8f215020, 0x6a5e2ef0
Caught exception in looking up space point ptr... 0x6a5e2ef0, 0x60d0b120
Caught exception in looking up space point ptr... 0x1aa8d520, 0x4b0fe4b0
Caught exception in looking up space point ptr... 0x4b0fe4b0, 0x8f215020
Caught exception in looking up space point ptr... 0x4f7863c0, 0x4f786550
Caught exception in looking up space point ptr... 0x4f786550, 0x4f7863c0
Caught exception in looking up space point ptr... 0x4f7863c0, 0x6f2481c0
Caught exception in looking up space point ptr... 0x6f2481c0, 0x4f7863c0
Caught exception in looking up space point ptr... 0x4f7863c0, 0x4f7866e0
Caught exception in looking up space point ptr... 0x4f7866e0, 0x4f7863c0
Caught exception in looking up space point ptr... 0xb332d40, 0xb332bb0
Caught exception in looking up space point ptr... 0xb332bb0, 0xb332d40
Caught exception in looking up space point ptr... 0xb332d40, 0x87400730
Caught exception in looking up space point ptr... 0x87400730, 0xb332d40
Caught exception in looking up space point ptr... 0x6193f8a0, 0x6193f580
Caught exception in looking up space point ptr... 0x6193f580, 0x6193f8a0
Caught exception in looking up space point ptr... 0x6193f8a0, 0x6193f710
Caught exception in looking up space point ptr... 0x6193f710, 0x6193f8a0
Caught exception in looking up space point ptr... 0x6193f8a0, 0x6193f260
Caught exception in looking up space point ptr... 0x6193f260, 0x6193f8a0
Caught exception in looking up space point ptr... 0x4f7c4ee0, 0x4f7c5200
Caught exception in looking up space point ptr... 0x4f7c5200, 0x4f7c4ee0
Caught exception in looking up space point ptr... 0x4f7c4ee0, 0x4f7c5070
Caught exception in looking up space point ptr... 0x4f7c5070, 0x4f7c4ee0
Caught exception in looking up space point ptr... 0x2c37c650, 0x2c37c330
Caught exception in looking up space point ptr... 0x2c37c330, 0x2c37c650
Caught exception in looking up space point ptr... 0x2c37c650, 0x2c37c010
Caught exception in looking up space point ptr... 0x2c37c010, 0x2c37c650
Caught exception in looking up space point ptr... 0x2c37c650, 0x2c37c4c0
Caught exception in looking up space point ptr... 0x2c37c4c0, 0x2c37c650
Caught exception in looking up space point ptr... 0x2c3a2300, 0x2c3a1fe0
Caught exception in looking up space point ptr... 0x2c3a1fe0, 0x2c3a2300
Caught exception in looking up space point ptr... 0x2c3a2300, 0x2c3a1cc0
Caught exception in looking up space point ptr... 0x2c3a1cc0, 0x2c3a2300
Caught exception in looking up space point ptr... 0x2c3a2300, 0x2c3a2170
Caught exception in looking up space point ptr... 0x2c3a2170, 0x2c3a2300
Caught exception in looking up space point ptr... 0x51d97dc0, 0x19f4d670
Caught exception in looking up space point ptr... 0x19f4d670, 0x51cd44f0
Caught exception in looking up space point ptr... 0x3c64dfb0, 0x3c64e140
Caught exception in looking up space point ptr... 0x3c64e140, 0x3c64dfb0
Caught exception in looking up space point ptr... 0x3c64dfb0, 0x3c64e2d0
Caught exception in looking up space point ptr... 0x3c64e2d0, 0x3c64dfb0
Caught exception in looking up space point ptr... 0x3c64dfb0, 0x469bf260
Caught exception in looking up space point ptr... 0x469bf260, 0x3c64dfb0
Caught exception in looking up space point ptr... 0x8f092d20, 0x8f092eb0
Caught exception in looking up space point ptr... 0x8f092eb0, 0xa86f450
Caught exception in looking up space point ptr... 0x6c318750, 0x384d8460
Caught exception in looking up space point ptr... 0x384d8460, 0x6c318750
Caught exception in looking up space point ptr... 0x877e3140, 0x877e2e20
Caught exception in looking up space point ptr... 0x877e2e20, 0x877e3140
Caught exception in looking up space point ptr... 0x877e3140, 0x877e2b00
Caught exception in looking up space point ptr... 0x877e2b00, 0x877e3140
Caught exception in looking up space point ptr... 0x877e3140, 0x877e2fb0
Caught exception in looking up space point ptr... 0x877e2fb0, 0x877e3140
Caught exception in looking up space point ptr... 0x6e83d410, 0x6e83d5a0
Caught exception in looking up space point ptr... 0x6e83d5a0, 0x4f4777c0
Caught exception in looking up space point ptr... 0xefe56a0, 0x6a5a9670
Caught exception in looking up space point ptr... 0x6a5a9670, 0xefe56a0
Caught exception in looking up space point ptr... 0xefe56a0, 0x6151dda0
Caught exception in looking up space point ptr... 0x6151dda0, 0xefe56a0
Caught exception in looking up space point ptr... 0x1a811520, 0xbac3ea0
Caught exception in looking up space point ptr... 0xbac3ea0, 0x1a811520
Caught exception in looking up space point ptr... 0x496fc650, 0x496fc970
Caught exception in looking up space point ptr... 0x496fc970, 0x496fcc90
Caught exception in looking up space point ptr... 0x4f647400, 0x48154d80
Caught exception in looking up space point ptr... 0x48154d80, 0x4f647400
++++>>>> total num hits: 521162, num free: 506975
04-Feb-2026 02:15:01 GMT  Closed output file "np02vd_raw_run042433_0161_df-s04-d1_dw_0_20260203T194320_reco_stage0_20260204T021501_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         370.535       418.468       500.336       412.394       30.1495        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       8.8338e-05    0.000140932   0.000244206   0.000148118   4.25616e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                120.068       165.438       233.609       162.669       26.5393        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.540902      0.747531       1.0033       0.727165      0.146526        19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0165238     0.016775      0.0170646     0.0167538    0.000154243      19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0164991     0.0188237     0.0547925     0.0167658    0.00848336       19     
produce:beamevent:BeamEvent                      0.000134055   0.000243237   0.00102508    0.000187108   0.000191454      19     
produce:pdvddaphne:DAPHNEReaderPDVD                11.8009       17.8934       22.8186       17.3521       2.64283        19     
produce:ophit:OpHitFinder                         0.0418426     0.0680351     0.0859867     0.0679531     0.0099758       19     
produce:wclsdatavd:WireCellToolkit                 140.845       185.822       213.939       194.769       21.3858        19     
produce:gaushit:GausHitFinder                      1.39121       2.32902       3.42357       2.35454      0.461644        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000678507   0.000975613   0.00135979    0.000964926   0.000172372      19     
produce:reco3d:SpacePointSolver                    16.957        28.7116       45.0037       28.1353       6.22306        19     
produce:hitpdune:DisambigFromSpacePoints          0.201297      0.412137      0.710093      0.424176      0.110354        19     
produce:cluster3d:Cluster3D                        2.01411       9.46942       36.227        4.25899       11.2118        19     
[art]:TriggerResults:TriggerResultInserter       3.7206e-05    6.22023e-05   0.000135811    5.198e-05    2.7659e-05       19     
end_path:out1:RootOutput                          8.192e-06    1.39823e-05   3.7302e-05    1.1023e-05    6.70337e-06      19     
end_path:out1:RootOutput(write)                    4.86065       7.4682        8.59608       7.79095       1.17516        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5400.61 MB
  Peak resident set size usage (VmHWM): 3496.54 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0161_df-s04-d1_dw_0_20260203T194320_reco_stage0_20260204T021501_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0161_df-s04-d1_dw_0_20260203T194320_reco_stage0_20260204T021501_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0161_df-s04-d1_dw_0_20260203T194320_reco_stage0_20260204T021501_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770163272.0,
    "core.end_time": 1770171304.0,
    "core.events": [
      18365,
      18371,
      18377,
      18383,
      18389,
      18395,
      18401,
      18407,
      18413,
      18419,
      18425,
      18431,
      18437,
      18443,
      18449,
      18455,
      18461,
      18467,
      18473
    ],
    "core.event_count": 19,
    "core.first_event_number": 18365,
    "core.last_event_number": 18473,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0161_df-s04-d1_dw_0_20260203T194320.hdf5"
    }
  ]
}
justIN time: 2026-02-04 07:36:50 UTC       justIN version: 01.06.00