justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303656.18@dunegpschedd01.fnal.gov

Jobsub ID303656.18@dunegpschedd01.fnal.gov
Workflow ID12717
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:56:25
SiteUK_QMUL
EntryDUNE_UK_London_QMUL_arcce02
Last heartbeat2026-02-04 02:04:16
From worker nodeHostnamecn106.htc.esc.qmul
cpuinfoIntel(R) Xeon(R) Gold 6248R CPU @ 3.00GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-04 00:10:56
Input filesvd-protodune:np02vd_raw_run042421_0025_df-s04-d1_dw_0_20260203T134433.hdf5
JobscriptExit code0
Real time1h (6662s)
CPU time51m (3090s = 46%)
Max RSS bytes2611306496 (2490 MiB)
Outputting started2026-02-04 02:01:59
Output files
Finished2026-02-04 02:04:16
Saved logsjustin-logs:303656.18-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

nt ptr... 0x3837f0d0, 0x38380140
Caught exception in looking up space point ptr... 0x38380140, 0x38380240
Caught exception in looking up space point ptr... 0x3d4ee760, 0x3d4eece0
Caught exception in looking up space point ptr... 0x3d4eece0, 0x3d4f0bc0
Caught exception in looking up space point ptr... 0x45335220, 0x45335710
Caught exception in looking up space point ptr... 0x45335710, 0x45335220
Caught exception in looking up space point ptr... 0x5949c8a0, 0x5a42c8f0
Caught exception in looking up space point ptr... 0x5a42c8f0, 0x3682abe0
Caught exception in looking up space point ptr... 0x390ec080, 0x390ec3a0
Caught exception in looking up space point ptr... 0x390ec3a0, 0x390ec080
Caught exception in looking up space point ptr... 0x3641a430, 0x3641a2e0
Caught exception in looking up space point ptr... 0x3641e700, 0x3641a430
Caught exception in looking up space point ptr... 0x51d28890, 0x405cecd0
Caught exception in looking up space point ptr... 0x51d28680, 0x51d28890
Caught exception in looking up space point ptr... 0x4582de40, 0x4583a510
Caught exception in looking up space point ptr... 0x4583a510, 0x4583a610
Caught exception in looking up space point ptr... 0x8fb4a90, 0x2f6d93d0
Caught exception in looking up space point ptr... 0x2f6d93d0, 0x8fb4a90
Caught exception in looking up space point ptr... 0x3bc2b860, 0x3bc2bab0
Caught exception in looking up space point ptr... 0x58292190, 0x3bc2b860
Caught exception in looking up space point ptr... 0x2ac59d00, 0x45345b50
Caught exception in looking up space point ptr... 0x45345b50, 0x4c7e9040
Caught exception in looking up space point ptr... 0x2ef28bd0, 0x2ef28490
Caught exception in looking up space point ptr... 0x2ef28490, 0x2ef28bd0
Caught exception in looking up space point ptr... 0x2ef27e00, 0x2ef27780
Caught exception in looking up space point ptr... 0x2ef27780, 0x2ef27e00
Caught exception in looking up space point ptr... 0x2ef27e00, 0x2ef25640
Caught exception in looking up space point ptr... 0x2ef25640, 0x2ef27e00
Caught exception in looking up space point ptr... 0x51cc1910, 0x51cbc4a0
Caught exception in looking up space point ptr... 0x51cc1b60, 0x51cc1910
Caught exception in looking up space point ptr... 0xae8b890, 0xae8b7a0
Caught exception in looking up space point ptr... 0xae8b7a0, 0xae8b890
Caught exception in looking up space point ptr... 0x4a6c8b40, 0x26b380a0
Caught exception in looking up space point ptr... 0x26b380a0, 0x4a6c8b40
Caught exception in looking up space point ptr... 0x390c9690, 0x390c94b0
Caught exception in looking up space point ptr... 0x390c94b0, 0x390c9690
Caught exception in looking up space point ptr... 0x4970d6b0, 0x4970d8b0
Caught exception in looking up space point ptr... 0x30125480, 0x4970d6b0
Caught exception in looking up space point ptr... 0x921b6d0, 0x921b540
Caught exception in looking up space point ptr... 0x921b540, 0x921b6d0
Caught exception in looking up space point ptr... 0x36284180, 0x36282ff0
Caught exception in looking up space point ptr... 0x36282ff0, 0x36284180
Caught exception in looking up space point ptr... 0x582cce90, 0x582d5390
Caught exception in looking up space point ptr... 0x582d5390, 0x582cce90
Caught exception in looking up space point ptr... 0x582cce90, 0x582ccd20
Caught exception in looking up space point ptr... 0x582ccd20, 0x582cce90
Caught exception in looking up space point ptr... 0x241bdd70, 0x241c1840
Caught exception in looking up space point ptr... 0x241c1840, 0x241c2530
Caught exception in looking up space point ptr... 0x4c7d60e0, 0x4c7d6380
Caught exception in looking up space point ptr... 0x4c7d6380, 0x4c7d60e0
Caught exception in looking up space point ptr... 0x4c7d60e0, 0x4c7d6230
Caught exception in looking up space point ptr... 0x4c7d6230, 0x4c7d60e0
Caught exception in looking up space point ptr... 0x5a432ee0, 0x5a432fe0
Caught exception in looking up space point ptr... 0x5a432fe0, 0x5a432ee0
Caught exception in looking up space point ptr... 0x5a432ee0, 0x5a4330e0
Caught exception in looking up space point ptr... 0x5a4330e0, 0x5a432ee0
Caught exception in looking up space point ptr... 0x3cf3d8f0, 0x5024d500
Caught exception in looking up space point ptr... 0x5024d500, 0x34e90a50
Caught exception in looking up space point ptr... 0x2ef1e730, 0x2ef157d0
Caught exception in looking up space point ptr... 0x2ef157d0, 0x2ef1ad20
Caught exception in looking up space point ptr... 0x38379a30, 0x38383290
Caught exception in looking up space point ptr... 0x38383290, 0x38379a30
Caught exception in looking up space point ptr... 0x38379a30, 0x38379ef0
Caught exception in looking up space point ptr... 0x38379ef0, 0x38379a30
Caught exception in looking up space point ptr... 0x2ac3efd0, 0x2ac3ecd0
Caught exception in looking up space point ptr... 0x2ac3ecd0, 0x2ac3edd0
++++>>>> total num hits: 50215, num free: 47980
04-Feb-2026 02:01:46 GMT  Closed output file "np02vd_raw_run042421_0025_df-s04-d1_dw_0_20260203T134433_reco_stage0_20260204T020146_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         176.983       242.111       311.429       239.149       30.4747        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       8.2933e-05    0.00010839    0.000236937   0.000103984   2.92149e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                82.0723       121.381       177.633       118.009       23.702         27     
produce:triggerrawdecoder:PDVDTriggerReader4      0.561547      0.678448      0.900775      0.653344      0.0792737       27     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.016588      0.0246434     0.056022      0.0227448     0.0093012       27     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0816667     0.083571      0.0986073     0.0829535    0.00311483       27     
produce:beamevent:BeamEvent                      9.0758e-05    0.000160082    0.0010411    0.000121856   0.000173822      27     
produce:pdvddaphne:DAPHNEReaderPDVD                14.1439       18.6393       24.3063       18.4748       2.50325        27     
produce:ophit:OpHitFinder                         0.0333637     0.0524034     0.0769246     0.0556636     0.0133201       27     
produce:wclsdatavd:WireCellToolkit                 66.2321       85.2929       108.69        82.9904       13.7005        27     
produce:gaushit:GausHitFinder                     0.529218       1.13789       1.86172       1.04879      0.385247        27     
produce:nhitsfilter:NumberOfHitsFilter           0.000296114   0.000472122   0.000870019   0.000469346   0.000125397      27     
produce:reco3d:SpacePointSolver                    3.98603       8.85453       12.8687       9.9874        2.77566        27     
produce:hitpdune:DisambigFromSpacePoints          0.0508599     0.128191      0.210636      0.122016      0.0487809       27     
produce:cluster3d:Cluster3D                       0.542474       1.62998       2.92738       1.50429      0.709917        27     
[art]:TriggerResults:TriggerResultInserter       2.2192e-05    3.29844e-05   7.1655e-05    3.0233e-05    1.08482e-05      27     
end_path:out1:RootOutput                          3.085e-06    1.08234e-05   0.000136869    4.674e-06    2.49719e-05      27     
end_path:out1:RootOutput(write)                    3.02273       4.18099       5.49873       4.74525      0.999506        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4520.36 MB
  Peak resident set size usage (VmHWM): 2611.31 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042421_0025_df-s04-d1_dw_0_20260203T134433_reco_stage0_20260204T020146_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042421_0025_df-s04-d1_dw_0_20260203T134433_reco_stage0_20260204T020146_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042421_0025_df-s04-d1_dw_0_20260203T134433_reco_stage0_20260204T020146_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770163867.0,
    "core.end_time": 1770170512.0,
    "core.events": [
      4097,
      4103,
      4109,
      4115,
      4121,
      4127,
      4133,
      4139,
      4145,
      4151,
      4157,
      4163,
      4169,
      4175,
      4181,
      4187,
      4193,
      4199,
      4205,
      4211,
      4217,
      4223,
      4229,
      4235,
      4241,
      4247,
      4253
    ],
    "core.event_count": 27,
    "core.first_event_number": 4097,
    "core.last_event_number": 4253,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42421
    ],
    "core.runs_subruns": [
      4242100001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042421_0025_df-s04-d1_dw_0_20260203T134433.hdf5"
    }
  ]
}
justIN time: 2026-02-04 06:06:24 UTC       justIN version: 01.06.00