justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297921.181@dunegpschedd02.fnal.gov

Jobsub ID297921.181@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:58:26
SiteUK_RAL-Tier1
EntryLIGO_UK_RAL_arc_ce03
Last heartbeat2026-02-04 01:19:21
From worker nodeHostnamedune001-1534682.0-lcg2547.gridpp.rl.ac.uk
cpuinfoAMD EPYC 7452 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit216000 (60 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:56:06
Input filesvd-protodune:np02vd_raw_run042433_0444_df-s04-d0_dw_0_20260203T213052.hdf5
JobscriptExit code0
Real time1h (4860s)
CPU time1h (4502s = 92%)
Max RSS bytes3399315456 (3241 MiB)
Outputting started2026-02-04 01:17:08
Output files
Finished2026-02-04 01:19:21
Saved logsjustin-logs:297921.181-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

. 0x43f98c50, 0x43f96fd0
Caught exception in looking up space point ptr... 0x67bdea80, 0x7897d0f0
Caught exception in looking up space point ptr... 0x7897d0f0, 0x6a0bfc50
Caught exception in looking up space point ptr... 0x62bcf2c0, 0xa674f2f0
Caught exception in looking up space point ptr... 0xa674f2f0, 0x62bcf2c0
Caught exception in looking up space point ptr... 0x7d36d1c0, 0x7d36d030
Caught exception in looking up space point ptr... 0x7d36d030, 0x7d36d1c0
Caught exception in looking up space point ptr... 0x47e12ce0, 0x7b31fce0
Caught exception in looking up space point ptr... 0x7b31fce0, 0x47e12ce0
Caught exception in looking up space point ptr... 0x94efaf30, 0x38ae7fd0
Caught exception in looking up space point ptr... 0x38ae7fd0, 0xa1aba4d0
Caught exception in looking up space point ptr... 0x44f33550, 0x485518e0
Caught exception in looking up space point ptr... 0x485518e0, 0x4531a2e0
Caught exception in looking up space point ptr... 0x408ba6a0, 0xb1f23dc0
Caught exception in looking up space point ptr... 0xb1f23dc0, 0xb1f23b20
Caught exception in looking up space point ptr... 0x673912d0, 0x67391030
Caught exception in looking up space point ptr... 0x67391030, 0x67391180
Caught exception in looking up space point ptr... 0x960fa120, 0x960f9c70
Caught exception in looking up space point ptr... 0x960f9c70, 0x960fa120
Caught exception in looking up space point ptr... 0xa1567d30, 0xa1567a90
Caught exception in looking up space point ptr... 0xa1567a90, 0xa1567d30
Caught exception in looking up space point ptr... 0x5a582dd0, 0x5a582c80
Caught exception in looking up space point ptr... 0x5a582c80, 0x5a582b30
Caught exception in looking up space point ptr... 0x5a599520, 0x5a592da0
Caught exception in looking up space point ptr... 0x5a592da0, 0x5a599520
Caught exception in looking up space point ptr... 0x67b6c350, 0x7593ab10
Caught exception in looking up space point ptr... 0x7593ab10, 0x67b6c350
Caught exception in looking up space point ptr... 0x888ca260, 0x888ca400
Caught exception in looking up space point ptr... 0x888ca400, 0x888ca550
Caught exception in looking up space point ptr... 0x6acbbee0, 0x39e5acc0
Caught exception in looking up space point ptr... 0x6769d3b0, 0x6acbbee0
Caught exception in looking up space point ptr... 0x9dd0ab90, 0x748760a0
Caught exception in looking up space point ptr... 0x748760a0, 0x9dd0a990
Caught exception in looking up space point ptr... 0x428e6bb0, 0xa5b023f0
Caught exception in looking up space point ptr... 0xa5b023f0, 0xa5b022f0
Caught exception in looking up space point ptr... 0x67bbd0f0, 0xab08c470
Caught exception in looking up space point ptr... 0xab08c470, 0x45dbf940
Caught exception in looking up space point ptr... 0x972365d0, 0x9723b4b0
Caught exception in looking up space point ptr... 0x9723b4b0, 0x97238550
Caught exception in looking up space point ptr... 0x659c1df0, 0x659c1ba0
Caught exception in looking up space point ptr... 0x659c1ba0, 0x659c20d0
Caught exception in looking up space point ptr... 0x826ac640, 0x826ab210
Caught exception in looking up space point ptr... 0x826ab210, 0x826ac640
Caught exception in looking up space point ptr... 0xaae28280, 0xaae27e50
Caught exception in looking up space point ptr... 0xaae27e50, 0xaae28280
Caught exception in looking up space point ptr... 0x59454220, 0x59454090
Caught exception in looking up space point ptr... 0x59454090, 0x59454220
Caught exception in looking up space point ptr... 0xa1c48620, 0x9f346550
Caught exception in looking up space point ptr... 0x5dc787b0, 0xa1c48620
Caught exception in looking up space point ptr... 0x673c5970, 0x71a1ad70
Caught exception in looking up space point ptr... 0x71a1ad70, 0x4075b650
Caught exception in looking up space point ptr... 0x3fbcf940, 0x3fbcf6a0
Caught exception in looking up space point ptr... 0x3fbcf6a0, 0x3fbcf940
Caught exception in looking up space point ptr... 0x91f0eff0, 0x91f0ec80
Caught exception in looking up space point ptr... 0x91f0ec80, 0x91f0eff0
Caught exception in looking up space point ptr... 0x6ec7fb50, 0x9541fb90
Caught exception in looking up space point ptr... 0x6a9898d0, 0x6ec7fb50
Caught exception in looking up space point ptr... 0x48569260, 0x485690d0
Caught exception in looking up space point ptr... 0x485690d0, 0x48569260
Caught exception in looking up space point ptr... 0x81625c70, 0x96aea100
Caught exception in looking up space point ptr... 0x96aea100, 0x6ee264d0
Caught exception in looking up space point ptr... 0xa515d6b0, 0xa515d560
Caught exception in looking up space point ptr... 0xa515d560, 0xa515d6b0
Caught exception in looking up space point ptr... 0xb5802be0, 0xb5802e80
Caught exception in looking up space point ptr... 0xb5802e80, 0xb5802d30
Caught exception in looking up space point ptr... 0x5bbaee90, 0x415d28d0
Caught exception in looking up space point ptr... 0x415d28d0, 0x5bbaee90
++++>>>> total num hits: 84186, num free: 80443
04-Feb-2026 01:16:49 UTC  Closed output file "np02vd_raw_run042433_0444_df-s04-d0_dw_0_20260203T213052_reco_stage0_20260204T011649_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         226.134       248.914       276.244       247.751       11.5146        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       0.00010297    0.00015161    0.000352462   0.000115781    7.458e-05       19     
produce:tpcrawdecoder:PDVDTPCReader                19.6832       20.2564       20.823        20.2731      0.293552        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.723707      0.762715      0.824373      0.757438      0.0319688       19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0234903     0.0305945     0.0655963     0.0245243     0.0103462       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0232707     0.0240356     0.0324212     0.0235139     0.0019912       19     
produce:beamevent:BeamEvent                      0.000129131   0.00036209    0.00147666    0.000334822   0.000275758      19     
produce:pdvddaphne:DAPHNEReaderPDVD                4.17412       4.64398       5.26139       4.62656      0.288873        19     
produce:ophit:OpHitFinder                         0.0447592     0.0558917     0.0654101     0.0562314    0.00439334       19     
produce:wclsdatavd:WireCellToolkit                 157.657       173.203        184.7        174.513       7.83136        19     
produce:gaushit:GausHitFinder                      1.87383       2.43139       2.97932       2.48831      0.319224        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000519593   0.000693555    0.0011352    0.000658253   0.000166006      19     
produce:reco3d:SpacePointSolver                    22.2953       34.2374       57.2578       35.0493       7.92696        19     
produce:hitpdune:DisambigFromSpacePoints          0.218997      0.352259      0.541068      0.342578      0.075539        19     
produce:cluster3d:Cluster3D                        2.70208       4.4533        7.1075        4.47706       1.02211        19     
[art]:TriggerResults:TriggerResultInserter        3.591e-05    6.22447e-05   0.000138171   4.8551e-05    2.86656e-05      19     
end_path:out1:RootOutput                          9.29e-06     3.41301e-05   0.000397522    1.236e-05    8.58661e-05      19     
end_path:out1:RootOutput(write)                    6.30912       8.37819       9.25805       8.52219      0.780996        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5536.53 MB
  Peak resident set size usage (VmHWM): 3399.32 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0444_df-s04-d0_dw_0_20260203T213052_reco_stage0_20260204T011649_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0444_df-s04-d0_dw_0_20260203T213052_reco_stage0_20260204T011649_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0444_df-s04-d0_dw_0_20260203T213052_reco_stage0_20260204T011649_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770162982.0,
    "core.end_time": 1770167812.0,
    "core.events": [
      50626,
      50632,
      50638,
      50644,
      50650,
      50656,
      50662,
      50668,
      50674,
      50680,
      50686,
      50692,
      50698,
      50704,
      50710,
      50716,
      50722,
      50728,
      50734
    ],
    "core.event_count": 19,
    "core.first_event_number": 50626,
    "core.last_event_number": 50734,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0444_df-s04-d0_dw_0_20260203T213052.hdf5"
    }
  ]
}
justIN time: 2026-02-04 07:43:11 UTC       justIN version: 01.06.00