justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303653.46@dunegpschedd01.fnal.gov

Jobsub ID303653.46@dunegpschedd01.fnal.gov
Workflow ID12717
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:52:24
SiteUK_Lancaster
EntryUBoone_UK_Lancaster_HEC_grendel_ce02
Last heartbeat2026-02-04 01:14:19
From worker nodeHostnamecomp22-07
cpuinfoIntel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:40:58
Input filesvd-protodune:np02vd_raw_run042421_0132_df-s04-d1_dw_0_20260203T144225.hdf5
JobscriptExit code0
Real time1h (5349s)
CPU time41m (2471s = 46%)
Max RSS bytes2721955840 (2595 MiB)
Outputting started2026-02-04 01:10:10
Output files
Finished2026-02-04 01:14:19
Saved logsjustin-logs:303653.46-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

bb0
Caught exception in looking up space point ptr... 0x410f4bb0, 0x410f4a80
Caught exception in looking up space point ptr... 0x4e258ef0, 0x4e2584c0
Caught exception in looking up space point ptr... 0x4e2584c0, 0x4e258ef0
Caught exception in looking up space point ptr... 0x2d23e750, 0x2bb7add0
Caught exception in looking up space point ptr... 0x2bb7add0, 0x3781d870
Caught exception in looking up space point ptr... 0x16c23480, 0x16c233a0
Caught exception in looking up space point ptr... 0x454db2c0, 0x16c23480
Caught exception in looking up space point ptr... 0x24d446a0, 0x501a40c0
Caught exception in looking up space point ptr... 0x501a40c0, 0x24d446a0
Caught exception in looking up space point ptr... 0x24d446a0, 0x2210a8e0
Caught exception in looking up space point ptr... 0x2210a8e0, 0x24d446a0
Caught exception in looking up space point ptr... 0x24d446a0, 0x23278690
Caught exception in looking up space point ptr... 0x23278690, 0x24d446a0
Caught exception in looking up space point ptr... 0x454d8db0, 0x454cd540
Caught exception in looking up space point ptr... 0x454cd540, 0x454d8db0
Caught exception in looking up space point ptr... 0x5217d4a0, 0x5217d6f0
Caught exception in looking up space point ptr... 0x5217d1b0, 0x5217d4a0
Caught exception in looking up space point ptr... 0x25124e30, 0x2512ed20
Caught exception in looking up space point ptr... 0x2512ed20, 0x25124e30
Caught exception in looking up space point ptr... 0x326bebe0, 0x1284e870
Caught exception in looking up space point ptr... 0x1284e870, 0x326bebe0
Caught exception in looking up space point ptr... 0x27a443b0, 0x127740c0
Caught exception in looking up space point ptr... 0x127740c0, 0x27a443b0
Caught exception in looking up space point ptr... 0x554508f0, 0x55452130
Caught exception in looking up space point ptr... 0x55452130, 0x55451bc0
Caught exception in looking up space point ptr... 0x3396c830, 0x33970470
Caught exception in looking up space point ptr... 0x33970470, 0x3396cab0
Caught exception in looking up space point ptr... 0x55c1ef60, 0x55c1f0f0
Caught exception in looking up space point ptr... 0x55c1f0f0, 0x55c1ef60
Caught exception in looking up space point ptr... 0x529f56f0, 0x63bf8b90
Caught exception in looking up space point ptr... 0x63bf8b90, 0x319c6890
Caught exception in looking up space point ptr... 0x319c6890, 0x11bb7410
Caught exception in looking up space point ptr... 0x11bb7410, 0xbceb560
Caught exception in looking up space point ptr... 0x258c6e00, 0x258b4620
Caught exception in looking up space point ptr... 0x258b4620, 0x258c2480
Caught exception in looking up space point ptr... 0x15163320, 0x3fbd0ae0
Caught exception in looking up space point ptr... 0x3fbd0ae0, 0x151634b0
Caught exception in looking up space point ptr... 0x2bc0b640, 0x2bc0b320
Caught exception in looking up space point ptr... 0x2bc0b320, 0x2bc0b640
Caught exception in looking up space point ptr... 0xe5738d0, 0xe5735b0
Caught exception in looking up space point ptr... 0xe5735b0, 0xe573a60
Caught exception in looking up space point ptr... 0x37e3a270, 0x30fe8310
Caught exception in looking up space point ptr... 0x33953870, 0x37e3a270
Caught exception in looking up space point ptr... 0x3fd23240, 0x4569c650
Caught exception in looking up space point ptr... 0x4569c650, 0x456a1ae0
Caught exception in looking up space point ptr... 0x456e7950, 0x12e24bc0
Caught exception in looking up space point ptr... 0x12e24bc0, 0x456e7950
Caught exception in looking up space point ptr... 0xe54df90, 0xe54de90
Caught exception in looking up space point ptr... 0xe54de90, 0x39fdc110
Caught exception in looking up space point ptr... 0x454df810, 0x454df9b0
Caught exception in looking up space point ptr... 0x454df9b0, 0x41d050a0
Caught exception in looking up space point ptr... 0x2dca2ad0, 0x4d1d2180
Caught exception in looking up space point ptr... 0x4d1d2180, 0x2dca2ad0
Caught exception in looking up space point ptr... 0x2b16a4d0, 0x31d0e820
Caught exception in looking up space point ptr... 0x31d0e820, 0x125bebf0
Caught exception in looking up space point ptr... 0x55669430, 0x55669560
Caught exception in looking up space point ptr... 0x55669560, 0x556692b0
Caught exception in looking up space point ptr... 0xe4f13f0, 0x41137120
Caught exception in looking up space point ptr... 0x41137120, 0xe4f14f0
Caught exception in looking up space point ptr... 0x2a93d590, 0x2a9362d0
Caught exception in looking up space point ptr... 0x2a9362d0, 0x2a93bc60
Caught exception in looking up space point ptr... 0x22abe410, 0x22abe310
Caught exception in looking up space point ptr... 0x22abe310, 0x22abe410
Caught exception in looking up space point ptr... 0x151b9320, 0x151b9000
Caught exception in looking up space point ptr... 0x151b9000, 0x151b9320
++++>>>> total num hits: 80047, num free: 76218
04-Feb-2026 00:53:44 GMT  Closed output file "np02vd_raw_run042421_0132_df-s04-d1_dw_0_20260203T144225_reco_stage0_20260204T005344_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         95.7879       158.041       368.331       125.374       66.5578        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       7.2457e-05    0.00013941    0.00025236    0.000130583   3.5944e-05       27     
produce:tpcrawdecoder:PDVDTPCReader                14.3244       48.2447       101.704       34.1988       33.1126        27     
produce:triggerrawdecoder:PDVDTriggerReader4      0.719411      0.919146       1.51476      0.816538      0.211289        27     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.023471      0.0322006     0.0667962     0.0304089    0.00994438       27     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.115949      0.141893      0.201237      0.149511      0.0246084       27     
produce:beamevent:BeamEvent                      0.00014672    0.000209122   0.00134636    0.00015924    0.000223565      27     
produce:pdvddaphne:DAPHNEReaderPDVD                5.18109       9.46687       18.8288       6.94788       4.27961        27     
produce:ophit:OpHitFinder                         0.0265754      0.03102      0.034474      0.0313548    0.00189491       27     
produce:wclsdatavd:WireCellToolkit                 61.6157       75.6396       282.921       64.6421       43.2802        27     
produce:gaushit:GausHitFinder                     0.484433      0.893483       1.26287      0.865053      0.189496        27     
produce:nhitsfilter:NumberOfHitsFilter           0.000206899   0.000318934   0.000452717   0.000310511   6.43206e-05      27     
produce:reco3d:SpacePointSolver                    4.75118       10.5267       32.6537       9.22496       5.47912        27     
produce:hitpdune:DisambigFromSpacePoints          0.0537367     0.107832      0.211806      0.104093      0.0379349       27     
produce:cluster3d:Cluster3D                       0.523663       8.65312       131.255       1.47451       24.9473        27     
[art]:TriggerResults:TriggerResultInserter       1.9263e-05    2.65292e-05   6.9211e-05     2.526e-05    8.81048e-06      27     
end_path:out1:RootOutput                          6.657e-06    1.76613e-05   0.000275373    7.133e-06    5.05938e-05      27     
end_path:out1:RootOutput(write)                    2.81026       3.35544       5.89862       3.17649      0.604825        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4632.52 MB
  Peak resident set size usage (VmHWM): 2721.96 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042421_0132_df-s04-d1_dw_0_20260203T144225_reco_stage0_20260204T005344_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042421_0132_df-s04-d1_dw_0_20260203T144225_reco_stage0_20260204T005344_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042421_0132_df-s04-d1_dw_0_20260203T144225_reco_stage0_20260204T005344_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770162067.0,
    "core.end_time": 1770166985.0,
    "core.events": [
      21455,
      21461,
      21467,
      21473,
      21479,
      21485,
      21491,
      21497,
      21503,
      21509,
      21515,
      21521,
      21527,
      21533,
      21539,
      21545,
      21551,
      21557,
      21563,
      21569,
      21575,
      21581,
      21587,
      21593,
      21599,
      21605,
      21611
    ],
    "core.event_count": 27,
    "core.first_event_number": 21455,
    "core.last_event_number": 21611,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42421
    ],
    "core.runs_subruns": [
      4242100001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042421_0132_df-s04-d1_dw_0_20260203T144225.hdf5"
    }
  ]
}
justIN time: 2026-02-04 04:27:04 UTC       justIN version: 01.06.00