justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303647.64@dunegpschedd01.fnal.gov

Jobsub ID303647.64@dunegpschedd01.fnal.gov
Workflow ID12717
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:46:24
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce515
Last heartbeat2026-02-03 23:37:02
From worker nodeHostnameb9p35p0043.cern.ch
cpuinfoAMD EPYC 9655P 96-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 22:47:20
Input filesvd-protodune:np02vd_raw_run042421_0140_df-s02-d0_dw_0_20260203T144642.hdf5
JobscriptExit code0
Real time47m (2836s)
CPU time45m (2732s = 96%)
Max RSS bytes2769129472 (2640 MiB)
Outputting started2026-02-03 23:34:37
Output files
Finished2026-02-03 23:37:02
Saved logsjustin-logs:303647.64-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

aught exception in looking up space point ptr... 0x525f0e70, 0x412af790
Caught exception in looking up space point ptr... 0x2938ea10, 0x525f0e70
Caught exception in looking up space point ptr... 0x3b1e4d70, 0x3b1e4aa0
Caught exception in looking up space point ptr... 0x3b1e4aa0, 0x304cd1f0
Caught exception in looking up space point ptr... 0x4883a3e0, 0x4884c810
Caught exception in looking up space point ptr... 0x4884c810, 0x4884c9a0
Caught exception in looking up space point ptr... 0x4884c9a0, 0x383bdf60
Caught exception in looking up space point ptr... 0x58a60cc0, 0x58a60f90
Caught exception in looking up space point ptr... 0x58a54ca0, 0x58a60cc0
Caught exception in looking up space point ptr... 0x5b1b7320, 0x5b1b71a0
Caught exception in looking up space point ptr... 0x5b1b71a0, 0x5b1b7320
Caught exception in looking up space point ptr... 0x23368e20, 0x2336f8a0
Caught exception in looking up space point ptr... 0x2825cd90, 0x23368e20
Caught exception in looking up space point ptr... 0xc44cd30, 0x557cc5f0
Caught exception in looking up space point ptr... 0x50daa820, 0xc44cd30
Caught exception in looking up space point ptr... 0x5b63cb10, 0x5b62a550
Caught exception in looking up space point ptr... 0x5b63cca0, 0x5b63cb10
Caught exception in looking up space point ptr... 0x20cbcdb0, 0x40f8e800
Caught exception in looking up space point ptr... 0x40f8e800, 0x58a08fa0
Caught exception in looking up space point ptr... 0x400623c0, 0x419ea7c0
Caught exception in looking up space point ptr... 0x419ea7c0, 0x400623c0
Caught exception in looking up space point ptr... 0x400623c0, 0x2c63cee0
Caught exception in looking up space point ptr... 0x2c63cee0, 0x400623c0
Caught exception in looking up space point ptr... 0x2a9714a0, 0x14aba9d0
Caught exception in looking up space point ptr... 0x388c4670, 0x2a9714a0
Caught exception in looking up space point ptr... 0x423d7500, 0x423cc8e0
Caught exception in looking up space point ptr... 0x423cc8e0, 0x47099880
Caught exception in looking up space point ptr... 0x374276e0, 0x85fa790
Caught exception in looking up space point ptr... 0x85fa790, 0x53a35760
Caught exception in looking up space point ptr... 0x53d36370, 0x53d361e0
Caught exception in looking up space point ptr... 0x53d36050, 0x53d36370
Caught exception in looking up space point ptr... 0x53af2ff0, 0x53ae1520
Caught exception in looking up space point ptr... 0x5f974060, 0x53af2ff0
Caught exception in looking up space point ptr... 0x53accb20, 0x49366fa0
Caught exception in looking up space point ptr... 0x53adedc0, 0x53accb20
Caught exception in looking up space point ptr... 0x57cd5bb0, 0x83ffab0
Caught exception in looking up space point ptr... 0x57cd5d40, 0x57cd5bb0
Caught exception in looking up space point ptr... 0x293baf70, 0x14f53190
Caught exception in looking up space point ptr... 0x14f53190, 0x14f53320
Caught exception in looking up space point ptr... 0x4717a8f0, 0x246bbf00
Caught exception in looking up space point ptr... 0x4717aa80, 0x4717a8f0
Caught exception in looking up space point ptr... 0x50d0da70, 0x525a7b90
Caught exception in looking up space point ptr... 0x525a7b90, 0x525a7d20
Caught exception in looking up space point ptr... 0x400a3230, 0x400a3550
Caught exception in looking up space point ptr... 0x597e68e0, 0x400a3230
Caught exception in looking up space point ptr... 0x425f59b0, 0x63ecceb0
Caught exception in looking up space point ptr... 0x425f5b40, 0x425f59b0
Caught exception in looking up space point ptr... 0x2ab9d2b0, 0x2ab8bfb0
Caught exception in looking up space point ptr... 0x2ab8bfb0, 0x2ab9d120
Caught exception in looking up space point ptr... 0x2af1aa30, 0x2ab97680
Caught exception in looking up space point ptr... 0x2ab97810, 0x2af1aa30
Caught exception in looking up space point ptr... 0x23fd6df0, 0x2234cf00
Caught exception in looking up space point ptr... 0x2234cf00, 0x23fd6c60
Caught exception in looking up space point ptr... 0x155f23a0, 0x5b527120
Caught exception in looking up space point ptr... 0x5b527120, 0x5b519ca0
Caught exception in looking up space point ptr... 0x35d14010, 0xa708720
Caught exception in looking up space point ptr... 0xa708720, 0x35d14010
Caught exception in looking up space point ptr... 0x3086b1e0, 0x3ac76360
Caught exception in looking up space point ptr... 0x3ac76360, 0x253009f0
Caught exception in looking up space point ptr... 0x253009f0, 0x3ac76030
Caught exception in looking up space point ptr... 0x3ac76030, 0x3086b1e0
Caught exception in looking up space point ptr... 0x3a559050, 0x3a5592d0
Caught exception in looking up space point ptr... 0x3a5592d0, 0x3a559520
Caught exception in looking up space point ptr... 0x4890e7f0, 0x3b533980
Caught exception in looking up space point ptr... 0x3b533980, 0x4890e7f0
++++>>>> total num hits: 314049, num free: 307716
04-Feb-2026 00:34:29 CET  Closed output file "np02vd_raw_run042421_0140_df-s02-d0_dw_0_20260203T144642_reco_stage0_20260203T233429_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         71.3103       103.203       124.26        110.122       17.4464        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                        5.912e-05    7.75411e-05   0.00020084     7.075e-05    2.62911e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                3.87806       4.50976       5.57552       4.46587      0.313173        27     
produce:triggerrawdecoder:PDVDTriggerReader4     0.00830909     0.012888      0.0456158     0.0109595    0.00717805       27     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.00042306    0.000642414    0.0016362    0.00061354    0.000251807      27     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.00112999    0.00186269     0.0033509    0.00165873    0.000499735      27     
produce:beamevent:BeamEvent                       6.473e-05    0.000118984   0.00077146     9.498e-05    0.000128822      27     
produce:pdvddaphne:DAPHNEReaderPDVD                1.85249       2.14921       2.34165       2.15434      0.124163        27     
produce:ophit:OpHitFinder                         0.0235959     0.0346486     0.0415774     0.034819      0.0040037       27     
produce:wclsdatavd:WireCellToolkit                 47.552        55.219        61.6088       55.4562       3.0825         27     
produce:gaushit:GausHitFinder                     0.519623       1.0895        1.69834       1.07604      0.300794        27     
produce:nhitsfilter:NumberOfHitsFilter            0.0001375    0.000221258   0.00033201    0.00022438    5.14918e-05      27     
produce:reco3d:SpacePointSolver                    4.6673        7.34893       13.3067       7.32427       2.06769        27     
produce:hitpdune:DisambigFromSpacePoints          0.0419967     0.173969      0.359449      0.185013      0.0821442       27     
produce:cluster3d:Cluster3D                       0.691763       28.1139       42.6926       36.2922       15.9061        27     
[art]:TriggerResults:TriggerResultInserter        2.653e-05    3.57826e-05    7.51e-05      3.43e-05     9.58932e-06      27     
end_path:out1:RootOutput                          4.51e-06     6.61444e-06    1.793e-05     5.75e-06     3.27848e-06      27     
end_path:out1:RootOutput(write)                    3.21164       4.5126        5.91914       4.48612      0.566385        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4768.05 MB
  Peak resident set size usage (VmHWM): 2769.13 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042421_0140_df-s02-d0_dw_0_20260203T144642_reco_stage0_20260203T233429_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042421_0140_df-s02-d0_dw_0_20260203T144642_reco_stage0_20260203T233429_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042421_0140_df-s02-d0_dw_0_20260203T144642_reco_stage0_20260203T233429_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770158846.0,
    "core.end_time": 1770161671.0,
    "core.events": [
      22746,
      22752,
      22758,
      22764,
      22770,
      22776,
      22782,
      22788,
      22794,
      22800,
      22806,
      22812,
      22818,
      22824,
      22830,
      22836,
      22842,
      22848,
      22854,
      22860,
      22866,
      22872,
      22878,
      22884,
      22890,
      22896,
      22902
    ],
    "core.event_count": 27,
    "core.first_event_number": 22746,
    "core.last_event_number": 22902,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42421
    ],
    "core.runs_subruns": [
      4242100001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042421_0140_df-s02-d0_dw_0_20260203T144642.hdf5"
    }
  ]
}
justIN time: 2026-02-04 07:44:41 UTC       justIN version: 01.06.00