justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303654.85@dunegpschedd01.fnal.gov

Jobsub ID303654.85@dunegpschedd01.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:52:26
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce510
Last heartbeat2026-02-04 00:57:22
From worker nodeHostnameb9p34p1010.cern.ch
cpuinfoAMD EPYC 9655 96-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-04 00:05:33
Input filesvd-protodune:np02vd_raw_run042433_0425_df-s04-d0_dw_0_20260203T212339.hdf5
JobscriptExit code0
Real time49m (2955s)
CPU time47m (2859s = 96%)
Max RSS bytes3654815744 (3485 MiB)
Outputting started2026-02-04 00:54:50
Output files
Finished2026-02-04 00:57:22
Saved logsjustin-logs:303654.85-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

tr... 0x3745fd40, 0x3fe59e20
Caught exception in looking up space point ptr... 0x8935f40, 0x3745fd40
Caught exception in looking up space point ptr... 0x2bcdfdb0, 0x44d4bdd0
Caught exception in looking up space point ptr... 0x2bcdfc20, 0x2bcdfdb0
Caught exception in looking up space point ptr... 0x145e51b0, 0x423fa020
Caught exception in looking up space point ptr... 0x423fa020, 0x5c5a1950
Caught exception in looking up space point ptr... 0x43078580, 0x19f575b0
Caught exception in looking up space point ptr... 0x19f575b0, 0x43078580
Caught exception in looking up space point ptr... 0x307b6030, 0x307a79b0
Caught exception in looking up space point ptr... 0x307a79b0, 0x5ae058d0
Caught exception in looking up space point ptr... 0x557b9290, 0x557c60d0
Caught exception in looking up space point ptr... 0x55799b70, 0x557abfa0
Caught exception in looking up space point ptr... 0x557abfa0, 0x557b9290
Caught exception in looking up space point ptr... 0x2b946da0, 0x300f7d00
Caught exception in looking up space point ptr... 0x300f7d00, 0xf35f480
Caught exception in looking up space point ptr... 0x747bbe20, 0x59d77d20
Caught exception in looking up space point ptr... 0x59d77d20, 0x747bbf50
Caught exception in looking up space point ptr... 0x13a1b2c0, 0x13a1b190
Caught exception in looking up space point ptr... 0x13a1b190, 0x6fa6ff10
Caught exception in looking up space point ptr... 0x3b716ca0, 0x3b716ef0
Caught exception in looking up space point ptr... 0x3b7225a0, 0x3b716ca0
Caught exception in looking up space point ptr... 0x5a8ff730, 0x50529180
Caught exception in looking up space point ptr... 0x50529180, 0x5a902070
Caught exception in looking up space point ptr... 0x4f7feff0, 0x4f8005d0
Caught exception in looking up space point ptr... 0x3a0267b0, 0x4f7feff0
Caught exception in looking up space point ptr... 0x13e27210, 0x13e268b0
Caught exception in looking up space point ptr... 0x13e268b0, 0x15c0e9b0
Caught exception in looking up space point ptr... 0x4fbe8e30, 0x16f57d30
Caught exception in looking up space point ptr... 0x16f57d30, 0x6ff1afc0
Caught exception in looking up space point ptr... 0x632a5cf0, 0x2f9aa480
Caught exception in looking up space point ptr... 0x749720b0, 0x632a5cf0
Caught exception in looking up space point ptr... 0x139fe8d0, 0x65ae2ba0
Caught exception in looking up space point ptr... 0x139fe740, 0x139fe8d0
Caught exception in looking up space point ptr... 0x8438e760, 0x621cb590
Caught exception in looking up space point ptr... 0x621cb590, 0x621cb720
Caught exception in looking up space point ptr... 0x842d9760, 0x682e5f30
Caught exception in looking up space point ptr... 0x682e5f30, 0x682e5da0
Caught exception in looking up space point ptr... 0x397ab320, 0x83c63f50
Caught exception in looking up space point ptr... 0x83c63f50, 0x3c548520
Caught exception in looking up space point ptr... 0x3c6abcf0, 0x469db250
Caught exception in looking up space point ptr... 0x469db250, 0x13f709a0
Caught exception in looking up space point ptr... 0x13a2ace0, 0x13a2ae70
Caught exception in looking up space point ptr... 0x13a2ae70, 0x13a2b000
Caught exception in looking up space point ptr... 0x383f6fc0, 0x66fa4490
Caught exception in looking up space point ptr... 0x66fa4490, 0x342bddc0
Caught exception in looking up space point ptr... 0x454a2520, 0x5c696090
Caught exception in looking up space point ptr... 0x454a26b0, 0x454a2520
Caught exception in looking up space point ptr... 0x6adf0ab0, 0xa867910
Caught exception in looking up space point ptr... 0x6adf0c40, 0x6adf0ab0
Caught exception in looking up space point ptr... 0x894d320, 0x4fc00010
Caught exception in looking up space point ptr... 0x894d4b0, 0x894d320
Caught exception in looking up space point ptr... 0x1a3c4260, 0x1a3b1980
Caught exception in looking up space point ptr... 0x1a3d6820, 0x1a3c4260
Caught exception in looking up space point ptr... 0x11a5fd20, 0x11a71fc0
Caught exception in looking up space point ptr... 0x11a51130, 0x11a5fd20
Caught exception in looking up space point ptr... 0x635490e0, 0x6177b630
Caught exception in looking up space point ptr... 0x6177b630, 0x8abdb430
Caught exception in looking up space point ptr... 0x3ce52430, 0x36cdd9a0
Caught exception in looking up space point ptr... 0x36cdd9a0, 0x3ce52430
Caught exception in looking up space point ptr... 0x3ce52430, 0x55a95070
Caught exception in looking up space point ptr... 0x55a95070, 0x3ce52430
Caught exception in looking up space point ptr... 0x3de08be0, 0x3de08640
Caught exception in looking up space point ptr... 0x3de08640, 0x3de08a00
Caught exception in looking up space point ptr... 0x11b02660, 0x498b38d0
Caught exception in looking up space point ptr... 0x498b38d0, 0x11b02660
Caught exception in looking up space point ptr... 0x4e80ac90, 0x4e80b9a0
Caught exception in looking up space point ptr... 0x4e80b9a0, 0x4e80ac90
++++>>>> total num hits: 541006, num free: 531759
04-Feb-2026 01:54:42 CET  Closed output file "np02vd_raw_run042433_0425_df-s04-d0_dw_0_20260203T212339_reco_stage0_20260204T005442_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         104.997        152.7        176.826       160.774       20.6153        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                        6.701e-05    9.28558e-05   0.000152718   8.5518e-05    2.15694e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                6.34955       6.69383       7.58365       6.67097      0.277821        19     
produce:triggerrawdecoder:PDVDTriggerReader4     0.00701433     0.0110769     0.0314555    0.00983051     0.0050155       19     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000360007   0.000565403   0.000958357   0.000540185   0.000149335      19     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.000302571   0.000447173   0.000702107   0.000399786   0.000113256      19     
produce:beamevent:BeamEvent                      7.4291e-05    0.000138661   0.000829266   9.9278e-05    0.000163734      19     
produce:pdvddaphne:DAPHNEReaderPDVD                1.85991       2.10413       2.3677        2.08089      0.127107        19     
produce:ophit:OpHitFinder                         0.0284868     0.0329576     0.0405492     0.0320842    0.00361854       19     
produce:wclsdatavd:WireCellToolkit                 75.2656       83.1642       91.2246       83.254        5.15125        19     
produce:gaushit:GausHitFinder                     0.715857       1.78184       2.47791       1.87238      0.474347        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000193899   0.000314096   0.000740904   0.000277524   0.000135464      19     
produce:reco3d:SpacePointSolver                    7.66418       13.0992       19.1084       13.8011       3.3339         19     
produce:hitpdune:DisambigFromSpacePoints          0.0961546     0.254176      0.352403      0.262638      0.0707175       19     
produce:cluster3d:Cluster3D                        1.02007       39.886        53.2146       50.2496       19.7009        19     
[art]:TriggerResults:TriggerResultInserter       3.1117e-05    4.74565e-05   0.000106078   4.4636e-05    1.60819e-05      19     
end_path:out1:RootOutput                          4.937e-06    8.32658e-06   2.8292e-05     6.89e-06     5.38125e-06      19     
end_path:out1:RootOutput(write)                    4.64927       5.63016       6.52448       5.53926      0.613442        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5740.07 MB
  Peak resident set size usage (VmHWM): 3654.82 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0425_df-s04-d0_dw_0_20260203T212339_reco_stage0_20260204T005442_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0425_df-s04-d0_dw_0_20260203T212339_reco_stage0_20260204T005442_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0425_df-s04-d0_dw_0_20260203T212339_reco_stage0_20260204T005442_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770163542.0,
    "core.end_time": 1770166484.0,
    "core.events": [
      48460,
      48466,
      48472,
      48478,
      48484,
      48490,
      48496,
      48502,
      48508,
      48514,
      48520,
      48526,
      48532,
      48538,
      48544,
      48550,
      48556,
      48562,
      48568
    ],
    "core.event_count": 19,
    "core.first_event_number": 48460,
    "core.last_event_number": 48568,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0425_df-s04-d0_dw_0_20260203T212339.hdf5"
    }
  ]
}
justIN time: 2026-02-04 05:56:34 UTC       justIN version: 01.06.00