justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297921.38@dunegpschedd02.fnal.gov

Jobsub ID297921.38@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:58:26
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce512
Last heartbeat2026-02-04 00:46:03
From worker nodeHostnameb9g24p4713.cern.ch
cpuinfoAMD EPYC 7302 16-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:47:34
Input filesvd-protodune:np02vd_raw_run042433_0226_df-s03-d1_dw_0_20260203T200802.hdf5
JobscriptExit code0
Real time55m (3357s)
CPU time54m (3251s = 96%)
Max RSS bytes3601985536 (3435 MiB)
Outputting started2026-02-04 00:43:33
Output files
Finished2026-02-04 00:46:03
Saved logsjustin-logs:297921.38-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ptr... 0x3c983450, 0x4a087c50
Caught exception in looking up space point ptr... 0x14eb86f0, 0x14eb8260
Caught exception in looking up space point ptr... 0x14eb8260, 0x14eafc20
Caught exception in looking up space point ptr... 0x6884c2a0, 0x320c2bf0
Caught exception in looking up space point ptr... 0x6884c5c0, 0x6884c2a0
Caught exception in looking up space point ptr... 0xf7ca230, 0x28fd5d80
Caught exception in looking up space point ptr... 0x28fd5d80, 0x28fd60a0
Caught exception in looking up space point ptr... 0x7ff46360, 0x742bd4e0
Caught exception in looking up space point ptr... 0x742bd4e0, 0x7ff46360
Caught exception in looking up space point ptr... 0x28df1af0, 0x14ae5330
Caught exception in looking up space point ptr... 0x14ae5330, 0x28df1af0
Caught exception in looking up space point ptr... 0x445f65d0, 0x445f62b0
Caught exception in looking up space point ptr... 0x445f62b0, 0x3515a4e0
Caught exception in looking up space point ptr... 0x60b83590, 0x29383610
Caught exception in looking up space point ptr... 0x29383610, 0x60b83590
Caught exception in looking up space point ptr... 0x36f7a040, 0x36f7a140
Caught exception in looking up space point ptr... 0x36f7a140, 0x36f7a040
Caught exception in looking up space point ptr... 0x8545f710, 0x8545f480
Caught exception in looking up space point ptr... 0x8545f480, 0x85469980
Caught exception in looking up space point ptr... 0x3c256410, 0x3c2565a0
Caught exception in looking up space point ptr... 0x3c2447b0, 0x3c256410
Caught exception in looking up space point ptr... 0x5df182a0, 0x5df18430
Caught exception in looking up space point ptr... 0x5df05ce0, 0x5df182a0
Caught exception in looking up space point ptr... 0x5e4e9590, 0x160a35e0
Caught exception in looking up space point ptr... 0x160a35e0, 0x160a3450
Caught exception in looking up space point ptr... 0x393217c0, 0x39321620
Caught exception in looking up space point ptr... 0x13f4bd00, 0x393217c0
Caught exception in looking up space point ptr... 0x2e759800, 0x2e7473d0
Caught exception in looking up space point ptr... 0x2e7473d0, 0x2e747240
Caught exception in looking up space point ptr... 0x32345120, 0xbd53b60
Caught exception in looking up space point ptr... 0x32344e80, 0x32345120
Caught exception in looking up space point ptr... 0x154c5b10, 0x15499de0
Caught exception in looking up space point ptr... 0x15499de0, 0x154c58c0
Caught exception in looking up space point ptr... 0x3624fb40, 0x575dc8e0
Caught exception in looking up space point ptr... 0x3624fcd0, 0x3624fb40
Caught exception in looking up space point ptr... 0x6805dca0, 0x32a88e20
Caught exception in looking up space point ptr... 0x6805de30, 0x6805dca0
Caught exception in looking up space point ptr... 0x43ae0bc0, 0x43ae0710
Caught exception in looking up space point ptr... 0x43ae0ee0, 0x43ae0bc0
Caught exception in looking up space point ptr... 0xf209ed0, 0x365f27b0
Caught exception in looking up space point ptr... 0xf20a060, 0xf209ed0
Caught exception in looking up space point ptr... 0x57005b00, 0x56ff3860
Caught exception in looking up space point ptr... 0x56ff3860, 0x17e50f10
Caught exception in looking up space point ptr... 0x3c8a1820, 0x3c8a1970
Caught exception in looking up space point ptr... 0x3c8a1970, 0x3c8a1ac0
Caught exception in looking up space point ptr... 0x6d7ce610, 0x28ed6110
Caught exception in looking up space point ptr... 0x28ed6110, 0xf9014a0
Caught exception in looking up space point ptr... 0x5cba5840, 0x1608bf40
Caught exception in looking up space point ptr... 0x18cd2480, 0x5cba5840
Caught exception in looking up space point ptr... 0x28b2d3f0, 0x37274f40
Caught exception in looking up space point ptr... 0x37274f40, 0x28b2d3f0
Caught exception in looking up space point ptr... 0x7431d510, 0x595f6b40
Caught exception in looking up space point ptr... 0x595f6b40, 0x85939b20
Caught exception in looking up space point ptr... 0x2e7bd050, 0x745a0960
Caught exception in looking up space point ptr... 0x745a0960, 0x745a07d0
Caught exception in looking up space point ptr... 0x3721ef70, 0x4e298220
Caught exception in looking up space point ptr... 0x4e298220, 0x3721ef70
Caught exception in looking up space point ptr... 0x3a576e70, 0x3a576c90
Caught exception in looking up space point ptr... 0x3a576c90, 0x3a576e70
Caught exception in looking up space point ptr... 0xf33b900, 0x81c964b0
Caught exception in looking up space point ptr... 0x81c964b0, 0x6c6edaa0
Caught exception in looking up space point ptr... 0x60b81440, 0x39c40900
Caught exception in looking up space point ptr... 0x39c40900, 0x64a04110
Caught exception in looking up space point ptr... 0x344497e0, 0x11a94720
Caught exception in looking up space point ptr... 0x11a94720, 0x344497e0
Caught exception in looking up space point ptr... 0x309bc880, 0x3a1dad40
Caught exception in looking up space point ptr... 0x3a1dad40, 0x56e95c00
++++>>>> total num hits: 513263, num free: 502859
04-Feb-2026 01:43:25 CET  Closed output file "np02vd_raw_run042433_0226_df-s03-d1_dw_0_20260203T200802_reco_stage0_20260204T004325_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         131.323       172.925       234.147       181.594       29.233         19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       5.2641e-05    0.00010776    0.000295196   8.0631e-05    6.34604e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                6.19937       7.27751       10.537        7.21533       0.95761        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.0100636     0.014918      0.0416139     0.0135484    0.00658848       19     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000338377   0.000671736   0.00104172    0.000626032    0.0002011       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.000395248   0.000532074   0.000892347   0.00048551    0.000135214      19     
produce:beamevent:BeamEvent                      6.1052e-05    0.000150386   0.000921558   0.000101882   0.000187426      19     
produce:pdvddaphne:DAPHNEReaderPDVD                2.20811       2.51475        3.157        2.44841      0.226947        19     
produce:ophit:OpHitFinder                         0.022623      0.0308647     0.0397707     0.0299875    0.00396738       19     
produce:wclsdatavd:WireCellToolkit                 100.05        109.922       129.37        106.378       9.17092        19     
produce:gaushit:GausHitFinder                     0.951866        1.706        3.49203       1.56081      0.632741        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000279016   0.000402452   0.000923288   0.000359367   0.000142783      19     
produce:reco3d:SpacePointSolver                    7.65657       13.6979       21.9309       12.9479       3.68252        19     
produce:hitpdune:DisambigFromSpacePoints          0.107922      0.243179       0.41106      0.224282      0.0697602       19     
produce:cluster3d:Cluster3D                        1.61522       31.7273       60.6217       48.2462       25.2279        19     
[art]:TriggerResults:TriggerResultInserter        2.737e-05    3.38077e-05   8.8252e-05    3.0331e-05    1.30627e-05      19     
end_path:out1:RootOutput                          4.22e-06     7.23758e-06   2.1051e-05     6.01e-06     3.68717e-06      19     
end_path:out1:RootOutput(write)                    4.57403       5.7599        7.69509       5.8941       0.825198        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5546.13 MB
  Peak resident set size usage (VmHWM): 3601.99 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0226_df-s03-d1_dw_0_20260203T200802_reco_stage0_20260204T004325_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0226_df-s03-d1_dw_0_20260203T200802_reco_stage0_20260204T004325_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0226_df-s03-d1_dw_0_20260203T200802_reco_stage0_20260204T004325_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770162469.0,
    "core.end_time": 1770165807.0,
    "core.events": [
      25773,
      25779,
      25785,
      25791,
      25797,
      25803,
      25809,
      25815,
      25821,
      25827,
      25833,
      25839,
      25845,
      25851,
      25857,
      25863,
      25869,
      25875,
      25881
    ],
    "core.event_count": 19,
    "core.first_event_number": 25773,
    "core.last_event_number": 25881,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0226_df-s03-d1_dw_0_20260203T200802.hdf5"
    }
  ]
}
justIN time: 2026-02-04 06:06:49 UTC       justIN version: 01.06.00