justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297921.80@dunegpschedd02.fnal.gov

Jobsub ID297921.80@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:58:26
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce510
Last heartbeat2026-02-04 00:40:13
From worker nodeHostnameb9p35p0023.cern.ch
cpuinfoAMD EPYC 9655P 96-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:53:22
Input filesvd-protodune:np02vd_raw_run042433_0095_df-s02-d0_dw_0_20260203T191812.hdf5
JobscriptExit code0
Real time44m (2674s)
CPU time43m (2606s = 97%)
Max RSS bytes3624472576 (3456 MiB)
Outputting started2026-02-04 00:37:58
Output files
Finished2026-02-04 00:40:13
Saved logsjustin-logs:297921.80-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

g up space point ptr... 0x3ac37c00, 0x8139fe40
Caught exception in looking up space point ptr... 0x12978e40, 0x12979090
Caught exception in looking up space point ptr... 0x12979090, 0x9bf5810
Caught exception in looking up space point ptr... 0x51b1fc60, 0x51b1fdb0
Caught exception in looking up space point ptr... 0x51b1fdb0, 0x51b1fc60
Caught exception in looking up space point ptr... 0x45b498d0, 0x81e66230
Caught exception in looking up space point ptr... 0x81e66230, 0x9e57fa0
Caught exception in looking up space point ptr... 0x9e57fa0, 0x400a4730
Caught exception in looking up space point ptr... 0x45b7e2f0, 0x45b5bd00
Caught exception in looking up space point ptr... 0x45b5bd00, 0x4e7da350
Caught exception in looking up space point ptr... 0xdff9450, 0xdff4800
Caught exception in looking up space point ptr... 0x18fc05c0, 0xdff9450
Caught exception in looking up space point ptr... 0x6d7bd640, 0x6d7cfa70
Caught exception in looking up space point ptr... 0x3b9adfc0, 0x6d7bd640
Caught exception in looking up space point ptr... 0x7dc4530, 0x28eafc20
Caught exception in looking up space point ptr... 0x543c8350, 0x7dc4530
Caught exception in looking up space point ptr... 0xfa4b210, 0x4f38f120
Caught exception in looking up space point ptr... 0xdfee5c0, 0xfa4b210
Caught exception in looking up space point ptr... 0x4b5650e0, 0x4b565370
Caught exception in looking up space point ptr... 0x4b54c460, 0x4b5650e0
Caught exception in looking up space point ptr... 0x80ddb50, 0x8147230
Caught exception in looking up space point ptr... 0x5cdbccd0, 0x17408ac0
Caught exception in looking up space point ptr... 0x17408ac0, 0x80ddb50
Caught exception in looking up space point ptr... 0x111ad520, 0x6ff7f680
Caught exception in looking up space point ptr... 0x6ff7f680, 0x6ff7f4f0
Caught exception in looking up space point ptr... 0x29bbac40, 0x29baa110
Caught exception in looking up space point ptr... 0x29baa110, 0x29bba920
Caught exception in looking up space point ptr... 0x69f3fc80, 0x69f3d2e0
Caught exception in looking up space point ptr... 0x69f3d2e0, 0x69f3d530
Caught exception in looking up space point ptr... 0x3ddfffe0, 0x3ef78470
Caught exception in looking up space point ptr... 0x3ef78470, 0x3ef730f0
Caught exception in looking up space point ptr... 0x7c070790, 0xca7a530
Caught exception in looking up space point ptr... 0xca7a530, 0xca7a7c0
Caught exception in looking up space point ptr... 0x6d673d40, 0x6463d080
Caught exception in looking up space point ptr... 0x6d673a20, 0x6d673d40
Caught exception in looking up space point ptr... 0x5dc938f0, 0x4567a640
Caught exception in looking up space point ptr... 0x4567a640, 0x4567a320
Caught exception in looking up space point ptr... 0x6ff9ab60, 0x3754e3f0
Caught exception in looking up space point ptr... 0x15e9dd10, 0x6ff9ab60
Caught exception in looking up space point ptr... 0x4b7729a0, 0x182e94f0
Caught exception in looking up space point ptr... 0x366d4260, 0x4b7729a0
Caught exception in looking up space point ptr... 0x10bcf660, 0x5cc5a720
Caught exception in looking up space point ptr... 0x5cc5a720, 0x10bcf340
Caught exception in looking up space point ptr... 0x108437a0, 0x7a1ef350
Caught exception in looking up space point ptr... 0x7a1ef350, 0x5a2ee3f0
Caught exception in looking up space point ptr... 0x6d771320, 0x1276a290
Caught exception in looking up space point ptr... 0x6d75ea40, 0x6d771320
Caught exception in looking up space point ptr... 0x356c7960, 0x5ec237a0
Caught exception in looking up space point ptr... 0x5ec237a0, 0x3b39f070
Caught exception in looking up space point ptr... 0x663c4930, 0x8908ac70
Caught exception in looking up space point ptr... 0x8908ac70, 0x8908aae0
Caught exception in looking up space point ptr... 0x37d53ce0, 0x37d53e70
Caught exception in looking up space point ptr... 0x37d30100, 0x37d53ce0
Caught exception in looking up space point ptr... 0x3a92f040, 0x18f6b4e0
Caught exception in looking up space point ptr... 0x8141400, 0x3a92f040
Caught exception in looking up space point ptr... 0x5f4b9180, 0x6abe9b80
Caught exception in looking up space point ptr... 0x5f4b94a0, 0x5f4b9180
Caught exception in looking up space point ptr... 0xfaac890, 0x1964a790
Caught exception in looking up space point ptr... 0xfaaca20, 0xfaac890
Caught exception in looking up space point ptr... 0x40e9bc90, 0x40e9be00
Caught exception in looking up space point ptr... 0x40e9be00, 0x40e9bc90
Caught exception in looking up space point ptr... 0x9f723b0, 0x3c9758d0
Caught exception in looking up space point ptr... 0x3c9758d0, 0x9f723b0
Caught exception in looking up space point ptr... 0x75f2ee0, 0x4045fc00
Caught exception in looking up space point ptr... 0x4045fc00, 0x827b5bb0
Caught exception in looking up space point ptr... 0x4208f470, 0x45fb6a60
Caught exception in looking up space point ptr... 0x45fb6a60, 0x4208f470
++++>>>> total num hits: 502012, num free: 494480
04-Feb-2026 01:37:52 CET  Closed output file "np02vd_raw_run042433_0095_df-s02-d0_dw_0_20260203T191812_reco_stage0_20260204T003752_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         96.6216       138.073       163.71        139.796       20.2042        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                        4.763e-05    7.08281e-05   0.000149301    6.844e-05    1.98079e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                3.95783       5.04502       6.42374       4.54301      0.885427        19     
produce:triggerrawdecoder:PDVDTriggerReader4     0.00712068     0.0113019      0.02004     0.00891075    0.00428846       19     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000305751   0.000438658   0.000607942   0.000451111   6.58955e-05      19     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.000268541   0.000393673   0.000583542   0.000370161   8.53889e-05      19     
produce:beamevent:BeamEvent                       5.701e-05    0.000108531   0.000733963    7.782e-05    0.000147726      19     
produce:pdvddaphne:DAPHNEReaderPDVD                1.64114       1.93586       2.23038        1.978       0.200366        19     
produce:ophit:OpHitFinder                         0.0223056     0.0316955     0.0392914     0.0330839    0.00426552       19     
produce:wclsdatavd:WireCellToolkit                 70.5108       78.2271       87.3397       77.5496       4.63928        19     
produce:gaushit:GausHitFinder                     0.654122       1.4486        2.0044        1.47528      0.365365        19     
produce:nhitsfilter:NumberOfHitsFilter           0.00015713    0.000244587   0.000380901   0.000218491   6.99829e-05      19     
produce:reco3d:SpacePointSolver                    6.71604       13.5936       26.1841       12.5932       5.65731        19     
produce:hitpdune:DisambigFromSpacePoints          0.100239      0.207383      0.310828      0.192216      0.0603486       19     
produce:cluster3d:Cluster3D                        1.18596       31.8609       49.1336       44.2104       20.1534        19     
[art]:TriggerResults:TriggerResultInserter        2.552e-05    3.44727e-05   7.2491e-05     3.297e-05    9.83318e-06      19     
end_path:out1:RootOutput                          4.66e-06     6.24105e-06    1.863e-05     5.61e-06      2.972e-06       19     
end_path:out1:RootOutput(write)                    4.52148       5.67545       7.00988       5.61799      0.653691        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5683.26 MB
  Peak resident set size usage (VmHWM): 3624.47 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0095_df-s02-d0_dw_0_20260203T191812_reco_stage0_20260204T003752_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0095_df-s02-d0_dw_0_20260203T191812_reco_stage0_20260204T003752_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0095_df-s02-d0_dw_0_20260203T191812_reco_stage0_20260204T003752_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770162811.0,
    "core.end_time": 1770165473.0,
    "core.events": [
      10836,
      10842,
      10848,
      10854,
      10860,
      10866,
      10872,
      10878,
      10884,
      10890,
      10896,
      10902,
      10908,
      10914,
      10920,
      10926,
      10932,
      10938,
      10944
    ],
    "core.event_count": 19,
    "core.first_event_number": 10836,
    "core.last_event_number": 10944,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0095_df-s02-d0_dw_0_20260203T191812.hdf5"
    }
  ]
}
justIN time: 2026-02-04 05:56:28 UTC       justIN version: 01.06.00