justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297921.160@dunegpschedd02.fnal.gov

Jobsub ID297921.160@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:58:26
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce507
Last heartbeat2026-02-04 01:15:03
From worker nodeHostnameb9p17p9437.cern.ch
cpuinfoIntel(R) Xeon(R) Silver 4216 CPU @ 2.10GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:56:09
Input filesvd-protodune:np02vd_raw_run042433_0329_df-s03-d0_dw_0_20260203T204710.hdf5
JobscriptExit code0
Real time1h (4593s)
CPU time1h (4495s = 97%)
Max RSS bytes3396206592 (3238 MiB)
Outputting started2026-02-04 01:12:43
Output files
Finished2026-02-04 01:15:03
Saved logsjustin-logs:297921.160-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

t ptr... 0x7f2718c0, 0x7f26e1b0
Caught exception in looking up space point ptr... 0x5aba4d00, 0x5aba4fe0
Caught exception in looking up space point ptr... 0x5aba4fe0, 0x11b1d770
Caught exception in looking up space point ptr... 0x15fc4e00, 0x15fc56a0
Caught exception in looking up space point ptr... 0x15fc56a0, 0x15fc3990
Caught exception in looking up space point ptr... 0x599174f0, 0x59917660
Caught exception in looking up space point ptr... 0x59917660, 0x599174f0
Caught exception in looking up space point ptr... 0x1211f070, 0x12137200
Caught exception in looking up space point ptr... 0x12137200, 0x1213dcf0
Caught exception in looking up space point ptr... 0x2a1ac920, 0x2a1ab6d0
Caught exception in looking up space point ptr... 0x2a1ad4f0, 0x2a1ac920
Caught exception in looking up space point ptr... 0x1419c410, 0x69beb190
Caught exception in looking up space point ptr... 0x69beb190, 0x1419c410
Caught exception in looking up space point ptr... 0x52843ed0, 0x52843d80
Caught exception in looking up space point ptr... 0x52843d80, 0x52843ed0
Caught exception in looking up space point ptr... 0x4b2de2e0, 0x4b2de410
Caught exception in looking up space point ptr... 0x4b2de410, 0x4b2de2e0
Caught exception in looking up space point ptr... 0x295ec710, 0x112fe720
Caught exception in looking up space point ptr... 0x112fe720, 0x3ab83520
Caught exception in looking up space point ptr... 0x45377a10, 0x453778e0
Caught exception in looking up space point ptr... 0x453778e0, 0x45377a10
Caught exception in looking up space point ptr... 0x37a27420, 0x5286d940
Caught exception in looking up space point ptr... 0x5286d940, 0x37a2f650
Caught exception in looking up space point ptr... 0x5acc07b0, 0x5acc04c0
Caught exception in looking up space point ptr... 0x5acc04c0, 0x5acc07b0
Caught exception in looking up space point ptr... 0x93ed750, 0x93edae0
Caught exception in looking up space point ptr... 0x93edae0, 0x93ed750
Caught exception in looking up space point ptr... 0x7f282f00, 0x7f282d70
Caught exception in looking up space point ptr... 0x7f282d70, 0x7f27ec50
Caught exception in looking up space point ptr... 0x17343c40, 0x17346480
Caught exception in looking up space point ptr... 0x17346480, 0x37e6e030
Caught exception in looking up space point ptr... 0x2a1d08b0, 0x17d249f0
Caught exception in looking up space point ptr... 0x17d249f0, 0x2a1d3670
Caught exception in looking up space point ptr... 0x2a01f840, 0x3a428830
Caught exception in looking up space point ptr... 0x3a428830, 0x72ed33b0
Caught exception in looking up space point ptr... 0x36d824f0, 0x36d83f10
Caught exception in looking up space point ptr... 0x36d83f10, 0x36d824f0
Caught exception in looking up space point ptr... 0x4e62e0c0, 0x4e62db30
Caught exception in looking up space point ptr... 0x4e62db30, 0x4e62e0c0
Caught exception in looking up space point ptr... 0x4e62e0c0, 0x4e62df70
Caught exception in looking up space point ptr... 0x4e62df70, 0x4e62e0c0
Caught exception in looking up space point ptr... 0x2d3cb760, 0x2d3c99c0
Caught exception in looking up space point ptr... 0x2d3c99c0, 0x2d3c3f30
Caught exception in looking up space point ptr... 0x141ea720, 0x7ea3df40
Caught exception in looking up space point ptr... 0x7ea3df40, 0x141ea720
Caught exception in looking up space point ptr... 0x7f270600, 0x7f270300
Caught exception in looking up space point ptr... 0x7f270300, 0x7f270470
Caught exception in looking up space point ptr... 0xe6dc850, 0xe6dca30
Caught exception in looking up space point ptr... 0xe6cec60, 0xe6dc850
Caught exception in looking up space point ptr... 0x7e86ac80, 0x7e86ad80
Caught exception in looking up space point ptr... 0x7e86ad80, 0x7e86ac80
Caught exception in looking up space point ptr... 0x39f5d040, 0x39f5ceb0
Caught exception in looking up space point ptr... 0x39f5ceb0, 0x39f5d040
Caught exception in looking up space point ptr... 0x33a08570, 0x33a09500
Caught exception in looking up space point ptr... 0x33a09500, 0x33a07610
Caught exception in looking up space point ptr... 0x2d3f5c90, 0x43e7e890
Caught exception in looking up space point ptr... 0x43e7e890, 0x2d3f5c90
Caught exception in looking up space point ptr... 0x45e76810, 0x467863e0
Caught exception in looking up space point ptr... 0x467863e0, 0x45e76810
Caught exception in looking up space point ptr... 0x61dfdb00, 0x61dfd920
Caught exception in looking up space point ptr... 0x61dfd920, 0x61dfdb00
Caught exception in looking up space point ptr... 0x5accbbd0, 0x5accba80
Caught exception in looking up space point ptr... 0x5accba80, 0x5accbbd0
Caught exception in looking up space point ptr... 0x1694e500, 0x455e02e0
Caught exception in looking up space point ptr... 0x455e02e0, 0x1694e500
Caught exception in looking up space point ptr... 0x2a56a3b0, 0x2a56a470
Caught exception in looking up space point ptr... 0x2a56a470, 0x17515e80
++++>>>> total num hits: 111074, num free: 107525
04-Feb-2026 02:12:35 CET  Closed output file "np02vd_raw_run042433_0329_df-s03-d0_dw_0_20260203T204710_reco_stage0_20260204T011235_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         211.845       236.747       289.175       236.848       20.8872        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       7.5036e-05    0.000124446   0.000275784   0.00010338    4.95497e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                8.31153       9.80103       13.045        9.70436       1.13175        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.0100815     0.0211458     0.0794367     0.0153744     0.0162737       19     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000641903   0.00075617    0.00102907    0.000746336   9.57464e-05      19     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.000540032   0.000732481    0.0012094    0.000669073   0.000157948      19     
produce:beamevent:BeamEvent                      8.7519e-05    0.000198892   0.00113549    0.000139083   0.000223994      19     
produce:pdvddaphne:DAPHNEReaderPDVD                5.53529       6.25443       7.88823       5.86342      0.786963        19     
produce:ophit:OpHitFinder                         0.0369535     0.0519606     0.0791669     0.0462938     0.0130719       19     
produce:wclsdatavd:WireCellToolkit                 155.475       176.002       217.344       167.109       16.3429        19     
produce:gaushit:GausHitFinder                      1.70307       2.20821       2.68108       2.26525      0.268243        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000635659   0.000749551   0.00108793    0.00071704    0.000125141      19     
produce:reco3d:SpacePointSolver                    21.9613       31.6961       41.1919       30.3614       5.09261        19     
produce:hitpdune:DisambigFromSpacePoints          0.250562      0.373473      0.667687       0.33614      0.121193        19     
produce:cluster3d:Cluster3D                        2.28271       3.61027       6.64867       3.22615       1.10454        19     
[art]:TriggerResults:TriggerResultInserter       3.0856e-05    4.44417e-05   0.00014227    3.8448e-05    2.37303e-05      19     
end_path:out1:RootOutput                          7.733e-06    1.17296e-05   3.2518e-05     8.906e-06    6.80678e-06      19     
end_path:out1:RootOutput(write)                    6.11019       6.66895       8.96254       6.45882      0.610737        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5363.4 MB
  Peak resident set size usage (VmHWM): 3396.21 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0329_df-s03-d0_dw_0_20260203T204710_reco_stage0_20260204T011235_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0329_df-s03-d0_dw_0_20260203T204710_reco_stage0_20260204T011235_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0329_df-s03-d0_dw_0_20260203T204710_reco_stage0_20260204T011235_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770162981.0,
    "core.end_time": 1770167557.0,
    "core.events": [
      37514,
      37520,
      37526,
      37532,
      37538,
      37544,
      37550,
      37556,
      37562,
      37568,
      37574,
      37580,
      37586,
      37592,
      37598,
      37604,
      37610,
      37616,
      37622
    ],
    "core.event_count": 19,
    "core.first_event_number": 37514,
    "core.last_event_number": 37622,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0329_df-s03-d0_dw_0_20260203T204710.hdf5"
    }
  ]
}
justIN time: 2026-02-04 06:07:26 UTC       justIN version: 01.06.00