justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297923.2@dunegpschedd02.fnal.gov

Jobsub ID297923.2@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 23:00:26
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce507
Last heartbeat2026-02-04 01:13:32
From worker nodeHostnameb9p17p9437.cern.ch
cpuinfoIntel(R) Xeon(R) Silver 4216 CPU @ 2.10GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:57:15
Input filesvd-protodune:np02vd_raw_run042433_0091_df-s04-d0_dw_0_20260203T191643.hdf5
JobscriptExit code0
Real time1h (4441s)
CPU time1h (4337s = 97%)
Max RSS bytes3430449152 (3271 MiB)
Outputting started2026-02-04 01:11:17
Output files
Finished2026-02-04 01:13:32
Saved logsjustin-logs:297923.2-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

r... 0x6ac38c40, 0x6ac38a60
Caught exception in looking up space point ptr... 0x6ac38a60, 0x6ac38c40
Caught exception in looking up space point ptr... 0x16da9f70, 0x137f3dc0
Caught exception in looking up space point ptr... 0x137f3dc0, 0x16da9f70
Caught exception in looking up space point ptr... 0x6ef0cca0, 0x6ef0cda0
Caught exception in looking up space point ptr... 0x6ef0cda0, 0x6ef30d30
Caught exception in looking up space point ptr... 0x75980e70, 0x75980d20
Caught exception in looking up space point ptr... 0x75980d20, 0x75980e70
Caught exception in looking up space point ptr... 0x75980e70, 0x75980bd0
Caught exception in looking up space point ptr... 0x75980bd0, 0x75980e70
Caught exception in looking up space point ptr... 0x2c85be90, 0x2c85a4b0
Caught exception in looking up space point ptr... 0x2c85a4b0, 0x2c85aa70
Caught exception in looking up space point ptr... 0x6f09f8b0, 0x6f0a8bd0
Caught exception in looking up space point ptr... 0x6f0a8bd0, 0x6ef2ab20
Caught exception in looking up space point ptr... 0x3aebafb0, 0x32f05640
Caught exception in looking up space point ptr... 0x32f05640, 0x32f054f0
Caught exception in looking up space point ptr... 0x3548e340, 0x79e45ed0
Caught exception in looking up space point ptr... 0x79e45ed0, 0x3548e040
Caught exception in looking up space point ptr... 0x3b205230, 0x45dc9480
Caught exception in looking up space point ptr... 0x45dc9480, 0x5b364c30
Caught exception in looking up space point ptr... 0x5b364c30, 0xd04eb70
Caught exception in looking up space point ptr... 0x819b3720, 0x819baf70
Caught exception in looking up space point ptr... 0x819baf70, 0x819bb060
Caught exception in looking up space point ptr... 0x3cc74ff0, 0x3cc74f20
Caught exception in looking up space point ptr... 0x3cc74f20, 0x3cc74ff0
Caught exception in looking up space point ptr... 0x72331580, 0x4fed9580
Caught exception in looking up space point ptr... 0x4fed9580, 0x72331580
Caught exception in looking up space point ptr... 0x3ad9e780, 0x3ad9db00
Caught exception in looking up space point ptr... 0x3ad9db00, 0x3ad9e780
Caught exception in looking up space point ptr... 0x3cc60ff0, 0x3cc60ea0
Caught exception in looking up space point ptr... 0x3cc60ea0, 0x3cc60d80
Caught exception in looking up space point ptr... 0x3cc60d80, 0x7235b7c0
Caught exception in looking up space point ptr... 0x7235b7c0, 0x7235b910
Caught exception in looking up space point ptr... 0x11af39e0, 0x18770440
Caught exception in looking up space point ptr... 0x18770440, 0x11af39e0
Caught exception in looking up space point ptr... 0x603b36e0, 0x603b3400
Caught exception in looking up space point ptr... 0x603b3400, 0x603b3160
Caught exception in looking up space point ptr... 0x453c1b00, 0x453c1a00
Caught exception in looking up space point ptr... 0x453c1a00, 0x453c1b00
Caught exception in looking up space point ptr... 0x50551da0, 0x50550580
Caught exception in looking up space point ptr... 0x50550580, 0x50551da0
Caught exception in looking up space point ptr... 0x72352ad0, 0x72359680
Caught exception in looking up space point ptr... 0x72359680, 0x72359bc0
Caught exception in looking up space point ptr... 0x64253f20, 0x64254240
Caught exception in looking up space point ptr... 0x64254240, 0x64253f20
Caught exception in looking up space point ptr... 0x64253f20, 0x642546f0
Caught exception in looking up space point ptr... 0x642546f0, 0x64253f20
Caught exception in looking up space point ptr... 0x3ccaed10, 0x3ccaded0
Caught exception in looking up space point ptr... 0x3ccaded0, 0x3ccaed10
Caught exception in looking up space point ptr... 0x36f9dc70, 0x36f9da70
Caught exception in looking up space point ptr... 0x36f9da70, 0x36f9dc70
Caught exception in looking up space point ptr... 0x370e5820, 0x370d7b10
Caught exception in looking up space point ptr... 0x370d7b10, 0x370e5820
Caught exception in looking up space point ptr... 0xc053150, 0x4c4105a0
Caught exception in looking up space point ptr... 0x4c4105a0, 0xc052e60
Caught exception in looking up space point ptr... 0x3cc613e0, 0x3cc61290
Caught exception in looking up space point ptr... 0x3cc61290, 0x3cc61680
Caught exception in looking up space point ptr... 0x354b05e0, 0x354b0500
Caught exception in looking up space point ptr... 0x354b0500, 0x354b06c0
Caught exception in looking up space point ptr... 0x2c85bce0, 0x2c858800
Caught exception in looking up space point ptr... 0x2c858800, 0x2c85bce0
Caught exception in looking up space point ptr... 0x5c209d00, 0x603ae1d0
Caught exception in looking up space point ptr... 0x603ae1d0, 0x5c209d00
Caught exception in looking up space point ptr... 0x370c3f90, 0x370c3cf0
Caught exception in looking up space point ptr... 0x370c3cf0, 0x370c3f90
Caught exception in looking up space point ptr... 0x41e461d0, 0x38491510
Caught exception in looking up space point ptr... 0x38491510, 0x41e461d0
++++>>>> total num hits: 94009, num free: 89551
04-Feb-2026 02:11:09 CET  Closed output file "np02vd_raw_run042433_0091_df-s04-d0_dw_0_20260203T191643_reco_stage0_20260204T011109_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         203.365       228.82        254.329       227.597        12.81         19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       7.2074e-05    0.000110931   0.000342234   9.6521e-05    5.86702e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                8.38802       9.94388       11.4381       10.0789      0.919492        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.0091834     0.0227241     0.0587757     0.0226014     0.0132747       19     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000503901   0.000705976   0.00122413    0.000672385   0.000152372      19     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.000447594   0.000665978   0.00121113    0.000622387   0.000160676      19     
produce:beamevent:BeamEvent                      8.9388e-05    0.000185237   0.00134613    0.00012452    0.000274864      19     
produce:pdvddaphne:DAPHNEReaderPDVD                5.49832       6.22776       8.00164       5.92047      0.722752        19     
produce:ophit:OpHitFinder                         0.0371711     0.0539003     0.0810693     0.0462731     0.0139938       19     
produce:wclsdatavd:WireCellToolkit                 154.389       171.513       192.477       172.538       9.62498        19     
produce:gaushit:GausHitFinder                      1.35035       2.1832        3.02822       2.17244      0.403858        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000490363   0.000710347   0.00162524    0.000664737   0.000237049      19     
produce:reco3d:SpacePointSolver                    15.9116       28.0546       42.4101       27.1532       6.6157         19     
produce:hitpdune:DisambigFromSpacePoints          0.207129      0.341983      0.532926      0.322592      0.0893758       19     
produce:cluster3d:Cluster3D                        2.07963       3.85693       17.5056       3.05219       3.27031        19     
[art]:TriggerResults:TriggerResultInserter       2.9596e-05    4.62085e-05   0.000195582   3.5144e-05    3.59757e-05      19     
end_path:out1:RootOutput                          6.388e-06    1.15814e-05   7.1969e-05     7.74e-06     1.43035e-05      19     
end_path:out1:RootOutput(write)                    5.79022       6.57132       8.53435       6.44118      0.579631        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5393.01 MB
  Peak resident set size usage (VmHWM): 3430.45 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0091_df-s04-d0_dw_0_20260203T191643_reco_stage0_20260204T011109_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0091_df-s04-d0_dw_0_20260203T191643_reco_stage0_20260204T011109_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0091_df-s04-d0_dw_0_20260203T191643_reco_stage0_20260204T011109_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770163049.0,
    "core.end_time": 1770167471.0,
    "core.events": [
      10384,
      10390,
      10396,
      10402,
      10408,
      10414,
      10420,
      10426,
      10432,
      10438,
      10444,
      10450,
      10456,
      10462,
      10468,
      10474,
      10480,
      10486,
      10492
    ],
    "core.event_count": 19,
    "core.first_event_number": 10384,
    "core.last_event_number": 10492,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0091_df-s04-d0_dw_0_20260203T191643.hdf5"
    }
  ]
}
justIN time: 2026-02-04 06:05:11 UTC       justIN version: 01.06.00