justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303647.185@dunegpschedd01.fnal.gov

Jobsub ID303647.185@dunegpschedd01.fnal.gov
Workflow ID12717
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:46:24
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce512
Last heartbeat2026-02-03 23:31:48
From worker nodeHostnameb9g24p4713.cern.ch
cpuinfoAMD EPYC 7302 16-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 22:47:10
Input filesvd-protodune:np02vd_raw_run042421_0067_df-s04-d0_dw_0_20260203T140717.hdf5
JobscriptExit code0
Real time42m (2539s)
CPU time40m (2429s = 95%)
Max RSS bytes2601402368 (2480 MiB)
Outputting started2026-02-03 23:29:30
Output files
Finished2026-02-03 23:31:48
Saved logsjustin-logs:303647.185-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

Caught exception in looking up space point ptr... 0x525d8d80, 0x525d8bf0
Caught exception in looking up space point ptr... 0xffcb500, 0x12631560
Caught exception in looking up space point ptr... 0x12631560, 0xffcb500
Caught exception in looking up space point ptr... 0x148e68a0, 0x148e6a30
Caught exception in looking up space point ptr... 0x148e6a30, 0x148e68a0
Caught exception in looking up space point ptr... 0x164a8850, 0xffe96f0
Caught exception in looking up space point ptr... 0xffe96f0, 0x164a8630
Caught exception in looking up space point ptr... 0x525d4730, 0x525d48c0
Caught exception in looking up space point ptr... 0x525d48c0, 0x525d4730
Caught exception in looking up space point ptr... 0x525d8f10, 0x525d90a0
Caught exception in looking up space point ptr... 0x525d90a0, 0x525d8f10
Caught exception in looking up space point ptr... 0x525d56d0, 0x525d5860
Caught exception in looking up space point ptr... 0x525d5860, 0x525d56d0
Caught exception in looking up space point ptr... 0x5bce7570, 0x5bce7700
Caught exception in looking up space point ptr... 0x5bce7700, 0x1261dd70
Caught exception in looking up space point ptr... 0x164abcf0, 0x5bce4370
Caught exception in looking up space point ptr... 0x5bce4370, 0x5bce4500
Caught exception in looking up space point ptr... 0x25008a10, 0x32882580
Caught exception in looking up space point ptr... 0x32882580, 0x25008a10
Caught exception in looking up space point ptr... 0x12614c30, 0x12614d60
Caught exception in looking up space point ptr... 0x12614d60, 0x12614b00
Caught exception in looking up space point ptr... 0x3287cde0, 0x5bce5180
Caught exception in looking up space point ptr... 0x5bce5180, 0x3287cde0
Caught exception in looking up space point ptr... 0x3289aa50, 0x5bce4b40
Caught exception in looking up space point ptr... 0x5bce4b40, 0x3287c390
Caught exception in looking up space point ptr... 0x207ccbd0, 0x207cecb0
Caught exception in looking up space point ptr... 0x207cecb0, 0x207ccbd0
Caught exception in looking up space point ptr... 0x207ccbd0, 0x207cca90
Caught exception in looking up space point ptr... 0x207cca90, 0x207ccbd0
Caught exception in looking up space point ptr... 0x540cc400, 0x32299a10
Caught exception in looking up space point ptr... 0x32299a10, 0x540cc400
Caught exception in looking up space point ptr... 0x95adcd0, 0x95adb40
Caught exception in looking up space point ptr... 0x95adb40, 0x95adcd0
Caught exception in looking up space point ptr... 0x148e7840, 0x148e79d0
Caught exception in looking up space point ptr... 0x148e79d0, 0x148e7840
Caught exception in looking up space point ptr... 0x1583f1e0, 0x16377f80
Caught exception in looking up space point ptr... 0x16377f80, 0x1583f1e0
Caught exception in looking up space point ptr... 0x525d7f70, 0x525d8100
Caught exception in looking up space point ptr... 0x525d8100, 0x525d7f70
Caught exception in looking up space point ptr... 0x525d6670, 0x525d6800
Caught exception in looking up space point ptr... 0x525d6800, 0x525d6670
Caught exception in looking up space point ptr... 0x148e7b60, 0x148e7cf0
Caught exception in looking up space point ptr... 0x148e7cf0, 0x148e7b60
Caught exception in looking up space point ptr... 0x525d85b0, 0x525d8740
Caught exception in looking up space point ptr... 0x525d8740, 0x525d85b0
Caught exception in looking up space point ptr... 0x525d6030, 0x4d904060
Caught exception in looking up space point ptr... 0x4d904060, 0x525d6030
Caught exception in looking up space point ptr... 0x148e7200, 0x148e7390
Caught exception in looking up space point ptr... 0x148e7390, 0x4d909e20
Caught exception in looking up space point ptr... 0x148e6bc0, 0x148e6d50
Caught exception in looking up space point ptr... 0x148e6d50, 0x148e6bc0
Caught exception in looking up space point ptr... 0x5f5e5ed0, 0x5f5e5da0
Caught exception in looking up space point ptr... 0x5f5e5da0, 0x5f5e5ed0
Caught exception in looking up space point ptr... 0x4d8f0c20, 0x4d8f34c0
Caught exception in looking up space point ptr... 0x4d8f34c0, 0x525d5d10
Caught exception in looking up space point ptr... 0x42158790, 0x421585c0
Caught exception in looking up space point ptr... 0x421585c0, 0x421569c0
Caught exception in looking up space point ptr... 0x24ed5670, 0x24ed58d0
Caught exception in looking up space point ptr... 0x24ed58d0, 0x24ed5670
Caught exception in looking up space point ptr... 0x2a950080, 0x2a94f2f0
Caught exception in looking up space point ptr... 0x2a94f2f0, 0x2a950080
Caught exception in looking up space point ptr... 0x4d8eca60, 0x4d8ec640
Caught exception in looking up space point ptr... 0x4d8ec640, 0x4d8eca60
Caught exception in looking up space point ptr... 0x10e2e030, 0x10e2cc60
Caught exception in looking up space point ptr... 0x10e2cc60, 0x10e2c510
++++>>>> total num hits: 122107, num free: 118115
04-Feb-2026 00:29:23 CET  Closed output file "np02vd_raw_run042421_0067_df-s04-d0_dw_0_20260203T140717_reco_stage0_20260203T232923_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         75.332        91.5095       123.028       92.7748       9.71519        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       4.6201e-05    0.000101339   0.000190654   9.1932e-05    3.62326e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                3.97481       5.13293       21.6194       4.36603       3.33402        27     
produce:triggerrawdecoder:PDVDTriggerReader4     0.00983616     0.0184262     0.0596745     0.0141512     0.0107813       27     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000384678   0.000555992   0.000935998   0.00053491    0.000142194      27     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.00133131    0.00179474    0.00270486    0.00168555    0.000334117      27     
produce:beamevent:BeamEvent                      5.4341e-05    0.00014097    0.000873357   9.7082e-05    0.00015426       27     
produce:pdvddaphne:DAPHNEReaderPDVD                2.25295       2.83928       3.4631        2.83739      0.350516        27     
produce:ophit:OpHitFinder                         0.0241628     0.0360099     0.0487029     0.0355208    0.00569451       27     
produce:wclsdatavd:WireCellToolkit                 58.164        66.5324       75.4439       66.9299       4.65363        27     
produce:gaushit:GausHitFinder                     0.510492      0.929809       1.43186      0.938511      0.239085        27     
produce:nhitsfilter:NumberOfHitsFilter           0.000178804   0.000295525   0.000606012   0.000293496   8.87724e-05      27     
produce:reco3d:SpacePointSolver                    4.48626       10.2226       20.3478       9.85061       3.39341        27     
produce:hitpdune:DisambigFromSpacePoints          0.053925      0.107086      0.208577      0.0948838     0.0364787       27     
produce:cluster3d:Cluster3D                       0.904319       1.90734       5.21523       1.5809       0.943979        27     
[art]:TriggerResults:TriggerResultInserter       2.0651e-05    3.67811e-05   0.000146253    3.024e-05    2.28545e-05      27     
end_path:out1:RootOutput                          3.76e-06     8.72648e-06   3.5901e-05     7.781e-06    6.0362e-06       27     
end_path:out1:RootOutput(write)                    3.11278       3.75939       5.01696       3.61871      0.488561        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4545.29 MB
  Peak resident set size usage (VmHWM): 2601.4 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042421_0067_df-s04-d0_dw_0_20260203T140717_reco_stage0_20260203T232923_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042421_0067_df-s04-d0_dw_0_20260203T140717_reco_stage0_20260203T232923_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042421_0067_df-s04-d0_dw_0_20260203T140717_reco_stage0_20260203T232923_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770158842.0,
    "core.end_time": 1770161364.0,
    "core.events": [
      10912,
      10918,
      10924,
      10930,
      10936,
      10942,
      10948,
      10954,
      10960,
      10966,
      10972,
      10978,
      10984,
      10990,
      10996,
      11002,
      11008,
      11014,
      11020,
      11026,
      11032,
      11038,
      11044,
      11050,
      11056,
      11062,
      11068
    ],
    "core.event_count": 27,
    "core.first_event_number": 10912,
    "core.last_event_number": 11068,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42421
    ],
    "core.runs_subruns": [
      4242100001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042421_0067_df-s04-d0_dw_0_20260203T140717.hdf5"
    }
  ]
}
justIN time: 2026-02-04 07:45:57 UTC       justIN version: 01.06.00