justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297920.40@dunegpschedd02.fnal.gov

Jobsub ID297920.40@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:54:27
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce505
Last heartbeat2026-02-04 00:16:06
From worker nodeHostnameb9p35p0031.cern.ch
cpuinfoAMD EPYC 9655P 96-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:37:53
Input filesvd-protodune:np02vd_raw_run042433_0153_df-s04-d1_dw_0_20260203T194017.hdf5
JobscriptExit code0
Real time36m (2171s)
CPU time34m (2072s = 95%)
Max RSS bytes3394674688 (3237 MiB)
Outputting started2026-02-04 00:14:05
Output files
Finished2026-02-04 00:16:06
Saved logsjustin-logs:297920.40-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

. 0x3bdecc90, 0x3bdecd90
Caught exception in looking up space point ptr... 0x4216cca0, 0x4216d090
Caught exception in looking up space point ptr... 0x4216d090, 0x4216cca0
Caught exception in looking up space point ptr... 0x1187bd50, 0x1187bbb0
Caught exception in looking up space point ptr... 0x1187bbb0, 0x1187bd50
Caught exception in looking up space point ptr... 0x4c1b7060, 0x4c1b6f10
Caught exception in looking up space point ptr... 0x4c1b6f10, 0x4c1b7060
Caught exception in looking up space point ptr... 0x4c1b7060, 0x4c1b6980
Caught exception in looking up space point ptr... 0x4c1b6980, 0x4c1b7060
Caught exception in looking up space point ptr... 0x4c1b7060, 0x4c1b6dc0
Caught exception in looking up space point ptr... 0x4c1b6dc0, 0x4c1b7060
Caught exception in looking up space point ptr... 0x46c339e0, 0x46c18dc0
Caught exception in looking up space point ptr... 0x46c18dc0, 0x46c339e0
Caught exception in looking up space point ptr... 0x5b8a2f30, 0x100d8890
Caught exception in looking up space point ptr... 0x3b68a290, 0x5b8a2f30
Caught exception in looking up space point ptr... 0x17f28450, 0x17f28350
Caught exception in looking up space point ptr... 0x17f28350, 0x17f28450
Caught exception in looking up space point ptr... 0x15227a60, 0x67135a70
Caught exception in looking up space point ptr... 0x67135a70, 0x3ae02280
Caught exception in looking up space point ptr... 0x7d8bf540, 0x7d8bfdf0
Caught exception in looking up space point ptr... 0x7d8bfdf0, 0x7d8bf540
Caught exception in looking up space point ptr... 0x333184e0, 0x333188d0
Caught exception in looking up space point ptr... 0x333188d0, 0x33318630
Caught exception in looking up space point ptr... 0x46c14e90, 0x46c14b70
Caught exception in looking up space point ptr... 0x46c14b70, 0x46c14e90
Caught exception in looking up space point ptr... 0x5fac1050, 0x31894c30
Caught exception in looking up space point ptr... 0x31894c30, 0x5fac0d90
Caught exception in looking up space point ptr... 0x3c4b07d0, 0x3c4b04a0
Caught exception in looking up space point ptr... 0x3c4b04a0, 0x66b11720
Caught exception in looking up space point ptr... 0x122e8a60, 0x122e6cc0
Caught exception in looking up space point ptr... 0x122e6cc0, 0x3c094eb0
Caught exception in looking up space point ptr... 0x3c094eb0, 0x122e8d80
Caught exception in looking up space point ptr... 0x122e8d80, 0x122e8a60
Caught exception in looking up space point ptr... 0x37575720, 0x37575360
Caught exception in looking up space point ptr... 0x37575360, 0x65fb5d70
Caught exception in looking up space point ptr... 0x3b8d1890, 0x129224f0
Caught exception in looking up space point ptr... 0x129224f0, 0x3b8d1890
Caught exception in looking up space point ptr... 0x5f7cb310, 0x3d3412a0
Caught exception in looking up space point ptr... 0x3d3412a0, 0x60ba8e60
Caught exception in looking up space point ptr... 0x81880330, 0x81880e70
Caught exception in looking up space point ptr... 0x81880e70, 0x81880330
Caught exception in looking up space point ptr... 0x5f7d0450, 0x5f7d0640
Caught exception in looking up space point ptr... 0x5f7d0640, 0x5f7d0450
Caught exception in looking up space point ptr... 0x5e1c8210, 0x5e1c7f00
Caught exception in looking up space point ptr... 0x5e1c7f00, 0x5e1c8210
Caught exception in looking up space point ptr... 0x4dbdfc30, 0x4dbd5420
Caught exception in looking up space point ptr... 0x4dbd5420, 0x4dbdfc30
Caught exception in looking up space point ptr... 0x4dbe7640, 0x4dc089c0
Caught exception in looking up space point ptr... 0x4dc089c0, 0x4dc08b50
Caught exception in looking up space point ptr... 0x5e2ac680, 0x5e2ac360
Caught exception in looking up space point ptr... 0x5e2ac360, 0x5e2ac4f0
Caught exception in looking up space point ptr... 0x36204500, 0x36204390
Caught exception in looking up space point ptr... 0x36204390, 0x36204500
Caught exception in looking up space point ptr... 0x197e81e0, 0x197dc3b0
Caught exception in looking up space point ptr... 0x197e85d0, 0x197e81e0
Caught exception in looking up space point ptr... 0x5f7aeca0, 0x5f7ae810
Caught exception in looking up space point ptr... 0x5f7ae810, 0x5f7aeca0
Caught exception in looking up space point ptr... 0x19813e60, 0x19813080
Caught exception in looking up space point ptr... 0x19813080, 0x19801c10
Caught exception in looking up space point ptr... 0x3712e3c0, 0x3712e910
Caught exception in looking up space point ptr... 0x3712e910, 0x3712e5a0
Caught exception in looking up space point ptr... 0x610def20, 0x7624a930
Caught exception in looking up space point ptr... 0x7624a930, 0x610def20
Caught exception in looking up space point ptr... 0x63329b50, 0x63329d30
Caught exception in looking up space point ptr... 0x63329d30, 0x63329b50
Caught exception in looking up space point ptr... 0x65fbc570, 0x65fbc3e0
Caught exception in looking up space point ptr... 0x65fbc3e0, 0x65fbc570
++++>>>> total num hits: 87349, num free: 84019
04-Feb-2026 01:13:58 CET  Closed output file "np02vd_raw_run042433_0153_df-s04-d1_dw_0_20260203T194017_reco_stage0_20260204T001358_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         96.8861       111.503       126.191       111.157       7.29429        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                        5.807e-05    8.28763e-05   0.00023962     7.448e-05    3.76064e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                6.79357       7.50807       10.3714       7.31628       0.81362        19     
produce:triggerrawdecoder:PDVDTriggerReader4     0.00828358     0.0338786      0.15425      0.0148214     0.0356354       19     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000359751   0.000637351   0.00106408     0.0006106    0.000163983      19     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.00027757     0.0005514    0.00085868    0.00058508    0.000156493      19     
produce:beamevent:BeamEvent                       6.262e-05    0.000128907   0.00082136     8.959e-05    0.000164387      19     
produce:pdvddaphne:DAPHNEReaderPDVD                1.76395       2.10214       2.3394        2.11143      0.151237        19     
produce:ophit:OpHitFinder                         0.0239516     0.0346048     0.0428142     0.0363145    0.00508127       19     
produce:wclsdatavd:WireCellToolkit                 72.4216       80.9464       92.9337       79.7154       4.25309        19     
produce:gaushit:GausHitFinder                     0.637593       1.02842       1.88037      0.910006       0.33451        19     
produce:nhitsfilter:NumberOfHitsFilter           0.00015113    0.000264107    0.0005647    0.00023138    0.00011703       19     
produce:reco3d:SpacePointSolver                    7.05969       12.8413       21.9192       12.5999       3.68769        19     
produce:hitpdune:DisambigFromSpacePoints          0.0717911     0.150735      0.279819      0.127121      0.0628104       19     
produce:cluster3d:Cluster3D                       0.978602       1.81206       3.31281       1.68396       0.60792        19     
[art]:TriggerResults:TriggerResultInserter        2.317e-05    3.74163e-05   0.00010003     3.298e-05    1.59976e-05      19     
end_path:out1:RootOutput                          4.52e-06     6.80211e-06    2.436e-05     5.73e-06     4.2214e-06       19     
end_path:out1:RootOutput(write)                    4.24698       5.00121       5.64287       4.98086       0.4111         19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5424.23 MB
  Peak resident set size usage (VmHWM): 3394.67 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0153_df-s04-d1_dw_0_20260203T194017_reco_stage0_20260204T001358_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0153_df-s04-d1_dw_0_20260203T194017_reco_stage0_20260204T001358_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0153_df-s04-d1_dw_0_20260203T194017_reco_stage0_20260204T001358_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770161880.0,
    "core.end_time": 1770164040.0,
    "core.events": [
      17453,
      17459,
      17465,
      17471,
      17477,
      17483,
      17489,
      17495,
      17501,
      17507,
      17513,
      17519,
      17525,
      17531,
      17537,
      17543,
      17549,
      17555,
      17561
    ],
    "core.event_count": 19,
    "core.first_event_number": 17453,
    "core.last_event_number": 17561,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0153_df-s04-d1_dw_0_20260203T194017.hdf5"
    }
  ]
}
justIN time: 2026-02-04 06:07:01 UTC       justIN version: 01.06.00