Improve compaction #149

marco6 · 2024-07-29T09:56:30Z

Description

This PR rewrites the compaction step in a similar way upstream is doing. This way, the throughput should be a lot higher. The main differences from upstream are:

small batches instead of a single big one;
proper range boundaries to reduce search space;
in-transaction update of the compaction version.

This query does not use indexes right now. It's still way faster than what we have and what upstream has. Maybe we could improve things with a covering index if/when it's needed for the Watch/TTL query. Otherwise we might be ok with what we have.

Improvements to the previous implementation

Up to this point we've been doing compaction on each revision within the compaction range row by row. Looping over all revisions takes about 7s on an idle cluster, rare issues have been recorded where compaction takes minutes. With this compaction query improvement we are doing a compaction batch in around 40ms and a max <100ms on an idle cluster. (Note: the idle cluster only needs 1 batch to complete the compaction under normal circumstances).

This PR addresses: #145 and #146

pkg/kine/drivers/sqlite/schema.go

louiseschmidtgen · 2024-07-29T12:07:00Z

pkg/kine/logstructured/sqllog/sql.go

-	SupersededCount = 100
-	otelName        = "sqllog"
+	SupersededCount  = 100
+	compactBatchSize = 1000


Should we add a compaction txn timeout?

marco6 · 2024-07-29T12:28:22Z

Depends on #140

github-actions · 2024-07-29T12:44:25Z

Benchmark Result

goos: linux
goarch: amd64
pkg: github.com/canonical/k8s-dqlite/test
cpu: AMD EPYC 7763 64-Core Processor                
                    │     Before     │                After                │
                    │     sec/op     │    sec/op     vs base               │
Compaction/sqlite-4   4299.02µ ±  6%   42.16µ ± 12%  -99.02% (p=0.001 n=7)
Compaction/dqlite-4   5445.00µ ±  5%   43.40µ ±  2%  -99.20% (p=0.001 n=7)
Create/sqlite-4         1.316m ± 10%   1.419m ±  3%   +7.83% (p=0.017 n=7)
Create/dqlite-4         1.199m ±  8%   1.178m ±  5%        ~ (p=0.710 n=7)
Delete/sqlite-4         1.535m ±  9%   1.547m ±  9%        ~ (p=0.620 n=7)
Delete/dqlite-4         1.481m ± 34%   1.390m ± 10%        ~ (p=0.073 n=7)
Get/sqlite-4            278.9µ ±  2%   266.4µ ±  1%   -4.50% (p=0.001 n=7)
Get/dqlite-4            375.9µ ±  1%   357.6µ ±  2%   -4.86% (p=0.001 n=7)
Lease/sqlite-4          106.3µ ±  3%   102.3µ ±  1%   -3.81% (p=0.001 n=7)
Lease/dqlite-4          107.6µ ±  5%   101.2µ ±  3%   -5.98% (p=0.002 n=7)
Update/sqlite-4         1.500m ±  3%   1.546m ±  7%   +3.03% (p=0.001 n=7)
Update/dqlite-4         1.475m ±  4%   1.471m ±  9%        ~ (p=0.710 n=7)
geomean                 882.5µ         395.7µ        -55.16%

                    │        Before        │                    After                     │
                    │ page-cache-misses/op │ page-cache-misses/op  vs base                │
Compaction/sqlite-4           0.000 ± 0%               0.000 ± 0%       ~ (p=1.000 n=7) ¹
Compaction/dqlite-4           0.000 ± 0%               0.000 ± 0%       ~ (p=1.000 n=7) ¹
Create/sqlite-4               0.000 ± 0%               0.000 ± 0%       ~ (p=1.000 n=7) ¹
Create/dqlite-4               0.000 ± 0%               0.000 ± 0%       ~ (p=1.000 n=7) ¹
Delete/sqlite-4               0.000 ± 0%               0.000 ± 0%       ~ (p=1.000 n=7) ¹
Delete/dqlite-4               0.000 ± 0%               0.000 ± 0%       ~ (p=1.000 n=7) ¹
Get/sqlite-4                  0.000 ± 0%               0.000 ± 0%       ~ (p=1.000 n=7) ¹
Get/dqlite-4                  0.000 ± 0%               0.000 ± 0%       ~ (p=1.000 n=7) ¹
Lease/sqlite-4                0.000 ± 0%               0.000 ± 0%       ~ (p=1.000 n=7) ¹
Lease/dqlite-4                0.000 ± 0%               0.000 ± 0%       ~ (p=1.000 n=7) ¹
Update/sqlite-4               0.000 ± 0%               0.000 ± 0%       ~ (p=1.000 n=7) ¹
Update/dqlite-4               0.000 ± 0%               0.000 ± 0%       ~ (p=1.000 n=7) ¹
geomean                                  ²                         +0.00%               ²
¹ all samples are equal
² summaries must be >0 to compute geomean

                    │        Before        │                    After                     │
                    │ page-cache-spills/op │ page-cache-spills/op  vs base                │
Compaction/sqlite-4           0.000 ± 0%               0.000 ± 0%       ~ (p=1.000 n=7) ¹
Compaction/dqlite-4           0.000 ± 0%               0.000 ± 0%       ~ (p=1.000 n=7) ¹
Create/sqlite-4               0.000 ± 0%               0.000 ± 0%       ~ (p=1.000 n=7) ¹
Create/dqlite-4               0.000 ± 0%               0.000 ± 0%       ~ (p=1.000 n=7) ¹
Delete/sqlite-4               0.000 ± 0%               0.000 ± 0%       ~ (p=1.000 n=7) ¹
Delete/dqlite-4               0.000 ± 0%               0.000 ± 0%       ~ (p=1.000 n=7) ¹
Get/sqlite-4                  0.000 ± 0%               0.000 ± 0%       ~ (p=1.000 n=7) ¹
Get/dqlite-4                  0.000 ± 0%               0.000 ± 0%       ~ (p=1.000 n=7) ¹
Lease/sqlite-4                0.000 ± 0%               0.000 ± 0%       ~ (p=1.000 n=7) ¹
Lease/dqlite-4                0.000 ± 0%               0.000 ± 0%       ~ (p=1.000 n=7) ¹
Update/sqlite-4               0.000 ± 0%               0.000 ± 0%       ~ (p=1.000 n=7) ¹
Update/dqlite-4               0.000 ± 0%               0.000 ± 0%       ~ (p=1.000 n=7) ¹
geomean                                  ²                         +0.00%               ²
¹ all samples are equal
² summaries must be >0 to compute geomean

                    │    Before     │                After                 │
                    │ page-reads/op │ page-reads/op  vs base               │
Compaction/sqlite-4    112.10 ±  1%    12.21 ±   1%  -89.11% (p=0.001 n=7)
Compaction/dqlite-4    115.70 ±  1%    12.20 ±   1%  -89.46% (p=0.001 n=7)
Create/sqlite-4         14.01 ±  2%    13.87 ±   3%        ~ (p=0.122 n=7)
Create/dqlite-4         13.82 ±  1%    13.84 ±   2%        ~ (p=0.740 n=7)
Delete/sqlite-4         25.05 ±  5%    25.00 ±   1%        ~ (p=1.000 n=7)
Delete/dqlite-4         24.68 ±  0%    24.52 ±   1%   -0.65% (p=0.027 n=7)
Get/sqlite-4            11.95 ± 25%    14.96 ±   0%  +25.19% (p=0.003 n=7)
Get/dqlite-4            11.96 ±  0%    11.97 ±   0%        ~ (p=0.325 n=7)
Lease/sqlite-4        1600.0µ ± 19%   800.0µ ± 112%        ~ (p=0.051 n=7)
Lease/dqlite-4        1400.0µ ± 14%   800.0µ ±  50%  -42.86% (p=0.002 n=7)
Update/sqlite-4         24.93 ±  1%    24.65 ±   4%        ~ (p=0.209 n=7)
Update/dqlite-4         24.79 ±  2%    24.85 ±   2%        ~ (p=0.805 n=7)
geomean                 5.097          3.217         -36.87%

                    │     Before      │                  After                  │
                    │ page-writes/op  │ page-writes/op  vs base                 │
Compaction/sqlite-4   11550.0m ± 1%        208.0m ± 0%  -98.20% (p=0.001 n=7)
Compaction/dqlite-4    9259.0m ± 2%        187.2m ± 0%  -97.98% (p=0.001 n=7)
Create/sqlite-4          5.116 ± 0%         5.111 ± 0%        ~ (p=0.598 n=7)
Create/dqlite-4          4.183 ± 1%         4.181 ± 2%        ~ (p=0.875 n=7)
Delete/sqlite-4          5.327 ± 1%         5.339 ± 1%        ~ (p=0.901 n=7)
Delete/dqlite-4          4.396 ± 0%         4.393 ± 0%        ~ (p=0.401 n=7)
Get/sqlite-4             0.000 ± 0%         0.000 ± 0%        ~ (p=1.000 n=7) ¹
Get/dqlite-4             0.000 ± 0%         0.000 ± 0%        ~ (p=1.000 n=7) ¹
Lease/sqlite-4           0.000 ± 0%         0.000 ± 0%        ~ (p=1.000 n=7) ¹
Lease/dqlite-4           0.000 ± 0%         0.000 ± 0%        ~ (p=1.000 n=7) ¹
Update/sqlite-4          5.147 ± 1%         5.121 ± 1%        ~ (p=0.246 n=7)
Update/dqlite-4          4.156 ± 0%         4.157 ± 1%        ~ (p=0.640 n=7)
geomean                             ²                   -48.33%               ²
¹ all samples are equal
² summaries must be >0 to compute geomean

                    │      Before       │                  After                  │
                    │  sec-reading/op   │ sec-reading/op  vs base                 │
Compaction/sqlite-4    737.8µ ±   61%      122.7µ ±   6%  -83.37% (p=0.001 n=7)
Compaction/dqlite-4   615.00µ ±   28%      26.90µ ±  25%  -95.63% (p=0.001 n=7)
Create/sqlite-4        37.70µ ± 3017%      84.70µ ± 255%        ~ (p=0.456 n=7)
Create/dqlite-4        8.300µ ±   84%      8.800µ ±  39%        ~ (p=0.876 n=7)
Delete/sqlite-4        209.9µ ±  279%      283.4µ ±  51%        ~ (p=0.383 n=7)
Delete/dqlite-4        56.20µ ±   25%      54.10µ ±  13%        ~ (p=0.128 n=7)
Get/sqlite-4           91.20µ ±    3%      87.40µ ±   3%   -4.17% (p=0.007 n=7)
Get/dqlite-4           41.10µ ±   12%      32.50µ ±  19%  -20.92% (p=0.003 n=7)
Lease/sqlite-4         300.0n ±     ?      300.0n ±    ?        ~ (p=0.249 n=7)
Lease/dqlite-4          0.000 ±    0%       0.000 ±   0%        ~ (p=1.000 n=7) ¹
Update/sqlite-4        167.6µ ±    6%      162.0µ ± 258%        ~ (p=0.456 n=7)
Update/dqlite-4       105.00µ ±   18%      94.20µ ±  11%        ~ (p=0.209 n=7)
geomean                               ²                   -29.61%               ²
¹ all samples are equal
² summaries must be >0 to compute geomean

                    │      Before       │                  After                  │
                    │  sec-writing/op   │ sec-writing/op  vs base                 │
Compaction/sqlite-4   3140.000µ ±  6%       9.900µ ± 12%  -99.68% (p=0.001 n=7)
Compaction/dqlite-4     82.200µ ± 53%       6.800µ ±  7%  -91.73% (p=0.001 n=7)
Create/sqlite-4          1.069m ± 12%       1.177m ±  4%  +10.10% (p=0.017 n=7)
Create/dqlite-4          35.00µ ± 43%       33.20µ ± 25%        ~ (p=0.902 n=7)
Delete/sqlite-4          1.161m ± 11%       1.170m ± 11%        ~ (p=0.620 n=7)
Delete/dqlite-4          23.40µ ± 38%       21.50µ ± 29%        ~ (p=0.200 n=7)
Get/sqlite-4              0.000 ±  0%        0.000 ±  0%        ~ (p=1.000 n=7) ¹
Get/dqlite-4              0.000 ±  0%        0.000 ±  0%        ~ (p=1.000 n=7) ¹
Lease/sqlite-4            0.000 ±  0%        0.000 ±  0%        ~ (p=1.000 n=7) ¹
Lease/dqlite-4            0.000 ±  0%        0.000 ±  0%        ~ (p=1.000 n=7) ¹
Update/sqlite-4          1.071m ±  4%       1.113m ±  9%   +3.92% (p=0.005 n=7)
Update/dqlite-4          18.10µ ± 51%       21.60µ ± 30%        ~ (p=0.620 n=7)
geomean                               ²                   -48.96%               ²
¹ all samples are equal
² summaries must be >0 to compute geomean

                    │     Before      │                After                 │
                    │      B/op       │     B/op       vs base               │
Compaction/sqlite-4     84442.0 ±  2%     726.0 ± 65%  -99.14% (p=0.001 n=7)
Compaction/dqlite-4   141.290Ki ± 27%   4.325Ki ± 34%  -96.94% (p=0.001 n=7)
Create/sqlite-4         18.93Ki ±  0%   18.89Ki ±  2%        ~ (p=0.073 n=7)
Create/dqlite-4         17.65Ki ±  1%   17.65Ki ±  0%        ~ (p=0.301 n=7)
Delete/sqlite-4         28.43Ki ±  3%   28.35Ki ±  1%        ~ (p=0.456 n=7)
Delete/dqlite-4         25.27Ki ±  0%   25.27Ki ±  0%        ~ (p=1.000 n=7)
Get/sqlite-4            23.33Ki ±  0%   23.31Ki ±  0%   -0.08% (p=0.001 n=7)
Get/dqlite-4            21.11Ki ±  0%   21.11Ki ±  0%   +0.03% (p=0.001 n=7)
Lease/sqlite-4          12.33Ki ±  0%   12.33Ki ±  0%   +0.02% (p=0.001 n=7)
Lease/dqlite-4          12.33Ki ±  0%   12.33Ki ±  0%   +0.02% (p=0.001 n=7)
Update/sqlite-4         26.18Ki ±  0%   26.13Ki ±  2%   -0.16% (p=0.001 n=7)
Update/dqlite-4         23.43Ki ±  0%   23.42Ki ±  0%        ~ (p=0.138 n=7)
geomean                 26.65Ki         13.40Ki        -49.72%

                    │    Before     │                After                 │
                    │   allocs/op   │  allocs/op   vs base                 │
Compaction/sqlite-4   3380.00 ±  2%   28.00 ± 71%  -99.17% (p=0.001 n=7)
Compaction/dqlite-4    3283.0 ± 27%   105.0 ± 34%  -96.80% (p=0.001 n=7)
Create/sqlite-4         419.0 ±  0%   418.0 ±  4%        ~ (p=0.054 n=7)
Create/dqlite-4         336.0 ±  0%   336.0 ±  0%        ~ (p=0.192 n=7)
Delete/sqlite-4         664.0 ±  5%   659.0 ±  1%        ~ (p=0.191 n=7)
Delete/dqlite-4         446.0 ±  0%   446.0 ±  0%        ~ (p=1.000 n=7) ¹
Get/sqlite-4            538.0 ±  0%   535.0 ±  0%   -0.56% (p=0.001 n=7)
Get/dqlite-4            379.0 ±  0%   379.0 ±  0%        ~ (p=1.000 n=7)
Lease/sqlite-4          224.0 ±  0%   224.0 ±  0%        ~ (p=1.000 n=7) ¹
Lease/dqlite-4          224.0 ±  0%   224.0 ±  0%        ~ (p=1.000 n=7) ¹
Update/sqlite-4         611.0 ±  0%   606.0 ±  4%   -0.82% (p=0.001 n=7)
Update/dqlite-4         428.0 ±  0%   427.0 ±  0%   -0.23% (p=0.021 n=7)
geomean                 572.4         287.5        -49.77%
¹ all samples are equal

                    │        Before         │                    After                     │
                    │ network-bytes-read/op │ network-bytes-read/op  vs base               │
Compaction/dqlite-4          43.641Ki ± 37%           1.592Ki ± 35%  -96.35% (p=0.001 n=7)
Create/dqlite-4                 240.6 ±  0%             240.2 ±  0%        ~ (p=0.362 n=7)
Delete/dqlite-4               1.157Ki ±  0%           1.158Ki ±  0%   +0.08% (p=0.016 n=7)
Get/dqlite-4                  1.132Ki ±  0%           1.133Ki ±  0%        ~ (p=0.152 n=7)
Lease/dqlite-4                0.07280 ± 40%           0.03280 ± 27%  -54.95% (p=0.001 n=7)
Update/dqlite-4                 565.2 ±  0%             564.3 ±  0%   -0.16% (p=0.001 n=7)
geomean                         291.0                   146.7        -49.59%

                    │          Before          │                      After                      │
                    │ network-bytes-written/op │ network-bytes-written/op  vs base               │
Compaction/dqlite-4             1111.000 ±  1%               4.404 ±   5%  -99.60% (p=0.001 n=7)
Create/dqlite-4                    114.7 ±  1%               113.9 ±   1%        ~ (p=0.366 n=7)
Delete/dqlite-4                    423.5 ±  0%               423.5 ±   0%        ~ (p=0.228 n=7)
Get/dqlite-4                       80.53 ±  0%               80.50 ±   0%        ~ (p=0.085 n=7)
Lease/dqlite-4                   0.09600 ± 67%             0.01360 ± 476%  -85.83% (p=0.001 n=7)
Update/dqlite-4                    264.8 ±  0%               263.4 ±   0%   -0.53% (p=0.002 n=7)
geomean                            69.27                     19.85         -71.34%

Current status

goos: linux
goarch: amd64
pkg: github.com/canonical/k8s-dqlite/test
cpu: AMD EPYC 7763 64-Core Processor                
                    │    After     │
                    │    sec/op    │
Compaction/sqlite-4   42.16µ ± 12%
Compaction/dqlite-4   43.40µ ±  2%
Create/sqlite-4       1.419m ±  3%
Create/dqlite-4       1.178m ±  5%
Delete/sqlite-4       1.547m ±  9%
Delete/dqlite-4       1.390m ± 10%
Get/sqlite-4          266.4µ ±  1%
Get/dqlite-4          357.6µ ±  2%
Lease/sqlite-4        102.3µ ±  1%
Lease/dqlite-4        101.2µ ±  3%
Update/sqlite-4       1.546m ±  7%
Update/dqlite-4       1.471m ±  9%
geomean               395.7µ

                    │        After         │
                    │ page-cache-misses/op │
Compaction/sqlite-4           0.000 ± 0%
Compaction/dqlite-4           0.000 ± 0%
Create/sqlite-4               0.000 ± 0%
Create/dqlite-4               0.000 ± 0%
Delete/sqlite-4               0.000 ± 0%
Delete/dqlite-4               0.000 ± 0%
Get/sqlite-4                  0.000 ± 0%
Get/dqlite-4                  0.000 ± 0%
Lease/sqlite-4                0.000 ± 0%
Lease/dqlite-4                0.000 ± 0%
Update/sqlite-4               0.000 ± 0%
Update/dqlite-4               0.000 ± 0%
geomean                                  ¹
¹ summaries must be >0 to compute geomean

                    │        After         │
                    │ page-cache-spills/op │
Compaction/sqlite-4           0.000 ± 0%
Compaction/dqlite-4           0.000 ± 0%
Create/sqlite-4               0.000 ± 0%
Create/dqlite-4               0.000 ± 0%
Delete/sqlite-4               0.000 ± 0%
Delete/dqlite-4               0.000 ± 0%
Get/sqlite-4                  0.000 ± 0%
Get/dqlite-4                  0.000 ± 0%
Lease/sqlite-4                0.000 ± 0%
Lease/dqlite-4                0.000 ± 0%
Update/sqlite-4               0.000 ± 0%
Update/dqlite-4               0.000 ± 0%
geomean                                  ¹
¹ summaries must be >0 to compute geomean

                    │     After     │
                    │ page-reads/op │
Compaction/sqlite-4    12.21 ±   1%
Compaction/dqlite-4    12.20 ±   1%
Create/sqlite-4        13.87 ±   3%
Create/dqlite-4        13.84 ±   2%
Delete/sqlite-4        25.00 ±   1%
Delete/dqlite-4        24.52 ±   1%
Get/sqlite-4           14.96 ±   0%
Get/dqlite-4           11.97 ±   0%
Lease/sqlite-4        800.0µ ± 112%
Lease/dqlite-4        800.0µ ±  50%
Update/sqlite-4        24.65 ±   4%
Update/dqlite-4        24.85 ±   2%
geomean                3.217

                    │     After      │
                    │ page-writes/op │
Compaction/sqlite-4    208.0m ± 0%
Compaction/dqlite-4    187.2m ± 0%
Create/sqlite-4         5.111 ± 0%
Create/dqlite-4         4.181 ± 2%
Delete/sqlite-4         5.339 ± 1%
Delete/dqlite-4         4.393 ± 0%
Get/sqlite-4            0.000 ± 0%
Get/dqlite-4            0.000 ± 0%
Lease/sqlite-4          0.000 ± 0%
Lease/dqlite-4          0.000 ± 0%
Update/sqlite-4         5.121 ± 1%
Update/dqlite-4         4.157 ± 1%
geomean                            ¹
¹ summaries must be >0 to compute geomean

                    │      After      │
                    │ sec-reading/op  │
Compaction/sqlite-4   122.7µ ±   6%
Compaction/dqlite-4   26.90µ ±  25%
Create/sqlite-4       84.70µ ± 255%
Create/dqlite-4       8.800µ ±  39%
Delete/sqlite-4       283.4µ ±  51%
Delete/dqlite-4       54.10µ ±  13%
Get/sqlite-4          87.40µ ±   3%
Get/dqlite-4          32.50µ ±  19%
Lease/sqlite-4        300.0n ±    ?
Lease/dqlite-4         0.000 ±   0%
Update/sqlite-4       162.0µ ± 258%
Update/dqlite-4       94.20µ ±  11%
geomean                             ¹
¹ summaries must be >0 to compute geomean

                    │     After      │
                    │ sec-writing/op │
Compaction/sqlite-4   9.900µ ± 12%
Compaction/dqlite-4   6.800µ ±  7%
Create/sqlite-4       1.177m ±  4%
Create/dqlite-4       33.20µ ± 25%
Delete/sqlite-4       1.170m ± 11%
Delete/dqlite-4       21.50µ ± 29%
Get/sqlite-4           0.000 ±  0%
Get/dqlite-4           0.000 ±  0%
Lease/sqlite-4         0.000 ±  0%
Lease/dqlite-4         0.000 ±  0%
Update/sqlite-4       1.113m ±  9%
Update/dqlite-4       21.60µ ± 30%
geomean                            ¹
¹ summaries must be >0 to compute geomean

                    │     After     │
                    │     B/op      │
Compaction/sqlite-4     726.0 ± 65%
Compaction/dqlite-4   4.325Ki ± 34%
Create/sqlite-4       18.89Ki ±  2%
Create/dqlite-4       17.65Ki ±  0%
Delete/sqlite-4       28.35Ki ±  1%
Delete/dqlite-4       25.27Ki ±  0%
Get/sqlite-4          23.31Ki ±  0%
Get/dqlite-4          21.11Ki ±  0%
Lease/sqlite-4        12.33Ki ±  0%
Lease/dqlite-4        12.33Ki ±  0%
Update/sqlite-4       26.13Ki ±  2%
Update/dqlite-4       23.42Ki ±  0%
geomean               13.40Ki

                    │    After    │
                    │  allocs/op  │
Compaction/sqlite-4   28.00 ± 71%
Compaction/dqlite-4   105.0 ± 34%
Create/sqlite-4       418.0 ±  4%
Create/dqlite-4       336.0 ±  0%
Delete/sqlite-4       659.0 ±  1%
Delete/dqlite-4       446.0 ±  0%
Get/sqlite-4          535.0 ±  0%
Get/dqlite-4          379.0 ±  0%
Lease/sqlite-4        224.0 ±  0%
Lease/dqlite-4        224.0 ±  0%
Update/sqlite-4       606.0 ±  4%
Update/dqlite-4       427.0 ±  0%
geomean               287.5

                    │         After         │
                    │ network-bytes-read/op │
Compaction/dqlite-4           1.592Ki ± 35%
Create/dqlite-4                 240.2 ±  0%
Delete/dqlite-4               1.158Ki ±  0%
Get/dqlite-4                  1.133Ki ±  0%
Lease/dqlite-4                0.03280 ± 27%
Update/dqlite-4                 564.3 ±  0%
geomean                         146.7

                    │          After           │
                    │ network-bytes-written/op │
Compaction/dqlite-4               4.404 ±   5%
Create/dqlite-4                   113.9 ±   1%
Delete/dqlite-4                   423.5 ±   0%
Get/dqlite-4                      80.50 ±   0%
Lease/dqlite-4                  0.01360 ± 476%
Update/dqlite-4                   263.4 ±   0%
geomean                           19.85

pkg/kine/prepared/db.go

test/admission_test.go

louiseschmidtgen

Great work Marco, excited to watch this land! Let's test this against the k8s-snap and microk8s and view traces as discussed.

marco6 · 2024-08-01T13:26:08Z

Given the discussion we had with @ktsakalozos, it is best not to change schema in this version, so now this query depends on #136 instead.

I expect the query to take a little bit longer than before, as the index is not covering anymore for the deleted rows, but it is still an improvement over what we had before.

marco6 · 2024-08-02T16:41:46Z

Marking as ready since #136 was meged.

pkg/kine/drivers/generic/generic.go

pkg/kine/logstructured/sqllog/sql.go

marco6 force-pushed the marco6/improve-compaction branch from 728d136 to 1ee6dd4 Compare July 29, 2024 10:20