release-6933

What's Changed

• feat(pageserver): do space check before gc-compaction by @skyzh in #9250
• storage_controller: delete stale shards when deleting tenant by @erikgrinaker in #9333
• safekeeper: flush WAL on compute disconnect by @erikgrinaker in #9436
• Add logging of installed_extensions by @lubennikovaav in #9438
• refactor(pageserver): remove aux v1 code path by @skyzh in #9424
• pageserver: detect & warn on loading an old index which is probably the result of a bad generation by @jcsp in #9383
• Update Rust to 1.82.0 and mold to 2.34.0 by @arpad-m in #9445
• update rustls by @conradludgate in #9396
• 2024 oct new clippy lints by @conradludgate in #9448
• compute_ctl: Add endpoint that allows setting role grants by @devjv in #9395
• storcon: fix case where we might fail to send compute notifications after two opposite migrations by @jcsp in #9435
• Increase shared_buffers in test_subscriber_synchronous_commit. by @arssher in #9427
• Update ruff to much newer version by @cloneable in #9433
• compute_ctl: Add endpoint that allows extensions to be installed by @devjv in #9344
• walproposer: immediately exit if sync-safekeepers collected 0/0. by @arssher in #9442
• pageserver: disconnect when asking client to reconnect by @VladLazar in #9390
• [local_proxy]: install pg_session_jwt extension on demand by @conradludgate in #9370
• storcon: handle ongoing deletions gracefully by @VladLazar in #9449
• Take the collector name as argument when generating sql_exporter configs by @tristan957 in #9447
• Accept basebackup --gzip requests by @arpad-m in #9456
• bump pg-session-jwt version by @conradludgate in #9455
• Compute release 2024-10-20 by @ololobus in #9459

Full Changelog: release-6911...release-6933

release-6911

What's Changed

• proxy: split out the console-redirect backend flow by @conradludgate in #9270
• Ignore pg_dynshmem fiel when comparing directories by @knizhnik in #9374
• CI(build-build-tools): fix unexpected cancellations by @bayandin in #9357
• Add test for fixed storage broker issue by @arpad-m in #9311
• tests: make size comparisons more lenient by @VladLazar in #9388
• Also consider offloaded timelines for obtaining retain_lsn by @arpad-m in #9308
• Add more timing histogram and gauge metrics to the Neon extension by @MMeent in #9116
• Switch CI, Storage and Proxy to Debian 12 (Bookworm) by @bayandin in #9170
• pageserver: stabilize & refine controller scale test by @jcsp in #8971
• Shut down timelines during offload and add offload tests by @arpad-m in #9289
• [local_proxy] update api for pg_session_jwt by @conradludgate in #9359
• proxy: Migrate jwt module away from anyhow by @cloneable in #9361
• Remove redundant code by @knizhnik in #9373
• Generate sql_exporter config files with Jsonnet by @tristan957 in #9368
• fix(pageserver): use a buffer for basebackup; add aux basebackup metrics log by @skyzh in #9401
• Add newly added metrics from #9116 to exports by @MMeent in #9402
• Fix jsonnet_files wildcard by @tristan957 in #9403
• Add jsonnetfmt targets by @tristan957 in #9404
• don't trigger workflows in two repos by @fcdm in #9340
• pageserver: improve handling of archival_config calls during Timeline shutdown by @jcsp in #9415
• proxy: format imports with nightly rustfmt by @cloneable in #9414
• storcon: use the same trace fields for reconciler and results by @VladLazar in #9410
• tests: stabilize test_timelines_parallel_endpoints by @jcsp in #9413
• Remove allow(clippy::unknown_lints) by @arpad-m in #9416
• Add support of extensions for v17 (part 2) by @lubennikovaav in #9389
• Activate timelines during unoffload by @arpad-m in #9399
• refactor(pageserver): upgrade remote_storage to use hyper1 by @skyzh in #9405
• proxy: merge AuthError and AuthErrorImpl by @cloneable in #9418
• Fix beta clippy warnings by @arpad-m in #9419
• Fix typo in sql_exporter generator by @tristan957 in #9420
• Fix some sql_exporter metrics for PG 17 by @tristan957 in #9406
• readme: fix build instructions for debian 12 by @problame in #9371
• Check access_count in lfc_evict by @knizhnik in #9407
• storage_scrubber: fixes to garbage commands by @jcsp in #9409
• proxy: move the connection pools to separate file by @awarus in #9398
• Add config variable for timeline offloading by @arpad-m in #9421
• Add support of extensions for v17 (part 3) by @lubennikovaav in #9430
• test_runner: Update the README.md to build neon with 'testing' by @awarus in #9437
• Releases/2024 10 17 compute kq only by @lubennikovaav in #9451

Full Changelog: release-6872...release-6911

release-6872

What's Changed

• Fix update of statistic for LFC/prefetch by @knizhnik in #9272
• proxy: rename console -> control_plane, rename web -> console_redirect by @conradludgate in #9266
• proxy: Move module base files into module directory by @cloneable in #9297
• Make sure BufferTag padding bytes are cleared in hash keys by @hlinnaka in #9292
• feat(compute_tools): Create JWKS Postgres roles without attributes by @devjv in #9031
• Replace BUFFERTAGS_EQUAL compatibility macro with new-style function by @hlinnaka in #9294
• Improve logging on changes in a compute's status by @tristan957 in #9265
• storage_broker: update hyper and tonic again by @arpad-m in #9299
• vm-monitor: Ignore LFC in postgres cgroup memory threshold by @sharnoff in #8668
• pgxn: Fix compiler warnings, and enable them by @hlinnaka in #9290
• Remove non-existent entries from .dockerignore by @hlinnaka in #9209
• Remove get_self_dir() by @tristan957 in #9306
• Fix compiler warnings on macOS by @bayandin in #9319
• storcon: apply all node status changes before handling transitions by @VladLazar in #9281
• Enable all pyupgrade checks in ruff by @tristan957 in #9323
• Add timeline offload mechanism by @arpad-m in #8907
• tests: Remove redundant log lines when starting an endpoint by @hlinnaka in #9316
• remote_storage: add DownloadOpts::byte_(start|end) by @erikgrinaker in #9293
• tests: Reduce noise from logging renamed files by @hlinnaka in #9315
• proxy: Unalias hyper1 and replace one use of hyper0 in test by @cloneable in #9324
• storcon: make observed state updates more granular by @VladLazar in #9276
• test_runner: fix path to tpc-h queries by @bayandin in #9327
• safekeeper: add evicted_timelines gauge. by @arssher in #9318
• Add /installed_extensions endpoint to collect statistics about extension usage. by @lubennikovaav in #8917
• pageserver: add direct io config to virtual file by @yliang412 in #9214
• tests: Use ThreadedMotoServer python class to launch mock S3 server by @hlinnaka in #9313
• tests: Remove redundant log lines when stopping storage nodes by @hlinnaka in #9317
• added workflow Report Workflow Stats by @fedordikarev in #9330
• Add partial support of extensions for v17: by @lubennikovaav in #9322
• local_proxy: integrate with pg_session_jwt extension by @conradludgate in #9086
• Move PgBenchInitResult.EXTRACTORS to a private module constant by @tristan957 in #9334
• Improve some typing in test_runner by @tristan957 in #9303
• add local-proxy suffix to wake-compute requests, respect the returned port by @conradludgate in #9298
• storage controller: use 'infra' JWT scope for node registration by @jcsp in #9343
• CODEOWNERS: assign storage_scrubber to storage by @erikgrinaker in #9346
• storage_scrubber: fix faulty assertion when no timelines by @erikgrinaker in #9345
• Move path-related fixtures into their own module by @tristan957 in #9304
• tests: allow a log warning in test_cli_start_stop_multi by @jcsp in #9320
• Enable cargo caching in check-codestyle-rust by @tristan957 in #9307
• Add compute_config_dir fixture by @tristan957 in #9352
• Use valid type hints for Python 3.9 by @tristan957 in #9349
• Introduce "quota" ErrorKind by @awarus in #9300
• tests: stabilize test_storage_controller_heartbeats by @jcsp in #9347
• proxy: Demote some errors to warnings in logs by @cloneable in #9354
• ci: gh-workflow-stats-action v0.1.4: remove debug output and proper pagination by @fedordikarev in #9356
• Test versions mismatch by @a-masterov in #9167
• CI(run-python-test-set): allow to skip missing compatibility snapshot by @bayandin in #9365
• proxy: refactor auth backends by @conradludgate in #9271
• Storage & Compute release 2024-10-14 by @vipvap in #9372

New Contributors

• @devjv made their first contribution in #9031
• @fedordikarev made their first contribution in #9330

Full Changelog: release-6812...release-6872

release-6812

What's Changed

• Expose more granular wait event data to the user by @MMeent in #9163
• proxy: Add timeout to webauth confirmation wait by @cloneable in #9227
• pageserver: tweak logging of "became visible" for layers by @jcsp in #9224
• storcon: ignore deleted timelines on new location catch-up by @VladLazar in #9244
• storcon: add metric for long running reconciles by @VladLazar in #9207
• fix(test): only test num of L1 layers in compaction smoke test by @skyzh in #9186
• tests: sync_after_each_test -> sync_between_tests by @jcsp in #9239
• Fix metric name of the 'getpage_wait_seconds_bucket' metric by @hlinnaka in #9242
• Change parentheses to clarify conditions in walproposer by @dotdister in #9180
• test_runner: bump requests and psycopg2-binary by @bayandin in #9248
• Add compute_tools/ to the list of paths that trigger an E2E run on a PR by @hlinnaka in #9251
• Use hyper 1.0 and tonic 0.12 in storage broker by @arpad-m in #9234
• Suppress wal lag timeout warnings right after tenant attachment by @arpad-m in #9232
• Remove unnecessary dependencies from postgis-build image by @hlinnaka in #9211
• chore: smaller layer changes by @koivunej in #9247
• vm-monitor: Upgrade axum from 0.6 to 0.7 by @hlinnaka in #9257
• Rename hyper 1.0 to hyper and hyper 0.14 to hyper0 by @arpad-m in #9254
• pageserver: tweak oversized key read path warning by @VladLazar in #9221
• safekeeper: fix panic in debug_dump. by @arssher in #9097
• Revert hyper and tonic updates by @arpad-m in #9268
• chore: remove unnecessary comments in compute/Dockerfile.compute-node by @davidgomes in #9253
• chore: makes some onboarding document improvements by @davidgomes in #9216
• tests: Refactor NeonCli test class by @hlinnaka in #9195
• Cargo.toml: enable sso for aws-config by @erikgrinaker in #9261
• pageserver: use conditional GET for secondary tenant heatmaps by @erikgrinaker in #9236
• storcon: do timeline creation on all attached location by @VladLazar in #9237
• remote_storage: add head_object integration test by @erikgrinaker in #9274
• neon walsender_hooks: take basebackup LSN directly. by @arssher in #9263
• proxy: add local-proxy to compute image by @conradludgate in #8823
• proxy: exclude triple logging of connect compute errors by @awarus in #9277
• safekeeper: remove local WAL files ignoring peer_horizon_lsn. by @arssher in #8900
• Storage & Compute release 2024-10-07 by @vipvap in #9291

New Contributors

• @awarus made their first contribution in #9277

Full Changelog: release-6777...release-6812

release-6777

What's Changed

• fix: ignore tonic to resolve advisories by @skyzh in #9230
• Update aws SDK crates by @arpad-m in #9233
• Fix compute metrics collector config by @hlinnaka in #9235
• Compute release 2024-10-02 (2) by @lubennikovaav in #9249

Full Changelog: release-6773...release-6777

release-6773

What's Changed

• Bump default neon extension version to 1.5 by @hlinnaka in #9188
• pageserver: refactor immediate_gc into TenantManager by @jcsp in #9183
• test_compatibility: fix the regexes detecting the version by @a-masterov in #9205
• tests: longer timeout in test_delete_timeline_client_hangup by @jcsp in #9161
• Misc python test cleanup by @hlinnaka in #9194
• add proxy-protocol header disable option by @conradludgate in #9203
• Bump vm-builder v0.29.3 -> v0.35.0 by @petuhovskiy in #9208
• proxy: auth broker by @conradludgate in #8855
• Remove unnecessary dev package from compute image by @hlinnaka in #9210
• storage controller: drop out of blocking compute notification loop if migration origin becomes unavailable by @jcsp in #9147
• Upgrade OpenTelemetry and other tracing crates by @cloneable in #9200
• tests: make test_multi_attach more stable by @jcsp in #9202
• feat(compute): adds pg_session_jwt extension to compute image by @davidgomes in #8888
• storage_scrubber: retry on index deletion failures by @jcsp in #9204
• proxy: ignore typ and use jwt.alg rather than jwk.alg by @conradludgate in #9215
• Use debian packages for cmake again by @hlinnaka in #9212
• Add new compute metrics to sql exporter by @hlinnaka in #9190
• docs: add aux file v2 RFC by @skyzh in #9115
• safekeeper: Fix a log message of HTTP worker by @shinyaaa in #9213
• Fix post_apply_config() by @lubennikovaav in #9220
• pageserver: remove TenantConfOpt::TryFrom<toml_edit::Item> by @erikgrinaker in #9219
• Fix small memory accounting bug in libpagestore by @MMeent in #9223
• tests: wait walreceiver on sks to be gone on 'immediate' ep restart. by @arssher in #9099
• Backpressure: reset ps display after it is done. by @arssher in #8980
• Fix the size of the perf counters shared memory array by @hlinnaka in #9226
• Compute release 2024-10-02 by @lubennikovaav in #9228

New Contributors

• @davidgomes made their first contribution in #8888

Full Changelog: release-6745...release-6773

release-6745

What's Changed

• CI: Leave out unnecessary build files from binary artifact by @hlinnaka in #9135
• tests: remove patching up of az id column by @VladLazar in #8968
• CI(promote-images): fix prod ECR auth (#9131) by @bayandin in #9146
• fix(pageserver): passthrough partition cancel error by @skyzh in #9154
• Fix compute_logical_snapshot_files for v14 by @tristan957 in #9155
• chore: Silence clippy warning with nightly by @hlinnaka in #9157
• Forbid creation of child timelines of archived timeline by @arpad-m in #9122
• CI: run compatibility tests on Postgres 17 by @bayandin in #9145
• fix(compute_ctl): race condition in configurator by @petuhovskiy in #9162
• pageserver: rename control plane client & chunk validation requests by @jcsp in #8997
• fix(pageserver): categorize client error in basebackup metrics by @skyzh in #9110
• Fix flakiness of test_timeline_copy. by @arssher in #9061
• proxy: make local-proxy config by signal with PID, refine JWKS apis with role caching by @conradludgate in #9164
• safekeeper: skip syncfs on start if --no-sync is specified. by @arssher in #9166
• libs: improve logging on PG connection errors by @VladLazar in #9130
• Dump backtrace if a core dump is called just "core" by @hlinnaka in #9125
• Use long options when calling initdb by @tristan957 in #9174
• fix(pageserver): LSN lease edge cases around restarts/migrations by @yliang412 in #9055
• proxy: add jwks endpoint to control plane and mock providers by @conradludgate in #9165
• pageserver: remove AdjacentVectoredReadBuilder and bump minmimum io_buffer_alignment to 512 by @yliang412 in #9175
• WalProposer: Read WAL directly from buffers by @MMeent in #9171
• tests: Reduce the # of iterations in safekeeper::test_random_schedules by @hlinnaka in #9182
• fix(pageserver): abort process if fsync fails by @skyzh in #9108
• neon_local: Use clap derive macros to parse the CLI args by @hlinnaka in #9103
• Set disk quotas on bind in compute_ctl by @petuhovskiy in #8936
• CI(benchmarking): fix Nightly Benchmarks by @bayandin in #9178
• CI(benchmarking): fix setting LD_LIBRARY_PATH by @bayandin in #9191
• Storage & Compute release 2024-09-30 by @vipvap in #9196

Full Changelog: release-6715...release-6745

release-6715

What's Changed

• chore(#9077): cleanups & code dedup by @problame in #9082
• Move the patch to compute by @a-masterov in #9120
• test: Make test_hot_standby_feedback more forgiving of slow initialization by @hlinnaka in #9113
• Little test startup speedups by @hlinnaka in #9094
• pageserver: handle decompression outside vectored read_blobs by @yliang412 in #8942
• Catch Cancelled and don't print a warning for it by @arpad-m in #9121
• Fix compiler warnings on macOS by @bayandin in #9128
• test: Make test_lfc_resize more robust by @hlinnaka in #9117
• Remove TenantState::Loading by @hlinnaka in #9118
• chore(docker-compose): fix typo in readme by @Damian972 in #9133
• fix(test): storage scrubber should only log to stdout with info by @skyzh in #9067
• storcon: add tags to scheduler logs by @VladLazar in #9127
• Re-export git-version from utils and remove as direct dep by @cloneable in #9138
• storcon: do az aware scheduling by @VladLazar in #9083
• storage controller: make proxying of GETs to pageservers more robust by @jcsp in #9065
• docs(rfc): Independent compute release flow by @ololobus in #8881
• Try building images using Debian 12 "Bookworm" for PG17 by @MMeent in #9132
• storcon: include timeline ID in LSN waiting logs by @VladLazar in #9141
• fix(pageserver): handle lsn lease requests for unnormalized lsns by @yliang412 in #9137
• Compute release 2024-09-25 by @lubennikovaav in #9151

New Contributors

• @Damian972 made their first contribution in #9133

Full Changelog: release-6694...release-6715

release-6694

What's Changed

• Bump vendor/postgres to include extenision path fix by @lubennikovaav in #9076
• Fix init of WAL page header at startup by @hlinnaka in #8914
• Move load_timeline_metadata into separate function by @arpad-m in #9080
• Check submodule forward progress by @hlinnaka in #8949
• Misc dependency cleanup by @hlinnaka in #9092
• storcon: update compute hook state on detach by @VladLazar in #9045
• walredo: add a ping method by @problame in #8952
• compact_level0_phase1: remove final traces of value access mode config by @problame in #8935
• change HTTP method to comply with spec by @nikitakalyanov in #9100
• feat(pageserver): filter keys with gc-compaction by @skyzh in #9004
• test: Downgrade info message on removing empty directories by @hlinnaka in #9093
• Add views for metrics about pageserver requests by @hlinnaka in #9008
• Maintain number of used pages for LFC by @knizhnik in #9088
• NOAI by @MMeent in #9084
• Fix compiler warnings with nightly rustc about elided lifetimes having names by @hlinnaka in #9105
• Move files related to building compute image into compute/ dir by @hlinnaka in #9090
• scrubber: retry when missing index key in the listing by @yliang412 in #8873
• Add regression tests for a cloud-based Neon instance by @a-masterov in #8681
• storcon: improve initial shard scheduling by @VladLazar in #9081
• proxy: check if IP is allowed during webauth flow by @cloneable in #9101
• Update PostgreSQL 17 from 17rc1 to 17.0 by @MMeent in #9119
• Storage & Compute release 2024-09-23 by @vipvap in #9095
• CI(promote-images): fix prod ECR auth by @bayandin in #9131

Full Changelog: release-6667...release-6694

release-6667

What's Changed

• Grant access to pg_show_replication_origin_status for neon_superuser by @tristan957 in #8978
• Fix PG17's extension modifications by @MMeent in #9010
• Remove misc unused code by @hlinnaka in #9014
• Generate pgbench data on the server by @tristan957 in #8977
• Little python test cleanup by @hlinnaka in #9018
• storcon: forward requests from stepped down instance to the current leader by @VladLazar in #8954
• CI: fix release pipeline by @bayandin in #9017
• Remove unnecessary dependencies by @hlinnaka in #9000
• Fix test code related to migrations by @tristan957 in #9019
• Rename some S3 usages to "remote storage" in exposed messages by @arpad-m in #8999
• CI: don't fetch the whole git history if it's not required by @bayandin in #9021
• refactor(walredo): process launch & kill-on-error machinery by @problame in #8951
• add neon_local subcommand for the broker & use that from regression tests by @problame in #8948
• Refactor ImageLayerWriter to avoid passing a Timeline to finish() by @hlinnaka in #9028
• proxy: make sql-over-http max request/response sizes configurable by @cloneable in #9029
• CI(deploy): fix deploy to staging and prod by @bayandin in #9030
• Add v17 to revisions.json by @tristan957 in #9027
• proxy: mock cplane usable without allowed-ips table by @cloneable in #9046
• neon_local start: parallel startup to break cyclic dependency by @problame in #8950
• fix(storage-scrubber): log version after initialize the logger by @skyzh in #9049
• Remove misc unused code by @hlinnaka in #9054
• Remove leftover dead code and other cleanup in config.rs by @hlinnaka in #9053
• safekeeper: add wal_last_modified to debug_dump. by @arssher in #8994
• pageserver: simple fix for vectored read image layer skip by @VladLazar in #9026
• impr(pageserver): log image layer creation by @skyzh in #9050
• proxy: remove auth info from http conn info & fixup jwt api trait by @conradludgate in #9047
• pageserver: wait for lsn lease duration after transition into AttachedSingle by @yliang412 in #9024
• refactor(pageserver): check layer map valid in one place by @skyzh in #9051
• Bump vendor/postgres by @arssher in #8986
• safekeeper: fix OpenAPI spec by @bayandin in #9066
• pageserver: add counters for started smgr/getpage requests by @problame in #9069
• Skip running clippy in --release mode. by @hlinnaka in #9073
• fix(pageserver): receive body error now 500 by @skyzh in #9052
• utils: leaky bucket should only report throttled if the notify queue is blocked on sleep by @conradludgate in #9072
• pageserver: suppress noisy "layer became visible" logs by @jcsp in #9064
• pageserver: throttling: per-tenant metrics + more metrics to help understand throttle queue depth by @problame in #9077
• storage hotfix release 2024-09-20 by @problame in #9085

Full Changelog: release-6616...release-6667