Add vector service concept, and resource endpoint to handle vector search

[sd2k]

This cannibalizes some of #23 but focusses on the read path: adding a resource endpoint to the plugin to search a configured vector store using the configured embedding engine.

It adds:

• a store package for interacting with various vector stores
  • the interfaces are split into ReadVectorStore and WriteVectorStore so we don't need to implement everything right now (we could remove the write interface for now since it's not being used yet)
  • there's an implementation for the pending Grafana vectorapi here too; I can copy a qdrant one from a draft Grafana PR later on as another example
• an embed package for interacting with embedding engines
  • there's an implementation here for OpenAI embeddings (which vectorapi is also compatible with); more can be added as-and-when
• a vector package which wraps and combines objects from these two packages and utilises them to provide a user-friendly service to search for related text, handling the mapping from collection -> model and the embedding process
• an resource endpoint /vector/search, which frontend clients can use to do search for related objects in various collections
• another docker-compose service for the vectorapi to be deployed, and extra provisioning for the app including more config to talk to the vectorapi service. This will probably need to be removed for now though since vectorapi isn't OSS yet.

Some decisions made here:

• collection details are specified in config; specifically this means each collection has exactly one associated model and dimension
  • this is specified in config rather than anywhere else right now because the vector service is responsible for embedding the query using the same model and dimension as the vectors in the collection
  • in theory this could go somewhere else but we can't make many assumptions about what each vector DB implementation lets you store, so config is a fairly safe denominator
  • ideally this would be shared with whatever sync service we end up using, so that they both use the same embedding model for each collection, and the same collection names for each type of object.
• we have interfaces for vector.Store and embed.Embedder so we can have various concrete implementations, such as Qdrant/Milvus/Weaviate for the store or OpenAI/custom APIs for the embedder. This PR only adds a single one for each though, and we may end up abstracting over stores in vectorapi instead
• there's a collection in the provisioned config named grafana:core:dashboards. The naming convention here is completely open to bikeshedding. The idea is to scope collections similarly to the way Grafana Live channels are scoped (the : can be replaced with / for consistency)

Testing this is a little tricky right now because we don't have open images for vectorapi, or a process for getting data into the vectorapi store. One way to do so is:

• build vectorapi using make
• run docker-compose up in this repository
• run poetry run python -m src.dashboards.index --directory ./src/dashboards/output from the vector-playground directory of the llm-experiment-lab repository, on the vectordb-playground-vectorapi branch. This will send a bunch of dashboards into the grafana:core:dashboards collection, ready for searching
• call the resource endpoint with curl http://localhost:3000/api/plugins/grafana-llm-app/resources/vector/search -d '{"collection": "grafana:core:dashboards", "text": "Mimir usage", "limit": 3}'

[csmarchbanks]

Let's get this going and iterate on it as we get more data into vector dbs!

[yoziru]

alright lgtm, provisioning needs some better docs or maybe less nesting but the core concepts work well 👍