feat: merge staging (vector store, doc saver/loader) into main #23
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This commit introduces the RedisVectorStore class, extending VectorStore to enable vector storage and retrieval using Memorystore Redis.
…20) * fix: fix linter and encoding of MemorystoreChatMessageHistory class. * fix: use relative path for __init__ imports
* feat(vector store): Add usage demo and enhance Redis integration - Add Jupyter notebook in `docs/` for using langchain vector store with Memorystore Redis. - Update `vector_store.py` for improved performance and reliability with Redis. - Modify `__init__.py` to reflect package structure changes and new functionalities. - Include `requirements.txt` and `setup.py` for easy installation and dependency management. - Add `state_of_the_union.txt` in `docs/` as a sample dataset for the notebook demo. * incorporate review feedback * added missing license header * excluded the test file from license check * reformatted source files * incorporated review feedback * fixed lint errors * fixed more lint errors * removed key_prefix argument from vectorstore * incorporated review feedback * fixed formatting errors * fixed a bad merge * suppress mypy errors for setuptools * trying mypy.ini * remove setup.py and requirements * add numpy dependency
* feat: add MemorystoreDocumentSaver class * fix: use variable in the worker pool of the Cloud Build integration * fix: update doc saver for json encoding * fix: align key_prefix behavior * fix: add back key_prefix if keys_or_ids is given * fix: fix indent of the comments * fix: get encoder directly from the Redis client * fix: add pipeline support for Redis client * fix: reorder prefix and content_field in constructor * fix: insert prefix to all given document IDs * fix: fix tests prefix and chat encoding
…es (#22) * feat(vector store): Add usage demo and enhance Redis integration - Add Jupyter notebook in `docs/` for using langchain vector store with Memorystore Redis. - Update `vector_store.py` for improved performance and reliability with Redis. - Modify `__init__.py` to reflect package structure changes and new functionalities. - Include `requirements.txt` and `setup.py` for easy installation and dependency management. - Add `state_of_the_union.txt` in `docs/` as a sample dataset for the notebook demo. * incorporate review feedback * added missing license header * excluded the test file from license check * reformatted source files * incorporated review feedback * fixed lint errors * fixed more lint errors * removed key_prefix argument from vectorstore * incorporated review feedback * fixed formatting errors * fixed a bad merge * suppress mypy errors for setuptools * trying mypy.ini * remove setup.py and requirements * add numpy dependency * refactor(vector-store): enhance validation and initialization processes - Add validation for unsupported data types and negative vector sizes in VectorIndexConfig. - Refactor field_name parameters in HNSWConfig and FLATConfig to support type hinting with optional strings. - Enhance RedisVectorStore initialization by enforcing type checks for client, index_name, and embedding_service. - Introduce an optional 'ids' parameter in methods to allow explicit document identifiers, improving document management. - Adjust method documentation and error messages for clarity and consistency. * Incorporated review feedback * fixed formatting * incorporated review feedback * formatting * fixed bugs * fix formatting * added vector store unittests * actualy added vector store tests * fix formatting * fix: fix styles and formats --------- Co-authored-by: Craig Chi <craigchi@google.com>
averikitsch
requested changes
Feb 15, 2024
PingXie
reviewed
Feb 16, 2024
| """ | ||
| if ids: | ||
| doc_ids = [self._key_prefix + doc_id for doc_id in ids] | ||
| self._redis.delete(*doc_ids) |
There was a problem hiding this comment.
Should we consider pipelining as well?
There was a problem hiding this comment.
DEL is a single command that accepts multiple keys at once. So I'm not sure how pipeline can help here? I updated it to batch delete though, PTAL, thanks!
averikitsch
approved these changes
Feb 16, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:
Fixes #12 , #14 , #15, #19