Content-Length: 311769 | pFad | http://github.com/vllm-project/vllm-spyre/pull/179

57 Add maybe_update_max_tokens for class SpyrePlatform by gmarinho2 · Pull Request #179 · vllm-project/vllm-spyre · GitHub
Skip to content

Add maybe_update_max_tokens for class SpyrePlatform #179

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 4 commits into
base: main
Choose a base branch
from
Draft

Add maybe_update_max_tokens for class SpyrePlatform #179

wants to merge 4 commits into from

Conversation

gmarinho2
Copy link
Contributor

FIX: #148

Currently the class SpyrePlatform uses the default max_tokens set in the OpenAI frontend code. This new method selects the warmup shape that fits the prompt and has the biggest shape['new_tokens']. This way SpyrePlatform will use the maximum number possible for token generation when max_tokens is not set in the request body.

@gmarinho2
add maybe_update_max_tokens for SpyrePlatform
083b986 
Signed-off-by: Gabriel Marinho <gmarinho@ibm.com>
@github-actions GitHub Actions
Copy link

👋 Hi! Thank you for contributing to vLLM support on Spyre.
Just a reminder: Make sure that your code passes all the linting checks, otherwise your PR won't be able to be merged. To do so, first install the linting requirements, then run format.sh and commit the changes. This can be done with uv directly:

uv sync --frozen --group lint --active --inexact

Or this can be done with pip:

uv pip compile --group lint > requirements-lint.txt
pip install -r requirements-lint.txt
bash format.sh

Now you are good to go 🚀

gmarinho2 added 3 commits May 22, 2025 14:44
@gmarinho2
add maybe_update_max_tokens for SpyrePlatform
4c41b06 
Signed-off-by: Gabriel Marinho <gmarinho@ibm.com>
@gmarinho2
add validation to check if self._warmup_shapes is None
189e6f6 
Signed-off-by: Gabriel Marinho <gmarinho@ibm.com>
@gmarinho2
add validation to check if self._warmup_shapes is None
671ed65 
Signed-off-by: Gabriel Marinho <gmarinho@ibm.com>
@wallashss
Copy link
Collaborator

I think you should probably set this PR as draft and wait for the feedback of the vllm community on vllm-project/vllm#18557. This changes depends on that.

@gmarinho2 gmarinho2 marked this pull request as draft May 22, 2025 19:15
maxdebayser
maxdebayser approved these changes May 22, 2025
View reviewed changes
Copy link
Collaborator

@maxdebayser maxdebayser left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, once the upstream PR gets merged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@maxdebayser maxdebayser maxdebayser approved these changes

@yannicks1 yannicks1 Awaiting requested review from yannicks1 yannicks1 will be requested when the pull request is marked ready for review yannicks1 is a code owner

@tdoublep tdoublep Awaiting requested review from tdoublep tdoublep will be requested when the pull request is marked ready for review tdoublep is a code owner

@nikolaospapandreou nikolaospapandreou Awaiting requested review from nikolaospapandreou nikolaospapandreou will be requested when the pull request is marked ready for review nikolaospapandreou is a code owner

@sducouedic sducouedic Awaiting requested review from sducouedic sducouedic will be requested when the pull request is marked ready for review sducouedic is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Incorrect default max_completion_tokens being set
3 participants
@gmarinho2 @wallashss @maxdebayser








ApplySandwichStrip

pFad - (p)hone/(F)rame/(a)nonymizer/(d)eclutterfier!      Saves Data!


--- a PPN by Garber Painting Akron. With Image Size Reduction included!

Fetched URL: http://github.com/vllm-project/vllm-spyre/pull/179

Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy