Integrate vllm with example Lora and Mistral

[lxning]

Description

Please read our CONTRIBUTING.md prior to creating your first pull request.

Please include a summary of the feature or issue being fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.

Fixes #(issue)

Type of change

Please delete options that are not relevant.

• Bug fix (non-breaking change which fixes an issue)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• New feature (non-breaking change which adds functionality)
• This change requires a documentation update

Feature/Issue validation/testing

Please describe the Unit or Integration tests that you ran to verify your changes and relevant result summary. Provide instructions so it can be reproduced.
Please also list any relevant details for your test configuration.

• Test Lora

cd serve/examples/large_models/vllm/lora
cat prompt.json
{
  "prompt": "A robot may not injure a human being",
  "max_new_tokens": 50,
  "temperature": 0.8,
  "logprobs": 1,
  "prompt_logprobs": 1,
  "max_tokens": 128,
  "adapter": "adapter_1"
}
python ../../utils/test_llm_streaming_response.py -m lora -o 50 -t 1 -n 1 --prompt-text "@prompt.json" --prompt-json
Tasks are completed
payload={'prompt': 'A robot may not injure a human being', 'max_new_tokens': 50, 'temperature': 0.8, 'logprobs': 1, 'prompt_logprobs': 1, 'max_tokens': 128, 'adapter': 'adapter_1'}
, output= or, through inaction, allow a human being to come to harm
o

• Test Mistral

cd serve/examples/large_models/vllm/mistral
cat prompt.json
{
  "prompt": "A robot may not injure a human being",
  "max_new_tokens": 50,
  "temperature": 0.8,
  "logprobs": 1,
  "prompt_logprobs": 1,
  "max_tokens": 128
}
python ../../utils/test_llm_streaming_response.py -m mistral -o 50 -t 1 -n 1 --prompt-text "@prompt.json" --prompt-json
Tasks are completed
payload={'prompt': 'A robot may not injure a human being', 'max_new_tokens': 50, 'temperature': 0.8, 'logprobs': 1, 'prompt_logprobs': 1, 'max_tokens': 128}
, output= or, through inaction, allow a human being to come to harm.

• Test pytest

pytest test_example_vllm.py::test_vllm_lora_mar
======================= test session starts ========================
platform linux -- Python 3.10.14, pytest-7.3.1, pluggy-1.4.0
rootdir: /home/ubuntu/serve
plugins: cov-4.1.0, mock-3.12.0
collected 1 item

test_example_vllm.py .                                       [100%]

================== 1 passed in 141.14s (0:02:21) ===================

Checklist:

• Did you have fun?
• Have you added tests that prove your fix is effective or that this feature works?
• Has code been commented, particularly in hard-to-understand areas?
• Have you made corresponding changes to the documentation?

[mreso]

LGTM

[mreso]

typo

[mreso]

vllm uses paged attention which typically allows for a larger batch sizes. We need to figure out a way to saturate the engine as setting batchSize == max_num_seqs will lead to under utilization.

[mreso]

We could use a similar strategy like for the micro-batching to always have enough requests for the engine available. Preferred would be an async mode which will just route all requests to the backend and gets replies asynchronously (as discussed earlier)

[mreso]

If we move the files and delete them we can run the test only once before we need to put them back manually. Can we use symbolic links instead?