Introduce outlines.models.transformers_multimodal - #33
Open
lapp0 wants to merge 1 commit into
Open
Conversation
lapp0
commented
Jun 19, 2024
| def __call__( # type: ignore | ||
| self, | ||
| prompts: Union[str, List[str]], | ||
| media: Union[str, Any], |
lapp0
force-pushed
the
multimodal-models
branch
2 times, most recently
from
June 19, 2024 20:02
11c0db1 to
a6c229e
Compare
|
Let’s call it |
lapp0
force-pushed
the
fix-mamba-integration
branch
from
July 15, 2024 09:05
f17913b to
48b6f8f
Compare
rlouf
force-pushed
the
fix-mamba-integration
branch
from
July 15, 2024 13:53
48b6f8f to
bf3694c
Compare
lapp0
force-pushed
the
fix-mamba-integration
branch
8 times, most recently
from
July 16, 2024 00:08
75dc370 to
acb0759
Compare
lapp0
force-pushed
the
multimodal-models
branch
4 times, most recently
from
July 19, 2024 11:56
bdf4097 to
653fe26
Compare
rlouf
reviewed
Jul 19, 2024
| return prompts, media | ||
|
|
||
| @classmethod | ||
| def _load_media(cls, media): |
There was a problem hiding this comment.
I'm not sure this should be part of the library?
Owner
Author
There was a problem hiding this comment.
Probably not, it's a convenience, but unnecessary. Removing.
| from outlines.processors import OutlinesLogitsProcessor | ||
|
|
||
|
|
||
| class TransformersMultiModal(Transformers): |
There was a problem hiding this comment.
Suggested change
| class TransformersMultiModal(Transformers): | |
| class TransformersVision(Transformers): |
| yield self._decode_generation(output_group_ids) | ||
|
|
||
|
|
||
| def transformers_multimodal( |
There was a problem hiding this comment.
Suggested change
| def transformers_multimodal( | |
| def transformers_vision( |
lapp0
force-pushed
the
multimodal-models
branch
9 times, most recently
from
July 19, 2024 14:22
9cc0775 to
56b5918
Compare
lapp0
force-pushed
the
multimodal-models
branch
17 times, most recently
from
July 19, 2024 15:50
6adb73b to
9ae6e70
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Docs: https://github.com/lapp0/outlines/blob/multimodal-models/docs/reference/models/multimodal.md
Done:
Todo:
MultiModalSequenceGeneratorAdaptertest_generate.py, current model is too expensive to be part of test suite.llava-hf/llava-v1.6-mistral-7b-hfImprove docs:
<image>token in prompt