Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add PreProcessor for VLM #57

Open
wants to merge 4 commits into
base: main-dev
Choose a base branch
from
Open

Conversation

wnma3mz
Copy link

@wnma3mz wnma3mz commented Jan 1, 2024

No description provided.

@ashvardanian
Copy link
Contributor

Thank you for contributions, @wnma3mz! Detaching the preprocessing code is probably the right thing to do. Give us a couple of days to merge it 🤗

@VoVoR
Copy link
Contributor

VoVoR commented Jan 5, 2024

@wnma3mz hey,

We appreciate your work on the PR!

I wanted to ask you to remove the changes from src/ dir and keep all the updates in scripts that are useful for onnx/coreml runtimes. We are using scr/ together with our pre-training code. So we didn't want to update it frequently.
We know it will be great to separate preprocessing and modeling into different classes, and did it already. You can expect it in the next release in a few weeks. But we did it in a different way a little bit.

Also, as far as I understand, you tested your script with model_fpath = "unum-cloud/uform-coreml-onnx," correct?

@wnma3mz
Copy link
Author

wnma3mz commented Jan 6, 2024

Thanks for your reply, I have deleted the changes in the src directory.

As you said, I tested it at scripts/example.py. Therefore, this part of the code will have an impact. When you push a new preprocessing, feel free to remind me to update scripts/example.py to make sure it works correctly.

@ashvardanian ashvardanian changed the base branch from main to main-dev January 11, 2024 20:13
@VoVoR
Copy link
Contributor

VoVoR commented Jan 15, 2024

@wnma3mz hi
I've tested the exmaple.py script with "model_fpath = 'unum-cloud/uform-coreml-onnx'" - it didn't work. And it shouldn't because "get_model" won't work with our coreml/onnx HF model card.
How did you exactly run the script? Can you push the working version by any chance so I can check it?

@wnma3mz
Copy link
Author

wnma3mz commented Jan 15, 2024

@VoVoR

I'm sorry for the trouble.
For the convenience of testing, I downloaded all the model file locally in advance. The file structure is as follows:

├── multilingual-v2.image-encoder.mlpackage
│   ├── Data
│   │   └── com.apple.CoreML
│   │       ├── model.mlmodel
│   │       └── weights
│   │           └── weight.bin
│   └── Manifest.json
├── multilingual-v2.image-encoder.mlpackage.zip
├── multilingual-v2.text-encoder.mlpackage
│   ├── Data
│   │   └── com.apple.CoreML
│   │       ├── model.mlmodel
│   │       └── weights
│   │           └── weight.bin
│   └── Manifest.json
├── multilingual-v2.text-encoder.mlpackage.zip
├── multilingual.image-encoder.onnx
├── multilingual.text-encoder.onnx
├── tokenizer.json
├── torch_config.json
└── torch_weight.pt

The current 'snapshot_download' function can interfere with testing due to network reasons, so I added the 'get_local_model' function for easy run.

If you have any other questions, please feel free to remind me

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants