-
-
Notifications
You must be signed in to change notification settings - Fork 455
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Exception when loading word embedding models with lines containing 2 words #5
Comments
@ricardopieper |
Yes, here it is: You can find more here: http://nilc.icmc.usp.br/embeddings |
One of the offending lines start with "R$ 0,00", it explodes because 0,00 can't be parsed to float. |
I changed the Fasttext model loader class with this code:
The idea is that, if we know the size of the word vectors, we can just load the last N splitted values and consider the rest as the word itself. |
made a pull request describing the fix #8 feel free to evaluate the solution |
Which pre-trained embeddings do you use and hitting this bug? |
@makcedward the same I mentioned earlier. Here's a bit more context: For portuguese word embeddings, we like to use USP's models (USP = Universidade de Sao Paulo, Brazil). They provide models with varied size. In particular we're using fasttext, though we could use any other (for our particular case, fasttext seems to be a bit better). The one I'm using is this one: http://143.107.183.175:22980/download.php?file=embeddings/fasttext/cbow_s50.zip You can find more here: http://nilc.icmc.usp.br/embeddings Also I'm afraid the same fix has to be applied to all other models (glove, word2vec, etc), but I haven't checked. |
After studying pre-trained embeddings from http://nilc.icmc.usp.br/embeddings, found that word2vec, glove and fasttext embeddings follow same fasttext's (FB official embeddings) file format. Will suggest to use FasttextAug to load those library. On the other hand,
Will apply the following change to read content correctly.
|
Some model files contain embeddings with multiple words (the NILC embeddings for portuguese) which causes the model loading code to explode. For instance, a line in the model file might contain this:
The same does not happen in Spacy, for instance.
I fixed it in my local dev environment, might make a pull request later.
The text was updated successfully, but these errors were encountered: