These exotic (non linear algebra based) embedding representations are often slow to take off unless they have an obvious use case.
The other one that I've always been curious of is Poincarré Embeddings [1] - where the embedding also has a hierarchical representation of the space.
There's issues with these becoming popular:
1. Querying the embeddings requires more math knowledge than just "lol cosine similarity". This also requires to write code for the query
2. You can often easily match the performance with regular embeddings by just adding dimensions and training more. So the advantage of exotic embeddings has to be with the information in the more complex mathematical abstraction.
So they need a killer usecase to become popular, it's hard to move the needle.
Do you think it makes sense to have a group of models, each with more ad-hoc embeddings, and coordinate them to respond according to the domain of the input?
Do multi-modal models use the same embedding type/structure for an image, sound, text?
I think your first question is open for people to explore.
The answer to the second is yes - it's all vector embeddings, and they're aligned to each other by finding a dataset that matches pairs (eg. images with captions)
The real use for exotic embeddings will have to be in analyzing the embeddings themselves I think, otherwise it's easier to shove normal vectors downstream into other models.
The other one that I've always been curious of is Poincarré Embeddings [1] - where the embedding also has a hierarchical representation of the space.
There's issues with these becoming popular:
1. Querying the embeddings requires more math knowledge than just "lol cosine similarity". This also requires to write code for the query
2. You can often easily match the performance with regular embeddings by just adding dimensions and training more. So the advantage of exotic embeddings has to be with the information in the more complex mathematical abstraction.
So they need a killer usecase to become popular, it's hard to move the needle.
[1]: https://arxiv.org/abs/1705.08039