Basically the title.
For me, a dense model around 20b, with an additional engram table of 10-20b, or a sparse MoE around 80b would be amazing.
Basically the title.
For me, a dense model around 20b, with an additional engram table of 10-20b, or a sparse MoE around 80b would be amazing.
Another 35B-A3B MoE or similar would be nice…
FYI, Yandex released an 80B MoE base model (i.e. not instruction tuned!) recently: https://huggingface.co/yandex/AliceAI-Foundation-80B-A3B-Base – haven’t experimented with it personally though.
I hope an instruction tuned will be released as well.