AI Accelerators and GPUs

 

We have Vicona running on our CPU but it is slow and the answers seem of lower quality.   Our current GPU does not have enough memory to run Vicona.   So we have a sudden interest in hardware to run LLMs.

I have ordered a NVIDIA RTX A6000 GPU which has 48GB.  One disadvantage of living on a tropical island is that it takes awhile for boats to bring stuff from Amazon.

This is a fair amount of money and it seems at least some day there will be AI accelerators just for running models that will be lower cost.   It is not clear they are a good answer today.   If anyone knows of an AI Accelerator that would work for Vicona please let me know and I would like to buy one and try it out..

The AI accelerators can do far more calculation with less hardware and power than GPUs.   For example, the Hailo-8™ M.2 AI Acceleration Module sounds amazing.  The Falcon-H8 uses several of these.   But it is not clear if getting a LLM like Vicona to run on this is possible.

The GroqCard™ Accelerator also sounds amazing.  It seems Groq has 7B LLaMA working on at least one of their accelerators.   It seems if LLaMA is working that Vicona should as well as it is the same model just with different weights.

The Axelera also sounds amazing.   Not sure what models it can run.

Also, Cerebras sounds amazing.  This is for data-centers and not end users.

Also, Graphcore BOW-200, looks amazing.   Again, probably for data-centers and not end users. 

The Habana Gaudi2 seems to work with LLMs.  Claims to be twice as fast as Nvidia A100 80GB.

AMD Instinct MI210 - Releasing 2023/3/22.  64GB RAM.    Peak FP16  of 181 TFLOPS, Peak FP32 of 45 TFLOPS.    Price maybe $31,210.  So nice but not cheap.


It seems clear that in the future AI accelerators will be the way to go.  Again if anyone knows of an AI accelerator that today can run a LLM with leading edge performance I would love to try it out (assuming cost is less than the GPU I just ordered). 


Comments

Post a Comment

Popular posts from this blog

Chatbot Arena and Leaderboard

Image
   The people at lmsys.org have Chatbot Arena   where you can give a question, see answers from two different bots and rate them.   Only after are you told which bots they were.    Using these human rated competitions between random bots, they can give ratings to the different bots.  As of May 3rd 2023 this is what their Leaderboard looked like: I do think that Vicuna is the best of these.   So I think the results are right.   And I like the method.   Hope they keep doing this.   It will make it easier to tell which is the best open source bot quickly.  Also interesting is their open source leaderboard .     
Read more

Vicuna Install

  Get Linux on Windows The official instructions for vicuna are here .   We had some trouble on Windows so the first thing we did was get Linux onto our Windows machine.   In a Power Window type:    wsl --install    After this we had to reboot the computer and then it will ask for a Linux user name and password.    After this the terminal icon will have an Ubuntu option for starting terminals.    Start a linux terminal and continue with everything below in linux. Get Vicuna git clone https://github.com/lm-sys/ FastChat.git cd FastChat pip3 install --upgrade pip  pip3 install -e . Get Weights cd mkdir vicuna.weights Use browser to go to https://huggingface.co/ eachadea/vicuna-13b/resolve/ main/ Click download next to each file.  Put these in directory vicuna.weights (note really saved to Windows directory and then moved from /mnt/c/..) Run Vicuna cd cd FastChat python3 -m fastchat.serve.cli --model-name ~/vicuna.weights --device cpu Results This got it running on our CPU.   We would
Read more

Open Source ChatBots

  ChatGPT   2022/11/30.   This was not open source but a challenge to the open source community. LLaMA 2023/02/23    Meta/Facebook.   You need to fill out some paperwork and get approved to download this code.  If you use a torrent and only want 13B be sure not to download the others as it gets huge.   Alpaca   2023/03/13   Stanford.  Created from LLaMA using training against ChatGPT.   The training cost was about $600.   GPT4ALL 2023/03   This has all the files needed to run and is open source.   A bit over 4GB for the download. OpenChatKit 2023/03 ColossalAI 2023/03 is another open source chatbot recently released. Dolly - 2023/03/24 trained on one machine in 30 minutes.   Dolly was the name of a cloned sheep.   Guanaco 2023/03  Raven-RWKV-7B   This uses RNN, or Recurrent Neural Network and not transformers like most LLMs. OpenAsisstant   To release Apr 15, 2023. Demo Vicuna 2023/03   You can try Vicuna here .  The paper on Vicuna explains that they trained on sample chat c
Read more