ChatGLM6b Install

 

The ChatGLM6b model has impressive results and can run on our current 12 GB GPU so we decided to download and install it while waiting for our new 48 GB GPU to come on the boat to Anguilla.   Note it was trained on both Chinese and English and is still impressive in English.

To download the model from HugginFace.co we made a unsecure script hugginface.download.py that you can download and then use.  It is unsecure as it will download and run code so don't use on a computer with any secrets or important stuff.  To our script to download maybe 12GB with all the weights etc. do:

 python3 hugginface.download.py THUDM/chatglm-6b

Then we used the English README file form GitHub as instructions and from the right directory to line up with the huggingface  weights did:

git clone https://github.com/THUDM/ChatGLM-6B

Our GPU can only handle INT8 version of this model so we had to change one line of  demo.cli.py to from
 

model = AutoModel.from_pretrained("THUDM/chatglm-6b", trust_remote_code=True).half().cuda()

To:
model = AutoModel.from_pretrained("THUDM/chatglm-6b", trust_remote_code=True).half().quantize(8).cuda()

 

Then we needed to install Nvidia Cuda so we went to:

https://developer.nvidia.com/cuda-downloads
 

We used type of OS as "Linux/X86-64/WSL-Ubuntu" as we have Microsoft's WSL for running Ubuntu on windows.
Then clicked "deb(network)" as type of installer and did what it said.  This took more than a half hour to download/install.


Then we could run it with:

cd ChatGLM-6B
python3 cli_demo.py 
 
Takes a minute to start up but then it runs fast and does well for a 6B model and only INT8. 
It does make up lots of stuff. 
  
 

Comments

Post a Comment

Popular posts from this blog

Amazingly easy to run LLMs with Ollama

Image
  There is a new tool called Ollama that makes it really easy to try out different LLMs on your local machine.   Here are several youtube videos about this.   Ollama - Local Models on your machine Sam Witteveen Running Mistral AI on your machine with Ollama Learn Data with Mark Ollama: The Easiest Way to RUN LLMs Locally Prompt Engineering Ollama on Linux: Easily Install Any LLM on Your Server Ian Wootten   It is so easy to run models with Ollama! To install Ollama on our Windows machine we opened a Windows/Linux terminal and entered:   curl https://ollama.ai/install.sh | sh   Then to run a model we can do things like:  ollama run llama2:70b   At https://ollama.ai/library you can see what models are available. After you click on a model you can click "tags" to see all the  different versions. Then you can click to copy the command to run that model.  I look for the largest one that fits in my 48 GB GPU.    At https://ollama.ai/library/llama2/tags there are different model
Read more

Chatbot Arena and Leaderboard

Image
   The people at lmsys.org have Chatbot Arena   where you can give a question, see answers from two different bots and rate them.   Only after are you told which bots they were.    Using these human rated competitions between random bots, they can give ratings to the different bots.  As of May 3rd 2023 this is what their Leaderboard looked like: I do think that Vicuna is the best of these.   So I think the results are right.   And I like the method.   Hope they keep doing this.   It will make it easier to tell which is the best open source bot quickly.  Also interesting is their open source leaderboard .     
Read more

Vicuna Install

  Get Linux on Windows The official instructions for vicuna are here .   We had some trouble on Windows so the first thing we did was get Linux onto our Windows machine.   In a Power Window type:    wsl --install    After this we had to reboot the computer and then it will ask for a Linux user name and password.    After this the terminal icon will have an Ubuntu option for starting terminals.    Start a linux terminal and continue with everything below in linux. Get Vicuna git clone https://github.com/lm-sys/ FastChat.git cd FastChat pip3 install --upgrade pip  pip3 install -e . Get Weights cd mkdir vicuna.weights Use browser to go to https://huggingface.co/ eachadea/vicuna-13b/resolve/ main/ Click download next to each file.  Put these in directory vicuna.weights (note really saved to Windows directory and then moved from /mnt/c/..) Run Vicuna cd cd FastChat python3 -m fastchat.serve.cli --model-name ~/vicuna.weights --device cpu Results This got it running on our CPU.   We would
Read more