LMSYS Org

Large Model Systems Organization (LMSYS Org) is an open research organization aiming to make AI more accessible by co-developing open models, datasets, systems, and evaluation tools. LMSYS's work encompasses research in both machine learning and systems, including the training of widely available large language models (LLMs) and the development of distributed systems to accelerate their training and inference.

The organization was founded in 2023 by students and faculty from the University of California, Berkeley in collaboration with the University of California, San Diego, and Carnegie Mellon University. LMSYS now also includes members Mohamed bin Zayed University of Artificial Intelligence, UAE. The organization primarily relies on university grants and donations to perform its work and access compute resources.

Vicuna-13B is an open-source chatbot trained by fine-tuning LLaMA on user-shared conversations collected from ShareGPT. Evaluation using GPT-4 shows Vicuna-13B achieves more than 90% quality of OpenAI ChatGPT and Google Bard while outperforming other models like LLaMA and Stanford Alpaca in more than 90% of cases. The code and weights, along with an online demo, are publicly available for non-commercial use.

LongChat-7B and LongChat-13B are LLMs with an extended context length of up to 16K tokens. Evaluation has shown the long-range retrieval accuracy of LongChat-13B is up to 2x higher than other long-context open models such as MPT-7B-storywriter (84K), MPT-30B-chat (8K), and ChatGLM2-6B (8k).

FastChat-T5 is an open-source chatbot trained by fine-tuning Flan-t5-xl (3B parameters) on 70K user-shared conversations collected from ShareGPT. It is based on an encoder-decoder transformer architecture, and can autoregressively generate responses to users' inputs.

Chatbot Arena is a benchmark platform LLM that features anonymous, randomized battles in a crowdsourced manner. Users chat with two anonymous models side-by-side and vote for which one is better. Users can undergo multiple turns of conversations before voting. The names of the models are revealed after voting. Conversations with identity keywords (e.g., ChatGPT, Bard, Vicuna) or any votes after the names are revealed do not count towards the Chatbot Arena leaderboard.

MT-bench is a set of challenging multi-turn open-ended questions for evaluating chat assistants. To automate the evaluation process, strong LLMs like GPT-4 act as judges and assess the quality of the models' responses.

LMSYS developed the Chatbot Arena Conversations dataset, containing 33K cleaned conversations with pairwise human preferences collected from Chatbot Arena between April and June 2023. Each sample includes two model names, their full conversation text, the user vote, the anonymized user ID, the detected language tag, the OpenAI moderation API tag, the additional toxic tag, and the timestamp. All personally identifiable information (PII) has been removed from the conversations.

FastChat is an open platform for training, serving, and evaluating LLM-based chatbots.