Code Llama

Code Llama is a family of large language models (LLMs) for generating code and natural language about code from either code or natural language prompts. Code Llama models are code-specialized versions of Llama 2, created by training the base model on code-specific datasets. They offer infilling capabilities, support for large input contexts, and zero-shot instruction following capabilities for programming tasks. Code Llama is free for research and commercial use and is being released under the same community license as Llama 2. Meta AI benchmark testing shows Code Llama outperforms other publicly available LLMs for code tasks.

Code Llama is available in three model sizes (7B, 13B, and 34B parameters) and three variants:

• Code Llama—base models designed for general code synthesis and understanding
• Code Llama - Python—models designed specifically for Python
• Code Llama - Instruct—models for instruction-following and safer deployment

All variants of Code Llama are autoregressive language models using optimized transformer architectures. Code Llama 7B and 13B support infilling text generation. All models were fine-tuned with up to 16K tokens and support up to 100K tokens at inference time.

Code Llama was released by Meta AI on August 24, 2023. Code Llama's release was accompanied by training information, model weights, and a research paper that goes into detail on its development and evaluation. The models were trained between January 2023 and July 2023 using custom libraries. Training and fine-tuning of the released models were performed using Meta's Research Super Cluster. Training all nine Code Llama models required 400K GPU hours of computation on hardware of type A100-80GB (TDP of 350-400W). All three model sizes were trained with 500B tokens of code and code-related data. The 7B and 13B base and instruct models have also been trained with fill-in-the-middle (FIM) capabilities, allowing them to insert code into pre-existing code, enabling tasks such as code completion.

The company says Code Llama can make workflows faster and more efficient for developers and lowers the barrier for new developers learning to code. It could be used as both a productivity and education tool, helping programmers produce more robust and better-documented software.

Code Llama's enhanced coding capabilities allow it to generate code and natural language about code. The models can also be used for code completion and debugging. Code Llama supports popular languages, including Python, C++, Java, PHP, Typescript (Javascript), C#, and Bash. The three model sizes provide different serving and latency requirements. The 7B model can be served on a single GPU. The 7B and 34B models are faster and better suited for low latency tasks (e.g., real-time code completion), whereas the 34B model returns the best results for improved coding assistance.

Code Llama is Meta AI's base code-specialized model designed for general code synthesis and understanding. While Code Llama is specialized for code-specific tasks, it isn’t appropriate as a foundation model for other tasks. Meta AI does not recommend using Code Llama or Code Llama-Python to perform general natural language tasks since neither model is designed to follow natural language instructions.

Diagram showing the training process of the three Code Llama Variants.

Code Llama - Python is a language-specialized variation of Code Llama, fine-tuned on 100B tokens of Python code. Meta AI developed Code Llama - Python as it is the most benchmarked language for code generation, and Python and PyTorch play an important role in the AI community.

Code Llama - Instruct is an instruction fine-tuned and aligned variation of Code Llama. Instruction tuning continues the training process, feeding the model a “natural language instruction” input and the expected output. This makes the model better at understanding what humans expect out of their prompts. Meta AI recommends using Code Llama - Instruct variants whenever using Code Llama for code generation since Code Llama - Instruct has been fine-tuned to generate helpful and safe answers in natural language.

Meta AI tested Code Llama’s performance against existing solutions, using two popular coding benchmarks—HumanEval and Mostly Basic Python Programming (MBPP). Meta AI states that Code Llama performs better than open-source, code-specific LLMs and outperformed Llama 2. For example, Code Llama 34B scored 53.7% on HumanEval and 56.2% on MBPP, the highest compared with other open solutions, and on par with ChatGPT.