Microsoft Reveals Phi-3: First in a New Wave of SLMs

Small Language Models or SLMs are the new kid on the block and refer to the much smaller language models that have made a buzz recently but have had much significance in propelling the industry to its current state. Unlike Large LMs, SLMs are less powerful and sophisticated LMs. They are also low cost considering most LLM model parameters range in the billions and even trillions in extreme cases.

Additionally, SLMs are cheaper than many LLMs in small-size versions, leading to reduced requirements for computational power or memory and lower consumption of energy. Thus, due to their lower cost and accessibility, SLMs are highly beneficial for businesses with no sufficient financial resources. SLMs are designed for particular tasks or areas where they are better than more general large models.

Microsoft Phi-3

Phi-3 marks a significant leap forward in small language models, propelling the evolution of model servers and the small AI stack. Microsoft recently announced Phi-3 as a game-changer in the field. Unveiled to the public on April 23, 2024, Phi-3 is not merely another incremental change but a unique model that significantly impacts the world of SLMs.

Why Phi-3 Is Essential

Phi-3 stands out due to its minimal parameter size, only one million, which is adequate for most practical applications. Its reduced computational power requirement allows it to run efficiently on smartphones, enhancing both performance and privacy. This smaller model size facilitates quicker and more comfortable implementation.

In the Microsoft Ecosystem

Phi-3 is accessible to a broad range of developers and IT professionals through platforms comparable to the Azure model gallery. Open-source model sites like Hugging Face and Ollama further support its distribution, making Phi-3 a superior choice for personal machine operation.

Technical Brilliance of Phi-3

Phi-3 has achieved top performance in the realm of open-context models. The Phi-3-mini variant, with 3.8 billion parameters, supports context lengths of 4K and 128K tokens—the first model to offer up to 128K tokens with minimal quality compromise. It is instruction-tuned for natural language communication, making it immediately usable. This model also features optimized support for ONNX Runtime, Windows DirectML, and cross-platform GPU compatibility.

A New Training Approach

Phi-3's training methodology is particularly innovative, inspired by children's learning methods. Researchers have used a curriculum that includes generating content from a list of over 3,000 phrases to mimic children's books, enhancing the model's coding and reasoning capabilities.

Safety First Model Design

Phi-3 models were developed following the Microsoft Responsible AI Standard—a company-wide set of requirements built on six principles: accountability, transparency, fairness, reliability and safety, privacy and security, and inclusiveness. The development process of Phi-3 models included rigorous safety measurements and evaluations, red-teaming, and sensitive use reviews, all guided by stringent security protocols. This comprehensive approach ensures that these models are responsibly developed, tested, and deployed, adhering strictly to Microsoft’s standards and best practices.

The Future of Small Language Models

Phi-3 is just the beginning of Microsoft's ventures into SLMs. Upcoming models like Phi-3-small (7B) and Phi-3-medium (14B) will expand the family, offering more options across the quality-cost spectrum.

In Conclusion

Microsoft's Phi-3 demonstrates the robust potential of small language models to deliver state-of-the-art performance with significantly less complexity in implementation and training. This advancement makes advanced AI technologies more accessible, promising to ignite a wave of innovation across the tech landscape.