Mistral unleashes Pixtral Large, upgrades Le Chat with image gen

Mistral, the French startup that made waves last year with a record-setting seed funding amount for Europe, has launched a slew of updates today including a new, large foundational model named Pixtral Large.

The company is further upgrading its free web-chased chatbot, Le Chat, adding image generation, web search, and an interactive “canvas,” matching the features of and turning it into a more serious and direct competitor to OpenAI’s ChatGPT.

As Mistral AI CEO and co-founder Arthur Mensch wrote on his account on the social network X, “At Mistral, we’ve grown aware that to create the best AI experience, one needs to co-design models and product interfaces. Pixtral was trained with high-impact front-end applications in mind and is a good example of that.”

Users who want to try out the new Le Chat features will need to enable them as beta features on the web interface. Note that Le Chat access does require a free Mistral, Google, or Microsoft account to use.

Pixtral Large — open source multimodal AI

Pixtral Large, Mistral’s new 124-billion-parameter model, builds upon its predecessor, Mistral Large 2, unveiled over the summer 2024, as well as its first multimodal model, Pixtral 12-B, released in September.

It includes a 123-billion-parameter decoder and a 1-billion-parameter vision encoder, enabling it to excel in both text and visual data processing.

Parameters, as you’ll recall, refer to the number of settings that govern a model’s inputs and outputs, with more parameters generally connoting a more capable, knowledgable and performant model.

According to a post by Mistral Head of Developer Relations Sophia Yang to her X account, Pixtral Large excels at “multilingual OCR [optical character recognition], reasoning, chart understanding, and more.” Yang included a screenshot of Pixtral Large in Le Chat analyzing a receipt uploaded by a user using OCR, showing its capabilities for ingesting and documenting expenses, as well as in this case, splitting a bill with a tip included.

With a context window of 128,000 tokens, Pixtral Large is able to handle up to 30 high-resolution images per input or around a 300-page book, again equivalent to leading OpenAI GPT series models.

The model demonstrates state-of-the-art performance across diverse benchmarks, including MathVista, DocVQA, and VQAv2, making it ideal for tasks like chart interpretation, document analysis, and image understanding.

While the model and weights are available for download freely on Hugging Face, they are released under a custom Mistral AI Research License, which specifies only non-commercial, research-focused applications.

Those looking to use it commercially will need to do so through Mistral’s API on its Le Platforme managed web service, or obtain a separate license from the company directly through a contact form, meaning it is not actually fully open source.

Still, by offering Pixtral Large, Mistral AI empowers researchers and developers to harness advanced multimodal AI while ensuring responsible and ethical use.

Le Chat comes for ChatGPT with rival matching features

At the center of Mistral’s AI tools is Le Chat, a free platform now enhanced with new features powered by Pixtral Large.

Designed for diverse use cases like research, ideation, and automation, Le Chat integrates text, vision, and interactive functionalities into a seamless productivity experience.

New Features of Le Chat:

1. Web Search with Citations: Users can supplement the AI’s knowledge with real-time web searches, complete with source citations for transparency.

2. Canvas for Ideation: This innovative interface allows users to create, modify, and collaborate on documents, presentations, and designs in an interactive new space that appears to the left of the chatbot interface.

As Yang wrote about it on X: Le Chat Canvas is “great for creative ideation. You can use Canvas to create documents, presentations, code, mockups… the list goes on.”

It comes just six weeks after OpenAI released its own Canvas sidebar interactive element for ChatGPT, which many viewed as a feature designed to rival Anthropic’s earlier Artifacts release for its Claude chatbot.

3. Advanced Document and Image Analysis: With Pixtral Large, Le Chat can now process and summarize complex PDFs, extracting insights from graphs, tables, equations, and more.

4. Image Generation: Through a partnership with separate image model startup Black Forest Labs, Le Chat now includes image generation capabilities powered by the Flux Pro model, enabling users to produce high-quality visuals directly in the chat interface. This is a clear answer to OpenAI’s DALL-E 3 integration in ChatGPT (both models from OpenAI, however) as well as the second big integration of Black Forest Labs’ new models into a leading AI foundation model provider’s offerings, following its earlier team-up with Elon Musk’s xAI to power image generation in that company’s Grok-2 chatbot available through X, the social network Musk also owns.

5. Task Agents for Automation: Customizable agents automate repetitive tasks like summarizing meeting minutes, processing invoices, or scanning receipts, saving users time and effort.

These features position Le Chat as a versatile AI assistant, capable of handling tasks traditionally requiring multiple tools.

Mistral AI highlights Le Chat’s comprehensive feature set and its accessibility compared to platforms like ChatGPT, Perplexity, and Claude. While competitors may require premium subscriptions for similar functionalities, Le Chat provides an integrated, multimodal experience entirely for free during its beta phase.

Mistral is coming to play hard

With Pixtral Large and the enhanced Le Chat, Mistral is flexing its research and development muscles.

Even as some in the tech industry believe that the cost of intelligence is being driven down and making life more difficult for model providers to find revenue streams, Mistral isn’t giving up on advancing its offerings to compete with the other leaders in the field, and doing so on fewer parameters — 124 billion compared to say, 405 billion from Meta’s latest Llama 3.1 release.

However, Mistral is still missing some of the advanced voice and audio features found on rivals such as OpenAI’s ChatGPT Advanced Voice Mode or Google’s Gemini Live.

A recent survey by Kong showed despite its technical prowess and varying open-source and proprietary offerings, usage of Mistral’s models and API by large enterprises remain far behind those of U.S.-based companies such as OpenAI, Anthropic, and Microsoft.

Yet with the recent presidential election and influence of xAI founder Elon Musk on President Trump, it is likely that the EU and those within it will look to Mistral as a means of accessing AI outside the control of the U.S. and its new, controversial leader.

Put another way: AI is rapidly becoming tied to nationalism and geopolitics, and Mistral finds itself in the perhaps advantageous position of being one of the best AI model providers Europe has yet cultivated.