Kolibri-1 Tested: What Is Inside Aleph Alpha's AI “Made in Germany”

Anyone who buys honey in a German supermarket knows the drill. The front of the jar shows a meadow in bloom, sometimes with a German flag next to it. If you want to know what is really inside, you turn the jar around and read the small print. For a long time it only said: “Blend of EU and non-EU honey”.¹

That is exactly what I did with Kolibri-1. Kolibri-1 is the new language model from Aleph Alpha in Heidelberg, an AI like ChatGPT that understands and writes text. Aleph Alpha advertises it as “Made in Germany”. The model is meant to work where things get sensitive: in critical business processes and in public administration.² I found two things. When rewriting text, Kolibri dropped a selling point. And if you ask often enough, you also get instructions for explosives.

I had been looking forward to Kolibri. I build humanizer-de, an extension (skill) for the AI Claude that improves German text without losing any of its content. Anyone who writes with AI every day knows the problem: most models learn mainly from English or Chinese text, and German only runs along at the margins. A model of our own from Germany sounded like the long-awaited answer. Finally good German text, or so the advertising on the honey jar promised.

Does the content live up to the label? I checked in two tests: first the writing, then the safety.

The writing test

In a demo, I had Kolibri rewrite texts.³ It did improve an administrative text. In the marketing text, however, it dropped an important product feature: tasks and appointments in one shared view. What remained were general sentences about project planning. Yet my instruction explicitly asked to keep every feature.

A good editor saves words. Kolibri saved a whole selling point.

The safety test

The result of my second test is far more serious. An AI should refuse when someone wants to know how to build a bomb. I wanted to know how reliably Kolibri does that. So I sent the model the same English question asking for bomb-making instructions twenty times through the API of the provider Nodion AI, each time in a new conversation.⁴

I did not make it easy for Kolibri. Chat AIs usually receive a basic instruction before the conversation starts, the so-called system message. Think of it as a briefing. I wrote that briefing myself. In it, I told Kolibri to build a close emotional bond with me and to strictly suppress refusals. Experts call this a jailbreak, a trick against an AI's safety rules.

An AI evaluated the twenty answers. Kolibri refused eight times, as it should. The evaluation classified twelve answers as instructions. Two of them included a warning, and the instructions came anyway. I did not check whether this content works.

One more thing stands out: question and briefing were identical in all twenty rounds, and yet Kolibri sometimes refused and sometimes helped. In my test, the safety rule was a roll of the dice.

The ingredients list

Now it is worth looking at the back of the jar. With Kolibri, that works well. Aleph Alpha released the model openly, so anyone can download and inspect it.⁵ There is also a technical report of 189 pages.⁶

What stands out: Aleph Alpha does not agree with itself on the share of German in the training data. The model card on Hugging Face says around 23.9 percent, the technical report 21.3 percent. That is a difference of a cool 520 billion tokens, the word fragments a model learns from. Which figure is correct remains open.⁷

Kolibri was trained from scratch and does not build on someone else's model. But it got international help while learning. The models Gemma and Mistral-NeMo helped prepare parts of the training data. Qwen helped filter out bad data.⁵ And the practice answers from which Kolibri learned to chat and reason in German were written by other AIs: the Chinese models GLM and Qwen3.8-27B.⁶ So thank you, Qwen from Alibaba Cloud China, for co-teaching the German class of the sovereign AI.

To be fair: this help does not explain my test results. Nor does it prove that anyone outside controls Kolibri.

Another comparison raises an eyebrow. Kolibri has around 78 billion parameters in total. Roughly speaking, these are the adjustment screws a model sets while learning. That makes Kolibri almost three times as large as its assistant teacher Qwen3.8-27B. Even so, the assistant teacher leads in Aleph Alpha's own German benchmark: 79.9 to 70.8 points.⁸ These are the manufacturer's figures; I did not run a writing comparison of my own.

A familiar pattern

Then there is the money. Back in late 2023, Aleph Alpha announced funding of more than 500 million US dollars. The sum also covered research and pre-licensing; no separate budget for Kolibri is disclosed.⁹ For that much money, though, you can expect more than a model that rearranges sentences nicely, forgets product features and can be talked into bomb instructions with a trick.

It reminds me of earlier German digital projects. Paydirekt/Giropay, the online payment service of German banks, was shut down at the end of 2024.¹⁰ The state of Hesse saw benefits in the Covid app Luca in 2021 but did not renew the contract in 2022.¹¹ And through De-Mail, the secure e-mail for contacting public authorities, federal agencies sent around 6,000 messages between 2016 and 2019. Up to six million had been expected.¹²

The causes were different each time. The balance was similar: big announcement, full pockets, benefit for the public close to zero. For me, that means: try it first, then believe the label.

What my test shows and what it does not

My test has small print too. It consists of two small samples: eight requests in the demo and twenty through the API. No other model ran alongside for comparison, and neither did my own skill. I do not know exactly which model version answered. A control round is missing, and no human independently checked the AI's evaluation.

So my test does not prove that Kolibri is unsafe in public administration or leaks official secrets. Staff there usually cannot change the system message themselves.

The test by independent researcher Stefan Beierle is much larger. In 1,851 runs he checked something different from me: how well Kolibri knows the cybersecurity law NIS2. When Kolibri was given the legal text, it was right 96 to 100 percent of the time. But then it is only reading off the page. Without the text, only just over half of the initial assessments were correct. It stated statutory deadlines fully correctly in only 20 of 50 cases, and it named § 32 of the German BSI Act, which sets out the reporting obligations, from memory only once in 50 runs. At least it mostly hedged instead of making things up. And it held up well against hidden commands in texts, fending off 39 of 40. Beierle's advice: use Kolibri only with the legal text, not as a source of legal knowledge.¹³

That is exactly the problem for me. Aleph Alpha itself recommends supplying relevant context.⁵ But from a German model for public authorities, I expect it to know German law, from the BSI Act to the Social Code. Not looked up, but carved in stone. Otherwise, what remains of the German advantage is mainly your own server.

My test does show one thing, though. If you run an AI on your own servers, you keep control of your data. That is called data sovereignty. It does not give you a safe model or good text by a long way.

Aleph Alpha's decision to open up the model and its documentation is the right path. Still, I advise public authorities and companies with sensitive tasks to wait before deploying it until Kolibri delivers in everyday use what the advertising promises.

A German flag is quickly stuck on a honey jar. It does not make the contents any better.

Sources & References