An AI chatbot is a dialogue system that understands and answers requests in natural language without anyone having mapped out the possible conversations in advance. The basis today is almost always a large language model that reads the conversation so far and produces the answer piece by piece. That is exactly where it differs from the classic bot: one follows a decision tree, the other composes.
This also moves the work. With a rule-based bot the effort sits in writing the dialogues. With an AI chatbot it sits in giving the model the right knowledge and the right boundaries.
How an answer comes about
The path through the system
- The user’s question goes to the application, not straight to the model.
- The application gathers matching information: excerpts from the knowledge base, data from the shop or CRM, the conversation so far.
- System instruction, retrieved information and question are assembled into a prompt.
- The model produces the answer.
- The application checks it, logs it and displays it, with a source reference where appropriate.
The crucial step is the second one. A language model knows nothing about your opening hours, your prices or the status of an order. Everything the bot says on those points had to be handed to it beforehand.
Where the knowledge comes from
The usual method for this is retrieval augmented generation: documents are split into sections, stored so they can be searched and retrieved to match each question. The model then composes from whatever it has just been handed.
The advantage is practical. When a piece of information changes, you change the document rather than the model. The answer can be traced back to a source, and that is precisely what you need when somebody asks why the bot claimed something.
How long the conversation and the retrieved excerpts may be together is capped by the context window. It is measured in tokens and is part of what drives running costs: every answer is billed by input and output volume.
Three ways to build one
| Build | How it works | Strength | Weakness |
|---|---|---|---|
| Rule-based | Fixed dialogue trees, keywords, buttons | Predictable, cheap, auditable | Fails on any unexpected phrasing |
| LLM-based | Language model with connected knowledge | Understands free language, covers many cases | Can be wrong, ongoing costs, needs more oversight |
| Hybrid | Rules for unambiguous flows, model for the rest | Combines reliability with language understanding | More work to design |
In practice the hybrid build is usually the viable one. Order tracking is an unambiguous process and belongs in a rule: take the number, query the interface, return the status. An openly phrased question about which variant of a product fits is not unambiguous, and that is where the model earns its keep.
Give the bot permission to operate tools itself and to plan several steps in sequence, and you have an AI agent. That is a different category of project, with considerably higher demands on permissions and oversight.
Where the technology holds up
- Recurring enquiries: questions support already answers daily and whose answers are documented.
- Search across large holdings: manuals, tariff structures, product catalogues. Someone who does not know what to search for gets further with a question than with a search box.
- Pre-qualification: taking down a request, classifying it and handing it on with a summary.
- Cover outside business hours: not as a replacement for people, but so that sorted enquiries are waiting in the morning.
It gets difficult wherever a wrong answer costs more than no answer: legal information, medical questions, binding commitments, price negotiations. There the human belongs in the loop, before the answer rather than after it.
What makes such a project hard
The state of the data. A chatbot is only as good as the documents it answers from. Contradictory or outdated content is not smoothed over, it is amplified. The honest first project step is therefore often tidying up the knowledge base.
Wrong answers. A model produces a fluent answer even when it does not know the answer. These AI hallucinations can be cut back considerably through connected knowledge, source references and a permitted non-answer, but not switched off. “I do not know that, let me put you through” has to be an acceptable outcome.
The handover. The exit to a human is the most important function in the whole system. It must be reachable at all times, carry the conversation with it and contain no dead ends.
Security. A bot that reads foreign content, uploaded documents or ticket histories for instance, is exposed to prompt injection. The more rights it holds on connected systems, the more serious that is.
Measurability. Without figures the benefit stays an assertion. Useful ones are the resolution rate without handover, the abandonment rate, the share of answers with a solid source, and how the ticket volume develops.
Duties when running one
A chatbot on a website touches three subjects at once.
Disclosure. Under the transparency rules of the EU AI Act, people must be able to tell that they are talking to a system. A quiet handover to “Julia from customer service” with a model behind it is not permissible.
Data protection. Whatever users type leaves the building when the model is hosted elsewhere. Place of processing, data processing agreements, retention of transcripts and the handling of personal data entered by accident belong settled before launch, not after.
Accessibility. A chat window is a user interface like any other: keyboard operation, focus handling, meaningful labels, announcement of new messages for screen readers, sufficient contrast, no time pressure. Many bought-in widgets fail exactly here. What needs checking is set out on our page about accessibility.
Frequently asked questions about AI chatbots
What separates an AI chatbot from a normal chatbot? The classic bot follows predefined paths and recognises keywords. The AI chatbot understands free phrasing and composes its answer afresh. The price is less predictability.
Can a chatbot replace support? It takes routine off the desk and shifts the work. It cannot replace support where judgement, goodwill or responsibility are called for. Relief on recurring questions is realistic, abolishing a role is not.
Does the model have to be trained on our data? In most cases no. Knowledge about your own company is connected through a searchable knowledge base rather than through training. That is cheaper, faster to update and easier to trace. Fine-tuning pays off for tone and format rather than for facts.
How much upkeep does such a system need? Some, permanently. Content ages, questions shift, models get retired. A chatbot is not a project with a handover, it is an operation with owners.
Conclusion
An AI chatbot is quick to set up technically and slow to get right editorially. What decides the benefit is rarely the model, it is the quality of the connected data, a clean route to a human, and whether somebody looks at the answers regularly. On top of that come duties that cannot be retrofitted: disclosure, data protection, usability for everyone. How we design applications where interfaces, logging and oversight fit together from the start is described on our page about custom software development. For an assessment of what AI really delivers in projects today, see the article AI in software development. What that means for an actual project, from the knowledge base to a clean handover to a human, is on our page about AI chatbot development. If you are weighing up whether this adds up in your case, work it through in a free consultation.