LLM and AI Application Security in Romania

The four risks that turn up on almost every project

Of the OWASP Top 10 for LLM applications, four come back on nearly every project. Prompt injection: instructions hidden in a document, e-mail or page that the assistant reads and then obeys. Unsafe output handling: the model's answer lands directly in HTML, a shell or a query. Sensitive data leakage: one user's context shows up in another's answer or in the logs. Excessive agency: the agent holds keys and rights for actions it should not take alone.

• Prompt injection, direct and through indexed content
• Model output treated as untrusted data, never as instructions
• Context isolated between users and between customers
• Least privilege for tools, with confirmation on irreversible actions
• Cost and rate limits per user and per key

What the attack looks like and how defence is layered

The surface is new but the economics of attack are familiar: cheap to try, automated and repeatable. An attacker plants instructions in a page you will index, then asks the assistant to summarise it and send the content elsewhere, or generates thousands of phrasings until one slips past a filter. Defence cannot be a single system prompt. We layer it: instructions separated from data, guardrails on input and output, PII redaction, access control evaluated at query time, prompt logging and automated red teaming with promptfoo and garak, run as a test on every release.

RAG: who is allowed to see what

In a RAG application the index becomes a copy of your documents, and the access filter has to be evaluated on every query, not when the document was indexed. We tie each fragment to its source and to the rights on that source, filter on metadata before generation and check the answer against the fragments the user was actually allowed to see. Documents, embeddings and the index stay on your infrastructure in Frankfurt; if you want no calls to outside providers, we run the embedding model on a GPU server. Citations in the answer make verification possible.

AI Act transparency, documentation and hours

Since 2 August 2026 the EU AI Act places new obligations on organisations that use AI systems; the most visible is transparency, because people must know when they are talking to an automated system. We document the model, the data entering the context, prompt retention, log access and active filters, on top of the GDPR basics: an Article 28 agreement, standard contractual clauses and PII redaction. Assessments are requested from the client panel and use your included hours; a larger red-teaming programme is billed separately, and hours do not carry over.

How we work

1. Request from the panel

   Request the assessment in DevOps Services: tell us which model you use, what data enters the context and who uses the application.

2. Scoping and risk modelling

   We map the flows: indexed sources, tools called by the agent, permissions and logs. We agree the test scenarios and the date you propose.

3. Testing and hardening

   We run automated and manual red teaming, then implement guardrails, PII redaction, access filters on RAG and rate limits.

4. Handover with repeatable tests

   You receive the report, the test suite that runs on every release and the documentation for transparency and for your GDPR file.

Use cases

Technology stack

• promptfoo
• garak
• Microsoft Presidio
• Qdrant
• LiteLLM
• OpenTelemetry
• HashiCorp Vault
• ModSecurity

Frequently asked questions

• What is prompt injection?

  Instructions slipped into data the model reads: a web page, a PDF, an e-mail, a fragment of the index. A model does not separate your instruction from text asking for something else, so it may follow both. Defence combines role separation, filters and minimal permissions for every tool.

• What are the risks of AI chatbots for businesses?

  Three main ones: leakage of data placed in the context, false answers presented with confidence, and actions taken by an agent that holds too many rights. Add uncontrolled cost when there are no per-user limits, and transparency duties towards the people who use the chatbot.

• What is the OWASP Top 10 for LLM applications?

  The public list of risks specific to language-model applications: prompt injection, unsafe output handling, sensitive data leakage, excessive agency, data poisoning and others. We use it as the checklist for the assessment and as the structure of the report.

• How do I stop a RAG chatbot from leaking customer data?

  Filter access at query time, not at indexing: each fragment carries the rights of its source, and the answer is built only from what the current user may see. Add PII redaction, prompt logging with a short retention period and keep the index on servers in the EU.

• What obligations does my company have under the AI Act?

  Since 2 August 2026 new obligations apply to organisations that use AI systems, the most visible being transparency: people must be told when they interact with an automated system. Document the model, the data in context and prompt retention. We deliver the technical part of that file.

• What should never be given to a chatbot?

  Data you are not allowed to send outside the EU, access secrets, and full personnel or patient files. If the information is needed, redact it first, replace it with identifiers and keep the model on your own infrastructure rather than on a public service.