A Practical Guide to Building a Self-Hosted RAG with FOSS Solutions
Artificial intelligence is rapidly becoming a practical tool for small and medium-sized enterprises (SMEs). Yet for many companies, the main challenge is not access to AI models. It is how to use AI with the company’s own knowledge while retaining control over its data.
For an SME, internal knowledge is often one of its most valuable assets: customer information, technical documentation, procedures, contracts, quotations, product specifications, commercial history and operational know-how. Sending all of this information to an external AI service may be convenient, but it also raises questions about privacy, ownership, compliance, vendor dependency and long-term control.
A Retrieval-Augmented Generation (RAG) architecture offers an alternative.
What is RAG and why does it matter?
A RAG system combines two capabilities: information retrieval and generative AI.
Instead of asking an AI model to answer a question solely from what it learned during training, a RAG retrieves relevant information from the company’s own data sources and provides that information to the model as context. The model then generates an answer based on the retrieved material.
In practical terms, an employee could ask:
“What are the warranty conditions for this product?”
The system can search the company’s technical documentation, commercial policies and previous records, retrieve the relevant information and generate an answer based on those sources.
This is particularly valuable for SMEs because it turns scattered organizational knowledge into a conversational knowledge layer.
The objective is not to replace the company’s existing ERP, CRM or databases. It is to provide an intelligent interface through which employees can interact with information already distributed across those systems.
Data sovereignty changes the architecture
A conventional approach is to send company information to a cloud AI provider. A sovereign approach asks a different question:
Where does the company’s data go, who controls it, and where does the AI computation take place?
A self-hosted RAG can keep documents, databases, embeddings and retrieval infrastructure inside the company’s own environment. The language model can also run locally using open-weight models.
For organizations with greater computational requirements, a hybrid approach is possible: sensitive information can remain inside the company’s infrastructure while selected workloads use open-weight models hosted by providers operating data centers under jurisdictions and contractual conditions acceptable to the company.
This creates a spectrum rather than a binary decision:
Fully local → Hybrid → Sovereign cloud/open-weight infrastructure
The appropriate choice depends on the sensitivity of the information, required performance, cost and available computing resources.
A practical FOSS architecture
A modern self-hosted RAG does not require a proprietary AI platform. A complete ecosystem can be assembled from mature Free and Open Source Software (FOSS) components.
One practical architecture is:
Business systems and data sources → n8n → Langflow → Retrieval layer → Local/Open-weight LLM → Answer
At the integration layer, n8n can connect the RAG to the existing business environment. It can retrieve information from APIs, databases, files, CRM and ERP systems, trigger workflows and automate ingestion pipelines.
At the center of the RAG, Langflow provides a visual environment for designing and experimenting with AI workflows. It can orchestrate document processing, embeddings, retrieval, reranking, prompts and model interactions without requiring every component to be developed from scratch.
For the language model itself, Ollama provides a straightforward way to run models locally. This makes it possible to experiment with different open-weight models without sending company information outside the infrastructure.
The architecture does not need to be tied permanently to one model. As models evolve, the SME can replace or complement its local model with another open-weight model, including models hosted in European data centers when additional computing capacity is required.
Choosing the right knowledge database
There is no single database that is best for every RAG.
OpenSearch is an attractive option when sophisticated search capabilities are important. Its combination of traditional search, vector search and reranking capabilities can be particularly useful when retrieving the most relevant documents from large and heterogeneous collections.
For companies already using PostgreSQL, PostgreSQL + pgvector can provide a simpler architecture. Vector embeddings can be stored alongside conventional relational data, reducing the number of systems that need to be maintained.
The choice becomes even more interesting when the information is highly interconnected.
Consider a CRM containing customers, subsidiaries, contacts, products, sales opportunities and relationships between companies. In this scenario, the problem may not simply be “find the most similar document.” The company may want to understand relationships and networks.
Graph capabilities can therefore complement vector retrieval when relationships between entities are important—for example, identifying connections between customers, suppliers, products, projects or organizations.
The important point is architectural flexibility: the RAG should adapt to the company’s information model, rather than forcing every type of information into the same database.
Connecting AI to the existing company
A self-hosted RAG becomes truly useful when it connects to the systems the company already uses.
Typical channels include:
- REST and GraphQL APIs
- SQL databases
- ERP and CRM connectors
- File systems and document repositories
- Email and collaboration platforms
- Webhooks and event-driven workflows
- CSV, PDF and office documents
- Custom internal applications
This is where n8n can play an important role. It acts as the integration and automation layer between the company’s operational systems and the AI infrastructure.
For example, when a new technical manual is uploaded, an n8n workflow can detect the document, process it, generate embeddings and update the RAG knowledge base automatically.
The same architecture can connect a CRM to the AI layer, allowing employees to ask questions about customers or commercial activity without manually searching multiple systems.
From infrastructure to business capability
The underlying infrastructure can also remain entirely open.
A Linux server can host the different services as Docker containers. Portainer or Komodo can simplify container management and deployment. If the company requires stronger infrastructure isolation or wants to consolidate several workloads, Proxmox can provide the virtualization layer for managing virtual machines and containers.
This creates a stack in which every layer can be replaced independently:
Proxmox → Linux → Docker → n8n / Langflow / databases / Ollama → AI models
There is no requirement to purchase a single proprietary platform that controls the entire architecture.
What can an SME actually do with it?
The applications are diverse.
Internal knowledge assistant: employees can ask questions about procedures, policies, manuals and historical documentation.
Technical support: service teams can retrieve relevant product specifications, troubleshooting procedures and previous cases.
Sales intelligence: a RAG connected to the CRM can help identify customer history, products, opportunities and related accounts.
Operations: employees can query logistics, purchasing or production documentation using natural language.
Management: executives can interact with selected business information through a controlled AI interface rather than manually navigating multiple systems.
Compliance and quality: internal policies, standards and procedures can become searchable and auditable knowledge sources.
These examples share a common principle: the value does not come from the language model alone. It comes from connecting AI to the organization’s proprietary knowledge.
Start small, retain control, evolve progressively
For an SME, building a sovereign AI environment does not mean creating a massive infrastructure project from day one.
A sensible strategy is to begin with a limited knowledge domain—for example, technical documentation or internal procedures. A small local model, PostgreSQL with pgvector and a basic Langflow workflow may be sufficient.
As the business case proves its value, the architecture can evolve toward OpenSearch, more powerful models, graph capabilities, additional data sources or hybrid infrastructure.
This incremental approach is one of the greatest advantages of an open architecture.
AI sovereignty is not about avoiding the cloud or rejecting commercial AI services. It is about maintaining the ability to decide where data lives, which models process it, how information flows through the organization and which providers the company depends upon.
For SMEs, a self-hosted RAG built from FOSS components provides a practical path toward that objective: an AI layer connected to the company’s existing systems, capable of growing with the business, while keeping its most valuable asset—its knowledge—under its own control.