Local AI Models for Business: Costs, Security, and Implementation

A local AI model runs its computations on a device or server within the infrastructure selected for the deployment, rather than through an external inference service. This requires deployable model weights, compatible software, and sufficient computing resources. For example, Google provides open-weight Gemma models for self-managed deployment, while Ollama supports running models locally. [1][2]

These terms describe different aspects of an architecture. On-premises refers to deployment on the organization’s own infrastructure. Self-hosted refers to managing the deployment yourself, including on a rented server. Renting a GPU in a third-party data center does not make it company-owned infrastructure simply because the company installs the model itself.

The model is only one component. A working AI system also needs to receive data, enforce access permissions, retrieve relevant documents, validate outputs, and pass results into existing applications.

A practical workflow might look like this:

This is an illustrative design, not a mandatory component list for every solution. Simple classification may not require knowledge-base retrieval. Financial calculations are better handled through verifiable formulas and code, leaving the model to work with text.

Local deployment makes it possible to design a workflow that does not send document contents to an external inference service. But this is a property of the complete architecture, not a guarantee that automatically comes with installing a model.

You need to check where document recognition, search, storage, automation, and logging take place. Even tools designed for local model execution can include cloud features: Ollama’s documentation separately explains how to disable them. [2]

The practical benefit is the ability to define and verify processing boundaries: which data the model can access, who can access the results, and which external connections are permitted.

Extracting document details, classifying support requests, or drafting a report does not always require the largest available model. The right question for a pilot is whether the selected configuration completes the specific task with an acceptable error rate.

We recommend defining the output format and acceptance criteria first. Instead of saying, “The model reads contracts well,” specify that it must “extract an agreed set of fields, identify the supporting source passages, and refer ambiguous cases to an employee.”

A cloud API can also support this type of system. The case for local deployment must therefore be established separately through data requirements, operational considerations, or economics.

With self-managed deployment, a company can retain a selected version of the model weights and configuration. Open-weight models such as Qwen and Ministral make this technically possible. Usage rights depend on the license for the specific release. [3][4]

For production workflows, we recommend versioning not only the model but also the instructions, knowledge base, software environment, and validation rules. Updates should pass testing before entering production. This supports controlled change management, but it does not make generation completely deterministic.

Local processing can continue without a connection to an external model. This can be useful for isolated environments and locations with unreliable connectivity. However, every required component and data source must also be available locally: a cloud CRM or remote knowledge base still creates a network dependency. [1][2]

The project should separately test internet outages, unavailable data sources, and server failures. Independence from an API provider is not a substitute for redundancy.

The following are candidates for a pilot, not a promise that any model will handle them without configuration.

Retrieval-augmented generation, or RAG, can support work with corporate knowledge. The system retrieves relevant material from a separate data store and includes it in the model’s request context. This is different from changing the model’s weights. [5]

Fine-tuning serves a different purpose: it changes model behavior through additional training. It is worth considering when testing identifies a persistent problem that instructions, data, and validation have not resolved. Having internal documents does not, by itself, mean that a model needs fine-tuning. [6]

Learn more about implementation: enterprise knowledge bases and RAG.

For knowledge-base projects, we recommend separately testing document freshness, access permissions, and the system’s ability to acknowledge insufficient information. RAG should not be treated as a guarantee of accurate answers.

A useful starting point is not to “replace the compliance specialist,” but to reduce manual searching and preparation without losing control over decisions.

Consider the following illustrative architecture, not a published Profitec client case study.

A company receives a questionnaire, supporting documents, and additional explanations. The proposed workflow allocates the work as follows:

In this design, the model should not be the sole source of sanctions information or the final authority on the interpretation of requirements. Current reference sources and specialized screening checks should be connected separately, with their source and update time recorded.

A confidently worded answer is not a reliability score. LLMs can produce convincing but incorrect information; OWASP identifies this as a distinct risk. [7]

For the pilot, we recommend measuring missed material discrepancies, false positives, and human review time separately. An overall “95% accuracy” figure can conceal unacceptable errors in rare but important cases.

Do not compare a local system with strong integrations and a knowledge base against a basic cloud chat without equivalent capabilities. That comparison measures differences in system design, not the advantages of where the model runs.

Choosing a model and choosing where to run it are two separate decisions. For example, Ministral 3 14B is available both as an open-weight model and through Mistral’s API. [4]

The following table is an architectural decision guide that should be tested against the customer’s requirements.

Enterprise cloud products should not automatically be equated with a public chat using unsuitable settings. For example, OpenAI states that Business, Enterprise, and API data is not used for training by default, and enterprise applications support connections to data sources. However, “not used for training” does not mean “not processed or stored”: those terms need to be checked separately. [8]

Hybrid is not a universal compromise either. If data cannot be sent outside the permitted environment, switching to a cloud model is not acceptable simply because the local model struggles. In the proposed architecture, the fallback should be another local processing step or referral to an employee.

Start with a shortlist of candidates, not a server purchase built around the largest model name available.

Examples for an evaluation shortlist as of the research reference date:

This is not a quality ranking or a claim that these models have been tested in Profitec projects. The characteristics come from the developers’ documentation. [1][3][4]

A model’s file size is not its total memory requirement. In addition to the weights, resources are needed for context, intermediate computation, and concurrent requests. For example, Ollama notes that parallel processing increases the memory allocated for context. [2]

The technical specification should therefore answer practical questions: how many users work simultaneously, how long the documents are, how long users can wait for a result, and what happens when capacity is exceeded.

For Mixture-of-Experts models, active parameters and total parameters also need to be distinguished. For example, the Gemma 4 26B A4B model card lists 25.2 billion total parameters and 3.8 billion active parameters. Estimating storage for all model weights from the active parameter count alone would be incorrect. [16]

A pilot should measure not just text-generation speed, but the time required to obtain a usable result.

For performance, distinguish time to first token, end-to-end request latency, queueing time, and throughput. For example, vLLM’s benchmarking tools support relevant performance metrics and their percentiles. [9]

We recommend evaluating the workflow across four groups of criteria:

Use the same source documents, output requirements, and evaluation rules. Keep the test set separate from the material used to tune instructions. Explicitly include poor scans, incomplete document sets, conflicting information, and the languages the business needs.

Test the local configuration with the actual quantization, context settings, and hardware intended for production. Test the cloud configuration with comparable tools and data.

A locally hosted model does not necessarily incur a charge to an external provider for each generated token. Computing still has costs, however: hardware or rental, electricity, operations, and support.

Begin by separating model inference costs from the total cost of the workflow.

Assume a monthly workload of 10,000 tasks, each consuming 20,000 input tokens and 2,000 billable output tokens, including reasoning tokens where they are billed as output. This totals 200 million input tokens and 20 million output tokens.

This is a standardized token-budget example, not an experiment using identical documents. Different models may tokenize text differently and consume different amounts of computation.

The figures use the published standard rates recorded for the original edition; the OpenAI examples use the short-context category. Pricing reference date: October 19, 2026. Prices are in US dollars, excluding taxes. The calculation excludes discounts, caching, regional surcharges, tools, OCR, storage, retries, and employee time. This table compares pricing, not model quality. [4][10][11]

For example, the Ministral calculation is:

The conclusion is limited but important: a substantial workload does not, by itself, justify buying a server. First check whether a suitable model offers lower inference costs.

Build a comparable cost model for local and cloud options over 12, 24, and 36 months:

For a local system, include hardware, implementation, energy, redundancy, support, and updates. For an API-based system, include implementation, model and tool calls, the infrastructure supporting your integrations, and maintenance. Include human review on both sides whenever it forms part of the workflow.

Do not double-count the equipment purchase and its full depreciation in the same cash-cost model. For an existing server, separately assess incremental costs, competing uses of capacity, and available headroom.

Then calculate:

Employee time also affects the economics. If a pilot actually demonstrates a 30-second reduction in review time across each of 10,000 tasks, that is approximately 83 hours per month. This is an arithmetic example, not a savings promise. Freed-up time also does not necessarily translate into an equivalent reduction in cash expenditure.

Under a simplified model, the monthly volume threshold can be estimated as:

Additional fixed costs include the difference in initial investment spread over the chosen time horizon, plus the difference in recurring fixed operating costs. Variable costs should include not only generation but also volume-dependent review and correction.

This formula only applies at comparable quality and within the capacity of the selected configuration. If the local option has higher fixed costs and does not save money per accepted task, there is no volume-based break-even point in its favor. If an additional server becomes necessary, the calculation must be revised.

Local deployment does not eliminate application risks. Prompt injection, sensitive information disclosure, and excessive tool permissions are risks for LLM systems, not just cloud products. [12]

We recommend checking access permissions before passing data to the model, limiting executable actions, protecting secrets and logs, and testing outbound connections and recovery procedures. Document contents should be treated as data—not as trusted instructions to change settings or export information. These measures align with the protective practices described by OWASP. [12][13]

Do not assume that regulation universally prohibits cloud processing. For example, HHS permits cloud use for electronic protected health information, or ePHI, with an appropriate business associate agreement, or BAA, and compliance with the other applicable HIPAA requirements. [14]

Likewise, installing a model on a company-owned server does not establish compliance with ISO/IEC 42001. The standard concerns an AI management system at the organizational level. [15]

Learn more about Profitec’s approach to AI governance and compliance.

Each project requires a review of applicable requirements with the people responsible for security, compliance, and legal matters. The right discussion is not, “The model is local, so everything is permitted.” It is about documented data flows, permissions, grounds for processing, and controls.

We recommend starting with an off-the-shelf product or a cloud pilot when data requirements allow it and the need for dedicated infrastructure has not yet been established.

Reasons to defer local deployment include infrequent workloads, no designated owner for ongoing operations, a significant quality shortfall, or an existing tool that already handles the process at an acceptable cost.

Also check whether AI is needed at all. Moving a field from one database to another or applying a known formula does not become more useful simply because an LLM is added.

Profitec AI provides local model implementation services and applies this approach in compliance projects. A practical starting point is one workflow with clearly defined data, an accountable employee, and a measurable outcome.

The project should define three stages.

The customer is paying for more than model installation. The deliverable is a working system that can be evaluated, integrated into operations, and maintained.

Implementation is justified when those requirements are validated in practice. The integrator’s role is to help the company move from a business process and its constraints to a verifiable solution—not simply to provide a server with a model installed.