Local AI with Jan, Organization AI on Microsoft Foundry, and a Data Leakage Filter Support

Why a hybrid AI architecture

The safest way to bring generative AI into an organization is not to pick one place for it to run, but to give every request the right place to land: on the laptop, in the organization’s cloud, or nowhere at all.

Most teams already use AI. The problem is where. Staff paste contracts, source code, student records, and financial drafts into whatever chatbot is open in the browser. That is shadow AI, and banning it rarely works because people simply move to their phones.

The pattern in this post answers with three layers that work together:

1. Local AI with Jan for private, offline, everyday work on the user’s own device, with an optional shared Qwen server in the building.
2. Organization AI on Microsoft Foundry for governed, high capability models, agents, and grounding on company data.
3. An AI security filter that sits between people and any model to stop sensitive data from leaving where it should stay.

The design goal is simple: make the safe path the easy path. If the approved tools are fast and good, people stop reaching for the unapproved ones.

The architecture at a glance

Three layers share one rule: sensitive data stays as close to the user as possible, and anything that leaves goes through a single filtered door.

[embed: node/33801a4a-303c]

Jan serves private work on the device; the gateway screens every outbound call and routes it to the on premises Qwen server or to Foundry, or blocks it; Purview labels and audits all three layers.

Layer 1: Local AI with Jan

Local AI handles the work that should never leave the device: drafting with confidential notes, summarizing internal files, and coding against private repositories.

Jan is an open source (Apache 2.0) desktop app for Windows, macOS, and Linux that downloads and runs models such as Qwen, Llama, Gemma, and gpt-oss fully offline. Three features make it a good enterprise building block:

• OpenAI compatible local API at http://127.0.0.1:1337/v1, powered by llama.cpp, so tools like VS Code extensions can call local models with no code change.
• MCP host support, so the local model can read approved folders and tools as an agent.
• Optional cloud providers, which is exactly the switch we must govern (see Layer 3).

Hardening checklist for Jan on managed devices

• Keep the API server bound to 127.0.0.1. Binding to 0.0.0.0 exposes the model to the whole network.
• Disable or remove cloud provider keys; route any cloud use through the organization gateway instead.
• Turn off CORS if no browser app needs the local API.
• Allow only a curated model list from an internal mirror, not arbitrary Hugging Face downloads.
• Deploy and update Jan through Intune so versions stay consistent.

Option: a shared local Qwen server

Laptops run small models well, but bigger Qwen models need a GPU. A shared on premises server gives a team stronger local reasoning while data still stays inside the building. Run it with vLLM or the llama.cpp server, expose an OpenAI compatible endpoint on the internal network, and add it in Jan as a custom provider.

Tier	Hardware	Qwen model that fits	Approx. memory (4-bit unless noted)	Good for
Laptop	16 GB RAM laptop	Qwen3 8B	~5 GB	Personal drafting, summaries
Laptop, high end	Apple Silicon, 32 GB unified memory	Qwen3 30B-A3B (MoE)	~17 to 19 GB	Coding, reasoning on the go
Team workstation	1x RTX 5090 (32 GB VRAM)	Qwen3 32B or 30B-A3B	~20 GB	A small team sharing one endpoint
Department server	1x RTX PRO 6000 Blackwell (96 GB VRAM)	Qwen3 32B at FP8	~35 GB, rest for KV cache	Many concurrent users, long context
Lab server	2x RTX PRO 6000 Blackwell (192 GB VRAM)	Qwen3 235B-A22B (MoE)	~120 GB	Near frontier quality, fully on premises

Two sizing notes. First, a MoE model needs memory for all its parameters, not just the active ones, so 30B-A3B still needs room for 30B weights. Second, context length eats VRAM through the KV cache, so leave headroom beyond the weights. Newer Qwen3.5 releases follow the same sizing logic. Figures are approximate; sources: Spheron Qwen3 sizing, VRLA Tech VRAM table, Qwen3 RAM guide.

Layer 2: Organization AI on Microsoft Foundry

Microsoft Foundry is where the organization runs work that needs frontier models, shared agents, or grounding on company data, all inside an Azure resource you already govern.

Quick naming note for readers: Azure AI Foundry is now Microsoft Foundry, and Azure AI Services became Foundry Tools. Older portal screens and docs may still show the previous names (Foundry Tools rebrand, Foundry guardrails docs).

What this layer contributes:

• Foundry Models: one catalog with models from OpenAI, Microsoft, Anthropic, Mistral, Meta, DeepSeek, and others, so you can swap models without rewriting apps (Azure AI solutions).
• Foundry Agent Service: the managed runtime for organization agents, with tool calling, remote MCP servers, memory, and publishing to Teams and Microsoft 365 Copilot (Ignite 2025 architect notes).
• Grounding: Azure AI Search over SharePoint, Fabric, or SQL, so answers cite approved sources instead of pasted text.
• Guardrails: configurable content filters and controls per model and per agent, managed in the Foundry Control Plane.

Baseline controls for this layer

• Private endpoints for Foundry, Search, and Storage; no public network access.
• Entra ID only, with managed identities for agents and Conditional Access for people. No shared API keys.
• One Foundry project per business domain, so data, agents, and costs stay separated.
• Diagnostic logs to Log Analytics and Defender for Cloud AI threat protection turned on.

For a Microsoft shop, this is also the natural place to connect Copilot Studio agents and line of business systems through the same governed endpoint.

Layer 3: The AI security filter

The filter is not one product but four checkpoints, so that a sensitive prompt is caught on the device, at the gateway, on the platform, or in the audit trail, whichever comes first.

Checkpoint 1: the endpoint. Microsoft Purview Endpoint DLP and the Purview browser extension (Edge, Chrome, Firefox) can warn or block users who paste or upload sensitive information into consumer AI sites such as public ChatGPT or Gemini. DSPM for AI shows which AI apps people actually use and which interactions touched sensitive data (Apps4.Pro DSPM guide). This is the shadow AI net.

Checkpoint 2: the AI gateway. Put Azure API Management in front of every model endpoint, both Foundry and the shared Qwen server. Every request passes the same pipeline:

1. Authenticate with Entra ID and tag the request with user, app, and data classification.
2. Detect and redact PII and secrets (Azure AI Language PII detection, or Microsoft Presidio for the on premises path).
3. Run Prompt Shields from Azure AI Content Safety to catch direct and indirect prompt injection.
4. Apply the routing policy (next section) and token quotas.
5. Log the request and the decision.

Checkpoint 3: the platform. With Purview enabled in the Foundry Control Plane, admins can apply inline DLP policies that evaluate prompts sent to Foundry apps and agents and block sensitive information before the model processes it. It runs on a pay as you go Purview subscription; the roadmap listed preview for May 2026 and GA for June 2026, so check the current status in your tenant (MC1304291, RM558565). Sensitivity labels, Insider Risk Management, and eDiscovery also cover Foundry interactions (Purview for Azure AI services).

Checkpoint 4: the record. Prompts and responses land in the Purview unified audit log and the AI activities view of Activity Explorer, and the Risky AI usage template flags prompt injection and protected material access into Defender XDR (Purview for generative AI).

One principle ties the four together: no device talks to a public AI API directly. Block direct egress to public model endpoints at the firewall or with Defender for Endpoint network protection, and let the gateway be the only door out.

Routing policy: where each request goes

The sensitivity label decides the destination, not the user’s preference: the more sensitive the data, the closer to the device the model runs.

Purview label	Example content	Allowed destination	Gateway action
Public	Published papers, marketing copy	Jan local, Qwen server, or Foundry	Log only
General	Internal how to guides, meeting notes	Jan local, Qwen server, or Foundry	Redact PII, log
Confidential	Contracts, source code, financial drafts	Jan local or Qwen server; Foundry only via approved, grounded agents	Redact PII and secrets, Prompt Shields, inline DLP
Highly Confidential	Personal data, health or student records, credentials	Jan local only, offline	Block any outbound call

A request flows like this: the user works in Jan or a Foundry backed app; anything leaving the device hits the gateway; the gateway reads the label, redacts, screens for injection, and then either sends it to the Qwen server, sends it to Foundry, or blocks it with a clear message that says why and where the user can run it instead. That last part matters. A block with a helpful alternative teaches people the policy; a silent block teaches them to find a workaround.

Threat model and controls

Each leakage path has a named control in a named layer, so no single control has to be perfect.

Threat	Where it happens	Primary control	Backup control
Staff paste secrets into a consumer chatbot	Browser	Purview Endpoint DLP and browser extension	DSPM for AI reporting, user coaching
Jan quietly sends data to a cloud provider	Device	Remove cloud keys, block direct egress	Gateway is the only allowed route
Local API exposed to the network	Device	Bind Jan to 127.0.0.1, host firewall	Intune compliance check
Prompt injection in a document or web page	Gateway and agents	Prompt Shields	Least privilege tools for agents
PII reaches a cloud model	Gateway	PII redaction	Purview inline DLP in Foundry
Agent overshares grounded data	Foundry	Sensitivity labels and scoped search indexes	Insider Risk Management
Unvetted model weights	Device and Qwen server	Curated internal model mirror	Hash checks before deployment
No evidence after an incident	All layers	Unified audit log	Gateway logs in Log Analytics

Rolling it out

Start with visibility, then give people a good safe option, and only then tighten the blocks.

1. See: turn on DSPM for AI and Endpoint DLP in audit mode to learn which AI apps people use and what data flows into them.
2. Offer: deploy hardened Jan with a small Qwen model to every managed laptop, so there is a private option on day one.
3. Centralize: stand up the Foundry project, private networking, and the API Management gateway with PII redaction and Prompt Shields.
4. Scale locally: add the shared Qwen server when local demand outgrows laptops, starting with the team workstation tier.
5. Enforce: switch DLP from audit to block, enable Purview inline DLP in Foundry, and close direct egress to public AI endpoints.

Closing thought

Good AI governance feels like good UX. When the local model is one click away, the organization model is better than the free chatbot, and the filter explains itself, people choose the safe path on their own. Jan gives privacy, Foundry gives power, and the filter makes sure neither becomes a leak.

I would love to hear how your team handles the local versus cloud decision. Share your patterns with the community. Off course you can change the foundry with your own AI API Key