In my previous post, I built a proof-of-concept using Microsoft Copilot Studio as a conversational AI interface for VMware infrastructure. It worked — you could ask questions in Teams and get real answers from live vCenter data. But it came with a significant caveat: everything flowed through Microsoft’s cloud.

This time, I wanted to see if I could achieve the same thing — or better — running entirely on-premises. No cloud APIs, no Microsoft licensing, no data leaving the network. Just a local LLM talking to my VMware APIs over localhost.

Spoiler: it works, and it’s surprisingly good.

Why go fully on-prem?

The Copilot Studio PoC proved the concept was sound, but a few things kept nagging at me:

• Data sovereignty — Every API response (VM names, host details, network topology) was flowing through Microsoft’s cloud. In a production environment with sensitive infrastructure, that’s a conversation stopper for many security teams.
• Licensing cost — Copilot Studio isn’t free. The per-user or per-message pricing adds up, especially for something that might become a daily operations tool.
• Network dependency — The setup required an on-premises data gateway (MAP gateway) to bridge my private network to Azure. If that gateway went down, or my internet connection dropped, the AI assistant was dead.
• Latency — Every question took a round-trip to the cloud and back. Not terrible, but noticeable.

What if I could run the entire stack locally?

Architecture comparison

Copilot Studio (cloud-dependent)

1

User → Teams → Copilot Studio (Azure) → MAP Gateway → MCP Server → vCenter/VCF APIs

• ✅ Polished UI (Teams integration)
• ✅ Microsoft’s GPT-4 model (excellent reasoning)
• ❌ Data flows through Azure
• ❌ Requires internet + VPN + gateway
• ❌ Per-user/per-message licensing
• ❌ 77 actions must be enabled one-by-one (click, click, click…)

On-Prem solution (zero cloud)

1

User → Web UI → Orchestrator → Ollama (local LLM) → MCP Server → vCenter/VCF APIs

• ✅ All data stays on-premises
• ✅ No internet required after initial setup
• ✅ No recurring licensing cost
• ✅ Full control over model, prompts, and behavior
• ✅ Sub-30-second responses with 8B model
• ❌ Requires dedicated VM resources (CPU + RAM)
• ❌ Smaller models can occasionally hallucinate
• ❌ No pre-built Teams integration (web UI only)

The setup

Hardware

I deployed a Photon OS 5.0 VM on my VCF cluster:

• 32 vCPUs — more cores = faster inference on CPU-only
• 128 GB RAM — enough to run the 70B model comfortably
• 300 GB disk — models are large (40GB for 70B)
• IP: 10.0.0.11 — same subnet as the MCP server

The MCP server (Windows Server 2022, 10.0.0.10) already runs the three FastAPI connectors from the Copilot Studio project:

• vCenter API on port 8080
• VCF Operations API on port 8081
• VCF Networks API on port 8082

Software stack

1. Ollama — Local LLM runtime, serves models via REST API
2. Llama 3.1 8B — Fast general-purpose model (~20-30s responses)
3. Hermes 3 — NousResearch model fine-tuned for tool calling
4. Nemotron 3 Nano 4B — NVIDIA’s agent-optimized model (structured output)
5. Llama 3.1 70B — Available for complex queries (~3-5 min on CPU)
6. Python orchestrator — Routes questions to the right APIs using tool-calling
7. Web UI — Chat interface with model selector and response timer

Available tools

The orchestrator exposes 24 tools to the LLM — a curated subset of the most useful operations from all three APIs:

vCenter (11 tools):

• List/search VMs, VM details, power state
• Host listing and resource usage
• Cluster summary, datastore capacity
• Active alarms, recent tasks
• Snapshot inventory and old snapshot detection

VCF Operations (7 tools):

• Environment summary and health
• Alerts (all, critical, top by severity)
• Resource search and recommendations
• Active symptoms

VCF Networks (6 tools):

• Entity search (VMs, switches, routers by name/IP)
• Network alerts
• NSX segment listing
• Host and cluster network view

How it works

The orchestrator is the brain. When you ask a question:

1. Your question goes to the orchestrator (FastAPI, port 8090)
2. The orchestrator sends it to Ollama with 24 tool definitions (API endpoints)
3. The LLM decides which tool(s) to call — it might pick vcenter_list_vms, ops_alerts, or multiple tools in parallel
4. The orchestrator executes those API calls against the MCP server
5. Results are fed back to the LLM
6. The LLM synthesizes a human-readable answer
7. The answer is returned to the web UI

The key innovation is Ollama’s tool-calling support in Llama 3.1+. The model doesn’t just generate text — it can decide to call specific functions with parameters, just like GPT-4 does in Copilot Studio.

Here’s the first successful response — the LLM queried vCenter and returned real snapshot data from my environment, all processed locally:

Installation on Photon OS

Setting up the LLM VM took about 30 minutes:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43

# Network configuration
cat > /etc/systemd/network/10-eth0.network << EOF
[Match]
Name=eth0

[Network]
Address=10.0.0.11/24
Gateway=10.0.0.1
DNS=10.0.0.10
DNS=8.8.8.8
EOF
systemctl restart systemd-networkd

# Install prerequisites
tdnf install -y tar gzip zstd git python3 python3-pip

# Install Ollama
curl -fsSL https://ollama.com/install.sh | sh

# Configure Ollama to listen on all interfaces
mkdir -p /etc/systemd/system/ollama.service.d
cat > /etc/systemd/system/ollama.service.d/override.conf << EOF
[Service]
Environment="OLLAMA_HOST=0.0.0.0"
EOF
systemctl daemon-reload
systemctl enable ollama
systemctl start ollama

# Pull models
ollama pull llama3.1:8b    # Fast daily driver (~5GB)
ollama pull llama3.1:70b   # Heavy hitter (~40GB)

# Deploy the orchestrator
cd /opt
git clone https://github.com/vidingeb/operational-intelligence.git
cd operational-intelligence/orchestrator
pip3 install -r requirements.txt

# Set up as services
# (orchestrator.service and orchestrator-ui.service)
systemctl enable orchestrator orchestrator-ui
systemctl start orchestrator orchestrator-ui

Scaling: from 20GB to 128GB

I started with a modest VM — 4 vCPUs and 20 GB RAM. That was enough to run the 3B model and prove the concept. But when I switched to the 70B model for better accuracy, response times ballooned to 6+ minutes. Not usable.

The fix was straightforward — hot-add resources in vCenter:

The final configuration — 32 vCPUs and 128 GB RAM — gives comfortable headroom for the 70B model while letting the 8B model respond in ~20 seconds. This is a dedicated “AI worker” VM that does nothing else.

Firewall gotcha on Photon OS

Photon OS ships with a strict iptables policy (default DROP on INPUT). Out of the box, the VM could reach the MCP server, but the MCP server couldn’t reach back. The fix:

1
2
3
4
5
6

# Allow all traffic from the local subnet
iptables -I INPUT 4 -s 10.0.0.0/24 -j ACCEPT

# Persist across reboots
mkdir -p /etc/systemd/scripts
iptables-save > /etc/systemd/scripts/ip4save

The web UI

The chat interface runs on port 8091 and includes:

• Model selector — Switch between ⚡ 8B (Fast), 🧠 70B (Smart), or 🚀 3B (Fastest) per question
• Response timer — Shows exactly how long each query takes
• Model tag — Each answer is tagged with which model generated it

In practice, I use 8B for 90% of queries (running VMs, snapshots, alerts) and switch to 70B only when I need deeper analysis or cross-domain correlation.

Real-world results

Here’s what it actually looks like in daily use:

“List running VMs” → 21 seconds, correct list of 9 powered-on VMs

“List powered-off VMs” → 21 seconds, complete list of 22 powered-off VMs with host distribution

“Do we have any snapshots?” → 31 seconds, correctly identified all snapshots across 3 VMs with dates and names, plus a recommendation to clean up old ones

The 8B model is surprisingly capable for structured data queries. It correctly picks the right API tool, parses the JSON response, and presents it clearly.

Model comparison

The beauty of running Ollama locally is that you can pull multiple models and switch between them per question. I tested five models for this project:

Hermes 3 vs Nemotron Nano — same question, different styles

Asking both models “list running VMs” reveals very different personalities:

Hermes 3 gives a verbose, detailed per-VM breakdown — every field the API returns (CPU, memory, guest OS, IP, VMware Tools status, host placement):

Nemotron 3 Nano formats the response as a clean table, intelligently filters out infrastructure nodes (ESXi hosts, management VMs), and adds context notes explaining its reasoning:

Both took ~60 seconds and picked the correct tool. The difference is in presentation intelligence — Nemotron was trained specifically for agent workflows and it shows. It understands that an operator asking “list running VMs” probably wants application VMs, not infrastructure components.

Qwen 2.5 7B — the surprise contender

Alibaba’s Qwen 2.5 7B is an interesting addition. At 58 seconds per response, it’s comparable to Hermes and Nemotron in speed, but it brings a unique perspective — it includes VMware Tools status in its output and proactively suggests next steps (“You might want to check why these VMs have outdated Tools…”).

It’s the most “operationally aware” of the smaller models, likely because Qwen’s training data includes a lot of infrastructure documentation. Worth pulling if you want an alternative to Hermes 3.

The model selector in the UI makes A/B testing trivial — same question, different model, instant comparison.

The sweet spot for interactive use is the 8B model for speed, or Nemotron Nano when you want smarter formatting. Switch to 70B only for complex multi-domain questions.

Copilot Studio vs On-Prem: the honest comparison

My take: If you’re in an environment where data sovereignty matters (government, healthcare, finance, defense), the on-prem solution is the clear winner. If you want polish and don’t mind the cloud dependency, Copilot Studio is easier to get started with.

For my home lab, I’m now using the on-prem solution daily. The 8B model with 20-30 second responses feels natural enough for operational queries, and knowing that my infrastructure data never leaves the network is a nice bonus.

Going beyond: Hermes Agent framework

The custom orchestrator works, but it’s essentially a “two-call loop” — the LLM picks a tool, the orchestrator calls the API, then the LLM synthesizes the answer. That’s fine for single questions, but real operations often need multi-step reasoning: “Find all VMs with snapshots older than 7 days, check if they’re powered off, and list the ones that are safe to clean up.”

Enter Hermes Agent by NousResearch — an open-source agent framework designed specifically for local LLMs. It goes several levels beyond a simple orchestrator:

• Persistent memory — Remembers context across sessions (“last time you asked about snapshots, there were 3 — now there are 5”)
• Task planning — Breaks complex questions into sub-tasks automatically
• Cron scheduling — “Check for critical alerts every 30 minutes and notify me”
• Multi-platform — Same agent accessible via CLI, Slack, Teams, Telegram, or web
• Skills system — Register custom tools (like our VMware API connectors) as reusable plugins
• Sub-agents — Delegate specialized tasks to child agents

Installation alongside the custom orchestrator

Hermes Agent installs cleanly on the same Photon OS VM and uses the same Ollama instance — no additional resources needed. Both can run simultaneously on different ports:

The key difference: the custom orchestrator is a stateless tool-caller (question → tool → answer), while Hermes Agent is a stateful agent (maintains context, plans multi-step tasks, learns from history).

For now, both coexist — the web UI is great for quick queries, and Hermes Agent handles complex operational workflows. The next step is registering the VMware API connectors as Hermes skills, giving it the same 24-tool inventory as the custom orchestrator but with agent-level intelligence on top.

What’s next

• GPU acceleration — Even a modest NVIDIA T4 or A2 would cut response times to 5-10 seconds. This is the single biggest improvement available.
• Hermes skills registration — Wire the 24 VMware API tools into Hermes Agent’s skill system for multi-step reasoning
• Streaming responses — Show tokens as they’re generated instead of waiting for the full answer
• Scheduled health checks — Use Hermes cron to run “daily infrastructure report” automatically
• Action execution — Allow the LLM to perform actions (power on/off VMs, create snapshots) with confirmation
• More data sources — External storage arrays, network switches, perimeter firewalls — everything in the data room should be queryable
• Teams/Slack integration — Hermes Agent supports multi-platform out of the box; configure when ready for production

The bigger picture: why isn’t this built into VMware already?

My original goal was simple: bring operational intelligence into the data room. Not just vCenter — but storage, networking, firewalls, the whole stack. Ask one question, get a unified answer across all systems.

After building this, I understand why VMware (and Broadcom) haven’t shipped this as a native feature in vCenter or VCF. The resource requirements are significant:

The resource problem:

• A useful LLM needs 8-128 GB of RAM dedicated to inference
• It needs substantial CPU (or GPU) that can’t be shared with production workloads
• On a typical 4-node VCF cluster, that’s resources you’d rather give to customer VMs

Embedding AI directly into the vCenter appliance would mean either degrading management performance or requiring customers to over-provision their management cluster. Neither is a great option.

The cloud approach (and its limitations):

The obvious solution — which is what VMware/Broadcom would likely pursue — is a cloud-hosted AI service. Ship telemetry and metadata to a VMware cloud endpoint, process it with a large model, return insights. This would work well for internet-connected clusters and avoids the local resource problem entirely.

But this completely fails for air-gapped environments — military, government, classified networks, critical infrastructure, or any “dark site” deployment where data cannot leave the premises. These are exactly the environments that would benefit most from AI-assisted operations, because they often have the most complex, multi-vendor stacks and the least staff to manage them.

The on-prem middle ground:

What I’ve built here is that middle ground. A dedicated “AI worker” VM that:

• Lives on the management network alongside vCenter and VCF components
• Has no internet access requirement (models are downloaded once)
• Can be extended to query any REST API in the data room
• Keeps all infrastructure data within the physical boundary

The trade-off is dedicating 32 CPUs and 128 GB RAM to AI inference — which on a modern 4-node cluster with 256+ cores and 2+ TB RAM total, is reasonable for the value it provides.

For organizations running air-gapped VCF deployments, this pattern — local LLM + API connectors + orchestrator — might be the only viable path to AI-assisted operations until hardware becomes cheap enough to embed directly into appliances.

Source code

The entire stack — orchestrator, web UI, API connectors, and Swagger specs — is available on GitHub:

🔗 github.com/vidingeb/operational-intelligence

The orchestrator/ folder contains everything needed for the on-prem AI assistant.