Two Intel Arc Pro B70 graphics cards will serve a 30-billion-parameter coding model at 880 tokens a second to a packed endpoint, or 111 tokens a second to one person waiting on it. The cards were never the bottleneck. Most of the work went into learning which of the “this doesn’t support Battlemage” errors were real, and which came down to a stale runtime version, a kernel bug, or a scheduler flag we hadn’t flipped.

The goal was an open-source, npm-installable inference server that picks the best engine for a one-card or two-card Battlemage rig. Two different workloads were in play. A shared endpoint serves many users at once, and there the useful measure is aggregate throughput across everyone. A single developer inside a coding agent is the opposite case, judged only by how fast that one stream runs. No single stack was best at both, and sorting that out reorganized the project.

The rig

Component	Detail
GPU	2× Intel Arc Pro B70 (Battlemage gen-2, BMG-G31, device ID 0x8086:0xe223)
Memory	32 GB GDDR6 ECC per card
Rated throughput	183.5 TFLOPS FP16 each, 367 TOPS INT8 each
Host	”Horizon,” openSUSE Tumbleweed, kernel ~6.x, KDE Wayland
CPU	Intel Core Ultra 9 285K
Board	MSI PRO Z890-A WIFI (BIOS 2.AC0, March 2026, the post-launch BIOS that fixed two of three Battlemage PCIe slots)
Container access	/dev/dri via card0/1/2 and renderD128/129/130; --device /dev/dri with the video and render group IDs (482 and 485 here)

When two B70s sit in a machine, what the software sees depends on the runtime you point at them. Intel’s NEO compute-runtime v2 UR adapter hits a Battlemage regression that collapses both cards into a single 59 GB composite Level-Zero device. The NEO release we eventually side-loaded enumerates them correctly as GPU.1 and GPU.2. One runtime seeing a single device while another sees two was the source of the first wave of “this doesn’t work” results, sitting on top of a hardware problem of its own.

After we flashed the board BIOS, one card negotiated PCIe Gen4 ×4 cleanly and the other sat at Gen1 ×8 with link equalization failing. The BIOS was asking for Gen1 on that slot. That is a known read-side performance ceiling on the second card, and there is no software fix for it short of a board or slot change. Everything below assumes that crooked PCIe topology and treats the Gen1 card as the slow one.

Stacks that didn’t work

Six obvious stacks failed before one worked, each for its own reason:

Stack	Model	What happened
llm-scaler-vllm 0.14, vLLM-XPU TP=2	gpt-oss-20b	torch.xpu saw one device, not two. The composite-device regression.
llm-scaler-vllm, MXFP4 on the composite	gpt-oss-20b	IGC mistagged BMG2 as XeLPG and rejected the dpas2 Xe2 intrinsics.
llm-scaler-vllm 1.4, BF16	Codestral-22B	Host RAM OOM. The 44 GB safetensors mmap blew past 62 GB of host memory.
llm-scaler-vllm 1.4, FP8	Qwen3-14B	vLLM-XPU FP8 weight-loader bug (QKVParallelLinear.weight_scale missing).
stock llama.cpp SYCL server	Qwen3-Coder-30B-A3B Q5_K_M	--parallel greater than 1 segfaulted in the MoE expert-routing kernel. Single-stream baseline: 6.1 tokens/s.
llama.cpp + Hal9000 cherry-picks	same	UR_RESULT_ERROR_DEVICE_LOST on the first concurrent request. The patches cover token-gen, not the routing kernel.

What finally worked was clean upstream llama.cpp-SYCL on Intel’s own intel/llm-scaler-vllm:1.4 runtime, the stack Intel ships for dual-Battlemage under the name Project Battlematrix. This is the v8 stack referenced throughout, and it landed 880.7 tokens/s.

With v8 fixed, we swept the per-server knobs. The peak config, 880.7 tokens/s aggregate at concurrency 2048, was Qwen3-Coder-30B-A3B at Q4_K_M, dual-card pipeline-split (-sm layer -ts 1,1), flash attention on, KV cache at q8_0 for both keys and values, an 8192 context, batch 2048, ubatch 512, and 256 parallel slots.

Change from the peak config	Result
Concurrency 4096 (2× batch)	453.7 tokens/s, down 48 percent. Queue saturates past 2× batch.
Batch 4096	Regressed. 2048 is optimal.
Ubatch 256	Down 22 percent. 512 is optimal.
Parallel 320	OOM. 256 slots is the ceiling at any context.
Parallel 512, context 4096	OOM. Slot overhead is non-linear; you can’t trade context for slots.
--threads-http sweep (8 / 16 / 32)	651 / 888 / 879. Sixteen HTTP threads is enough; below sixteen hurts.

Underneath all of this is a structural ceiling: v8 SYCL tops out at 256 parallel slots, regardless of context. It is per-slot scheduler overhead that runs out first, well before raw KV memory. Because parallel × context is not conserved, you cannot fork your way out of it by shrinking the context. Pushing past 256 would mean forking llama-server itself, weeks of work for an estimated 1.4 to 1.8× aggregate gain.

docker run --rm -d --name v8 \
  --device /dev/dri --group-add 482 --group-add 485 \
  -v /path/to/Qwen3-Coder-30B-A3B-Instruct-Q4_K_M.gguf:/models/m.gguf:ro \
  -p 18000:18000 \
  -e GGML_SYCL_DISABLE_OPT=1 \
  -e UR_L0_ENABLE_RELAXED_ALLOCATION_LIMITS=1 \
  -e GGML_SYCL_HOST_MEM_FALLBACK=ON \
  b70-llamacpp-sycl:v8 \
  -m /models/m.gguf --host 0.0.0.0 --port 18000 \
  --device SYCL0,SYCL1 -sm layer -ts 1,1 -ngl 999 \
  -fa 1 --cache-type-k q8_0 --cache-type-v q8_0 \
  --ctx-size 8192 --batch-size 2048 --ubatch-size 512 \
  --parallel 256 --no-warmup

OpenVINO actually supports the B70

The claim you read everywhere is that OpenVINO does not support the B70, and the 2026.2.0 release notes seem to confirm it by scoping the card to single-GPU inference on 20-to-30-billion-parameter LLMs. That phrasing sounds like a hard limit on the hardware.

Stock OpenVINO 2026.2.0 does fail to see the B70, but only one thing is wrong with it. The stock image bundles compute-runtime 24.48 from December 2024, and the BMG-G31 device ID was not added until compute-runtime 25.48 in December 2025. OpenVINO’s GPU plugin has no device allowlist at all; it filters only on Intel vendor, OpenCL 1.2 or better, and GPU type. It sees whatever NEO sees. The moment we side-loaded NEO 26.18, both B70s enumerated as GPU.1 and GPU.2. The “OpenVINO doesn’t support B70” story was always a bundled-runtime-version problem.

The side-load is a five-line Dockerfile on top of the stock dev image:

FROM openvino/ubuntu24_dev:2026.2.0
USER root
COPY debs/*.deb /tmp/neo/
RUN set -ex && \
    dpkg --remove --force-depends intel-level-zero-gpu && \
    rm -f /tmp/neo/intel-ocloc_*.deb && \
    dpkg -i /tmp/neo/*.deb && \
    rm -rf /tmp/neo

The .deb set to COPY in: intel-igc-core and intel-igc-opencl at 2.34.4, intel-opencl-icd and libze-intel-gpu1 at 26.18.38308.1, and libigdgmm12 at 22.10.0. Skip intel-ocloc; it conflicts with the OpenVINO base and is not needed for inference. Download them to the host first and COPY them in, because buildkit DNS fails fetching them on openSUSE.

With both cards visible, single-stream inference worked immediately. Qwen2.5-7B-Instruct at int4 ran at 147 tokens/s on GPU.1 and 144 tokens/s on GPU.2, symmetric across three runs each, within plus or minus 2.4 percent. That is roughly 78 percent faster than llama.cpp-SYCL single-stream on a similar-class model, on the same silicon.

Two stacks, two workloads

One test settled the architecture: the 30B model on a single B70 through OpenVINO’s ContinuousBatchingPipeline, set against v8 llama.cpp-SYCL on both B70s. The result flipped which stack we ship.

Concurrency	Aggregate (tokens/s)	Per-user (tokens/s)	Wall (256-token reply)
1	111.5	111.5	2.30 s
4	261.1	65.3	3.92 s
8	449.1	56.1	4.56 s
16	512.9	32.1	7.99 s
32	733.6	22.9	11.17 s

Set that against v8’s per-user numbers and the crossover becomes visible:

Concurrency	v8 SYCL (two cards) per-user	OpenVINO CB (one card) per-user
1	10 to 20 tokens/s	111.5 tokens/s
8	72.9 tokens/s	56.1 tokens/s
32	not tested	22.9 tokens/s
2048	0.43 tokens/s	VRAM-infeasible

OpenVINO CB on one B70 reaches 83 percent of v8 SYCL’s two-card aggregate ceiling, at far better per-user latency across the concurrency band an interactive agent actually lives in. A single-user coding agent runs at concurrency 1 to 32, where per-user throughput is all that matters. At the concurrency each stack was built for, the per-user gap is roughly 53×: 22.9 tokens/s on OpenVINO CB at concurrency 32 against 0.43 tokens/s on v8 at concurrency 2048. That is the cost of aggregate throughput. v8 buys its headline number by spreading one decode across thousands of streams.

Each stack owns a different workload. OpenVINO CB on a single B70 is the primary path for the npm CLI use case: one to thirty-two users, latency-sensitive, 734 tokens/s aggregate, 22.9 tokens/s per user at concurrency 32, and 111 tokens/s at concurrency 1. That leaves the second B70 free for spare capacity, a second model, or future tensor-parallel work. v8 SYCL across both cards earns its place only in heavy multi-tenant serving past 100 concurrent users, where it delivers 880 tokens/s aggregate at concurrency 2048 and 0.43 tokens/s per user. For the workload in front of us, OpenVINO CB is the one to run.

The dual-card wall, and why we stopped caring

If you wanted one big model spread across both cards through OpenVINO, here is why that does not work today. Four issues are open simultaneously against dual-card stateful LLM serving on the B70 in stock OpenVINO 2026.2.

• query_model_ratio (PR #34702). With two identical 32 GB GPUs, the ratio calculator hands GPU.1 the entire model and GPU.2 nothing. Validated by the submitter on two Arc Pro B50s with Qwen2.5-7B-Instruct-INT8 through OVMS.
• HETERO broken for large models (issue #33012). CL_INVALID_KERNEL_ARGS on the second GPU, from a NEO USM-with-discrete-GPU regression in the GPU plugin’s engine. Three Intel developers are assigned. Reproduces on every driver from 32.0.101.7028 through 32.0.101.8737.
• Singleton OpenCL context per device (ambeckley gists one and two). NEO crashes if you create more than one cl_context per device. Ships as source patches against oneDNN and OpenCL-CLHPP.
• Multi-GPU KV cache validation (genai PR #3496). Closed for inactivity, still needed. Without it, the cache validator counts only GPU.1’s memory and rejects the config.

The canary for all four is genai issue #3162, which tracks the same failure on two A770s, the B70’s sibling card. The most recent reproduction, on an OpenVINO nightly from May 8, fails identically to stock 2026.2. The one community result that works is SearchSavior/OpenArc, an OpenVINO-based OpenAI-compatible server that ships a Battlemage.Dockerfile. Its maintainer gets two B70s running Qwen3-8B in INT8 only, with the full four-patch bundle applied, at about 30 percent slower than single-card. INT4 plus dual-card plus HETERO is not known to work today.

OpenVINO’s MULTI plugin is a dead end of its own. It hardcodes a throw on stateful models, and any model with a KV cache is stateful, so it never even loads.

We stopped caring about the wall because of the two-stacks finding. Once OpenVINO CB on a single card delivered the per-user latency the CLI actually needs, the dual-card HETERO path stopped being load-bearing. Two independent single-card instances behind a load balancer give you both cards’ worth of throughput with the latency profile of single-card CB, and they sidestep the entire wall. That is the architecture we shipped.

The 100-second time-to-first-token

The latency numbers above are all decode. Prefill is a separate problem, and for a coding agent it dominates. Every turn, the agent sends the system prompt, the tool definitions, and a large slice of the repository, easily 60,000 tokens. Our first time-to-first-token measurement on a 60K prompt was 100 seconds, which sent us straight back to the scheduler.

The fix was a single flag. OpenVINO’s ContinuousBatchingPipeline SchedulerConfig has an enable_prefix_caching knob, block-level, persisting across requests, added back in OpenVINO 2024.4. Interactive agents are turn-2-dominant: the system prompt, tools, and repo context are identical across turns, and only the user’s new message is new prefill work. Claude Code, for reference, hits a 92 percent prefix-reuse rate. No production local-LLM agent runs 60K context without prefix caching. We simply had not turned the flag on.

There was a second, smaller fix sitting next to it. The default max_num_batched_tokens in the scheduler is 256, which is far too small for long-prompt prefill. Raising it to 2048 chunked the cold prefill better. Together the two fixes produced these numbers:

Test	Turn	TTFT	Wall (256-token reply)
Same 60K prompt, three runs	1, cold	81.3 s	84.3 s
Same 60K prompt, three runs	2, warm	1.5 s	4.4 s
Same 60K prompt, three runs	3, warm	1.5 s	4.4 s
Realistic agent (58K shared prefix + varying delta)	1	1.5 s	4.5 s
Realistic agent	2	1.5 s	4.4 s
Realistic agent	3	1.5 s	4.5 s

Prefix caching took turn-2 TTFT from 100 seconds to 1.5 seconds, a 54× speedup. The higher batched-tokens ceiling shaved the cold turn from 101 to 81 seconds, about 20 percent. The realistic-agent rows are the honest test, a shared 58K prefix with a varying delta, the way an agent actually behaves. Once that stable prefix is warm, every later turn on a full 60K context lands in about 5 seconds total: 1.5 seconds to first token, then roughly 3.5 seconds of decode. On dual B70 that is a usable coding-agent experience.

import openvino_genai as ov_genai

sch = ov_genai.SchedulerConfig()
sch.cache_size = 12                  # GB
sch.max_num_seqs = 32                # max concurrent streams
sch.dynamic_split_fuse = True
sch.enable_prefix_caching = True     # turn-2 TTFT drops to ~1.5s
sch.max_num_batched_tokens = 2048    # default is 256, far too small

pipe = ov_genai.ContinuousBatchingPipeline(
    "/path/to/qwen3-coder-30b-a3b-int4-ov",
    scheduler_config=sch, device="GPU.1",
)

# Warm the stable system+tools+repo prefix once at startup (~80s, one-time tax).
STABLE_PREFIX = build_stable_prefix()  # 50 to 58K tokens, aligned to block_size
warm = ov_genai.GenerationConfig(max_new_tokens=1)
pipe.generate(STABLE_PREFIX, warm)

# Every coding-agent turn now lands in ~5s; only the user delta is new prefill.
def chat(user_msg: str) -> str:
    cfg = ov_genai.GenerationConfig(max_new_tokens=2048)
    return pipe.generate(STABLE_PREFIX + user_msg, cfg)

User-visible action	Wall time
First session opens, server warms the prefix	~80 s, one-time
Every later turn, full 60K context	~5 s (1.5 s TTFT + 3.5 s decode for 256 tokens)
Same turn, 2K-token response	~6.5 s (1.5 s + 5 s decode)
Multi-turn conversation, prefix stays warm	5 to 7 s per turn

What we shipped

For the server itself, OpenVINO Model Server (OVMS) is the right layer. It exposes the same SchedulerConfig knobs as the Python pipeline, adds about 3 to 8 percent overhead over raw ContinuousBatchingPipeline, and ships native SSE streaming and bearer-token auth. The production shape is two single-card OVMS containers behind NGINX, one bound to GPU.1 and one to GPU.2, deliberately avoiding the MULTI:GPU.1,GPU.2 path where the open multi-Arc bugs live.

docker run -d --rm --name ovms-gpu1 \
  --device /dev/dri --group-add $(getent group render | cut -d: -f3) \
  -p 8001:8001 \
  -v /path/to/ov-ir:/models:ro \
  openvino-b70:2026.2-neo26.18 \
  ovms --rest_port 8001 --config_path /models/config.json

The tuned knobs ride in the graph config, which is where enable_prefix_caching, the cache size, and the batched-tokens ceiling all land:

node_options: {
  [type.googleapis.com/mediapipe.LLMCalculatorOptions]: {
    models_path: "./"
    plugin_config: '{ "KV_CACHE_PRECISION": "u8" }'
    enable_prefix_caching: true
    cache_size: 12
    max_num_batched_tokens: 2048
    max_num_seqs: 16
    device: "GPU.1"
    pipeline_type: LM_CB
  }
}

The endpoint is OpenAI-compatible at POST /v3/chat/completions. The one NGINX detail that will cost you an afternoon is streaming. A default reverse proxy buffers SSE, which destroys the token-by-token experience and quietly inflates latency. The upstream has to be least_conn rather than round-robin, because streamed requests imbalance under round-robin, and buffering has to be off:

upstream ovms_pool {
    least_conn;          # not round_robin: streamed requests imbalance under rr
    server 127.0.0.1:8001;
    server 127.0.0.1:8002;
    keepalive 32;
}
server {
    listen 8000;
    location /v1/ { rewrite ^/v1/(.*)$ /v3/$1 break; proxy_pass http://ovms_pool; include /etc/nginx/sse-proxy.conf; }
    location /v3/ { proxy_pass http://ovms_pool; include /etc/nginx/sse-proxy.conf; }
}

sse-proxy.conf is the part that actually fixes streaming:

proxy_http_version 1.1;
proxy_set_header Connection "";
proxy_buffering off;
proxy_cache off;
chunked_transfer_encoding on;
proxy_read_timeout 1h;
proxy_send_timeout 1h;

Note: Prefix cache is per-instance. With two OVMS containers, a user whose requests land on a different card each turn will never hit a warm prefix. Pin sessions to a card with ip_hash or hash $arg_session consistent instead of least_conn when cache hit-rate matters more than even load.

We skipped vLLM-XPU TP=2 entirely. Its prefix-cache support is “TBD” on the XPU roadmap, it has a documented crash history with --enable-prefix-caching, and even working it was predicted at 600 to 1100 tokens/s, under the OpenVINO CB throughput we already had. It shares the same oneCCL family as our v8 base, so there was no new information to gain by trying it.

Quants: the operational floor

Quantization choice on Battlemage SYCL has a measured floor. The B70-specific tg128 numbers in llama.cpp issue #21517 set it:

Quant	tg128 (tokens/s)	Memory-bandwidth utilization	Use
Q4_0	23.67	57%	yes
Q4_K_S	23.05	56%	yes
Q4_K_M	20.56	53%	default
IQ4_XS	17.52	40%	15% slower
Q5_K_M	13.78	41%	33% slower, marginal quality gain
Q6_K	13.83	48%	skip
Q8_0 (pre-fix)	4.88	21%	broken
Q8_0 (post PRs #21527 + #21638)	~15.24	n/a	still slow vs Q4_K_M
IQ4_NL	5.85	14%	broken

The defaults that fall out of this are narrow. Weight quant at Q4_K_M or UD-Q4_K_XL, never standalone Q5_K_M, IQ4_NL, or pre-fix Q8_0. KV cache at q8_0 for keys and values, which uses different kernels than the weight matmul. And GGML_SYCL_DISABLE_OPT=1 set until the Xe2 kernel corruption bug (#21893) closes.

What does not work

Three classes of thing are closed doors on this hardware today.

• The OpenVINO MULTI plugin, hardcoded to throw on stateful models.
• Anything built on Gated DeltaNet, linear attention, or Mamba: Qwen3-Coder-Next 80B-A3B, the Qwen3.6 family, Granite-4.0-H. The SYCL GATED_DELTA_NET kernel is broken, and we measured the slowdown directly: 153 tokens/s on Qwen3.6-35B-A3B against 880.7 on Qwen3-Coder-30B-A3B at the same config, a 5.8× regression.
• Anything that does not fit 64 GB at the quants that work: GLM-4.5-Air (73 GB at Q4_K_M), Qwen3-Coder-Plus 480B, Mixtral 8x22B, DeepSeek-V3. These wait on a third card or a pruned variant.

Two tooling paths are also closed. IPEX-LLM went read-only in January 2026. The Hal9000 SYCL patches that advertise a 47 percent MoE uplift measured at negative 14 percent here, because the work they carry is already merged upstream.

What is still open

Paths we have not closed, in rough priority for anyone picking this up:

• A custom OpenVINO build with the four patches applied (PR #34702, the engine.cpp one-liner from #33012, the two ambeckley gists, genai #3496). Reproduce the SearchSavior INT8 result, then test whether INT4 also works. Still the only path to a 60 GB model across two 32 GB cards.
• SGLang on dual B70. RadixAttention prefix-cache reuse could compound on the agentic workloads where OpenVINO’s linear prefix caching already helps. Battlemage support status is unknown and worth checking.
• The OpenVINO ggml backend now previewed in llama.cpp. Validated only on Core Ultra AI PCs so far, with “GPU unsupported” rows for Phi-3 and Qwen3-8B. A smoke test on the B70 would show whether it inherits OpenVINO’s single-card win with llama.cpp’s slot scheduler.
• ZLUDA, the CUDA-on-Intel translation layer, if it gets revived for Battlemage. A large ecosystem unlock if it lands.
• Forking llama-server to shrink per-slot state, the path past the 256-slot ceiling to an estimated 1.4 to 1.8× aggregate gain. Weeks of work.