Comparative Analysis: Architectural Foundations of Tencent Hy4, DeepSeek-V4, and Qwen 3.8 MoE Models

1. Tencent Hy4-Preview Architecture: Sparse Activation and 1M Context Engineering

In the intensifying three-way open-source MoE landscape between Tencent Hy4, Qwen 3.8, and DeepSeek V4, architectural efficiency at massive scale defines real-world serving viability.
Tencent Hy4-preview establishes its position in this competitive ecosystem through an aggressive sparse activation framework, high-capacity long-context handling, and tailored residual pathways.

770B Parameter Routing and Gated Sparse Attention with IndexCache

The foundational backbone of Tencent Hy4-preview comprises a massive 770B total parameter configuration, engineered to activate only 49B parameters per token.
This design achieves an ultra-sparse activation rate of approximately 6.3%, significantly reducing per-token computational overhead without compromising representational breadth.
The model structure spans 78 backbone layers, starting with 1 dense FFN layer followed by 77 MoE layers.
Routing across these MoE layers is governed by a fine-grained 256 routed expert architecture supplemented by 1 shared expert, dynamically routing tokens through a Top-8 routed expert activation scheme.
Long-sequence processing is supported by an expansive context window capacity of 1,048,576 tokens (1M), complemented by a generation capability of up to 64,000 completion tokens.
To sustainably process million-token contexts, Tencent Hy4-preview implements Gated DeepSeek Sparse Attention (Gated DSA).
This attention mechanism integrates IndexCache, which enables cross-layer sparse index reuse during 1M context serving to prevent redundant key-value memory lookups and accelerate decoding.

Identity Hyper-Connections and OpenRouter Performance Metrics

To preserve signal integrity and gradient propagation across its deep 78-layer stack, Tencent Hy4-preview incorporates Identity Hyper-Connections (iHC) directly onto the residual pathway, substantially expanding inter-layer information flow.
For inference acceleration, the model embeds a Native Multi-Token Prediction (MTP) layer scaling at 10B total parameters with 0.7B activated parameters dedicated to speculative decoding.
Officially released on August 28, 2026, under the permissive Apache License 2.0, the model weights are distributed across Hugging Face, ModelScope, GitCode, and CNB, including native FP8 quantization formats.
Post-training compression and quantization workflows are managed via the dedicated AngelSlim toolkit.
According to deployment data on OpenRouter, Tencent Hy4-preview operates at an inference throughput of 40 tokens/second with an average latency of 3.49s.
The platform's API pricing structure is set at $0.834 per 1M input tokens, $2.501 per 1M output tokens, and $0.042 per 1M cache read tokens.

Architectural / Operational Metric	Tencent Hy4-Preview Specification
Total / Active Parameters	770B Total / 49B Active (~6.3% activation rate)
Layer Backbone	78 Layers (1 Dense FFN Layer + 77 MoE Layers)
Expert Routing Configuration	256 Routed Experts + 1 Shared Expert (Top-8 Activation)
Context / Completion Capacity	1,048,576 Tokens (1M) Context / Up to 64,000 Completion Tokens
Attention & Residual Architecture	Gated DSA with IndexCache & Identity Hyper-Connections (iHC)
Speculative Decoding Module	Native MTP Layer (10B Total / 0.7B Active Parameters)
OpenRouter Throughput & Latency	40 tokens/sec Throughput; 3.49s Latency
OpenRouter Pricing (Input / Output / Cache)	$0.834 / $2.501 / $0.042 per 1M Tokens
License & Release Date	Apache License 2.0; Released August 28, 2026

Despite strong throughput metrics, official documentation and release notes from the Tencent Hy Team highlight a notable behavioral limitation: the model exhibits excessive self-verification (over-verification).
This over-verification tendency causes unnecessarily extended reasoning times during complex generation cycles.
Mitigating this behavior for latency-sensitive production environments requires targeted adjustments to system prompts and careful tuning of reasoning effort configurations.

2. DeepSeek-V4 Multimodal MoE Design and Latent Attention Mechanics

As the open-source MoE three-way battle intensifies alongside Tencent Hy4 and Qwen 3.8, DeepSeek's architectural paradigm establishes a distinct vector of efficiency and scale.
The foundation of this ecosystem rests on the flagship DeepSeek-V3 baseline, which established a 671B total parameter architecture activating only 37B parameters per token, achieving a lean ~5.5% activation rate.
The full DeepSeek-V3 Hugging Face checkpoint scales to 685B parameters, incorporating the 671B main model alongside a 14B Multi-Token Prediction (MTP) module.
Pre-trained across 14.8 trillion tokens, this foundational architecture required 2.664M H800 GPU hours out of 2.788M total H800 GPU hours allocated to the training run.

Multi-Head Latent Attention and Dynamic Load Balancing

DeepSeek's structural throughput relies centrally on Multi-head Latent Attention (MLA), an attention formulation engineered for low-overhead key-value (KV) cache compression.
By projecting key-value states into low-dimensional latent spaces, MLA drastically curbs memory footprint bottlenecks during auto-regressive generation without sacrificing context retrieval fidelity.
Complementing MLA, the MoE routing mechanism implements an auxiliary-loss-free dynamic load balancing strategy across its vast expert pool.
Traditional MoE routing penalizes load imbalance via auxiliary loss terms, which often degrades primary task performance; DeepSeek eliminates this penalty by dynamically adjusting expert routing thresholds.
In DeepSeek-V4, this framework is further augmented with Compressed Sparse Attention and manifold-constrained hyper-connections (mHC), maintaining representation stability across high-dimensional residual streams.

DeepSeek-V4 Multimodal Backbone and FP4/FP8 Precision Scheme

The multimodal evolution is embodied in the DeepSeek-V4-Flash-Vision-Exp configuration, comprising 285B total parameters with only 13B activated per token.
This parameter budget divides into a 284B parameter language backbone and a ~0.5B parameter vision aligner.
The underlying backbone spans 43 layers, orchestrating 256 routed experts where 6 experts are activated per token.
DeepSeek-V4 incorporates a knowledge distillation pipeline that directly transfers reflection and reasoning patterns from the DeepSeek-R1 series into the multimodal MoE routing graph.
The model supports an extensive 1,048,576-token context window.

Architectural Parameter	DeepSeek-V3 MoE Baseline	DeepSeek-V4-Flash-Vision-Exp
Total Parameters	671B (685B checkpoint with MTP)	285B (284B language + ~0.5B vision aligner)
Active Parameters per Token	37B (~5.5% activation rate)	13B
Layer Count	Noted in baseline specs	43 layers
Routed / Active Experts	MoE routing infrastructure	256 routed experts (6 active per token)
Context Window	Standard baseline context	1,048,576 tokens (1M context)
Precision Architecture	Standard mixed-precision baseline	FP4 experts, FP8 attention/norm/router, BF16 vision

Serving the 285B parameter multimodal model leverages a specialized FP4+FP8 mixed-precision scheme.
Under this layout, the MoE feed-forward experts operate in FP4 precision, attention projections, layer normalization, and routing gating mechanisms execute in FP8, and the vision tower processes spatial tokens in native BF16.
The complete DeepSeek-V4-Flash-Vision-Exp checkpoint is distributed across 48 shards, consuming ~168 GB of storage on disk and demanding ~202 GB of VRAM prior to KV cache allocation.
While the architecture supports an advertised 1,048,576-token context window, sustaining extreme context lengths introduces substantial memory overhead well beyond the baseline checkpoint allocation.
Deployment verification under vLLM PR #54566 has confirmed vision execution exclusively on NVIDIA hardware platforms, as ROCm and XPU backends currently lack the vision implementation runtime.

3. Qwen 3.8 Flagship Scale: 2.4T Fine-Grained MoE and Dual-Mode Reasoning

Within the three-way open-source MoE competition alongside Tencent Hy4 and DeepSeek V4, the Qwen 3.8 ecosystem anchors its performance on massive parameter scale, granular routing, and flexible execution architectures.
Documented by the Alibaba Cloud Qwen Team, this lineup demonstrates how massive parameter capacity can be balanced with operational efficiency through advanced sparse activation strategies.

2.4T Parameter Scale and Fine-Grained Expert Segmentation

At the pinnacle of the architectural family stands the Qwen3.8-2.4T-A95B configuration, which commands an aggregate of 2.4 trillion total parameters while routing exactly 95 billion activated parameters per token.
The broader ecosystem scales across multiple deployment tiers, encompassing the high-throughput Qwen3.8-Flash-Next at 180B parameters, the compact Qwen3.8-27B tier, alongside foundational MoE structural configurations such as the 235B-A22B and 30B-A3B variants.

A defining architectural attribute of the Qwen MoE design is its fine-grained expert segmentation mechanism.
Rather than relying on monolithic expert blocks, standard Feed-Forward Networks (FFNs) are partitioned into numerous smaller individual experts, such as an arrangement of 64 total experts utilizing 4 shared experts alongside 4 routed experts per forward pass.
This fine-grained division enables specialized parameter activation while preserving shared cross-domain representations across all tokens.
To accelerate convergence during training, the architecture leverages model upcycling initialization, transforming pre-trained dense checkpoints into sparse structures.
The efficiency trajectory of this sparse design builds on precedents established by earlier milestones like Qwen1.5-MoE-A2.7B, which demonstrated a 75% reduction in training costs and a 1.74x inference speedup over standard dense 7B baselines.

Model Tier / Configuration	Total Parameters	Activated Parameters	Architectural / Scaling Focus
Qwen3.8-2.4T-A95B	2.4 Trillion (2.4T)	95 Billion (95B)	Flagship scale with fine-grained routing and dense upcycling
Qwen3.8-Flash-Next	180 Billion (180B)	Dynamic MoE	High-speed, balanced inference tier
Qwen3.8-27B	27 Billion (27B)	Dynamic MoE	Efficient edge and local workstation deployment
Qwen3 MoE (235B-A22B)	235 Billion (235B)	22 Billion (22B)	Enterprise-scale sparse routing
Qwen3 MoE (30B-A3B)	30 Billion (30B)	3 Billion (3B)	High-efficiency sparse baseline
Qwen1.5-MoE-A2.7B Baseline	Sparse MoE Baseline	2.7 Billion (2.7B)	75% training cost reduction and 1.74x speedup vs. dense 7B

Dual-Mode Reasoning and Long-Context Stability

Qwen 3.8 integrates a dual-mode reasoning framework designed to handle diverse operational demands within a unified checkpoint.
The system facilitates seamless switching between an extended Thinking mode for complex step-by-step logic and a direct Non-Thinking Instruct mode for low-latency operational responses.

To maintain structural coherence across extended generation, the context window spans from a baseline of 131,072 tokens (131K) up to 1,048,576 tokens (1M tokens).
Long-context stability over these massive sequence lengths is sustained via Dual-chunk Attention combined with Grouped Query Attention (GQA), reducing memory overhead while preventing attention degradation.

Deploying this architecture in production pipelines requires attention to runtime-specific quirks.
In high-performance inference frameworks like SGLang and vLLM, default request preprocessing pipelines may drop the reasoning_content fields, necessitating verbatim message passing configurations to maintain multi-step tool execution fidelity.
Furthermore, local execution environments utilizing rotated context management—such as default runtime setups in Ollama—require explicit manual parameter overrides to ensure correct long-context window scaling.

4. Cross-Architectural Structural Matrix and Open-Source Licensing Paradigms

The competitive landscape of open-source Mixture-of-Experts (MoE) models highlights distinct approaches to parameter activation density, routing topologies, attention mechanisms, and software licensing.
Evaluating Tencent Hy4-preview alongside DeepSeek MoE and Qwen 3.8 reveals critical trade-offs between computational overhead, memory management, and downstream deployment rights.

Parameter Density and Attention Architecture Matrix

Structural divergence begins with total parameter scale and dynamic activation efficiency.
Tencent Hy4-preview implements a 770B total parameter architecture that activates 49B active parameters per forward pass, maintaining an activation ratio of approximately ~6.3%.
In comparison, the DeepSeek MoE series operates with 671B total parameters and activates 37B active parameters, yielding an activation profile of roughly ~5.5%.
Qwen 3.8 MoE expands the computational ceiling with its largest flagship configuration reaching 2.4T total parameters with 95B active parameters.
Expert routing topologies illustrate differing methodologies for token distribution.
Both Hy4-preview and DeepSeek MoE implement a structured topology comprising 256 routed experts plus 1 shared expert, utilizing Top-8 routing to dispatch tokens across specialized pathways while maintaining common baseline representations.
Qwen 3.8 adopts a fine-grained routing plus shared expert mechanism to distribute computational load across its massive parameter base.
Attention design and context capacity further differentiate these architectures.
Hy4-preview utilizes a Gated DSA + IndexCache mechanism, supporting a long-context capacity of up to 1,048,576 tokens (1M).
DeepSeek MoE incorporates Multi-head Latent Attention (MLA) to optimize key-value memory requirements, managing context spans from 128,000 to 1,048,576 tokens.
Qwen 3.8 leverages GQA / Dual-chunk Attention, supporting context windows ranging from 131,072 to 1,048,576 tokens.

Model	Total / Active Parameters	Expert Topology	Attention Architecture	Context Window	License Model
Tencent Hy4-preview	770B total / 49B active (~6.3% activation)	256 routed + 1 shared (Top-8 routing)	Gated DSA + IndexCache	1,048,576 tokens (1M)	Apache License 2.0 (fully open)
DeepSeek MoE	671B total / 37B active (~5.5% activation)	256 routed + 1 shared (Top-8 routing)	Multi-head Latent Attention (MLA)	128,000 to 1,048,576 tokens	DeepSeek custom open license with commercial permissions
Qwen 3.8 MoE	2.4T total / 95B active (largest flagship configuration)	Fine-grained routing + shared expert mechanism	GQA / Dual-chunk Attention	131,072 to 1,048,576 tokens	Qwen custom commercial license

Licensing Strategy: Permissive Apache 2.0 vs Custom Commercial Terms

Beyond physical and computational topologies, governance and intellectual property terms govern how enterprise engineering teams integrate these foundational models.
Tencent Hy4-preview is distributed under the fully open Apache License 2.0, providing an unencumbered framework for broad commercial utilization, private modification, and secondary redistribution.
In contrast, DeepSeek MoE and Qwen 3.8 depart from standard permissive open-source definitions by employing proprietary governance structures.
DeepSeek MoE operates under a DeepSeek custom open license with commercial permissions, establishing specific operational parameters for commercial deployment.
Qwen 3.8 is released under a Qwen custom commercial license, defining custom compliance boundaries and usage criteria for enterprise adopters.
These varying license restrictions create distinct operational paths between universally permissive terms and vendor-specific commercial licensing frameworks.

5. Empirical Benchmark Performance and Blind Engineering Evaluations

The competitive landscape among open-source Mixture-of-Experts (MoE) models has shifted dramatically with the arrival of Tencent Hy4, Qwen 3.8, and DeepSeek V4.
Evaluating these architectures across rigorous standardized coding, agentic reasoning, and blind engineering tasks demonstrates how open-weight systems have reached empirical parity with leading commercial APIs.

Coding and Agentic Benchmarks: SWE-bench, Terminalbench, and GPQA

Across standardized software engineering evaluations, frontier open-source MoE architectures have demonstrated rapid capability gains, with SWE-bench Verified resolution rates across the three architectures reaching the mid-70% to 80% range.
Tencent Hy4-preview achieved an 82.9 resolution score on SWE-bench Multilingual and 65.7 on SWE-bench Pro.
On the Deep SWE benchmark, Hy4-preview posted a score of 64.3, outperforming DeepSeek-V4-Flash text parity at 59.3.
Terminal execution and environment interaction show a similar trend: Hy4-preview scored 85.4 on Terminalbench 2.1 compared to 83.9 for DeepSeek-V4-Flash parity.
In high-level academic and scientific reasoning, Hy4-preview registered a score of 92.3 on the GPQA Diamond benchmark.
Agentic execution evaluations highlight competitive standing in tool use and multi-step problem solving.
Hy4-preview recorded a Toolathlon Verified score of 74.1 and reached 37.1 on Apex Agents, compared to DeepSeek-V4-Flash-Vision-Exp at 36.5 on ApexBench.
On the rigorous Agents' Last Exam benchmark, DeepSeek-V4-Flash-Vision-Exp recorded a score of 27.3.
These empirical results confirm that coding and agentic performance parity across frontier open-source MoE models directly rivals closed commercial systems such as the Claude Sonnet tier.

Benchmark Metric	Tencent Hy4-preview	DeepSeek-V4-Flash / Vision-Exp
SWE-bench Multilingual (Resolved)	82.9	Mid-70% to 80% category range
SWE-bench Pro	65.7	—
Deep SWE	64.3	59.3 (Text Parity)
Terminalbench 2.1	85.4	83.9 (Parity Score)
GPQA Diamond	92.3	—
Toolathlon Verified	74.1	—
ApexBench / Apex Agents	37.1	36.5
Agents' Last Exam	—	27.3

Blind Expert Trials and Multimodal Vision OCR Performance

Real-world enterprise viability extends beyond synthetic benchmarks into human engineering evaluations and multimodal ingestion tasks.
Hy4 was co-designed directly with production platforms including Tencent CodeBuddy and WorkBuddy to handle complex document analysis and autonomous code generation workflows.
To assess real-world programming execution, a Tencent internal blind side-by-side evaluation deployed 163 domain experts across 203 engineering tasks.
In direct blind testing against GLM 5.3, Hy4-preview secured an average score of 2.99 versus 2.92 for GLM 5.3, recording 46.8% wins, 12.8% ties, and 40.4% losses.
When evaluated against Kimi K3, Hy4-preview maintained an average score of 2.99 versus 2.94 for Kimi K3, achieving 51.2% wins, 7.9% ties, and 40.9% losses according to internal Tencent expert ratings.
In multimodal vision capabilities, DeepSeek-V4-Flash-Vision-Exp achieved an overall OCRBench score of 83.5% (835/1000) evaluated on a single GB200 NVL4 tray.
Its performance breakdown across OCR tasks showed high fidelity in Key Information Extraction at 92.5%, Document VQA at 90.0%, and Scene Text recognition at 89.5%.
However, specific structural limitations remain in multimodal edge cases.
Hugging Face model card evaluations indicate that DeepSeek-V4-Flash-Vision-Exp exhibits reduced accuracy on handwriting recognition at 58.0% and handwritten math at 49.0%.

6. Speculative Decoding Systems and High-Throughput Multi-Framework Deployment

Native Multi-Token Prediction and Fused Speculative Drafting

Inference efficiency in frontier open-source Mixture-of-Experts architectures increasingly relies on eliminating standalone draft model overhead.
The integration of the Tencent Hy4 native MTP layer and DeepSeek MTP modules delivers over 1.5x throughput (TPS) improvement across high-concurrency environments without requiring external speculative drafters.
By utilizing fused draft modules, the target model repository acts as its own speculative drafter directly inside vLLM runtimes, minimizing memory bandwidth bottlenecks and pipeline latency.

On NVIDIA GB200 infrastructure, DeepSeek DSpark speculative decoding demonstrates exceptional verification efficiency.
Benchmark data reveals a mean generation length of 2.99 tokens per forward pass and an overall acceptance rate of 66.3%.
Verification fidelity across draft token sequences maintains strong acceptance dynamics, registering 83.9% at the first draft position, 66.5% at the second position, and 50.7% at the third position.

Multimodal inference imposes distinct tokenization constraints during speculative execution.
For DeepSeek-V4-Flash-Vision-Exp, image tokens preprocess at a maximum cap of 387 prompt tokens per image, bound by a minimum pixel threshold of 147,456.
Operational constraints outlined in official vLLM speculative decoding recipes dictate that recommended speculative draft depth is capped at 3 tokens, despite the underlying architecture possessing a 5-token trained width.

Model Family	Speculative / MTP Mechanism	Supported Runtimes & Frameworks	Key Metrics & Deployment Constraints
Tencent Hy4	Native MTP layer (fused draft module)	vLLM, SGLang	>1.5x TPS uplift without standalone draft models; prebuilt container support
DeepSeek (V3 / V4-Flash-Vision-Exp)	DeepSeek MTP / DSpark speculative decoding	vLLM, SGLang, Huawei Ascend MindIE	66.3% acceptance rate on GB200 (2.99 tokens/forward); depth capped at 3 tokens (5-token trained width)
Alibaba Qwen (Qwen3 / Qwen 3.8)	Multi-engine execution support	vLLM, SGLang, TensorRT-LLM, Ollama, llama.cpp	Heterogeneous runtime flexibility across datacenter and localized engines

Enterprise Deployment Across vLLM, SGLang, and Heterogeneous Runtimes

Production serving recipes for Tencent Hy4 provide turnkey acceleration through official containerized environments.
Hy4 production deployment is supported via prebuilt Docker images in vLLM (vllm/vllm-openai:hy4-preview) and SGLang (lmsysorg/sglang:hy4-preview), enabling immediate validation of MoE routing kernels.

DeepSeek-V3 extends enterprise serving flexibility across heterogeneous hardware stacks.
Official runtime guidelines detail full support for multi-node Tensor Parallelism and Pipeline Parallelism across NVIDIA and AMD GPU clusters in SGLang and vLLM, as well as native integration on Huawei Ascend MindIE accelerators.

The Qwen ecosystem maintains the broadest serving versatility.
Both Qwen3 and Qwen 3.8 support multi-engine serving across SGLang, vLLM, TensorRT-LLM, Ollama, and llama.cpp, allowing seamless transitions between high-throughput cloud inference clusters and lightweight local runtimes.

Despite these multi-framework deployment capabilities, advanced memory management features face active constraints.
According to vLLM speculative decoding recipes and SGLang serving guidelines, Prefill-Decode (PD) disaggregation and CPU or filesystem KV-cache offload remain experimental or unsupported on initial multimodal deployment recipes.