Mar 14, 2026 · San Francisco, CA
We implemented 4 GPU kernels using Helion DSL targeting NVIDIA B200 (Blackwell) GPUs: 1. causal_conv1d: Depthwise causal 1D convolution (core Mamba/Mamba-2 component). Key optimizations: eliminated padding allocation by using hl.load with extra_mask for inline zero-padding, per-shape configs with large block_sizes (2048-4096), deep pipelining (num_stages=7), and L2 cache grouping. 2. gated_deltanet_chunk_fwd_h: Inter-chunk state recurrence for Gated DeltaNet. Sequential bottleneck kernel with two hl.dot operations per chunk. Key optimizations: dot_precision="tf32" (~1.5x), exp2 instead of exp (hardware fast-path), gating diff instead of k (fewer FLOPs), fused hl.dot(acc=state) accumulation, per-shape configs with higher warps for tiny BH=1 shapes, and range_num_stages=[0,3] for inner loop pipelining (~1.3x). 3. gated_deltanet_recompute_w_u: Recomputes W and U via two matrix multiplications per chunk. Achieved 2.4x speedup using persistent_blocked PID with num_sm_multiplier=16, maxnreg=32, aggressive range_unroll_factors=[4], and l2_groupings=[16]. 4. gated_deltanet_chunk_fwd_o: Intra-chunk forward pass with 3 dot products (local QK attention + global state projection). Optimized with dot_precision="tf32", exp2 fast math, and num_warps=16.
Here's how each kernel works: 1. FP8 Quantization (fp8_quant) Memory-bound, fully parallel. Takes float32 activations, groups them (64-128 elements per group), finds the absolute max per group, computes a scale factor (absmax / 448.0), then divides and clamps to [-448, 448]. Our optimized version removes the baseline's 3x redundant abs/amax/divide operations — single pass per group. 2. Causal Conv1d (causal_conv1d) Memory-bound, fully parallel. Depthwise 1D convolution with causal (left) padding, used in Mamba/SSM architectures. Each channel is convolved independently with a tiny filter (W=3-4 taps): out[b,d,t] = bias[d] + sum_{k} weight[d,k] * x[b,d,t-W+1+k]. The filter is so small it fits in registers — the bottleneck is memory bandwidth. 3. GDN chunk_fwd_h (gated_deltanet_chunk_fwd_h) Compute-bound, sequential across chunks. The state recurrence kernel — maintains a hidden state h [K,V] and updates it chunk by chunk (C=64). Per chunk: (1) store h, (2) compute v_new = u - w @ h, (3) gate with exp(g_last - g_t), (4) decay h *= exp(g_last), (5) update h += k^T @ v_gated. The current VG6 version uses hl.dot with fp32 inputs and acc=state*decay to fuse the decay into the accumulator. Sequential bottleneck: NT chunks must run in order. 4. GDN chunk_fwd_o (gated_deltanet_chunk_fwd_o) Compute-bound, fully parallel. Computes outputs per chunk using the stored hidden states from chunk_fwd_h. Three matmuls per chunk: (1) inter = q @ h * exp(g) (cross-chunk state contribution), (2) intra = causal_mask(q @ k^T * exp(g_diff)) @ v_new (within-chunk attention), (3) output = inter + intra. The causal mask is built via o_t[:, None] >= o_t[None, :]. Embarrassingly parallel across all chunks. 5. GDN recompute_w_u (gated_deltanet_recompute_w_u) Compute-bound, fully parallel. Computes the WY-transformed keys and values needed by chunk_fwd_h. Per chunk: u = A @ (v * beta) and w = A @ (k * beta * exp(g)), where A is a [64,64] WY matrix. Two matmuls per chunk — load A once, loop over V chunks then K chunks. Feeds directly into chunk_fwd_h. Pipeline They chain together: recompute_w_u → chunk_fwd_h → chunk_fwd_o. All use chunk size C=64, K/V typically 64-128. FP8 quant and causal conv1d are independent standalone kernels.
causal_conv1d: Causal depthwise 1D convolution with static per-shape configs and a stable fused convolution path; this is the highest-confidence officially validated variant from my set. fp8_quant: Group-128 FP8 E4M3 quantization with exact-shape dispatch across the benchmark regimes; this was my fastest locally correct quantization variant. gated_deltanet_chunk_fwd_h: Inter-chunk state recurrence with a faster multi-chunk state pass while preserving the official-safe execution path; it improved the larger official benchmark cases substantially. gated_deltanet_chunk_fwd_o: Output kernel with a vectorized fast path that remained locally correct while recovering most of the speed lost in earlier safe rewrites. gated_deltanet_recompute_w_u: WY-transform forward kernel with a merged-matmul hot path for the dominant 64x64, H=3 cases plus shape dispatch for correctness elsewhere.
causal_conv1d - brief summary reading, it's functionality not known and performance not known fp8_quant_py - brief summary reading, it's functionality not known and performance not known gated_deltanet_chunk_fwd_h_py - brief summary reading, it's functionality not known and performance not known gated_deltanet_chunk_fwd_o_py - brief summary reading, it's functionality not known and performance not known gated_deltanet_recompute_w_u_py - brief summary reading, it's functionality not known and performance not known
--- Causal Depthwise Conv1D (Mamba/SSM) Core operator in Mamba state-space models. Each channel is convolved independently with a causal filter: y[b,d,t] = bias[d] + Σ weight[d,k] * x[b,d,t-k], left-zero-padded for causality. Single dynamic Helion kernel specializes only on filter width W (unrolls the inner loop) while B, D, N stay dynamic — cutting JIT compilations from 8 down to 3, solving a 12-min CI timeout. Config autotuned on Nebius B200. ---
I added two PRs for improvements: https://github.com/pytorch/helion/pull/1705 https://github.com/pytorch/helion/pull/1704
gated_deltanet_recompute_w_u/submission/submission.py in the repo computes the Gated DeltaNet WY-transform forward outputs (w, u) from raw inputs (k, v, beta, A, g). Functionally, it applies the per-chunk WY matrix A over sequence tiles of size 64, producing u = A @ (v * beta) and w = A @ (k * beta * exp(g)). The optimized kernel runs directly on the original [B, T, H, D] layout, fuses both output paths in one kernel, and computes the gating terms inside the kernel instead of materializing chunked or pre-scaled temporaries on the host. Performance-wise, the final promoted version is in the low single-digit microsecond range on the tuned benchmark shapes. On the official evaluator it measured about 7.07 us mean runtime with correctness passing. The main gains came from three structural improvements: removing chunk/unchunk materialization, removing pre-scaled 4D temporaries, and moving beta * exp(g) formation into the kernel hot path. The final round also showed that the best large-shape config uses block_sizes=[64, 64], which brought the tuned kernel down from about 7.97 us to about 7.10 us.
We removed all the extra compute in all the kernels. We adjusted tile size and autotuned the kernels and used hl.dot instead of separate outer products for the gated_deltanet_recompute_w_u_py.
Leaderboard name: bloomberg9383 All 5 kernels implemented in Helion DSL for Gated DeltaNet on B200: 1. fp8_quant: FP8 group quantization — computes per-group absmax and scales, quantizes to float8_e4m3fn. Straightforward elementwise kernel. 2. causal_conv1d: Causal depthwise 1D convolution — eliminated input padding by computing directly on raw input with clamped indices and validity masking, halving memory bandwidth. Uses [1,512] block tiling with 1 warp for maximum S-dimension coalescing. ~10.5μs geomean across benchmark shapes. 3. gated_deltanet_chunk_fwd_h: Inter-chunk state recurrence (h = decay * h + key^T * value) — sequential across chunks with hl.dot() matmuls. Autotuned per-shape configs with pointer indexing. 4. gated_deltanet_chunk_fwd_o: Intra/inter-chunk output computation with causal attention masking — uses hl.dot() for all matmuls (QK^T, attention*V, state contribution). Optimized with persistent_blocked scheduling, block_ptr indexing, and range flattening. ~17.7μs geomean. 5. gated_deltanet_recompute_w_u: WY-transform forward pass — parallel matmul-heavy kernel using hl.dot() for matrix products. Autotuned with per-shape block sizes and warp counts. Key optimizations: removed baseline redundant computations, replaced element-wise accumulation with hl.dot(), per-shape autotuned configs with static_shapes=True, persistent_blocked PID scheduling for L2 locality.
1. Causal Conv1d — Depthwise Causal Convolution Implements the core Mamba/Mamba-2 causal convolution where each channel is convolved independently with past context. The kernel tiles across batch, channels, and sequence length with filter width (W=4) specialized at compile time for loop unrolling. Boundary handling uses masked loads (hl.load with extra_mask) to avoid host-side padding. Autotuned configs optimize block sizes (1024–4096), L2 cache grouping, and pipeline stages for the memory-bound pattern on B200. 2. Gated DeltaNet Chunk Forward H — Inter-Chunk Recurrence Implements the sequential bottleneck (Eq. 8), maintaining a K×V state across 64-timestep chunks. Parallelized across batch×head while processing chunks sequentially. Each step applies delta correction, exponential decay gating, and state accumulation. Uses IEEE float32 dot precision to prevent numerical drift. Dynamic shape fallback ensures correctness. 3. Gated DeltaNet Chunk Forward O — Output Computes per-chunk outputs combining inter-chunk state queries and intra-chunk causal attention (Eq. 9). Chunks run fully in parallel. Optimizations include exp2, fused hl.dot(..., acc=) accumulation, and B200 ACF scheduling. 4. Gated DeltaNet Recompute W/U — WY Transform Computes WY-transformed keys/values via two batched matmuls per chunk (Eq. 4–7). Uses exp2 gating and persistent-interleaved PID scheduling for high SM utilization. Single config passes all 12 shapes within the 12-minute compile limit.
KernalForge is our Helion hackathon project for optimizing production-style GPU kernels on NVIDIA B200 hardware. We built and benchmarked all five required kernels, using shape-specific Helion configurations, remote B200 tuning, compile-aware optimization, and submission-safe kernel design so the code is not only fast in steady state but also practical under real evaluation-time constraints. In parallel, we contributed our learnings back to the Helion ecosystem through an upstream PR that adds deployment guidance for compile-budget management and new example kernels derived from this work. causal_conv1d We implemented a causal depthwise 1D convolution kernel in Helion for state-space style sequence models. The work focused on moving causal boundary handling into the kernel, reducing wrapper overhead, and tuning the kernel structure for efficient per-channel filtering on B200 GPUs. fp8_quant We implemented a per-token-group FP8 E4M3 quantization kernel in Helion. The kernel computes per-group absmax scaling factors and quantized outputs, targeting the quantization patterns used in modern LLM inference pipelines. gated_deltanet_chunk_fwd_h We implemented the Gated DeltaNet inter-chunk state recurrence kernel in Helion. The kernel maintains chunk state across the sequence, applies gated updates, and writes the recurrent state efficiently for long-context sequence modeling workloads. gated_deltanet_chunk_fwd_o We implemented the Gated DeltaNet output kernel in Helion, combining chunk-local causal interactions with inter-chunk state contributions. The work focused on expressing the DeltaNet output path clearly in Helion while tuning it for low-latency execution on B200. gated_deltanet_recompute_w_u We implemented the Gated DeltaNet WY-transform recomputation kernel in Helion. The kernel reconstructs the intermediate W and U terms from chunk-local inputs using matrix-style updates, with tuning aimed at minimizing latency while keeping the submission path practical. Special-track PR Description We also prepared an upstream Helion contribution based on this project. The PR adds deployment and autotuning guidance for managing compile budget and cold-start cost, and contributes new upstream examples and tests for several of the kernels explored during the hackathon. PR 1: Multi-Fidelity Autotuner (multi-fidelity-autotuner branch) Speeds up autotuning by filtering out bad configs cheaply before expensive benchmarks. Evaluates candidates in stages with increasing benchmark precision (fewer reps → more reps), eliminating the bottom fraction at each stage. Only survivors get the full-cost evaluation. Wraps any existing search algorithm (PatternSearch, LFBOTreeSearch, etc.) — inspired by Successive Halving. Usage: HELION_AUTOTUNER=MultiFidelitySearch HELION_MULTI_FIDELITY_INNER=PatternSearch --- PR 2: Grid Search Autotuner (grid-search-autotuner branch) — https://github.com/pytorch/helion/pull/1712 Exhaustively enumerates the entire config search space (Cartesian product of all parameter values) and benchmarks every combination to find the global optimum. For large spaces, randomly samples up to max_configs (500 default). Unlike heuristic searches that might miss the best config, this guarantees complete coverage when the space is small enough. Usage: HELION_AUTOTUNER=GridSearch PR 3: We also prepared an upstream Helion contribution based on this project. The PR adds deployment and autotuning guidance for managing compile budget and cold-start cost, and contributes new upstream examples and tests for several of the kernels explored during the hackathon.
implemented all 4 scored Helion kernels for Gated DeltaNet and causal conv1d, optimized for B200 GPUs with per-shape autotuned configs. causal_conv1d: Zero-copy causal depthwise conv: eliminated external padding via in-kernel index clamping/masking, avoiding redundant memory traffic from torch.cat. ~2.9x speedup over naive padded approach. TMA + 6-stage pipelining, hl.specialize(W) for compile-time unrolling. gated_deltanet_chunk_fwd_h: Inter-chunk state recurrence with sequential scan, maintaining [K,V] hidden state in registers. hl.dot with accumulator for state updates, multi-stage pipelining, tuned loop orders, variable static_ranges handling. gated_deltanet_recompute_w_u: WY-transform kernel computing w and u as two fused dot products sharing a single A matrix load. Mixed tensor_descriptor/pointer indexing, hl.specialize for K, V, and chunk dims. gated_deltanet_chunk_fwd_o: Output kernel combining inter-chunk (q@h) and intra-chunk (causal attention). Early inter-chunk computation for better pipelining, causal masking via hl.arange, persistent_interleaved pid with num_sm_multiplier=8 for full SM utilization.
Problem 3 (gated_deltanet_chunk_fwd_h): Implemented inter-chunk state recurrence kernel for Gated DeltaNet using Helion DSL with hardcoded configs optimized for B200.
I autotuned the kernels and tried to do some DMA-Compute overlapping, though that wasn't really effective.
Not sure where the f8_quant submission is supposed to go, so here it is: https://github.com/dpiresearch/Helion_20260314/blob/main/helion/fp8_quant_py/submission.py For causal_conf1d Tuned the Helion execution config so the kernel uses the GPU more effectively: num_warps: 1 → 4 More threads per block (4×32 = 128), so more parallelism and better occupancy. num_stages: 1 → 2 More pipeline stages so load/store and compute can overlap better (better memory latency hiding). For the gated_deltanet_* kernels Here’s a concise summary of what the gated_deltanet\* submission kernels improve over the reference implementations. 1. gated_deltanet_chunk_fwd_h Reference (reference.py): PyTorch eager, sequential over chunks. For each chunk it: Keeps full chunk tensors k_c, w_c, u_c, g_c in memory Does v_new_c[:, c] = u_c[:, c] - w_c[:, c] @ h and h = h * exp(g_last) + k_c[:, c].T @ v_gated Loops over c in Python; lots of intermediates and global memory traffic Improvements in submission (submission.py): Single Helion kernel with explicit tiling: tiles over (B*H, V) with block [1, 8] and over time T with chunk size C = 64. Recurrent state in registers: keeps state [K, V] per (b, h) and updates it chunk-by-chunk with hl.dot for the matmuls (e.g. w @ state, k_adj.T @ diff), avoiding full chunk-sized matmuls in global memory. Specialized sizes: K and V are hl.specialize(...) so the compiler can optimize for fixed dimensions. Stable config: One helion.Config (e.g. num_warps=4, num_stages=2) for all shapes to stay within the leaderboard timeout while still being GPU-friendly. So the main improvement is moving from a chunk-by-chunk Python loop with big PyTorch matmuls to one tiled Helion kernel that keeps the recurrence in small state and uses hl.dot for the core math. 2. gated_deltanet_chunk_fwd_o Reference: Reshapes to chunks and does: o_inter = (q_c @ h) * exp(g_c) (inter-chunk) Full C×C qk = q_c @ k_c.T * exp(g_diff) with causal mask, then o = (o_inter + qk @ v_c) * scale Large temporary tensors for the full chunk–chunk attention. Improvements in submission: Tiled over (BH, T) with block [1, C] so each program works on one chunk of time for one (b, h). Same math, better mapping: Intra-chunk: qk = hl.dot(q_tile, k_tile.T), causal mask and g_diff, then sim = where(causal, qk * exp(g_diff), 0) and local_out = hl.dot(sim, v_tile). Inter-chunk: global_out = hl.dot(q_s, h[c_idx]) with q_s = q_tile * exp(g_vals). Output: (global_out + local_out) * scale. Helion dot primitives instead of large PyTorch matmuls, so the work is expressed as smaller, cache-friendly ops. Config: e.g. num_warps=8, num_stages=4 for more parallelism on this matmul-heavy kernel. So the improvement is preserving the exact chunk-fwd-o formula while executing it in a tiled, dot-based way that fits the GPU and avoids huge intermediates. 3. gated_deltanet_recompute_w_u Reference: Reshapes to (B, NT, C, H, K/V) and does two batched matmuls: u_c = A_c @ (v_c * beta_c) w_c = A_c @ (k_c * (beta_c * exp(g_c))) then permute/reshape back to (B, T, H, K/V). Improvements in submission: No big batched matmuls: the matmul is expressed as two explicit passes over the inner dimension ci in 0..C-1 and ci in C-1..0, each accumulating: w_acc += a_col[:, None] * (k_ci * coeff_ci * decay_ci)[None, :] u_acc += a_col[:, None] * (v_ci * coeff_ci)[None, :] Averaging: w_out = (w_acc1 + w_acc2) * 0.5, u_out = (u_acc1 + u_acc2) * 0.5. That gives a symmetric (forward+backward) order of summation, which can improve numerical behavior (e.g. cancellation) compared to a single pass. Tiling: over (B*H, T) with block [1, C], so each tile handles one chunk and the inner loop is over C with small accumulators (hl.zeros([rt, K]), hl.zeros([rt, V])), which can stay in registers or fast memory. Config: e.g. num_warps=4, num_stages=2 for stability under the 12-minute leaderboard limit. So the improvement is replacing one-shot batched matmuls with a tiled, two-pass accumulation that is both GPU-friendly and numerically more stable. Cross-cutting improvements Static shapes and configs: All three use static_shapes=True and SHAPE_CONFIGS keyed by (B, T, H, K, V) so the right kernel/config is chosen per shape without autotuning on the bot. Single config for timeout: Comments note that B200-tuned or autotuned configs hit the 12-minute leaderboard timeout; using one “safe” config per kernel improves reliability. IEEE dot precision: All use dot_precision="ieee" for correctness. Optional ACF: Comments point to advanced_controls_file for further tuning (e.g. booster pack ACFs) once a baseline is correct and stable. In short: the gated_deltanet submissions improve over the references by turning chunk-wise PyTorch loops and large matmuls into single, tiled Helion kernels that use small state, explicit dots, and (for recompute_w_u) a two-pass summed form for better numerics and GPU utilization.
We implemented four kernels for the Helion challenge on NVIDIA B200 GPUs, prioritizing FP32 accumulation for precision and 3D tiling for better hardware occupancy. causal_conv1d: This is a causal depthwise 1D convolution used in Mamba and Mamba-2 architectures. We optimized this by using F.pad before the kernel launch and caching the weights and bias in the SRAM (the GPU's fast on-chip memory) and Registers. This resulted in execution times ranging from 0.03ms to 0.09ms on the B200. gated_deltanet_chunk_fwd_h: This kernel handles the inter-chunk state recurrence. We achieved performance gains by removing redundant hl.dot (matrix multiplication) calls and streamlining the math used for the gating mechanism. It runs in 0.005ms to 0.08ms. gated_deltanet_chunk_fwd_o: This kernel computes the final output. We utilized B200-specific Advanced Control Files (ACFs) to tune hardware parameters and implemented lazy-compilation to avoid timing out on the 12-minute leaderboard limit. Performance ranges from 0.02ms to 0.06ms. gated_deltanet_recompute_w_u: This handles the forward WY-transform. We achieved a speedup by vectorizing the matrix multiplications (processing multiple data points at once) and scaling the K and V vectors before performing a single hl.dot pass. This takes 0.13ms to 0.36ms. Take a look at our cool dashboard in the repo for screenshots and instructions on how to run it! :)