KernelForge

KernalForge is our Helion hackathon project for optimizing production-style GPU kernels on NVIDIA B200 hardware. We built and benchmarked all five required kernels, using shape-specific Helion configurations, remote B200 tuning, compile-aware optimization, and submission-safe kernel design so the code is not only fast in steady state but also practical under real evaluation-time constraints. In parallel, we contributed our learnings back to the Helion ecosystem through an upstream PR that adds deployment guidance for compile-budget management and new example kernels derived from this work. causal_conv1d We implemented a causal depthwise 1D convolution kernel in Helion for state-space style sequence models. The work focused on moving causal boundary handling into the kernel, reducing wrapper overhead, and tuning the kernel structure for efficient per-channel filtering on B200 GPUs. fp8_quant We implemented a per-token-group FP8 E4M3 quantization kernel in Helion. The kernel computes per-group absmax scaling factors and quantized outputs, targeting the quantization patterns used in modern LLM inference pipelines. gated_deltanet_chunk_fwd_h We implemented the Gated DeltaNet inter-chunk state recurrence kernel in Helion. The kernel maintains chunk state across the sequence, applies gated updates, and writes the recurrent state efficiently for long-context sequence modeling workloads. gated_deltanet_chunk_fwd_o We implemented the Gated DeltaNet output kernel in Helion, combining chunk-local causal interactions with inter-chunk state contributions. The work focused on expressing the DeltaNet output path clearly in Helion while tuning it for low-latency execution on B200. gated_deltanet_recompute_w_u We implemented the Gated DeltaNet WY-transform recomputation kernel in Helion. The kernel reconstructs the intermediate W and U terms from chunk-local inputs using matrix-style updates, with tuning aimed at minimizing latency while keeping the submission path practical. Special-track PR Description We also prepared an upstream Helion contribution based on this project. The PR adds deployment and autotuning guidance for managing compile budget and cold-start cost, and contributes new upstream examples and tests for several of the kernels explored during the hackathon. PR 1: Multi-Fidelity Autotuner (multi-fidelity-autotuner branch) Speeds up autotuning by filtering out bad configs cheaply before expensive benchmarks. Evaluates candidates in stages with increasing benchmark precision (fewer reps → more reps), eliminating the bottom fraction at each stage. Only survivors get the full-cost evaluation. Wraps any existing search algorithm (PatternSearch, LFBOTreeSearch, etc.) — inspired by Successive Halving. Usage: HELION_AUTOTUNER=MultiFidelitySearch HELION_MULTI_FIDELITY_INNER=PatternSearch --- PR 2: Grid Search Autotuner (grid-search-autotuner branch) — https://github.com/pytorch/helion/pull/1712 Exhaustively enumerates the entire config search space (Cartesian product of all parameter values) and benchmarks every combination to find the global optimum. For large spaces, randomly samples up to max_configs (500 default). Unlike heuristic searches that might miss the best config, this guarantees complete coverage when the space is small enough. Usage: HELION_AUTOTUNER=GridSearch PR 3: We also prepared an upstream Helion contribution based on this project. The PR adds deployment and autotuning guidance for managing compile budget and cold-start cost, and contributes new upstream examples and tests for several of the kernels explored during the hackathon.