Malte Landgren

Orate is a library for Programmatic Grammar over Inference of LLMs. It unifies structured output, toolcalling under one abstraction, @program, which headline feature is that it allows you to constrain the fields you extract with arbitrary python predicate. Instead of only typing, now we can also guide the output with logic. The library handles compiling your decorated program schema into a grammar, and how it should recompile when needed depending on how the execution goes (as fields can reference other fields!). These programs are also composable, and it then also handles switching grammars. Today only rudimentary optimizations are in place, like grammar caching so we don't always need to recompute.