HeimdaLLM allows you to execute untrusted output from a large language model, specifically in the domain of SQL. The name originates from Heimdall, a Norse god who serves as a guardian of Asgard and prevents destructive forces from entering.

HeimdaLLM is a validation framework which can be used to guarantee that an LLM’s SQL output is safe to execute, by restricting selected tables and requiring specific constraints. Github mentioned HeimdaLLM in their developer blog as one of upcoming tools for LLM safety. Star it on Github here.

Simply put, you can ask an LLM to generate a SQL query, validate it through HeimdaLLM, and guarantee that the output is safe to run on your database.

Ensuring safety at a critical component can be a challenging task, particularly when dealing with untrusted code. I accomplished this with HeimdaLLM by using a unique integration of a restricted grammar (written from scratch with safety in mind), and a paranoid parser. The grammar restricts what operations are considered syntactically valid, which makes it easy to disable entire swaths of functionality. The parser builds on top of this by analyzing the parse tree to ensure things like specific JOIN or WHERE constraints are satisfied.

Finally, I defined an API for developers to specify a validator for their untrusted SQL queries:

For example, the validator above ensures that:

• The query is constrained by customer.customer_id (requester_identities)

• Any other WHERE and JOIN constraints are allowed (parameterized_constraints)

• Any columns may be selected (select_column_allowed)

• JOINs may only include the customer, rental and payment tables, and only on specific columns (allowed_joins)

• The query must only return a maximum of 10 rows (max_limit)