mspandit.github.io

Context Free Grammar Parsing

Tue, 14 Apr 2026 21:14:00 +0000

The binary tree generator operates similarly to a shift-reduce parser, but right now, there is no grammar to use in parsing!

Starting with a GeneratorState consisting of an empty stack, the code performs shift and reduce steps for every input token. The result is all possible binary trees over—all possible decompositions of—the input sequence.

A context-free grammar consists of productions, or rules. The left-hand side (LHS) of each production is a nonterminal symbol. For a grammar in Chomsky Normal Form, the right-hand side (RHS) is a pair of terminal or nonterminal symbols. (Terminal symbols are simply those that never occur on the LHS of a production.)

In a shift-reduce parser, those rules control whether the shift and reduce steps happen.

To parse using a CFG, I’ll modify the code to “check with” a grammar before shifting or reducing. If the input token corresponds to a terminal grammar symbol, it can be pushed on to the stack. If the two items on the top of the stack correspond to symbols on the RHS of a production, then the nonterminal on the LHS of the production can be pushed on to the stack.

Binary Tree Modifications

The first modification will be to store labels (grammar symbols) in our BinaryTree:

pub enum BinaryTree<T: Token> {
    Terminal { label: String, token: T },
    Nonterminal {
        label: String,
        left: Box<BinaryTree<T>>,
        right: Box<BinaryTree<T>>
    },
}

Next, we’ll add a label method on BinaryTree to retrieve the label:

    pub fn label(&self) -> String {
        match self {
            BinaryTree::Terminal { label, .. } => label.clone(),
            BinaryTree::Nonterminal { label, .. } => label.clone(),
        }
    }

Grammar Representation

The grammar will store rules mapping tokens to terminal symbols as “unigrams,” and rules mapping pairs of symbols to nonterminal symbols as “digrams.”

Note that in a grammar, tokens can correspond to multiple terminal symbols, and pairs of symbols can correspond to multiple nonterminal symbols. These represent “alternative interpretations” of the input sequence. Therefore, the mapping is to Vec<String> not String.

pub struct Grammar<T: Token> {
    unigrams: HashMap<T, Vec<String>>, // token -> terminal label
    // (left_label, right_label) -> nonterminal label
    digrams: HashMap<(String, String), Vec<String>>,
}

I’ve added a couple of elementary grammars to demonstrate the parser. One is a simplified version of an expression grammar from Unit 1 of Bill MacCartney’s SippyCup project. (This project has been a source of great inspiration for me!) This grammar parses expressions consisting of the characters 1, 2, 3, 4, -, +, and *. It allows - to be used as a unary operator for negation, or a binary operator for subtraction:

    pub fn expression() -> Self {
        let mut unigrams = HashMap::default();
        unigrams.insert('1', vec!["E".to_string()]);
        unigrams.insert('2', vec!["E".to_string()]);
        unigrams.insert('3', vec!["E".to_string()]);
        unigrams.insert('4', vec!["E".to_string()]);
        unigrams.insert('-', vec![
            "UnOp".to_string(),
            "BinOp".to_string()
        ]);
        unigrams.insert('+', vec!["BinOp".to_string()]);
        unigrams.insert('*', vec!["BinOp".to_string()]);
        let mut digrams = HashMap::default();
        digrams.insert(("UnOp".to_string(), "E".to_string()), vec![
            "E".to_string()
        ]);
        digrams.insert(("E".to_string(), "BinOp".to_string()), vec![
            "EBO".to_string()
        ]);
        digrams.insert(("EBO".to_string(), "E".to_string()), vec![
            "E".to_string()
        ]);
        Self { unigrams, digrams }
    }

The other is an elementary natural language grammar:

    pub fn sentence() -> Self {
        let mut unigrams = HashMap::default();
        unigrams.insert("the", vec!["Det".to_string()]);
        unigrams.insert("cat", vec!["N".to_string()]);
        unigrams.insert("sat", vec!["V".to_string()]);
        unigrams.insert("on", vec!["P".to_string()]);
        unigrams.insert("mat", vec!["N".to_string()]);
        let mut digrams = HashMap::default();
        digrams.insert(("Det".to_string(), "N".to_string()), vec![
            "NP".to_string()
        ]);
        digrams.insert(("V".to_string(), "NP".to_string()), vec![
            "VP".to_string()
        ]);
        digrams.insert(("P".to_string(), "NP".to_string()), vec![
            "PP".to_string()
        ]);
        digrams.insert(("V".to_string(), "PP".to_string()), vec![
            "VP".to_string()
        ]);
        digrams.insert(("NP".to_string(), "VP".to_string()), vec![
            "S".to_string()
        ]);
        Self { unigrams, digrams }
    }

Processing Modifications

The modified process_fn will check with the grammar using lookup_terminals before putting an input token on the stack. It will then create copies of the stack for every terminal symbol the token corresponds to.

fn process_fn<T: Token>(token: T, grammar: &Grammar<T>)
-> impl Fn(Vec<Stack<T>>, Stack<T>) -> Vec<Stack<T>> {
    move |new_stacks, current_stack| {
        grammar.lookup_terminals(& token).map_or(
            new_stacks.clone(),
            |t_labels| t_labels.iter().fold(
                new_stacks,
                |mut new_stacks, terminal_label| {
                    new_stacks.append(&mut current_stack.clone()
                        .shift_reduce(
                            BinaryTree::Terminal {
                                label: terminal_label.clone(),
                                token: token.clone()
                            },
                            grammar
                        )
                    );
                    new_stacks
                }
            )
        )
    }
}

The modified shift_reduce function will check with the grammar using lookup_nonterminals before reducing new terminal and the tree on the top of the stack. It will then create copies of the stack for every nonterminal symbol the pair corresponds to:

    pub fn shift_reduce(
        mut self: Self,
        tree: BinaryTree<T>,
        grammar: & Grammar<T>
    ) -> Vec<Self> {
        match self.pop() {
            None => {
                self.push(tree); // shift only
                vec![self]
            },
            Some(popped_tree) => {
                let r = self.clone(); // for recursion later
                // restore the popped tree
                self.push(popped_tree.clone());
                self.push(tree.clone()); // shift
                let new_stacks = vec![self.clone()];

                match grammar.lookup_nonterminals(
                    &(popped_tree.label(), tree.label())
                ) {
                    None => new_stacks,
                    Some(new_nonterminal_labels) => {
                        new_nonterminal_labels.iter().fold(
                            new_stacks,
                            |mut new_stacks, new_nonterminal_label| {
                                let new_nonterminal = BinaryTree::Nonterminal {
                                    label: new_nonterminal_label.clone(),
                                    left: Box::new(popped_tree.clone()),
                                    right: Box::new(tree.clone()),
                                };
                                new_stacks.append(&mut r.clone()
                                    .shift_reduce(new_nonterminal, grammar)
                                );
                                new_stacks
                            }
                        )
                    }
                }
            }
        }
    }

Tokenizeable

I want to clean up the code by introducing a new trait, Tokenizeable:

trait Tokenizeable<T: Token> {
    fn tokenize(self) -> impl Iterator<Item = T>;
}

The implementation of this trait for str will call the chars method, and the implementation for Vec<& str> will call into_iter:

impl Tokenizeable<char> for &str {
    fn tokenize(self) -> impl Iterator<Item = char> {
        self.chars()
    }
}

impl<'a> Tokenizeable<&'a str> for Vec<&'a str> {
    fn tokenize(self) -> impl Iterator<Item = &'a str> {
        self.into_iter()
    }
}

Now generate can operate on any sequence that can iterate over tokens:

fn generate<T: Token>(
    input_sequence: impl Tokenizeable<T>,
    grammar: &Grammar<T>
) -> Vec<BinaryTree<T>> {
    generate_stacks(input_sequence.tokenize(), grammar)
        .filter_stacks()
        .tops()
}

Calling it becomes more straightforward, too:

fn main() {
    let x = generate("-1+2*4", &Grammar::expression());
    println!("{} trees", x.len());
    for t in x {
        println!("{}", t);
    }
    let word_sequence = vec!["the", "cat", "sat", "on", "the", "mat"];
    let x = generate(word_sequence, &Grammar::sentence());
    println!("{} trees", x.len());
    for t in x {
        println!("{}", t);
    }
}

This version of the code generates five trees for the expression example. This is to be expected, because the grammar is ambiguous and allows various orders of operation for the different operators.

The sentence grammar is deterministic, so the code generates a single tree for the example.

More Functional Binary Tree Generation

Mon, 13 Apr 2026 15:40:00 +0000

Generator State

In the last version of our binary tree generator, generate_stacks was iterating over the input sequence, and updating a Vec<Stack> in every iteration. This Vec<Stack> represents the state of the generator. To make our code more functional, we represent this state using its own type, and provide a default state consisting of a single empty stack:

pub struct GeneratorState<T: Token>(Vec<Stack<T>>);

impl<T: Token> Default for GeneratorState<T> {
    fn default() -> Self {
        Self(vec![Stack::default()])
    }
}

The state has a method to process a Token to produce a new state.

This method calls the higher-order process_fn to create a new function stack_fn to process stacks with the token. process starts with a default (empty) Vec<Stack>, and iterates over the stacks in the current state. At each iteration, it calls stack_fn which produces one or more new stacks. The set of new stacks produced from all of the current stacks forms the new state.

    pub fn process(self: Self, token: T) -> Self {
        let stack_fn = process_fn(token);
        Self(
            self.0.into_iter().fold(
                Vec::default(),
                stack_fn
            )
        )
    }

The function returned by process_fn creates a new BinaryTree::Terminal from the token and passes it to shift_reduce on the current stack. The stacks returned by shift_reduce are appended to the set of new stacks.

fn process_fn<T: Token>(token: T) -> impl Fn(Vec<Stack<T>>, Stack<T>) -> Vec<Stack<T>> {
    move |mut new_stacks, current_stack| {
        new_stacks.append(& mut current_stack
            .shift_reduce(BinaryTree::Terminal(token.clone()))
        );
        new_stacks
    }
}

shift_reduce replaces PoppingIterator with recursion over the stack. It accepts a new tree which, when initially called, is the Terminal for the new token.

It attempts to pop a tree off the stack. If there was none, then it pushes the new tree on the stack and returns a Vec containing only itself. This is all that happens for the first input token in the sequence.

If there was a tree on the stack, the method must put the stack representing the lazy (shift) choice and one or more stacks representing eager (reduce) choices in the new state.

The shift choice restores the tree that was popped and also pushes the new tree on the stack, and adds it to the new stacks for the new state.

The reduce choice creates a new nonterminal containing the tree that was popped and the new tree. It then recursively calls shift_reduce, passing the new nonterminal.

    pub fn shift_reduce(mut self: Self, tree: BinaryTree<T>)
    -> Vec<Self> {
        match self.pop() {
            None => {
                self.push(tree); // shift only
                vec![self]
            },
            Some(popped_tree) => {
                let r = self.clone(); // for recursion later
                // restore the popped tree
                self.push(popped_tree.clone());
                self.push(tree.clone()); // shift
                let mut new_stacks = vec![self.clone()];

                let new_nonterminal = BinaryTree::Nonterminal(
                    Box::new(popped_tree),
                    Box::new(tree),
                );
                new_stacks.append(&mut r
                    .shift_reduce(new_nonterminal)
                ); // reduce
                new_stacks
            }
        }
    }

Finally, we make filter_stacks and tops into methods on GeneratorState

    pub fn tops(self: Self) -> Vec<BinaryTree<T>> {
        self.0.into_iter().map(|stack| stack.top()).collect()
    }

    pub fn filter_stacks(self: Self) -> Self {
        Self(
            self.0.into_iter()
                .filter(|stack| 1 == stack.len())
                .collect()
        )
    }

generate_stacks was calling the chars method, but that method is only suitable for character sequences. We replace its argument with any sequence that implements an iterator over Token types. Starting with the default GeneratorState, it folds the input sequence into new states iteratively by calling the process method.

fn generate_stacks<T: Token>(input_sequence: impl Iterator<Item = T>)
-> GeneratorState<T> {
    input_sequence.fold(
        GeneratorState::default(),
        |gen_state, input| {
            gen_state.process(input)
        }
    )
}

We generalize the argument of generate similarly, and use the new methods on the GeneratorState:

fn generate<T: Token>(input_sequence: impl Iterator<Item = T>)
-> Vec<BinaryTree<T>> {
    generate_stacks(input_sequence)
        .filter_stacks()
        .tops()
}

These changes make the design more functional without altering the operation of the program.

From Binary Trees to Grammar Derivations

Sat, 11 Apr 2026 16:46:00 +0000

Déjà Vu

If you have studied natural language processing (NLP) then generation of binary trees might have given you a feeling of déjà vu. Here’s why:

NLP often involves using a context-free grammar (CFG) to decompose a sentence. We use the rules (“productions”) of the grammar to generate a parse tree, also called a derivation. In general, CFGs are non-deterministic or ambiguous, meaning multiple derivations can be produced. (It is certainly for a CFG to be deterministic, in which case a single derivation can be produced—computer programming languages, unlike natural languages, conform to a deterministic CFG.)

Shift-reduce parsing is a flexible method for generating derivations of a CFG from an input text. Remember the choices in the generation of binary trees: “lazy” to put something on the stack, and “eager” to create new non-terminals and put them on copies of the stack. The “lazy choice” directly corresponds to the “shift” step of a shift-reduce parser. The “eager” choice corresponds to the “reduce” step.

In the latest version of the Rust code, I’ve replaced use of the term eager with reduce, and use of the term lazy with shift.

From Characters to Tokens

The code currently generates binary trees from sequences of characters. In NLP, derivations of CFGs are usually generated from sequences of words. We’d like to accommodate these possibilities—and more.

For example, we might expect a speech recognition system to produce a sequence of words from an audio signal. However, there can be uncertainty about which word was uttered, because different words can sound similar, especially in the presence of background noise. Therefore, the speech recognition system is likely to produce a sequence of probability distributions over words.

For this reason, I’ve modified the code to treat the input as a sequence of tokens. In Rust, we can allow tokens to be of non-character types using generics. However, we do need to display and clone tokens. Therefore, we introduce the Token trait. This trait itself does not require implementation of any methods, but it refers to types that implement the required Clone and Display methods:

trait Token: Clone + Display {}

The terminals of the binary tree will now consist of tokens, any type that implements the Token trait:

enum BinaryTree<T: Token> {
    Terminal(T),
    Nonterminal(Box<BinaryTree<T>>, Box<BinaryTree<T>>),
}

One way to interpret this is: the Rust compiler will generate, on the fly, versions of the BinaryTree storing different types of terminals, as long as those types implement the Token trait. (We say they “are Token.”) It will also generate versions of any methods on BinaryTree that depend on what is stored.

The built-in char and str types already implement Display and Clone. We still need to “mark” them as implementing Token with a single line each:

impl Token for char {}
impl Token for &str {}

generate_stacks was calling the chars method, but that method is only suitable for character sequences. We replace its argument with any sequence that implements an iterator over Token types. We generalize the argument of generate similarly:

fn generate_stacks<T: Token>(input_sequence: impl Iterator<Item = T>)
-> Vec<Stack<T>> {
    input_sequence.fold(
        vec![Stack::default()],
        |stacks, input| {
            stacks.into_iter().fold(
                Vec::default(),
                shift_reduce_fn(input)
            )
        }
    )
}

fn generate<T: Token>(input_sequence: impl Iterator<Item = T>)
-> Vec<BinaryTree<T>> {
    tops(filter_stacks(generate_stacks(input_sequence)))
}

In main, we test this capability by passing sequences of characters as well as words:

fn main() {
    let x = generate("abcde".chars());
    println!("{} trees", x.len());
    for t in x {
        println!("{}", t);
    }
    let word_sequence = vec!["the", "cat", "sat", "on", "the", "mat"];
    let x = generate(word_sequence.into_iter());
    println!("{} trees", x.len());
    for t in x {
        println!("{}", t);
    }
}

This version successfully generates 42 trees for the sequence of words!

Generating Binary Trees

Sat, 11 Apr 2026 02:43:00 +0000

How would you generate all possible binary trees from a sequence of inputs?

Trees and Binary Trees

First, what even is a binary tree? In computer science, a tree data structure models “one-to-many” relationships. For example, a mother can have multiple children, but a child can have only one (biological) mother. The branch of an actual tree might have many leaves, but any leaf is connected to a single branch. Abstractly, some “nodes” in a tree are “parents” or “branches” to other nodes, their “children” (which can themselves be “parents” or “branches”). Eventually, there are “children” with no children of their own, the “leaves” of the tree.

In a binary tree, a parent has exactly two children. Binary trees are interesting because they are recursively defined, but still very simple. A tree in which parents might have more than two children (an “\(n\)-ary tree”) can always be converted into a binary tree by introducing “intermediate” or “latent” parents. This is related to the fact that context-free grammars can always be converted to Chomsky normal form. For this reason, I’ll be referring to “branch” nodes as “nonterminal” and “leaf” nodes as “terminal.”

Tree data structures are valuable for modeling component-part relationships. Therefore, when we generate “all possible binary trees from a sequence of inputs,” we are generating “all possible decompositions of a sequence of inputs.”

Generating Trees

This is a non-trivial task. There are solutions for a sequence of a given length. But I am more interested in an incremental solution where you don’t know the length of the sequence in advance. Let’s consider various cases:

For an empty sequence of inputs, no binary trees are possible.
For the first input, for example a, a tree consisting of a terminal is possible. I’ll simply write this tree as a.
For the second input, for example b, a tree consisting of a nonterminal, with the two inputs as its children, is possible. I’ll write this tree as (a b) The parentheses signal the presence of a nonterminal with the children between them.

For more than two inputs, things get complicated.

For a sequence of three inputs, two trees are possible: (a (b c)) and ((a b) c)
For a sequence of four inputs, five trees are possible: (a (b (c d))), ((a (b c)) d), (a ((b c) d)), ((a b) (c d)), and (((a b) c ) d)

You can see that the idea is to “connect” an input to, or “pair” an input with its prior (left) or subsequent (right) neighbor at different times during the construction of the neighbor. In the most “eager” case, we pair every input with its left neighbor immediately and get a tree like (((a b) c) d). In the most “lazy” case, we pair every input with its right neighbor, if there is one. This means we must wait to do any connections until the last input is read. This gives us a tree like (a (b (c d))).

Every time we get an input, there is a choice of being eager or lazy. A different sequence of choices will result in a different tree, so all possible sequences must be generated. This means that for intermediate inputs, we need to “split” the state of the tree generation to represent both choices.

An Algorithmic Approach

A stack is a suitable data structure in which to store a sequence of choices. Our approach will be to create multiple stacks representing different sequences of choices in tree generation:

We will start with a single empty stack
For the first input, we will create a terminal and put it on top of the stack for later use. This represents the choice to be lazy. (There is no possibility to be eager with the first input because there is no prior input to pair with.)
For any subsequent input, we will create a terminal for the input and a copy of the stack. The original (“lazy stack”) will represent the choice to be lazy for this input. The copy (“eager stack”) will represent the choice to be eager.
1. We will put the new terminal on top of the lazy stack.
2. We will pop a prior tree off the top of the eager stack, and create a nonterminal combining the tree and the new terminal
3. With this new nonterminal, too, there is a choice to be eager or lazy. So we will create a copy of the popped stack. The original will represent the choice to be lazy. The copy will represent the choice to be eager, so we will do the above iteratively until we cannot pop any more trees off the stack.
For the second input, we will end up with two stacks. For additional inputs, we will do the above for all the stacks.

Once the last input is processed as described above, there will be a large number of stacks:

Some will have multiple trees stacked on them. These were created in anticipation of additional input that would pair the trees. These stacks can be filtered out.
The remainder will have a single binary tree on top, representing a different sequence of decisions. These will be the possible binary trees on the input.

This process is illustrated below for four inputs. Commas separate trees on a stack. Grey lines trace which stacks are copies of which other stacks.

Implementation in Rust

We can try to represent a binary tree data structure corresponding to characters in Rust as follows:

enum BinaryTree {
    Terminal(char),
    Nonterminal(BinaryTree, BinaryTree)
}

Unfortunately, this causes the compiler error recursive type BinaryTree has infinite size.

The reason is that when a function creates a local variable of type BinaryTree, Rust needs to know its size at compilation time so it can be put on the stack.

The compiler suggests storing references to other binary trees, because the size of any reference is known at compilation time:

enum BinaryTree {
    Terminal(char),
    Nonterminal(Box<BinaryTree>, Box<BinaryTree>),
}

At the top level, we have a function that borrows a string (because it won’t need to modify it), generates stacks, filters them, and returns the top of the remaining stacks.

fn generate(input_sequence: &str) -> Vec<BinaryTree> {
    tops(filter_stacks(generate_stacks(input_sequence)))
}

Stacks are represented using Rust’s standard Vec type, which supports push and pop operations as you would expect a stack to support:

struct Stack(Vec<BinaryTree>);

The generate_stacks function uses the chars method to acquire a character iterator on the input string. Iterator types have a powerful method, fold. The idea of fold is to “fold” all elements of the iterator into some result, which can be of any type. fold takes two arguments:

an initial value for the result (which would be returned if the iterator was empty)
a closure or function. This must accept two arguments: the current value for the result, and the next element from the iterator. It must return an updated value for the result.

(A similar method, reduce, is more restrictive. You need not provide an initial value, because it uses the first element as the initial value. However, because the iterator might be empty, reduce returns an Option of the same type as the iterator.)

By using fold wherever possible, you will find that your code becomes more comprehensible, better organized, and more amenable to parallel processing.

In the case of generate_stacks, the result to generate is a list of stacks.

The closure, in turn, acquires an iterator on the current list of stacks and folds it into a new list of stacks. The function used in this fold is generated at run-time by a higher-order function lazy_eager_fn which takes the current character as input.

fn generate_stacks(input_sequence: & str) -> Vec<Stack> {
    input_sequence.chars().fold(
        vec![Stack::default()],
        |stacks, input| {
            stacks.into_iter().fold(
                Vec::default(),
                lazy_eager_fn(input)
            )
        }
    )
}

lazy_eager_fn returns a function that takes the list of stacks being generated and one of the prior stacks generated from the previous input. Based on the “eager” choice, multiple stacks may be added to the list, so the work of generating them is delegated to eager. Based on the “lazy” choice from the prior stack, a new stack is added to the list with a terminal containing the current input.

fn lazy_eager_fn(current: char) -> impl Fn(Vec<Stack>, Stack)
-> Vec<Stack> {
    move |mut new_stacks, mut stack| {
        // eager may produce multiple stacks
        new_stacks.append(& mut eager(stack.clone(), current));
        // lazy
        stack.0.push(BinaryTree::Terminal(current));
        new_stacks.push(stack); // transfer ownership
        new_stacks
    }
}

Recall that eager must pop trees off the stack iteratively. For this reason, we define a new method on a Stack called popping_iter which consumes (takes ownership of) the stack and returns an PoppingIterator for it. The PoppingIterator implements the Iterator trait. Each of its elements is a tuple consisting of a stack element and a copy of the stack that element was just popped off.

struct PoppingIterator(Stack);

impl Iterator for PoppingIterator {
    type Item = (BinaryTree, Stack);

    fn next(&mut self) -> Option<Self::Item> {
        self.0.0.pop().and_then(|tree| Some((
            tree, // The tree that was just popped off the stack
            self.0.clone(), // Copy of the stack after pop
        )))
    }
}

impl Stack {
    fn popping_iter(self: Self) -> PoppingIterator {
        PoppingIterator(self)
    }
}

The tuples returned by the PoppingIterator can now be folded into a new list of stacks. However, a nonterminal must also be updated in every iteration. So eager starts with an initial tuple consisting of an empty list of stacks, and a terminal for the current input. In every iteration, the list of stacks (initially empty) is updated and the binary tree (initially a terminal for the current input) gets updated.

You can see that for the first input, the PoppingIterator immediately returns None because the stack is empty. eager returns an empty list of stacks. It is as if lazy_eager_fn only created a lazy version of the stack, which is the behavior we want.

fn eager0(new: (Vec<Stack>, BinaryTree), popped: (BinaryTree, Stack))
-> (Vec<Stack>, BinaryTree) {
    let mut new_stacks = new.0;
    let new_tree = new.1;
    let tree = popped.0;
    let stack = popped.1;
    let new_nonterminal = BinaryTree::Nonterminal(
        Box::new(tree),
        Box::new(new_tree),
    );
    let mut new_stack = stack;
    new_stack.0.push(new_nonterminal.clone());
    new_stacks.push(new_stack);
    (new_stacks, new_nonterminal)
}

fn eager(stack: Stack, current: char) -> Vec<Stack> {
    stack.popping_iter().fold(
        (Vec::default(), BinaryTree::Terminal(current)),
        eager0
    ).0
}

Once stacks have been generated for all of the inputs, filter_stacks returns the ones with a single element:

fn filter_stacks(stacks: Vec<Stack>) -> Vec<Stack> {
    stacks.into_iter()
        .filter(|stack| 1 == stack.0.len())
        .collect()
}

tops returns the binary trees on the top of those stacks:

fn tops(stacks: Vec<Stack>) -> Vec<BinaryTree> {
    stacks.iter()
        .map(|stack| stack.0[0].clone())
        .collect()
}

Once trees are generated, they can be printed out by implementing the Display trait for BinaryTree. The main function prints out the 14 trees for the sequence abcde.

impl Display for BinaryTree {
    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
        match self {
            BinaryTree::Terminal(c) => write!(f, "{c}"),
            BinaryTree::Nonterminal(left, right) =>
                write!(f, "({left} {right})"),
        }
    }
}

fn main() {
    let x = generate("abcde");
    println!("{} trees", x.len());
    for t in x {
        println!("{}", t);
    }
}

14 trees
((((a b) c) d) e)
(((a b) c) (d e))
((a b) ((c d) e))
(((a b) (c d)) e)
((a b) (c (d e)))
(a (((b c) d) e))
((a ((b c) d)) e)
(a ((b c) (d e)))
(((a (b c)) d) e)
((a (b c)) (d e))
(a (b ((c d) e)))
(a ((b (c d)) e))
((a (b (c d))) e)
(a (b (c (d e))))

This version of the full code is here.

Why Rust?

Wed, 08 Apr 2026 19:01:00 +0000

For decades, I have enjoyed web application development in Ruby and observed the evolution of JavaScript from its earliest days.

As an AI developer, I started with Common Lisp, but was later forced to use Python heavily, even though I dislike its syntax. I did significant experiments in Go, Haskell and Julia (which readers of this blog are familiar with).

I’ve recently settled on the use of Rust. Here’s why:

The Rust compiler enforces strong typing and strict ownership rules. This eliminates, at compile time, many errors that would otherwise get deferred to run-time. Such errors made it more difficult to maintain older Ruby code that was crashing in production.

Python recently introduced type hints, but without enforcement, and when dealing with legacy code, there is a slippery slope to the quagmire of dynamic typing. (In my experience, type enforcement made TypeScript a huge improvement over JavaScript.)

The Rust toolchain incorporates and enforces the latest innovations in dependency management. This makes it easy to get a codebase running in a new environment.

AI researchers are to be commended for publishing their work in open source repositories (usually of Python code). However, I have wasted many hours trying to figure out which versions of Python and dependent libraries their code successfully ran on, and getting those libraries installed in my environment. UV promises to bring Python dependency management to modern standards. It is written in Rust 😀.

The smart pointer system, in combination with strict ownership rules, allow error-free management of memory allocated on the heap, but encourage allocation on the stack. And management of the stack is orders of magnitude faster than management of the heap.

I remember that garbage collection essentially freed programmers (in Java, .NET, Ruby, Python, Go, and Julia) from concerns about memory allocation. When first learning Rust, I remember thinking, “Oh 💩, I have to concern myself with the stack vs. the heap again.” Eventually, I concluded that the speed of execution and elegance of design afforded by the Rust approach are well worth the additional concern.

The rules mentioned above make the learning curve for Rust very steep. There are reports of developers utterly frustrated by their first exposure to Rust. They struggle with the compiler, especially one of its components, the borrow checker. I was one of these developers. These struggles manifest in hours spent compiling, editing, and recompiling.

What really tipped me over to Rust was the integration of an LSP server in Visual Studio Code. (This integration is available in other editors as well.) This integration gives instantaneous feedback, exposing errors while you type code.

Rust supports type inference. This makes code more concise. It makes Rust a bit easier to adopt for developers accustomed to dynamically-typed languages. But it can also make it harder to understand. LSP integration makes inferred types explicit. This aids development—without adding any unnecessary verbosity to the code. It also helps explain errors related to mismatched types.

When the Rust compiler enforces strict rules, it is able to make certain assumptions about a program and its execution. Based on these assumptions, numerous compiler optimizations become possible, whether for speed, memory usage, or even power efficiency.

I’ve spent most of my career enjoying (and indeed promoting) the benefits of object-oriented programming to manage complexity in large, long-lived systems. Along with much of the developer community, I learned to prefer composition over inheritance. In the last couple of years, I have found myself returning to my Common Lisp roots, enjoying and promoting the benefits of functional programming. (CoffeeScript emphasizes the functional aspects of JavaScript with a more concise syntax.) Here, Rust seems to strike an ideal balance. It supports algebraic data types, iterators, closures, and monads. However, unlike Haskell, a “pure functional” programming language, Rust makes it easy to put println! statements pretty much anywhere, for basic debugging 😌.

I know at least two diehards who have devoted decades-long careers to software development in C++. When I ask them why they haven’t adopted Rust, their answers are usually related to habits, an ecosystem of trusted, mature tools, and a community of relationships that allows them to produce and maintain high-quality code at large scale. This is perfectly understandable. However, for software developers with the curiosity to explore a new programming language and the opportunities it affords, I encourage you to work past the steep learning curve of Rust.

What is a Monad (in Rust)?

Mon, 06 Apr 2026 19:40:00 +0000

Below is my modified transcription of the video What is a Monad? - Computerphile in which a presentation by Dr. Graham Hutton is filmed and edited by Sean Riley. In my revision, I demonstrate a monad in Rust rather than Haskell.

The concept of monads was invented in mathematics in the 1960s, and then it was rediscovered in computer science in the 1990s. It gives you a new way of thinking about programming with effects. This may be one of the most important new ideas in programming languages in the last 25 years. So that’s what we’re going to be looking at today—programming with monads.

We’re going to come at this using an example of writing a function that evaluates simple expressions. And I’m going to use Rust for this, but we’re going to use it in a very simple way, and I’m going to explain everything as we’re going along.

We’re going to start by defining a simple datatype for the kind of expressions that we’re going to be evaluating. So, we’ll use the enum keyword in Rust, which introduces a new data type, and then we’re going to define a new data type for expressions. There’s two things that an expression can be. It can either be an integer value, so we’ll write that down—we have Val of an i64. Or, it can be the division of two sub-expressions. So we’ve got two variants here in our data type—we’ve got Val, which builds expressions from integers, and we’ve got Div, which builds expressions from two sub-expressions.

enum Expr {
    Val(i64),
    Div(Expr, Expr)
}

However, the Rust compiler does not like this:

error[E0072]: recursive type `Expr` has infinite size
 --> src/lib.rs:1:1
  |
1 | enum Expr {
  | ^^^^^^^^^
2 |     Val(i64),
3 |     Div(Expr, Expr)
  |         ---- recursive without indirection
  |
help: insert some indirection (e.g., a `Box`, `Rc`, or `&`) to break the cycle
  |
3 |     Div(Box<Expr>, Expr)
  |         ++++    +

In order to allocate a new value of an Expr type on the stack, Rust needs to know its size at compilation time. This is not possible if it is defined recursively. The compiler is suggesting that sub-expressions live on the heap where they will be allocated at run-time. We specify this by putting them inside Box pointers:

enum Expr {
    Val(i64),
    Div(Box<Expr>, Box<Expr>)
}

To reiterate what what’s actually going on here, we’re declaring a new data type called Expr, and it’s got two new variants—one called Val, which takes an integer parameter, and one called Div, which takes two sub-expressions as parameters. Expressions are built up from integer values using a simple division operator. Many of you may not be familiar of this kind of syntax, so let’s have a couple of examples of values of this data type, so that we make sure everyone’s on the same page.

I’m going to draw a little table. On the left-hand side, I’m going to have what we would normally write down in mathematics. And then on the right-hand side, we’ll think how would you translate this into a value in this Rust data type?

Let’s have three simple examples here—we’ll have one, and we’ll have six divided by two, and let’s do one more example, we’ll have six divided by three divided by one.

Mathematics	Rust
1
6 ÷ 2
6 ÷ (3 ÷ 1)

These simple expressions are built up from integers using a division operator. But, we’re writing Rust programs today, so let’s think how do these things actually get represented as values of our expression data type? The first one is very simple—if we want to represent the value one, we just need to use the Val tag, so write Val of one.

Mathematics	Rust
1	`Expr::Val(1)`
6 ÷ 2
6 ÷ (3 ÷ 1)

If we want to have an expression like six divided by two, well it’s a division, so we have a Div at the top level, and then we have two values—we have Val six, and Val two.

Mathematics	Rust
1	`Expr::Val(1)`
6 ÷ 2	`Expr::Div(Box::new((Expr::Val(6), Expr::Val(2))))`
6 ÷ (3 ÷ 1)

The last one will need two divisions, three Val constructors, and then a bunch of brackets.

Mathematics	Rust
1	`Expr::Val(1)`
6 ÷ 2	`Expr::Div(Box::new((Expr::Val(6), Expr::Val(2))))`
6 ÷ (3 ÷ 1)	`Expr::Div(Box::new(Expr::Val(6)), Box::new(Expr::Div(Box::new(Expr::Val(3)), Box::new(Expr::Val(1)))))`

Indeed, the following test passes:

    #[test]
    fn test_expr_type() {
        let one = Expr::Val(1);
        let six_div_two = Expr::Div(Box::new(Expr::Val(6)), Box::new(Expr::Val(2)));
        let six_div_three_div_one = Expr::Div(
            Box::new(Expr::Val(6)),
            Box::new(Expr::Div(
                Box::new(Expr::Val(3)),
                Box::new(Expr::Val(1)),
            ))
        );
    }

We’ve got simple expressions built up from integers using division. Let’s write a function to evaluate these expressions.

We’re going to write an function that takes an expression as input, and what it’s going to give back is the integer value of that expression. And there’s going to be two cases here, because we have two kinds of expressions. We have a case for values, and we need to figure out what to do with that, which we’ll do in a moment. And then we have a case for division, and we need to think what to do with that. (Note that, in Rust, we borrow the expression rather than taking ownership of it in this function, because we are treating it as “read-only.”)

fn evaluate(x: & Expr) -> i64 {
    match x {
        Expr::Val(_) => todo!(),
        Expr::Div(expr, expr1) => todo!(),
    }
}

So, we’ve got the skeleton here of a function, and then we just need to fill in the details. So, how do you evaluate an integer value? Well, that’s very simple, you just give back the number—so if I had Val of one, it’s value is just one. And then how do I evaluate a division? Well, these two expressions here, expr1 and expr2, these could be as complicated as you wish. So, we need to evaluate these recursively. So what we would do, is evaluate the first one, expr1, and that will give us an integer. And then we’ll evaluate the second one, expr2, and that will give us another integer. And then, all we need to do is divide one by the other.

fn evaluate(x: & Expr) -> i64 {
    match x {
        Expr::Val(number) => *number,
        Expr::Div(expr1, expr2) => evaluate(expr1) / evaluate(expr2),
    }
}

This simple function evaluates expressions built up from integers using division — we just have a simple recursive function, two cases, and everything looks fine.

But there’s a problem with this function, and the problem is that it may crash - because if you divide a number by zero, then that’s undefined, so this function will just crash.

    #[test]
    fn demo_crash() {
        let _result = evaluate(
            Expr::Div(
                Box::new(Expr::Val(1)),
                Box::new(Expr::Val(0))
            )
        );
    }

running 1 test

thread 'tests::demo_crash' (2035286) panicked at src/lib.rs:9:36:
attempt to divide by zero

We don’t want our programs to crash, so what do we do to fix this problem?

First of all, we’re going to define a safe version of the division operator, which doesn’t crash anymore. That’s basically the root of the problem here—division by zero gives an undefined result, and the function is going to crash. So, let’s define a safe version of the division operator. We’re going to define a function called safediv, and it’s going to take a couple of integers, and it’s going to give back an Option of an integer. Option is one way that we deal with things that can possibly fail in Rust. The function signature here is not i64 and i64 returning i64, it’s i64 and i64 returning Option<i64>, because division may fail. And we’ll see how this Option type works in a moment.

So, how do we actually define safediv? We take two integers, n and m, and then what we’ll do is check—is the second of these zero? Because that’s the case when things would go wrong. So, if m happened to be zero, then we will give back the result None. None is one of the variants of the Option type. If m is not zero, what we’re going to do is give back the result of dividing. Some is another variant of the Option type—Option only has two variants, None, which represents things that have gone wrong, or in our case division by zero, and Some, which represent things that have gone fine. In this case, we actually just get back the result of dividing one number by the other.

fn safediv(n: i64, m: i64) -> Option<i64> {
    if 0 == m {
        None
    } else {
        Some(n / m)
    }
}

We now have a safe version of the division operator, which is explicitly checking for the case when the function would have crashed. So this doesn’t crash anymore, it returns one of two values—either None if things go wrong, or Some of the division if things have gone fine.

    #[test]
    fn demo_safediv() {
        assert!(safediv(6, 0).is_none());
        match safediv(6, 3) {
            Some(result) => assert_eq!(2, result),
            None => assert!(false),
        }
    }

With this safe division operator, we can rewrite our little evaluate function to make sure that it doesn’t crash.

Our new evaluator is going to have a slightly different type than before. The original function just took an expression as input, and then it gave back an integer. But that could crash. The new evaluator takes an expression as input as before, but now it gives you an optional integer, because it could fail, it could have division by zero.

We’ll do the two cases again. In the base case, we can’t just return number this time, because we’ve got to return a Option value. And there’s only two things we could return, either None or Some, and in this case the right thing to do is to return Some of number, because if you evaluate a value then that’s always going to succeed, so we use a success tag, which is Some, and then we have the integer value sitting in here.

        Expr::Val(number) => Some(*number),

If we have a division, we need to do a bit more work, because when we evaluate expr1 that may fail, when we evaluate expr2 that may fail, and then when we do the division that may fail. So, we’re going to need to do a little bit of checking and management of failure.

So, what we’re going to do, is when we evaluate a division, first of all, we’ll do a case analysis on the result of evaluating expr1. And that could be one of two things—it could either be None, in which case we’re going to do something, or we could get Some number, in which case we’re going to do something. So, there’s two cases to consider—when we evaluate the first parameter expr1, either it succeeds or it fails. So in the failure case, if we get back None, the only sensible thing to do is just to say, well if evaluation of expr1 fails, the evaluation of the whole division fails. So we’ll just return None as well. In the Some case, then we need to evaluate the second parameter expr2. So, what we’re going to do is do another case analysis, we’ll do a match evaluate of expr2, and then again there’s two possible outcomes which we could have here—either we could have None, which means it failed, or we could have Some of m, some other number, in which case we’ve succeeded. Then again, we need to think what do we do in each of these two cases. So, in the first case, if the evaluation of expr2 fails, the only sensible thing to do is say, well, we fail as well. In the second case, we’ve now got to successfully evaluated expressions—expr1 has given the result n, expr2 has given the result m, and now we can do the safe division. So, in this case we just do safediv.

fn evaluate(x: Expr) -> Option<i64> {
    match x {
        Expr::Val(number) => Some(number),
        Expr::Div(expr1, expr2) => {
            match evaluate(*expr1) {
                None => None,
                Some(n) => match evaluate(*expr2) {
                    None => None,
                    Some(m) => safediv(n, m)
                },
            }
        },
    }
}

Now we have a working evaluator. We started off with a two-line program, which kind of did the essence of evaluation, but it didn’t check for things going wrong—it didn’t check for a division by zero. Now we’ve fixed the problem completely, we have a program which works, this program will never crash, it will always give a well-defined result, either None or Some, but there’s a bit of a problem with this program, in that it’s a bit too long. It’s a bit too verbose, there’s quite a lot of noise in here, I can hardly see what’s going on anymore, because it’s all of this management of failure.

How can we make this function better? And how can we make it more like the original function, that didn’t work, but still maintain the fact that this actually does the right thing? And the idea here, is we’re going to observe a common pattern.

When you look at this function, you can see quite clearly we’re doing the same thing twice—we’re doing two matches. What we’re doing, is doing a match on the result of evaluating expr1, and if it’s None we give back None, and if it’s Some, we do something with the result. And then we do exactly the same thing with the result of evaluating expr2. If that gives None we give back None, and if it’s a Some, we do something with it. So, a very common idea in computing is when you see the same things multiple times, you abstract them out, and have them as a definition. And that’s what we’re going to do here.

Let’s draw a little picture first, to capture the pattern which we’ve seen twice. We’re doing a match on something, so let me just draw it as an ellipsis—we don’t know what’s in there,

match ... {
    None => None
    Some(x) => ...(x)
}

And, there’s two cases—if it’s None, we give back None, and if we get Some of some value x, then we’re going to process it in some way, we’re going to apply some other function to x. So, this is the pattern which we’ve seen twice. In the first case, we had eval of expr1 sitting here, and in the second case, we had eval of expr2 sitting here, but this is the same pattern that we see two times in the new evaluator which we’ve just written.

We can abstract this out as a definition. We’re going to give names to these ellipses. The first ellipsis is an Option value—it’s going to be either None or a Some, so we’ll call it m. And this second ellipsis is going to be a function, it’s going process the result in the case we’re successful, so we’ll call this f. So, we can turn this picture here into a definition now, and then we can use it to make our program simpler.

fn and_then(m: Option<i64>, f: impl Fn(i64) -> Option<i64>)
-> Option<i64> {
    match m {
        None => None,
        Some(x) => f(x),
    }
}

This function looks at what the Option value is—if it’s None, we’ll give back None, and if it’s Some of x, we’ll apply the function to it. We’ve just captured the pattern, which we’ve seen twice, by a definition now. So, we have some Option value, and then, or in sequence with, some function f, and all we’re going to do is look at what the value of the Option is—if it’s failed, we’ll fail, if it succeeds, we pass the result to the function f. It’s just the idea of abstracting out a common pattern as a definition. So, now we can use this definition to make our program simpler.

Let’s rewrite our evaluator once again. The type will remain the same—it takes in an expression as input, and it’s going to give back an Option value, as before. But the definition is going to be a bit simpler this time. If we evaluate a value, we’re going to do something. If we evaluate a division, we’re going to do something.

fn evaluate(x: Expr) -> Option<i64> {
    match x {
        Expr::Val(number) => todo!(),
        Expr::Div(expr1, expr2) => todo!(),
    }
}

In the base case, we will return Some of number. In the division case, we’re going to evaluate expr1 first, and then if that’s successful, using our little and_then function, we’re going to feed the result into a lambda expression. That lambda expression is going to take the result, n, that comes from evaluating expr1, and then it’s going to evaluate expr2. If that’s successful, we’re going to feed that result, m, into a lambda expression. So, if these two things are both successful, we’ll have two values, n and m, and then all we do is call safediv with them, then close the brackets.

fn evaluate(x: & Expr) -> Option<i64> {
    match x {
        Expr::Val(number) => Some(*number),
        Expr::Div(expr1, expr2) => and_then(
            evaluate(expr1),
            |n| and_then(
                evaluate(expr2),
                |m| safediv(n, m)
            )
        )
    }
}

This function is equivalent to the version that had all the nested matches. But, that’s all been abstracted away now—it’s all been kind of abstracted into and_then, and safediv.

This is a nicer program now, but still I’m not entirely happy with this. There’s still some complexity in there—we’re still using the funny lambda notation, we’re still using and_then. Maybe I can make it even simpler? Fortunately, Rust provides an and_then function as a method on the Option enum. Let’s write this function using that method, and then we’ll come back to what this has all got to do with monads. This will be our final program. We take an expression as input, and it’s going to return an Option of an integer. The base case will not change, so we’ll get Some of number. But the recursive case is going to become a bit simpler, because we use the and_then method.

fn evaluate(x: & Expr) -> Option<i64> {
    match x {
        Expr::Val(number) => Some(*number),
        Expr::Div(expr1, expr2) => evaluate(expr1)
            .and_then(|n| evaluate(expr2)
                .and_then(|m| safediv(n, m))
            )
    }
}

If I evaluate a division, I’m going to take the result n, from the result of evaluating expr1, if it’s successful. Then, we’ll take the result m, from the result of evaluating expr2, if that’s successful. And then, we will call safediv. And this is our final program, and I’m much happier with this one. I mean, it looks kind of similar to the original program, a similar level of complexity, but all the failure management is now handled for us automatically. The failure is happening behind the scenes behind the and_then method, and with safediv, but we don’t need to see that when we’re reading this program. This is a much nicer program than the last one, because we’ve kind of abstracted away from a lot of the detail.

So, you can look at a program like this, and I’ve hardly mentioned the word monads in the last ten minutes, you can say what’s this actually got to do with monads? Well, what we’ve actually done, is we’ve rediscovered what’s known as the Option monad. The Option monad is three things—it’s the Option type, or really the Option type constructor, because it takes a parameter. So you can have an Option of an integer, or option of a Boolean, or option of whatever you like. And then it’s two functions—it’s the function called Some, and it’s the and_then method which we introduced. And we can think about what are the types of these things? Some takes a thing of any old type, T—could be an integer, could be a Boolean, could be whatever you like. And it converts it into a Option value. So, in our case, this just took an integer like five, and we return Option of five.

And what it gives you, is a bridge between the pure world of values here, and the impure world of things that could go wrong—so it’s a bridge from pure to impure, if you like. And what and_then does, is it gives you a way of sequencing things—so you give it something which can fail, an Option<T>, and then you give it a function that tells you what to do with that T, if you succeed—so a T to Option<U>. And then, finally, what you’re going to get back is a Option<U>. Okay, and this is all that a monad is essentially—a monad is some kind of type constructor, like Option, or List, or something else, as there’s many other examples, together with two functions that have these types here. So, what we’ve essentially done is rediscovered what’s called the Option monad.

We seem to have gone through quite a lot of steps, to write in the end quite a simple program. What was the actual point here? So there’s four points which I would like to emphasize here.

So, the first point, is that the same idea we’ve seen works for other effects as well—it’s not specific to Option, which captures failure. The same idea can be used with other kinds of effects like input/output, like mutable state, like reading from environments, like writing to log files, non-determinism. All sorts of other things which you think of as being effects in programming languages fit exactly the same pattern. So, monads kind of give you a uniform framework for thinking about programming with effects.

Another important point is that it supports separation of pure code from effects. Pure functions just take inputs, and produce outputs, they don’t have any kind of side effects at all. But you need to have side effects to write real programs. So, what monads give you is a way of doing impure things, like proper side effects, like input/output, separately from pure functions like division.

Another important point here, is that the use of the effects is explicit in the types. When I wrote the evaluator which didn’t fail, it took an expression as input, and it delivered an Option of an integer. So, the Option in the type is telling me that this program may fail. So, this is the idea of being explicit about what kind of effects, or side effects, that your programs can have in the types. And this is a very, very powerful idea.

And the last thing is, it’s a little bit strange, but it’s particularly interesting, it’s the idea of writing functions that work for any effect—we might call this kind of effect polymorphism. So, a simple example of this would be maybe you have a sequence of things, which can have some effects, and you want to run them all one after the other. You could write a generic function, in a language like Haskell which supports monads, which would take a sequence of effects of any kind, any monadic type, and it would run them for you. So this is a very, very powerful idea, and languages like Haskell have libraries of kind of generic effect functions, which are very useful. So, that’s basically all I want to say.

Just going back to the start, I think the idea of programming with monads is one of the most important developments in programming languages in the last 25 years—I find this particularly fascinating. We’ve only really touched on the surface here, and if you want to know a little bit more, I can do a bit of a plug—I have a new book which came out fairly recently, Programming in Haskell, and this has got a chapter specifically about this, which goes into much more detail. I’ve only really touched on the surface, there’s lots of things I didn’t say, which maybe you need to know to write real programs using this stuff. So, you could have a look in the book to find out more about that.

Exploding and Vanishing Gradients

Sat, 24 Dec 2022 14:45:00 +0000

We train neural networks by gradually adjusting a large number (hundreds of billions in the latest large language models) of weights \(W_i\). In a backpropagation pass, the loss is calculated, and the weights are adjusted, starting at the output layer and proceeding to the input layer. For each weight, we calculate a partial derivative or gradient. Roughly, the gradient value tells us how rapidly the loss grows (or shrinks) based on an incremental change to the weight—assuming all other weights are held to their current value. The magnitude of the gradient reflects how “sensitive” the loss is to an incremental change in the weight.

If a gradient happens to become extremely small or zero, then the loss is insensitive to incremental changes in the weight. There is no point in continuing to make incremental changes to this weight. In addition, logically, the loss will be insensitive to incremental changes to some weights in earlier layers, because those layers feed their outputs to this one. Those weights will also change slowly or stop changing entirely. This is the vanishing gradient problem. The network stops training without minimizing the loss.

If a gradient happens to become extremely large, then the loss is overly sensitive to incremental changes in the weight. Incremental changes in the weight will cause a large fluctuation in the loss value, but fail to achieve the desired reduction in the magnitude of the loss. In addition, logically, the loss will be overly sensitive to incremental changes to some weights in earlier layers, because those layers feed their outputs to this one. Changes to those weights will also fail to minimize the loss. This is the exploding gradient problem. During training, the network loss tends to oscillate around its minimum value.

In a very deep neural network, if the magnitude of the weight matrix \(||W_i|| > 1.0\), then the gradients of the loss with respect to the weights will be likely to explode. If the magnitude of the weight matrix \(||W_i| < 1.0\), then the gradients of the cost with respect to the weights will be likely to vanish.

An unrolled recurrent neural network is like a deep network that uses the same set of weights in every “layer.” Consequently, the ability to train the network is highly sensitive to the magnitude of the weights.

Long Short-Term Memory (LSTM)

Sat, 17 Dec 2022 07:26:00 +0000

LSTMs are a special kind of RNN able to learn long-term dependencies. “Remembering information for long periods of time is practically their default behavior, not something they struggle to learn.”

The output of an LSTM is fed back into its input (or fed to a copy of the LSTM) as in a standard RNN.

The (input and) output of an LSTM is divided into two parts

\(h_t\) The previous \(h_{t-1}\) is scaled by a factor (the “forget gate”) ranging from 0.0 (“let nothing through”) to 1.0 (“let everything through”). Some amount (“input gate” multiplied by candidate values) is then added to it to store new information and generate \(h_t\). Because \(h_{t-1}\) is minimally processed, “It’s very easy for information to just flow along it unchanged.” If the forget gate is at 1.0, and the input gate is at 0.0, then \(h_t = h_{t-1}\)
\(y_t\) A sigmoid layer scales \(y_t\) for the update. This scaled value is multiplied by \(tanh(h_t)\), which generates values between -1.0 and 1.0 to output only the parts we decided to.

Long Short-Term Memory

Sequence to Sequence Learning with Neural Networks

Understanding LSTM Networks

Recurrent Neural Networks

Fri, 16 Dec 2022 14:26:00 +0000

RNNs accept an input vector \(x\) and generate an output vector \(y\). However, the value of \(y\) is influenced not only by the most recent \(x\), but also on the entire history of inputs \(x\) that have been fed in.

\[y \neq f(x)\] \[y = f(x_0, x_1, ... x_n)\]

An RNN maintains internal “hidden state” \(h\) that gets updated every time a new input \(x_i\) is presented. The RNN’s trainable parameters include

\(W_{hh}\), which is multiplied by \(h\) when it is updated,
\(W_{xh}\), which is multiplied by \(x\) when \(h\) is updated, and
\(W_{hy}\), which is multiplied by \(h\) when \(y\) is calculated.

In a standard RNN,

\[h_t = tanh(W_{hh}h_{t-1} + W_{xh}x_t)\] \[y_t = W_{hy}h_t\]

You can see from the above that \(y_t\) depends on \(h_t\) which depends on \(h_{t-1}\) and \(x_t\). Consequently, RNNs can be thought of as “multiple copies of the same network, each passing a message to a successor.”

(A 2-layer recurrent network can be formed by supplying the output of one RNN as the input of the other.)

Long-Term Dependencies

One appeal of RNNs is that they connect previous inputs to a current task. In standard RNNs, the repeating module will have a very simple structure such as a single \(tanh()\). If the gap (measured in number of inputs) between a relevant previous input and the place it is needed is small, then standard RNNs can learn to use the previous input. However, the gap between a relevant previous inputs and the place it is needed can become very large, and traditional RNNs can become unable to learn to connect the information. They suffer from short-term memory

Vanishing/Exploding Gradient

Gradients are used to update a neural network’s weights. During backpropagation through a recurrent neural network, the network calculates its gradient with respect to the gradients in the later step. If the gradients in the later step are less than 1.0, then the gradients may shrink to zero exponentially for earlier steps, resulting in no learning from early inputs. If the gradients in the later step are greater than 1.0, then the gradients may explode exponentially for earlier steps, resulting in unstable weights based on early inputs.

Cloud AI Service Survey

Thu, 13 Sep 2018 14:41:00 +0000

This is a survey of artificial intelligence capabilities provided by Amazon Web Services, Google Cloud, IBM, and Microsoft.

Automatic Machine Learning

Google Cloud Auto ML

Machine Learning Model Training and Deployment

Amazon SageMaker

Jupyter Notebook, API

Amazon SageMaker is an online environment for defining, training, testing, and deploying machine learning models. It gives you a Jupyter notebook from which to do development, and a library to import. The library provides

good integration with Amazon’s Elastic Cloud Compute (EC2) service so that as your models can be run on a variety of hardware and software infrastructures, whether during development or deployment.
hyperparameter tuning, so that you need not resort to brute-force grid searches through the space of hyperparameters
good integration with Amazon’s Simple Storage Service (S3), where many data scientists store training and testing data anyway.
a number of standard machine learning algorithms, but you can also develop your own using TensorFlow or MxNet.

Amazon Machine Learning

Web Application, API

For a lightweight introduction to machine learning, there is Amazon Machine Learning. You specify that it should import data from AWS data sources including S3, Redshift, or RDS. You get some visual tools for previewing the data.

You specify one of the columns in your data as a target attribute that the system will learn to predict. It automatically divides your data into a training set and a test set. It uses the training set to train the model, and then tells you how the model did on the test set, and lets you set a prediction threshold. You need not deal with any complexities of the model itself.

Once you’re happy with the model, you can deploy it for real-time (on-demand) predictions through the web interface, or for batch prediction on data in a file or in S3.

Google Cloud Machine Learning (ML) Engine

Command-line interface, Tensorboard

Google Cloud ML Engine includes freely downloadable components for developing machine learning models on a development system. The models can use scikit-learn, XGBoost, Keras, or TensorFlow. Once your models are running locally, you can copy your data into the cloud, and train your model on a cloud instance or in distributed fashion across multiple cloud instances.

Cloud ML Engine can also run in a mode that tunes hyperparameters on your model.

Once the model is trained, you can deploy it to perform prediction on new data. Cloud ML Engine supports “online” prediction via a REST API, or batch prediction.

IBM Machine Learning

Command-line interface, REST API

IBM Watson Studio

Web application

IBM Watson Studio lets you upload data, cleanse and refine it, and visualize it to discover patterns and trends.

Jupyter notebooks or RStudio let you analyze the data.

Built-in models let you classify image or natural language data. A graphical model builder lets you define a Spark ML model.

The service lets you run experiments in parallel and automates evaluation of model performance under various hyperparameter configurations.

It lets you accelerate training by distributing models across multiple servers and using multiple GPUs.

Image Processing (Computer Vision)

Google Cloud Vision

REST API

When you submit an image through its REST API, Google Cloud Vision

classifies it into thousands of categories, providing scores for the most likely ones,
detects faces within it,
detects topical entities including celebrities and logos,
determines the likelihood of several types of inappropriate content, and
reads printed words contained within it

Google Cloud AutoML Vision lets you upload images and labels and uses this data to train a recognition model. There is also an option to upload images and have human beings label them.

Amazon Rekognition

REST API

When you submit an image through its REST API, Amazon Rekognition will identify objects, celebrities, text, and activities, as well as detect any inappropriate content. It also provides face recognition and analysis (sex, eyes open/closed, glasses, facial hair, happiness and age range).

Microsoft Azure Computer Vision

Microsoft Azure Custom Vision

Microsoft Azure Face

Microsoft Azure Content Moderator

IBM Visual Recognition

Video Intelligence (Computer Vision)

Google Cloud Video Intelligence

Amazon Rekognition

REST API

When you submit a video through its REST API, Amazon Rekognition will identify objects, people, their paths, celebrities, scenes, and activities, as well as detect any inappropriate content. It also provides face recognition and analysis (sex, eyes open/closed, glasses, facial hair, happiness and age range).

mspandit.github.io

Context Free Grammar Parsing

Binary Tree Modifications

Grammar Representation

Processing Modifications

Tokenizeable

More Functional Binary Tree Generation

Generator State

From Binary Trees to Grammar Derivations

Déjà Vu

From Characters to Tokens

Generating Binary Trees

Trees and Binary Trees

Generating Trees

An Algorithmic Approach

Implementation in Rust

Why Rust?

What is a Monad (in Rust)?

Exploding and Vanishing Gradients

Long Short-Term Memory (LSTM)

Recurrent Neural Networks

Long-Term Dependencies

Vanishing/Exploding Gradient

Cloud AI Service Survey

Automatic Machine Learning

Machine Learning Model Training and Deployment

Image Processing (Computer Vision)

Video Intelligence (Computer Vision)

HR Hiring

Natural Language Intent Classification

Natural Language Entity Recognition

Natural Language Sentiment Analysis

Natural Language Syntactic Parsing

Natural Language Content Categorization

Natural Language Identification

Natural Language Knowledge Extraction

Speech Recognition (Speech-to-Text)

Speech Synthesis (Text-to-Speech)

Speaker Identification

Natural Language Translation