Attention, conversation, and power grid strategies

Your new Strategy Toolkit newsletter (July 22, 2024)

(1) One thing after another, only better

Arguably the most influential paper on AI published in the last 10 years, “Attention is All You Need”* provided the necessary new thinking to unstick how neural net ideas were applied, introducing the Transformer.

For strategists, two things to note: first, the simplicity of the new idea, i.e. how self-attention can be used to generate a representation of a sequence, a powerful abstraction; and second, that it took a team of brilliant colleagues, not just an individual, to advance science. Referencing 9 of the 11 authors, Google highlighted their “equal contribution.” Each may not be a household name, but their contributions are used everyday.

“Self-attention, sometimes called intra-attention is an attention mechanism relating different positions of a single sequence in order to compute a representation of the sequence. Self-attention has been used successfully in a variety of tasks including reading comprehension, abstractive summarization, textual entailment and learning task-independent sentence representations.

“End-to-end memory networks are based on a recurrent attention mechanism instead of sequence-aligned recurrence and have been shown to perform well on simple-language question answering and language modeling tasks.

“To the best of our knowledge, however, the Transformer is the first transduction model relying entirely on self-attention to compute representations of its input and output without using sequence-aligned RNNs or convolution.”*

* Vaswani, A. et al, “Attention is All You Need,” arXiv:1706.03762 [cs.CL] (original June 12 2017); https://doi.org/10.48550/arXiv.1706.03762