What are Transformers?
To quote 3Blue1Brown, a transformer is a just a very specific type of neural network (Deep Learning). When prompting a Large Language Model, it will attempt to find the token that is most likely to occur to finish the sentence. Machine Learning is a very flexible approach to building a function, it doesn’t require a specific set of procedures (snippets of code) to produce an output, instead it has a bunch of tunable parameters. I interpret these to be the weights and biases for a deep learning model. In the case of supervised learning, we feed the network a bunch of inputs and their corresponding correct outputs and use an algorithm called backpropogation to tune the weights and biases in such a way that the deep learning model can gradually minimize the error of its predictions.
Tokenization and Embeddings
Say we want to process a chunk of text like this
The first thing we want to do is break up these words, a process known as tokenization and then create vector embeddings from them. Vector Embeddings are ways to preserve the semantic meanings of each token in a higher dimensional vector space. To get a feel for how vector embeddings work, we can play around with a local embedding model and see what happens when we add or subtract specific words. Adding two vectors and finding the nearest vectors of the resulting sum should combine the semantic meaning of each
# E("Water") + E("Ship")
$ python3 transformer.py
[('vessel', 0.8715493083000183),
('sea', 0.8330526351928711),
('boat', 0.8327040672302246),
('ocean', 0.8029822111129761),
('ships', 0.8028886914253235),
('vessels', 0.7998801469802856),
('boats', 0.7984598278999329),
('waters', 0.7883374094963074)]We get words that are sort of semanatically similar to them. But how does this even work? The current model that I am using represents each vector embedding in a dimensional space which is of course impossible to visualize. But we can get a sense geometrically with the dot product. Remember, the dot product of two vectors encodes a sense of directionality between them. If two vectors are facing the same direction then their dot product will be positive, if they are perpendicular then their dot product would be close to or equal to zero.
If they are facing opposite directions their dot product would be negative. Take for example the embeddings for the words “sushi”, “japan”, “germany”. What would happen if we were to say take their dot products? Well sushi is from japan and not from germany so I would assume that in this dimensional space encoding the semantics of each word, the dot product between “sushi” and “japan” would be greater than “germany” and “sushi”.
python3 transformer.py # E(Sushi) * E(Germany)
-0.038159132
python3 transformer.py # E(Sushi) * E(Japan)
6.0519114The same goes with addition and subtraction of these embedding vectors in this higher dimensional vector space. If two vectors are pointing in the same direction, then adding them should produce a vector further along that direction. This is why in the example above we see that adding the words “water” and “ship” produce embeddings very similar to those words contextually.
It’s important to note that higher dimensional vector spaces are capable of encoding more and more information than merely the representation of the word in a vacuum. It can encode context surrounding the word, like how we interpret the meanings of specific words in a sentence differently depending on its relation, position, and meaning when reading a book or talking to someone.
Input Encodings and Positional Encodings
When we first process the text above, we merely take each of its vector embeddings in isolation from the model’s vocabulary. Essentially each of these vectors represent the model’s current understanding of the word in isolation. That is, no context surrounding each of the word or anything. When implementing my transformer, I want to seperate it out into tokenizer, embeddings, and then positional encoding.
The formula for positional encoding depends on the position of the token with respect to the sequence and also the even/oddness of the encodings in the vector embeddings.
def tokenize(prompt) -> list[str]:
words = prompt.split()
return words
def embed(tokens, embedding_model):
embeddings = []
for token in tokens:
embeddings.append(embedding_model[token])
return np.array(embeddings)
def positional_encoding(tokens):
def pe(pos, i):
exponent = 2 * (i // 2) / EMBEDDING_DIMENSION
if i % 2 == 0:
return np.sin(pos / (10000 ** exponent))
else:
return np.cos(pos / (10000 ** exponent))
positional_encodings = []
for pos in range(len(tokens)):
position_encoding = []
for i in range(EMBEDDING_DIMENSION):
position_encoding.append(pe(pos, i))
positional_encodings.append(np.array(position_encoding))
return np.array(positional_encodings)The goal of the entire network is to enable the model to tweak and tune its parameters so that each vector embedding can soak up and internalize the context surrounding each word in addition to its meaning! That is when predicting the next token, it is to be able to assign a probability to each word in the model’s vocabulary and then take the highest probability token and append it on to its response. How do we actually get these probabilities? Well we use something called a softmax to normalize all the output of every single number in the resultant vector to a real number between such that they all add up to . The process is very elegant, consider an arbitrary vector of real numbers
Attention
When we first prompt a message, each word/token of the prompt merely encodes the surface level definition of the models vocabulary. Take for example the context of “buck”.
- It will cost you a “buck”!
- I saw a really beautiful “buck” on my hike up the mountain yesterday!
The embeddings are the same for the word “buck” in each (For my specific embedding model)
[-0.67531 0.10425 0.33199 -0.17162 0.46126 0.13706
-1.16 -0.0031324 -0.21076 -0.084427 -0.52776 0.90209
-0.095148 0.028555 0.2243 -0.23107 0.81729 0.049866
-0.51744 -0.41996 -0.15688 -0.45597 0.25738 -0.24272
0.43153 -0.78803 -0.26952 -0.11614 0.36117 -0.88136
1.0305 -0.15528 0.15564 0.16281 -0.066141 -0.054714
-0.10931 -0.37815 0.63336 -0.24165 -0.44231 0.44146
-0.9849 -0.061133 0.28721 -0.59582 -0.080311 -0.65274
-0.25996 1.1327]If we call the first prompt , then we have a matrix of size , where each row corresponds to a word in the original prompt and each entry is the embeddings in the space . After calculating the positional encodings for each token we add those encodings as well onto the matrix.
def predict(prompt: str, embedding_model) -> str:
tokens = tokenize(prompt)
embeddings = embed(tokens, embedding_model)
positional_encodings = positional_encoding(tokens)
model_input = embeddings + positional_encodings
...Now we want to compute a query vector, this vector encodes the context of the words in front of the word. Essentially what we want is to take our embedding matrix and multiply it through a learned query weight matrix to produce a query matrix
The output dimensions of the query are seemingly arbitrary, I have settled on but you can choose anything that you deem appropriate. Similarily we will want to do the same thing for our key matrix and value matrix. We finally arrive to the famous attention equation
Notice what exactly is doing? Its computing a dot product for every single word in the key matrix and every single word in the value matrix. What does this tell us? Remember, the dot product is a measure of how similar the directions of two vectors are. You can imagine that doing this will result in finding combinations of words that are similar or related to one another. Which means that the queries will be answered by the keys. That is, each word can ask its preceding words how important so to speak it is with the current word.
def attention_block(prompt: str, embedding_model, W_Q, W_K, W_V) -> str:
tokens = tokenize(prompt)
embeddings = embed(tokens, embedding_model)
positional_encodings = positional_encoding(tokens)
model_input = embeddings + positional_encodings
Q = model_input @ W_Q
K = model_input @ W_K
V = model_input @ W_V
attention = (
softmax((Q @ K.T) / np.sqrt(QUERY_DIMENSION)) @ V
) # The attention block!