Raul Chedrese
Machine Learning from Scratch in Elixir
April 8, 2023
It’s an exciting time for machine learning. What get’s me most excited though is the ability to easily leverage machine learning for building product features. The Elixir community has recently made big strides in making this easy. Tools like Nx, Axon, and Bumblebee make it easy to integrate machine learning into Elixir applications. If you’re anything like me though you may find it difficult to use these tools because you lack intuition on how they work under the hood. Let’s explore machine learning in Elixir without these tools.
Predict
So how do you “do” machine learning? The first thing to understand is that machine learning is all about making predictions. We might be trying to predict the price of a stock tomorrow, whether an image contains a cat, or which set of pixels should represent a sentence. All machine learning follows the same three steps: Predict, Compare, and Learn.To start making predictions we need to create, or otherwise acquire, a machine learning model and then pass it input data to make predictions. This is our first machine learning model:
input * weight = prediction
Simple right? You can think of
input
here as the information we know,
weight
as the information the model has learned, and
prediction
the information we want to know.
Let’s turn this model into Elixir code.
defmodule Model do
def run(input, weight) do
input * weight
end
end
Let’s make some predictions! What should we predict? Let’s try to predict how much money a movie will make based on its budget. Our input will be the budget, $100,000, and the weight we use will be 2.5. We’ll talk much more about how to pick a weight later.
Model.run(100_000, 2.5)
250000.0
Our model tells us that a film with a $100,000 budget will make $250,000! Is this a good prediction? Probably not but how do we know?
Compare
It’s critical that we have a way to calculate how accurate our predictions are. If we don’t know how wrong our model is we can’t know how to improve it. To assess how accurate our model is we run it for an input that we already know the correct output for.Let’s look up a movie that actually had $200,000 budget and compare its world wide gross to what our model predicted. According to IMDB, Mad Max, released in 1980, had a budget of $200,000 and made a total of $99,750,000
input = 200_000
weight = 2.5
known_result = 99_750_000
prediction = Model.run(input, weight)
delta = prediction - known_result
error = (prediction - known_result) ** 2
IO.puts("Delta: #{delta}")
IO.puts("Error: #{error}")
Delta: -9.925e7
Error: 9.8505625e15
The
delta
tells us how far off our prediction is. We also calculate an
error
value. We will need the error value when we talk about training. For now it’s enough to know that there exist multiple ways to calculate error and the one we are using here is known as “mean squared error”.
<br />Clearly our model isn’t very good if it is resulting in such a high error value. We need to adjust our
weight
value based on what we just
learned from this example.
Learn
The learning part of “Machine Learning” is all about updating ourweight
based on the
error
we calculate from making a prediction. Our model “learns” by making predictions and updating the
weight
until our
error
gets to 0. As with calculating error there exist multiple ways to calculate the amount to change the weight. The method we will use here is called gradient decent and it’s much simpler than it sounds.
First we’ll calculate the amount to change the weight,
weight_delta
, and then subtract that amount from our current weight. The weight delta is calculated by multiplying the
delta
, difference between our prediction and the known value, by the input. This “scales” the delta by the input and helps prevent the delta from being much to large. The delta could be a very large number even when the input is a small number.
weight_delta = delta * input
updated_weight = weight - weight_delta
19850000000002.5
We’ll add one more complication to our weight calculation,
alpha
. We’ll multiply our
weight_delta
by
alpha
.
alpha
is an arbitrary number that we can tweak how our model learns. We can increase
alpha
to make our model learn faster and decrease it to prevent our learning algorithm from overshooting.
alpha = 1
weight_delta = delta * input
updated_weight = weight - weight_delta * alpha
19850000000002.5
Here is the training function that combines all three steps to generate a new weight for our model.
defmodule ModelTrainer do
def train(weight, input, goal_prediction, current_prediction \\ nil)
def train(weight, _input, goal_prediction, goal_prediction) do
IO.puts("Training Complete, final weight: #{weight}")
end
def train(weight, input, goal_prediction, current_prediction) do
IO.puts(
"weight: #{weight}, current_prediction: #{current_prediction}, goal_prediction: #{goal_prediction}"
)
# Predict
alpha = 0.00000000001
prediction = Model.run(input, weight)
# Compare
delta = prediction - goal_prediction
_error = delta ** 2.0
# Learn
weight_delta = delta * input
updated_weight = weight - weight_delta * alpha
train(updated_weight, input, goal_prediction, prediction)
end
end
input = 200_000.0
starting_weight = 1.0
known_result = 99_750_000.0
ModelTrainer.train(starting_weight, input, known_result)
weight: 1.0, current_prediction: , goal_prediction: 9.975e7
weight: 200.1, current_prediction: 2.0e5, goal_prediction: 9.975e7
weight: 319.56, current_prediction: 4.002e7, goal_prediction: 9.975e7
...
weight: 498.74999999999994, current_prediction: 99749999.99999997, goal_prediction: 9.975e7
weight: 498.75, current_prediction: 99749999.99999999, goal_prediction: 9.975e7
Training Complete, final weight: 498.75
If we run this it will print out the learned weight,
498.75
. We can verify this by plugging it back into our model.
Model.run(200_000, 498.75)
99750000.0
Our model learned to predict the correct value! This machine learning model has some glaring limitations but hopefully it has given you some insight into how machine learning models work at basic level. Some of the biggest issues with our model is that it doesn’t support multiple inputs or outputs. Our
ModelTrainer
also only trains on a single example. Useful machine learning models are trained on many examples. We’ll address these issues and more in a future post.