Skip to main content
Crowdfunding
Python + AI for Geeks
Practice

Machine Learning Objectives - Optimization and Gradient Descent

Optimization in machine learning refers to the process of adjusting weights and biases to ensure that the model performs at its best.

The process of model learning involves the following steps:

  1. The model takes input data and calculates predicted values.

  2. A loss function is utilized to compute the difference or loss between the predicted values and the actual values.

  3. If the loss is significant, the model adjusts its weights and biases to make better predictions.

  4. This process is repeated, allowing the model to gradually improve the accuracy of its predictions.

Through this iterative process, a machine learning model, which initially may not predict answers accurately, learns to make increasingly accurate predictions.


Optimization Algorithms​

To enable a model to learn on its own, it must find ways to reduce loss.

Optimization algorithms serve this purpose.

The most commonly used optimization algorithm is Gradient Descent.


What is Gradient Descent?​

Gradient Descent is akin to finding the lowest point from a mountain.

Imagine starting at the top of a mountain.

A large loss value implies being at the mountain's peak, while a small loss value indicates being closer to the base.

Gradient Descent is the process of moving step by step until the lowest point (minimum loss) is reached.

The key concept here is the Gradient of the Loss Function.


When descending a mountain, finding the steepest path ensures a faster descent.

Similarly, in Gradient Descent, the gradient (or derivative) of the loss function is calculated to adjust the weights.

If the gradient is steep, the loss value is reduced quickly with larger steps; if it's shallow, the adjustments are smaller and more cautious, aiming for the optimal values.

Thus, Gradient Descent calculates the gradient of the loss function, and adjusts the weights and biases in the opposite direction (where the gradient decreases).

In the next lesson, we'll have a simple quiz to review the concepts we've covered so far.

Want to learn more?

Join CodeFriends Plus membership or enroll in a course to start your journey.