# 경사 하강법

둘러보기로 가기 검색하러 가기

## 노트

### 말뭉치

• If it is convex we use Gradient Descent and if it is concave we use we use Gradient Ascent.[1]
• When we use the convex one we use gradient descent and when we use the concave one we use gradient ascent.[1]
• Here, we are going to look into one such popular optimization technique called Gradient Descent.[2]
• In machine learning, gradient descent is used to update parameters in a model.[2]
• Let us relate gradient descent with a real-life analogy for better understanding.[2]
• Batch Gradient Descent: In this type of gradient descent, all the training examples are processed for each iteration of gradient descent.[2]
• Gradient descent is one of the most popular algorithms to perform optimization and by far the most common way to optimize neural networks.[3]
• At the same time, every state-of-the-art Deep Learning library contains implementations of various algorithms to optimize gradient descent.[3]
• By the end of this blog post, you’ll have a comprehensive understanding of how gradient descent works at its core.[3]
• We will intuitively by the means of gradient descent accomplish a task of rod balancing problem on our finger.[3]
• At each step, the weight vector (w) is altered in the direction that produces the steepest descent along with the error.[4]
• Summing over multiple examples in standard gradient descent requires more computation per weight update step.[4]
• Similar to batch gradient descent, stochastic gradient descent performs a series of steps to minimize a cost function.[5]
• Gradient Descent is an optimization algorithm used for minimizing the cost function in various machine learning algorithms.[6]
• Batch Gradient Descent: This is a type of gradient descent which processes all the training examples for each iteration of gradient descent.[6]
• But if the number of training examples is large, then batch gradient descent is computationally very expensive.[6]
• Hence if the number of training examples is large, then batch gradient descent is not preferred.[6]
• Since we need to calculate the gradients for the whole dataset to perform one parameter update, batch gradient descent can be very slow.[7]
• In mini-batch gradient descent, we calculate the gradient for each small mini-batch of training data.[7]
• Gradient descent is one of those “greatest hits” algorithms that can offer a new perspective for solving problems.[8]
• At a theoretical level, gradient descent is an algorithm that minimizes functions.[8]
• To run gradient descent on this error function, we first need to compute its gradient.[8]
• Below are some snapshots of gradient descent running for 2000 iterations for our example problem.[8]
• However, it still serves as a decent pedagogical tool to get some of the most important ideas about gradient descent across the board.[9]
• However, this gives you a very inaccurate picture of what gradient descent really is.[9]
• As depicted in the above animation, gradient descent doesn't involve moving in z direction at all.[9]
• A widely used technique in gradient descent is to have a variable learning rate, rather than a fixed one.[9]
• The gradient descent varies in terms of the number of training patterns used to calculate errors.[10]
• Each iteration of the gradient descent uses a single sample and requires a prediction for each iteration.[10]
• If the gradient descent is running well, you will see a decrease in cost in each iteration.[10]
• Gradient Descent is an optimization algorithm used to find a local minimum of a given function.[11]
• Gradient Descent finds a local minimum, which can be different from the global minimum.[11]
• Gradient Descent needs a function and a starting point as input.[11]
• As we can see, Gradient Descent found a local minimum here, but it is not the global minimum.[11]
• Gradient Descent is an iterative process that finds the minima of a function.[12]
• To get an idea of how Gradient Descent works, let us take an example.[12]
• Now let us see in detail how gradient descent is used to optimise a linear regression problem.[12]
• For simplicity, we take a constant slope of 0.64, so that we can understand how gradient descent would optimise the intercept.[12]
• Gradient descent is an optimization technique that can find the minimum of an objective function.[13]
• Now it's time to run gradient descent to minimize our objective function.[13]
• To keep things simple, let's do a test run of gradient descent on a two-class problem (digit 0 vs. digit 1).[13]
• When running gradient descent, we'll keep learning rate and momentum very small as the inputs are not normalized or standardized.[13]
• This process is called Stochastic Gradient Descent (SGD) (or also sometimes on-line gradient descent).[14]
• Gradient descent is by far the most popular optimization strategy used in machine learning and deep learning at the moment.[15]
• Gradient descent is an optimization algorithm that's used when training a machine learning model.[15]
• Gradient Descent is an optimization algorithm for finding a local minimum of a differentiable function.[15]
• The equation below describes what gradient descent does: b is the next position of our climber, while a represents his current position.[15]
• Gradient descent is an optimization technique commonly used in training machine learning algorithms.[16]
• With gradient descent, you'll simply look around in all possible directions and take a step in the steepest downhill direction.[16]
• Mini batch gradient descent allows us to split our training data into mini batches which can be processed individually.[16]
• On the other extreme, a batch size equal to the number of training examples would represent batch gradient descent.[16]
• This way is called Gradient Descent and it also follow our downhill strategy.[17]
• Gradient Descent is one of the most used machine learning algorithms in the industry.[18]
• And with a goal to reduce the cost function, we modify the parameters by using the Gradient descent algorithm over the given data.[18]
• Gradient descent was originally proposed by CAUCHY in 1847.[18]
• Gradient descent using Contour Plot.[18]
• Gradient descent is an optimization algorithm which is commonly-used to train machine learning models and neural networks.[19]
• Before we dive into gradient descent, it may help to review some concepts from linear regression.[19]
• While gradient descent is the most common approach for optimization problems, it does come with its own set of challenges.[19]
• gradient descent, SGD approximates the true gradient of \(E(w,b)\) by considering a single training example at a time.[20]
• This is where Gradient Descent comes into the picture.[21]
• We are first going to look at the different variants of gradient descent.[22]
• We will also take a short look at algorithms and architectures to optimize gradient descent in a parallel and distributed setting.[22]
• In machine learning, we use gradient descent to update the parameters of our model.[23]
• In this post you discovered gradient descent for machine learning.[24]
• Gradient Descent is an optimizing algorithm used in Machine/ Deep Learning algorithms.[25]
• The first stage in gradient descent is to pick a starting value (a starting point) for \(w_1\).[26]
• Note: When performing gradient descent, we generalize the above process to tune all the model parameters simultaneously.[26]
• Backtracking line search is another variant of gradient descent.[27]
• Gradient descent is a first-order iterative optimization algorithm for finding a local minimum of a differentiable function.[28]
• The basic intuition behind gradient descent can be illustrated by a hypothetical scenario.[28]
• Gradient descent can also be used to solve a system of nonlinear equations.[28]
• Below is an example that shows how to use the gradient descent to solve for three unknown variables, x 1 , x 2 , and x 3 .[28]