Module 3 — Calculus for Deep Learning

Calculus is one of the most important mathematical foundations of Deep Learning. Every neural network, including Transformer models, relies on derivatives and gradients for optimization through backpropagation.

Topics

Limits
Derivatives
Partial Derivatives
Chain Rule
Gradient
Jacobian
Hessian
Taylor Expansion

1. Limits

A limit describes the value that a function approaches as the input approaches a particular value.

Formula

\lim_{x\rightarrow a}f(x)

Example

\lim_{x\rightarrow0} \frac{\sin x}{x}=1

2. Derivatives

The derivative measures the rate of change of a function.

Formula

\frac{dy}{dx}

Definition

\frac{dy}{dx} = \lim_{h\rightarrow0} \frac{f(x+h)-f(x)}{h}

Common Derivatives

\frac{d}{dx}(x^n)=nx^{n-1}

\frac{d}{dx}(e^x)=e^x

\frac{d}{dx}(\ln x)=\frac1x

\frac{d}{dx}(\sin x)=\cos x

\frac{d}{dx}(\cos x)=-\sin x

3. Partial Derivatives

A partial derivative measures the change in a multivariable function with respect to one variable while keeping the others constant.

Formula

\frac{\partial f}{\partial x}

Example

f(x,y)=x^2+y^3

then

\frac{\partial f}{\partial x}=2x

\frac{\partial f}{\partial y}=3y^2

4. Chain Rule

The chain rule computes derivatives of composite functions.

Formula

\frac{dz}{dx} = \frac{dz}{dy} \cdot \frac{dy}{dx}

Multiple Variables

\frac{\partial z}{\partial x} = \frac{\partial z}{\partial u} \cdot \frac{\partial u}{\partial x}

This rule is the mathematical foundation of backpropagation.

5. Gradient

The gradient is a vector containing all partial derivatives of a function.

Formula

\nabla f(x) = \left[ \frac{\partial f}{\partial x_1}, \frac{\partial f}{\partial x_2}, \ldots, \frac{\partial f}{\partial x_n} \right]^T

Example

f(x,y)=x^2+y^2

\nabla f= \begin{bmatrix} 2x\\ 2y \end{bmatrix}

6. Jacobian Matrix

The Jacobian represents the first-order partial derivatives of a vector-valued function.

Formula

J= \frac{\partial(y_1,\ldots,y_m)} {\partial(x_1,\ldots,x_n)}

Expanded Form

J= \begin{bmatrix} \frac{\partial y_1}{\partial x_1} & \cdots & \frac{\partial y_1}{\partial x_n} \\ \vdots & \ddots & \vdots \\ \frac{\partial y_m}{\partial x_1} & \cdots & \frac{\partial y_m}{\partial x_n} \end{bmatrix}

7. Hessian Matrix

The Hessian is a matrix of second-order partial derivatives.

Formula

H= \nabla^2f(x)

Expanded Form

H= \begin{bmatrix} \frac{\partial^2f}{\partial x_1^2} & \frac{\partial^2f}{\partial x_1\partial x_2} \\ \frac{\partial^2f}{\partial x_2\partial x_1} & \frac{\partial^2f}{\partial x_2^2} \end{bmatrix}

The Hessian is widely used in optimization and curvature analysis.

8. Taylor Expansion

Taylor expansion approximates a function around a given point.

First-Order Taylor Expansion

f(x) \approx f(a) + f'(a)(x-a)

Second-Order Taylor Expansion

f(x) \approx f(a) + f'(a)(x-a) + \frac{f''(a)}{2!}(x-a)^2

General Taylor Series

f(x) = \sum_{n=0}^{\infty} \frac{f^{(n)}(a)} {n!} (x-a)^n

Why Calculus is Important in Transformers

Calculus enables Transformer models to learn by optimizing millions or even billions of parameters.

It is used in:

Gradient Descent
Backpropagation
Adam Optimizer
Loss Function Optimization
Neural Network Training
Attention Weight Updates
Large Language Models (LLMs)

Summary

Concept	Formula
Limit	$\lim_{x\rightarrow a}f(x)$
Derivative	$\dfrac{dy}{dx}$