Tensor Indexing & Slicing (Part 4)

PyTorch Tensor Indexing & Slicing: Practical Guide with Projects

Tensor indexing and slicing are fundamental PyTorch skills used throughout deep learning. Once you understand how to select, filter, modify, and rearrange tensor data, you can work more effectively with images, text sequences, batches, embeddings, attention masks, labels, and model outputs.

In this practical module, you will combine the indexing techniques learned throughout Module 4 to build several small PyTorch projects.

Projects Covered

Tensor Data Analyzer
Image Channel Extractor
Mini One-Hot Encoder
Transformer Attention Mask Generator
Batch Selection Utility
Sentence Token Filtering
Practical Tensor Filtering Workflows
Interview-Style Tensor Problems

Learning Objectives

After completing this module, you will be able to:

Build practical PyTorch projects using tensor indexing and slicing.
Select individual tensor elements, rows, columns, and sub-tensors.
Filter tensor values using Boolean masks.
Replace tensor values conditionally.
Find matching tensor indices with torch.nonzero().
Use torch.where() and masked_fill() for conditional operations.
Select rows and columns using index tensors.
Generate one-hot encoded labels using scatter_().
Extract image channels using tensor indexing.
Create causal attention masks for Transformer models.
Filter padding tokens from NLP sequences.
Select specific samples from a batch.
Understand how tensor indexing appears in real deep learning pipelines.

Why Tensor Indexing Matters in Deep Learning

Neural networks process large multidimensional tensors rather than individual values.

For example, an image batch may have the shape:

text
1(batch_size, channels, height, width)

A Transformer input may have the shape:

text
1(batch_size, sequence_length, embedding_dimension)

A classification model may produce:

text
1(batch_size, number_of_classes)

In each case, we frequently need to select only part of the tensor.

Typical operations include:

text
1Select a sample
2      ↓
3Select a channel
4      ↓
5Select a sequence
6      ↓
7Filter invalid values
8      ↓
9Modify selected values
10      ↓
11Generate model-ready tensors

Tensor indexing provides the mechanism for performing these operations efficiently.

Project 1: Tensor Data Analyzer

Project Overview

Suppose we have the marks of ten students stored inside a PyTorch tensor.

We want to:

Find the highest mark.
Find the lowest mark.
Calculate the average.
Select students scoring above 80.
Replace failing marks.
Find the indices of high-scoring students.

This project combines reduction operations, Boolean indexing, torch.where(), and torch.nonzero().

Step 1: Import PyTorch

python
1import torch

Step 2: Create the Dataset

python
1marks = torch.tensor([
2    85, 72, 90, 45, 68,
3    95, 77, 82, 39, 88
4])
5
6print(marks)

Output:

text
1tensor([85, 72, 90, 45, 68, 95, 77, 82, 39, 88])

Step 3: Find the Highest Mark

python
1highest = torch.max(marks)
2
3print(highest)

Output:

text
1tensor(95)

If you need the index of the highest mark:

python
1highest_index = torch.argmax(marks)
2
3print(highest_index)

Output:

text
1tensor(5)

The highest score is therefore located at index 5.

Step 4: Find the Lowest Mark

python
1lowest = torch.min(marks)
2
3print(lowest)

Output:

text
1tensor(39)

Step 5: Calculate Average Marks

Because marks is an integer tensor, convert it to a floating-point tensor before calculating the mean.

python
1average = torch.mean(marks.float())
2
3print(average)

Output:

text
1tensor(74.1000)

Step 6: Select Students Scoring Above 80

Boolean indexing makes filtering simple:

python
1high_scores = marks[marks > 80]
2
3print(high_scores)

Output:

text
1tensor([85, 90, 95, 82, 88])

The expression:

python
1marks > 80

creates a Boolean mask:

text
1True
2False
3True
4False
5False
6True
7False
8True
9False
10True

The tensor then uses this mask to select only matching elements.

Step 7: Find the Indices of High Scores

python
1indices = torch.nonzero(marks > 80, as_tuple=True)[0]
2
3print(indices)

Output:

text
1tensor([0, 2, 5, 7, 9])

Using as_tuple=True is often convenient when the result will be used directly for indexing.

For example:

python
1students = marks[indices]
2
3print(students)

Output:

text
1tensor([85, 90, 95, 82, 88])

Step 8: Replace Failing Marks

Suppose a mark below 50 should be replaced with 50.

python
1updated_marks = torch.where(
2    marks < 50,
3    torch.tensor(50),
4    marks
5)
6
7print(updated_marks)

Output:

text
1tensor([85, 72, 90, 50, 68, 95, 77, 82, 50, 88])

A simpler form also works because PyTorch can use a scalar value:

python
1updated_marks = torch.where(
2    marks < 50,
3    50,
4    marks
5)

Complete Tensor Data Analyzer

python
1import torch
2
3marks = torch.tensor([
4    85, 72, 90, 45, 68,
5    95, 77, 82, 39, 88
6])
7
8print("Marks:")
9print(marks)
10
11print("\nHighest:")
12print(torch.max(marks))
13
14print("\nLowest:")
15print(torch.min(marks))
16
17print("\nAverage:")
18print(torch.mean(marks.float()))
19
20print("\nScores Above 80:")
21print(marks[marks > 80])
22
23print("\nIndices Above 80:")
24print(torch.nonzero(marks > 80, as_tuple=True)[0])
25
26print("\nUpdated Marks:")
27print(torch.where(marks < 50, 50, marks))

What This Project Teaches

This project demonstrates several important PyTorch concepts:

text
1torch.max()
2torch.min()
3torch.mean()
4torch.argmax()
5Boolean indexing
6torch.nonzero()
7torch.where()

These techniques are useful when analyzing datasets, preprocessing training data, filtering predictions, and debugging machine learning pipelines.

Project 2: Image Channel Extractor

Project Overview

Images are commonly represented using multiple dimensions.

An RGB image can be represented as:

text
1Height × Width × Channels

For example:

text
1224 × 224 × 3

The three channels represent:

text
10 → Red
21 → Green
32 → Blue

However, many PyTorch computer vision models use the channel-first representation:

text
1Channels × Height × Width

Therefore:

text
1(H, W, C)

can be converted to:

text
1(C, H, W)

using permute().

Note that image libraries and datasets may use different layouts, so always inspect the tensor shape before assuming its dimension order.

Step 1: Create a Fake RGB Image

python
1import torch
2
3image = torch.randint(
4    0,
5    256,
6    (4, 4, 3),
7    dtype=torch.uint8
8)
9
10print(image)
11print(image.shape)

Output shape:

text
1torch.Size([4, 4, 3])

The 256 upper bound is exclusive, so this generates values from 0 through 255.

Step 2: Extract the Red Channel

python
1red = image[:, :, 0]
2
3print(red.shape)

Output:

text
1torch.Size([4, 4])

The indexing means:

text
1:   → select every row
2:   → select every column
30   → select channel 0

Step 3: Extract the Green Channel

python
1green = image[:, :, 1]
2
3print(green.shape)

Step 4: Extract the Blue Channel

python
1blue = image[:, :, 2]
2
3print(blue.shape)

Step 5: Convert HWC to CHW

If the image is currently:

text
1(H, W, C)

convert it to:

text
1(C, H, W)

using:

python
1image_chw = image.permute(2, 0, 1)
2
3print(image_chw.shape)

Output:

text
1torch.Size([3, 4, 4])

Step 6: Add a Batch Dimension

A CNN generally processes a batch of images.

The shape becomes:

text
1(B, C, H, W)

Add the batch dimension:

python
1batch = image_chw.unsqueeze(0)
2
3print(batch.shape)

Output:

text
1torch.Size([1, 3, 4, 4])

The transformation is:

text
1(H, W, C)
2        ↓
3(C, H, W)
4        ↓
5(B, C, H, W)

Complete Image Channel Extractor

python
1import torch
2
3image = torch.randint(
4    0,
5    256,
6    (4, 4, 3),
7    dtype=torch.uint8
8)
9
10print("Original shape:")
11print(image.shape)
12
13red = image[:, :, 0]
14green = image[:, :, 1]
15blue = image[:, :, 2]
16
17print("\nRed shape:")
18print(red.shape)
19
20print("\nGreen shape:")
21print(green.shape)
22
23print("\nBlue shape:")
24print(blue.shape)
25
26image_chw = image.permute(2, 0, 1)
27
28print("\nCHW shape:")
29print(image_chw.shape)
30
31batch = image_chw.unsqueeze(0)
32
33print("\nBatch shape:")
34print(batch.shape)

Project 3: Mini One-Hot Encoder

What Is One-Hot Encoding?

Machine learning models often work with categorical labels.

Suppose we have three classes:

text
10 → Dog
21 → Cat
32 → Bird

Instead of representing:

text
1Dog → 0
2Cat → 1
3Bird → 2

one-hot encoding represents them as:

text
1Dog  → [1, 0, 0]
2Cat  → [0, 1, 0]
3Bird → [0, 0, 1]

Each class receives its own position.

Step 1: Create Labels

python
1import torch
2
3labels = torch.tensor([
4    0,
5    2,
6    1,
7    0
8])
9
10print(labels)

Step 2: Create an Empty One-Hot Tensor

We have:

text
14 samples
23 classes

Therefore:

python
1one_hot = torch.zeros(
2    4,
3    3,
4    dtype=torch.float32
5)

Step 3: Use `scatter_()`

python
1one_hot.scatter_(
2    1,
3    labels.unsqueeze(1),
4    1
5)
6
7print(one_hot)

Output:

text
1tensor([
2    [1., 0., 0.],
3    [0., 0., 1.],
4    [0., 1., 0.],
5    [1., 0., 0.]
6])

Why `unsqueeze(1)`?

The labels initially have shape:

text
1[4]

After:

python
1labels.unsqueeze(1)

the shape becomes:

text
1[4, 1]

This matches the dimensional structure required for the scatter operation.

Complete One-Hot Encoder

python
1import torch
2
3labels = torch.tensor([0, 2, 1, 0])
4
5num_classes = 3
6
7one_hot = torch.zeros(
8    labels.shape[0],
9    num_classes
10)
11
12one_hot.scatter_(
13    1,
14    labels.unsqueeze(1),
15    1
16)
17
18print(one_hot)

This pattern is useful for understanding label encoding, although many PyTorch classification workflows use integer class labels directly with losses such as CrossEntropyLoss rather than manually converting them to one-hot tensors.

Project 4: Transformer Attention Mask Generator

Why Do Transformers Need Attention Masks?

Autoregressive Transformer models must prevent a token from attending to future tokens during generation.

Consider:

text
1I love deep learning

When predicting:

text
1deep

the model should not use information from:

text
1learning

A causal attention pattern looks like:

text
1✓ ✗ ✗ ✗
2✓ ✓ ✗ ✗
3✓ ✓ ✓ ✗
4✓ ✓ ✓ ✓

The lower-triangular region is allowed, while positions representing future tokens are masked.

Step 1: Create a Causal Mask

python
1import torch
2
3sequence_length = 4
4
5mask = torch.triu(
6    torch.ones(
7        sequence_length,
8        sequence_length,
9        dtype=torch.bool
10    ),
11    diagonal=1
12)
13
14print(mask)

Output:

text
1tensor([
2    [False,  True,  True,  True],
3    [False, False,  True,  True],
4    [False, False, False,  True],
5    [False, False, False, False]
6])

Here:

text
1False → allowed
2True  → masked

Step 2: Apply the Mask to Attention Scores

Suppose the model has attention scores:

python
1scores = torch.randn(4, 4)

Apply the causal mask:

python
1masked_scores = scores.masked_fill(
2    mask,
3    float("-inf")
4)
5
6print(masked_scores)

The future-token positions become:

text
1-inf

When these scores are passed through softmax, those positions receive zero attention probability.

For example:

python
1attention_weights = torch.softmax(
2    masked_scores,
3    dim=-1
4)

The result prevents attention to future positions.

Important Distinction: Boolean Mask Meaning

Different PyTorch APIs may interpret boolean masks differently depending on the API.

For example, with:

python
1scores.masked_fill(mask, float("-inf"))

True means:

text
1replace this position

Therefore, always verify what True and False mean for the specific operation you are using.

Complete Causal Mask Generator

python
1import torch
2
3sequence_length = 4
4
5causal_mask = torch.triu(
6    torch.ones(
7        sequence_length,
8        sequence_length,
9        dtype=torch.bool
10    ),
11    diagonal=1
12)
13
14print("Causal Mask:")
15print(causal_mask)
16
17scores = torch.randn(
18    sequence_length,
19    sequence_length
20)
21
22masked_scores = scores.masked_fill(
23    causal_mask,
24    float("-inf")
25)
26
27attention_weights = torch.softmax(
28    masked_scores,
29    dim=-1
30)
31
32print("\nAttention Weights:")
33print(attention_weights)

Project 5: Batch Selection Utility

Deep learning models usually process multiple samples simultaneously.

Suppose we have:

text
164 images
23 channels
3224 height
4224 width

The tensor shape is:

text
1(64, 3, 224, 224)

Create a Batch

python
1import torch
2
3images = torch.randn(
4    64,
5    3,
6    224,
7    224
8)
9
10print(images.shape)

Output:

text
1torch.Size([64, 3, 224, 224])

Select the First 16 Images

python
1batch = images[:16]
2
3print(batch.shape)

Output:

text
1torch.Size([16, 3, 224, 224])

Select the Last 16 Images

python
1batch = images[-16:]
2
3print(batch.shape)

Output:

text
1torch.Size([16, 3, 224, 224])

Select Specific Images

python
1indices = torch.tensor([
2    2,
3    5,
4    12,
5    31
6])
7
8selected = images[indices]
9
10print(selected.shape)

Output:

text
1torch.Size([4, 3, 224, 224])

This is an example of advanced or fancy indexing.

Project 6: Sentence Token Filtering

NLP models often process padded sequences.

Suppose:

text
10 = padding token

and the sequence is:

python
1tokens = torch.tensor([
2    101,
3    203,
4    0,
5    0,
6    345,
7    567,
8    0
9])

Remove Padding Tokens

python
1filtered = tokens[tokens != 0]
2
3print(filtered)

Output:

text
1tensor([101, 203, 345, 567])

The Boolean mask is:

python
1tokens != 0

which selects only non-padding tokens.

Using `masked_select()`

The same operation can be written as:

python
1filtered = torch.masked_select(
2    tokens,
3    tokens != 0
4)
5
6print(filtered)

Output:

text
1tensor([101, 203, 345, 567])

For simple filtering, Boolean indexing is usually easier to read.

Project 7: Tensor Filtering Pipeline

A common preprocessing pipeline can combine several indexing operations.

Suppose we have model predictions:

python
1scores = torch.tensor([
2    0.2,
3    0.9,
4    0.4,
5    0.95,
6    0.7
7])

We want predictions above a confidence threshold of 0.8.

Create a Mask

python
1mask = scores > 0.8
2
3print(mask)

Output:

text
1tensor([False, True, False, True, False])

Select High-Confidence Predictions

python
1high_confidence = scores[mask]
2
3print(high_confidence)

Output:

text
1tensor([0.9000, 0.9500])

Find Their Indices

python
1indices = torch.nonzero(
2    mask,
3    as_tuple=True
4)[0]
5
6print(indices)

Output:

text
1tensor([1, 3])

This type of workflow appears in object detection, classification, ranking, and recommendation systems.

Practical Tensor Indexing Patterns

Select the First Sample

python
1sample = batch[0]

Select the First Ten Samples

python
1samples = batch[:10]

Select the Last Sample

python
1sample = batch[-1]

Select a Specific Channel

python
1channel = image[0]

Select a Specific Pixel

For a channel-first image:

python
1pixel = image[0, 100, 100]

Select a Region of an Image

python
1crop = image[:, 50:150, 50:150]

Select a Sequence Segment

python
1sequence = input_ids[:, :128]

Select Specific Batch Elements

python
1indices = torch.tensor([0, 4, 8])
2
3selected = batch[indices]

Filter Positive Values

python
1positive = x[x > 0]

Replace Negative Values

python
1x = torch.where(x < 0, 0, x)

Find Matching Indices

python
1indices = torch.nonzero(
2    x > 10,
3    as_tuple=True
4)[0]

Real-World Applications

Tensor indexing and slicing appear throughout modern machine learning systems.

Field	Tensor Indexing Application
Computer Vision	Image cropping and channel extraction
CNNs	Batch and feature-map selection
NLP	Token filtering and sequence slicing
Transformers	Attention and causal masks
Classification	Selecting class predictions
Object Detection	Confidence-based filtering
Segmentation	Pixel and region masking
Reinforcement Learning	Selecting actions and Q-values
Recommendation Systems	Selecting user/item embeddings
Medical Imaging	Selecting image or volume regions
Time Series	Selecting temporal windows
Multimodal AI	Selecting image and text features

Interview Questions

Q1. What is the difference between indexing and slicing?

Indexing accesses a specific position.

python
1x[2]

Slicing selects a range.

python
1x[2:5]

For example:

text
1Indexing
2x[2]
3    ↓
4one element
5
6Slicing
7x[2:5]
8    ↓
9multiple elements

Q2. What is Boolean indexing?

Boolean indexing uses a Boolean tensor as a mask.

Example:

python
1x = torch.tensor([5, 10, 15, 20])
2
3result = x[x > 10]
4
5print(result)

Output:

text
1tensor([15, 20])

Only elements whose mask value is True are selected.

Q3. What is the difference between `masked_fill()` and `torch.where()`?

masked_fill() replaces positions where a mask is True with a specified value.

python
1x.masked_fill(mask, 0)

torch.where() chooses between two values based on a condition.

python
1torch.where(condition, value_if_true, value_if_false)

For example:

python
1result = torch.where(
2    x > 10,
3    100,
4    x
5)

Q4. What is the difference between `gather()` and `scatter()`?

gather() reads values according to indices.

text
1gather
2   ↓
3read
4   ↓
5output

scatter() writes values according to indices.

text
1source
2   ↓
3scatter
4   ↓
5target

A useful mental model is:

text
1gather  → collect
2scatter → place

Q5. Why is `unsqueeze()` used before `scatter_()` in one-hot encoding?

Suppose:

python
1labels.shape

is:

text
1[4]

The target one-hot tensor has shape:

text
1[4, 3]

Adding a dimension:

python
1labels.unsqueeze(1)

produces:

text
1[4, 1]

This provides the index structure needed for the scatter operation.

Q6. What does `torch.nonzero()` return?

It returns the indices of non-zero elements or elements satisfying a Boolean condition.

Example:

python
1x = torch.tensor([0, 5, 0, 8])
2
3indices = torch.nonzero(x)
4
5print(indices)

Output:

text
1tensor([
2    [1],
3    [3]
4])

For easier direct indexing, you can use:

python
1torch.nonzero(
2    x > 0,
3    as_tuple=True
4)[0]

Q7. Why do Transformer models use causal masks?

Autoregressive Transformers must prevent a token from accessing future tokens during training or generation.

A causal mask creates a structure such as:

text
1✓ ✗ ✗ ✗
2✓ ✓ ✗ ✗
3✓ ✓ ✓ ✗
4✓ ✓ ✓ ✓

This preserves the left-to-right dependency required for autoregressive prediction.

Q8. Why can `torch.mean()` fail on an integer tensor?

Some reduction operations, including torch.mean(), require a floating-point or complex dtype.

Therefore:

python
1x = torch.tensor([10, 20, 30])
2
3torch.mean(x)

can fail for an integer tensor.

Use:

python
1torch.mean(x.float())

instead.

Q9. What is the difference between `permute()` and indexing?

Indexing selects data.

python
1image[:, :, 0]

selects one channel.

permute() changes the order of dimensions.

python
1image.permute(2, 0, 1)

changes:

text
1(H, W, C)

into:

text
1(C, H, W)

Q10. Does Boolean indexing preserve the original tensor shape?

Not necessarily.

For example:

python
1A = torch.tensor([
2    [1, 2],
3    [3, 4]
4])
5
6print(A[A > 1])

Output:

text
1tensor([2, 3, 4])

The selected elements are returned as a one-dimensional tensor.

Common Mistakes

Mistake 1: Forgetting Floating-Point Conversion for Mean

Incorrect:

python
1marks = torch.tensor([80, 90])
2
3average = torch.mean(marks)

Use:

python
1average = torch.mean(marks.float())

Mistake 2: Using the Wrong Image Dimension

Do not assume every image tensor uses the same dimension order.

First inspect:

python
1print(image.shape)

If the image is:

text
1(H, W, C)

and your model expects:

text
1(C, H, W)

use:

python
1image = image.permute(2, 0, 1)

Mistake 3: Incorrect `scatter_()` Index Shape

Incorrect assumptions about the shape of the index tensor can cause scatter errors.

For one-hot encoding:

python
1labels.unsqueeze(1)

is commonly required.

Mistake 4: Confusing `gather()` and `scatter()`

Remember:

text
1gather  → reads
2scatter → writes

Mistake 5: Forgetting That Slice End Is Exclusive

python
1x[1:4]

selects:

text
11
22
33

It does not include index 4.

Mistake 6: Using Python `and` Instead of Tensor `&`

Incorrect:

python
1mask = (x > 5) and (x < 20)

Correct:

python
1mask = (x > 5) & (x < 20)

Use parentheses around individual conditions.

Mistake 7: Modifying a Tensor That Requires Gradients

In model training, direct in-place modifications can interact with autograd.

For example:

python
1x[x > 0] = 0

may be problematic when x participates in a computation graph requiring gradients.

When working with tensors that require gradients, understand whether an operation is in-place and whether autograd needs the original tensor values.

Mistake 8: Confusing Boolean Mask Semantics

For:

python
1x.masked_fill(mask, 0)

True means:

text
1replace this position

But other PyTorch APIs may define mask semantics differently. Always check the operation's expected mask meaning.

Best Practices for PyTorch Tensor Indexing

1. Inspect Tensor Shapes

Before complex indexing:

python
1print(tensor.shape)

Shape awareness prevents many errors.

2. Use Boolean Indexing for Simple Filtering

Instead of complicated loops:

python
1positive = x[x > 0]

3. Use `torch.where()` for Conditional Replacement

python
1result = torch.where(
2    x < 0,
3    0,
4    x
5)

4. Use `masked_fill()` for Mask-Based Replacement

python
1scores = scores.masked_fill(
2    mask,
3    float("-inf")
4)

This is especially useful for attention scores.

5. Use `index_select()` for Explicit Dimension-Based Selection

python
1rows = torch.tensor([0, 2])
2
3selected = torch.index_select(
4    A,
5    0,
6    rows
7)

6. Use `gather()` When Values Depend on Indices

gather() is useful when each row or batch element has its own index selection.

7. Use `scatter()` for Index-Based Placement

One-hot encoding is a classic example:

python
1one_hot.scatter_(
2    1,
3    labels.unsqueeze(1),
4    1
5)

8. Avoid Unnecessary Python Loops

Tensor operations are generally preferable for large datasets because PyTorch can execute them efficiently on CPUs and GPUs.

Instead of manually processing every value:

python
1for value in x:
2    ...

look for a vectorized tensor operation.

9. Be Careful with In-Place Operations

Operations ending with _, such as:

python
1scatter_()
2masked_fill_()

modify tensors in place.

Use them deliberately, especially when autograd is involved.

10. Keep Device and Dtype Consistent

Before combining tensors:

python
1print(x.device)
2print(x.dtype)

For example, index tensors and source tensors may need compatible devices and appropriate dtypes.

Practice Exercises

Beginner Exercise 1: Basic Tensor Filtering

Create:

python
1import torch
2
3x = torch.arange(1, 21)

Perform the following:

Select all even numbers.
Select all numbers greater than 10.
Select numbers between 5 and 15.
Replace numbers greater than 15 with 0.
Find the indices of numbers divisible by 3.
Reverse the tensor using torch.flip().

Beginner Exercise 2: Matrix Indexing

Create:

python
1A = torch.tensor([
2    [11, 12, 13],
3    [21, 22, 23],
4    [31, 32, 33]
5])

Perform:

Select 22.
Select the last row.
Select the second column.
Extract the top-left 2 × 2 matrix.
Extract the bottom-right 2 × 2 matrix.
Replace values greater than 20 with 0.

Intermediate Exercise 3: Batch Processing

Create:

python
1images = torch.randn(
2    32,
3    3,
4    64,
5    64
6)

Perform:

Select the first eight images.
Select images at indices [2, 7, 15].
Select the red channel.
Extract a 32 × 32 center crop.
Verify every resulting tensor shape.

Intermediate Exercise 4: One-Hot Encoding

Create:

python
1labels = torch.tensor([
2    2, 0, 1, 2, 1
3])

Assume there are three classes.

Create the one-hot representation using:

python
1scatter_()

Verify the output shape.

Advanced Exercise 5: Transformer Causal Mask

Create a sequence length of:

text
18

Generate a causal mask using:

python
1torch.triu()

Then:

Convert the mask to Boolean.
Generate random attention scores.
Apply masked_fill().
Apply softmax.
Verify that future-token probabilities are zero.

Advanced Exercise 6: Transformer Tensor Indexing

Create a tensor representing:

text
1batch_size = 4
2sequence_length = 8
3embedding_dimension = 16

python
1x = torch.randn(
2    4,
3    8,
4    16
5)

Perform:

Select the first sequence.
Select the first four tokens.
Select embedding dimensions 0 through 7.
Select batch elements 0 and 2.
Verify every resulting shape.

Mini Challenge: Build a Prediction Filter

Create:

python
1scores = torch.tensor([
2    0.15,
3    0.92,
4    0.73,
5    0.97,
6    0.41,
7    0.88
8])

Write a program that:

Creates a confidence mask using threshold 0.80.
Selects high-confidence predictions.
Finds their original indices.
Replaces low-confidence scores with 0.
Prints all results.

Expected high-confidence scores:

text
10.92
20.97
30.88

Module 4 Summary

Congratulations! You have completed Module 4: Tensor Indexing & Slicing.

You have now learned how PyTorch tensors can be accessed, filtered, modified, and selected using a wide range of indexing techniques.

The major concepts covered include:

Positive indexing
Negative indexing
Tensor slicing
Step slicing
Multi-dimensional indexing
Boolean masking
Advanced indexing
torch.where()
torch.nonzero()
torch.index_select()
torch.gather()
Tensor assignment
masked_fill()
masked_select()
scatter()
scatter_add()
torch.take()
torch.diagonal()
torch.triu()
torch.tril()
permute()
reshape()
flatten()
unsqueeze()
Batch selection
Image channel extraction
Token filtering
One-hot encoding
Transformer causal masking

These operations form an important part of the practical PyTorch programming workflow.

They are used when preparing datasets, processing images, manipulating embeddings, filtering predictions, creating attention masks, selecting batches, and implementing deep learning algorithms.

A strong understanding of tensor indexing also makes advanced PyTorch code much easier to read and debug.

Real-World PyTorch Workflow

A typical deep learning pipeline may look like:

text
1Raw Dataset
2     ↓
3Tensor Creation
4     ↓
5Tensor Indexing
6     ↓
7Filtering
8     ↓
9Reshaping
10     ↓
11Batch Construction
12     ↓
13Model Input
14     ↓
15Neural Network
16     ↓
17Predictions
18     ↓
19Indexing / Filtering
20     ↓
21Final Results

Tensor indexing appears throughout this entire pipeline.

Next Module: Module 5 — Tensor Mathematics & Linear Algebra

The next module moves from tensor manipulation to the mathematical operations that power neural networks, computer vision, Transformers, and large language models.

Topics Covered

Scalar Mathematics
Vector Mathematics
Matrix Mathematics
Tensor Mathematics
Matrix Multiplication
torch.matmul()
Dot Product
Inner Product
Outer Product
Cross Product
Matrix Inverse
Matrix Determinant
Matrix Rank
Matrix Trace
Eigenvalues
Eigenvectors
Singular Value Decomposition
SVD
L1 Norm
L2 Norm
Frobenius Norm
Vector and Matrix Distances
Euclidean Distance
Cosine Similarity
Batch Matrix Multiplication
torch.bmm()
Einstein Summation
torch.einsum()
Attention Score Computation
Similarity Search
PCA Fundamentals
Practical Linear Algebra Projects

These mathematical operations provide the foundation for understanding:

text
1Neural Networks
2      ↓
3CNNs
4      ↓
5RNNs
6      ↓
7Attention
8      ↓
9Transformers
10      ↓
11Vision Transformers
12      ↓
13Large Language Models
14      ↓
15Modern AI Architectures

PyTorch Tensor Indexing & Slicing: Practical Guide with Projects

Projects Covered

Learning Objectives

Why Tensor Indexing Matters in Deep Learning

Project 1: Tensor Data Analyzer

Project Overview

Step 1: Import PyTorch

Step 2: Create the Dataset

Step 3: Find the Highest Mark

Step 4: Find the Lowest Mark

Step 5: Calculate Average Marks

Step 6: Select Students Scoring Above 80

Step 7: Find the Indices of High Scores

Step 8: Replace Failing Marks

Complete Tensor Data Analyzer

What This Project Teaches

Project 2: Image Channel Extractor

Project Overview

Step 1: Create a Fake RGB Image

Step 2: Extract the Red Channel

Step 3: Extract the Green Channel

Step 4: Extract the Blue Channel

Step 5: Convert HWC to CHW

Step 6: Add a Batch Dimension

Complete Image Channel Extractor

Project 3: Mini One-Hot Encoder

What Is One-Hot Encoding?

Step 1: Create Labels

Step 2: Create an Empty One-Hot Tensor

Step 3: Use scatter_()

Why unsqueeze(1)?

Complete One-Hot Encoder

Project 4: Transformer Attention Mask Generator

Why Do Transformers Need Attention Masks?

Step 1: Create a Causal Mask

Step 2: Apply the Mask to Attention Scores

Important Distinction: Boolean Mask Meaning

Complete Causal Mask Generator

Project 5: Batch Selection Utility

Create a Batch

Select the First 16 Images

Select the Last 16 Images

Select Specific Images

Project 6: Sentence Token Filtering

Remove Padding Tokens

Using masked_select()

Project 7: Tensor Filtering Pipeline

Create a Mask

Select High-Confidence Predictions

Find Their Indices

Practical Tensor Indexing Patterns

Select the First Sample

Select the First Ten Samples

Select the Last Sample

Select a Specific Channel

Select a Specific Pixel

Select a Region of an Image

Select a Sequence Segment

Select Specific Batch Elements

Filter Positive Values

Replace Negative Values

Find Matching Indices

Real-World Applications

Interview Questions

Q1. What is the difference between indexing and slicing?

Q2. What is Boolean indexing?

Q3. What is the difference between masked_fill() and torch.where()?

Q4. What is the difference between gather() and scatter()?

Q5. Why is unsqueeze() used before scatter_() in one-hot encoding?

Q6. What does torch.nonzero() return?

Q7. Why do Transformer models use causal masks?

Q8. Why can torch.mean() fail on an integer tensor?

Q9. What is the difference between permute() and indexing?

Q10. Does Boolean indexing preserve the original tensor shape?

Common Mistakes

Mistake 1: Forgetting Floating-Point Conversion for Mean

Mistake 2: Using the Wrong Image Dimension

Mistake 3: Incorrect scatter_() Index Shape

Mistake 4: Confusing gather() and scatter()

Mistake 5: Forgetting That Slice End Is Exclusive

Step 3: Use `scatter_()`

Why `unsqueeze(1)`?

Using `masked_select()`

Q3. What is the difference between `masked_fill()` and `torch.where()`?

Q4. What is the difference between `gather()` and `scatter()`?

Q5. Why is `unsqueeze()` used before `scatter_()` in one-hot encoding?

Q6. What does `torch.nonzero()` return?

Q8. Why can `torch.mean()` fail on an integer tensor?

Q9. What is the difference between `permute()` and indexing?

Mistake 3: Incorrect `scatter_()` Index Shape

Mistake 4: Confusing `gather()` and `scatter()`

Mistake 6: Using Python `and` Instead of Tensor `&`

3. Use `torch.where()` for Conditional Replacement

4. Use `masked_fill()` for Mask-Based Replacement

5. Use `index_select()` for Explicit Dimension-Based Selection

6. Use `gather()` When Values Depend on Indices

7. Use `scatter()` for Index-Based Placement