Tensor Indexing & Slicing part3

Tensor Assignment, Masked Operations, Scatter, Diagonal & Triangular Tensors

Topics Covered

• Tensor assignment and in-place updates
• masked_fill() and masked_fill_()
• masked_select()
• scatter() and scatter_()
• scatter_add() and scatter_add_()
• torch.take()
• torch.diagonal()
• torch.triu()
• torch.tril()
• Transformer causal attention masks
• One-hot encoding with scatter operations
• Practical PyTorch tensor indexing techniques

Learning Objectives

After completing this chapter, you will be able to:

• Modify individual tensor elements and tensor regions.
• Assign values using integer indexing and slices.
• Replace tensor values conditionally with masked_fill().
• Extract elements using boolean masks with masked_select().
• Write values into tensors using scatter().
• Accumulate values using scatter_add().
• Understand the difference between gather() and scatter().
• Extract diagonal elements from matrices and higher-dimensional tensors.
• Create upper and lower triangular tensors.
• Build causal masks used by Transformer models.
• Apply advanced tensor indexing techniques in practical deep learning workflows.

Introduction

Tensor indexing is not only about reading values from a tensor. In real PyTorch applications, we frequently need to modify, filter, route, accumulate, and mask tensor data.

For example, a deep learning model may need to:

• Replace invalid values.
• Mask padding tokens.
• Prevent a Transformer from attending to future tokens.
• Convert class labels into one-hot vectors.
• Update selected positions in a tensor.
• Accumulate values at repeated indices.
• Extract diagonal values from attention or similarity matrices.
• Create upper or lower triangular matrices.

PyTorch provides specialized tensor indexing and manipulation operations for these tasks.

The operations covered in this chapter are especially important for PyTorch deep learning, Transformer architectures, attention mechanisms, NLP, computer vision, and tensor data processing.

Tensor Assignment

Tensor assignment directly changes values stored in a tensor.

This is useful when you know exactly which positions need to be updated.

Assigning a Single Element

pythonCopy
1import torch
2
3x = torch.tensor([10, 20, 30])
4
5x[1] = 99
6
7print(x)

Output:

textCopy
1tensor([10, 99, 30])

The element at index 1 changed from 20 to 99.

Assigning Multiple Elements

A slice can be replaced with another tensor.

pythonCopy
1x = torch.tensor([1, 2, 3, 4, 5])
2
3x[1:4] = torch.tensor([20, 30, 40])
4
5print(x)

Output:

textCopy
1tensor([ 1, 20, 30, 40,  5])

The slice x[1:4] contains indices 1, 2, and 3.

Assigning a Scalar to a Slice

PyTorch can broadcast a scalar across the selected region.

pythonCopy
1x = torch.tensor([1, 2, 3, 4, 5])
2
3x[1:4] = 0
4
5print(x)

Output:

textCopy
1tensor([1, 0, 0, 0, 5])

This is a simple example of using broadcasting during tensor assignment.

Assigning Rows and Columns

Consider a two-dimensional tensor:

pythonCopy
1A = torch.tensor([
2    [1, 2],
3    [3, 4]
4])

Replace an Entire Row

pythonCopy
1A[0] = torch.tensor([9, 9])
2
3print(A)

Output:

textCopy
1tensor([
2    [9, 9],
3    [3, 4]
4])

Replace an Entire Column

pythonCopy
1A[:, 1] = torch.tensor([100, 200])
2
3print(A)

Output:

textCopy
1tensor([
2    [  9, 100],
3    [  3, 200]
4])

The expression:

pythonCopy
1A[:, 1]

means:

• : → select every row
• 1 → select column 1

Assignment with Boolean Masks

Boolean masking can be combined with assignment.

pythonCopy
1x = torch.tensor([5, 10, 15, 20])
2
3x[x > 10] = 999
4
5print(x)

Output:

textCopy
1tensor([  5,  10, 999, 999])

Every value greater than 10 is replaced with 999.

This technique is useful for:

• Removing invalid values.
• Clipping data.
• Replacing outliers.
• Cleaning tensors.
• Preprocessing machine learning data.

masked_fill()

masked_fill() replaces values wherever a Boolean mask is True.

Basic syntax:

pythonCopy
1tensor.masked_fill(mask, value)

The operation returns a new tensor.

Basic Example

pythonCopy
1import torch
2
3x = torch.tensor([5, 10, 15, 20])
4
5mask = x > 10
6
7result = x.masked_fill(mask, 0)
8
9print(result)

Output:

textCopy
1tensor([ 5, 10,  0,  0])

The mask is:

textCopy
1[False, False, True, True]

Therefore, only the last two values are replaced.

Replacing Negative Values

pythonCopy
1x = torch.tensor([-5, 2, -3, 8])
2
3result = x.masked_fill(x < 0, 0)
4
5print(result)

Output:

textCopy
1tensor([0, 2, 0, 8])

This is similar to replacing negative values with zero during preprocessing.

Replacing Large Values

pythonCopy
1x = torch.tensor([20, 50, 100])
2
3result = x.masked_fill(x > 30, -1)
4
5print(result)

Output:

textCopy
1tensor([20, -1, -1])

masked_fill() vs masked_fill_()

There are two forms:

pythonCopy
1x.masked_fill(mask, value)

and:

pythonCopy
1x.masked_fill_(mask, value)

The version ending with _ performs the operation in-place.

Example:

pythonCopy
1x = torch.tensor([5, 10, 15])
2
3mask = x > 10
4
5y = x.masked_fill(mask, 0)
6
7print(x)
8print(y)

x remains unchanged, while y contains the masked result.

With the in-place version:

pythonCopy
1x = torch.tensor([5, 10, 15])
2
3x.masked_fill_(x > 10, 0)
4
5print(x)

Output:

textCopy
1tensor([5, 10, 0])

In-place operations should be used carefully when tensors participate in autograd computations because modifying values needed for gradient calculation can cause errors.

masked_select()

masked_select() extracts all elements for which the Boolean mask is True.

Syntax:

pythonCopy
1torch.masked_select(input, mask)

Basic Example

pythonCopy
1import torch
2
3x = torch.tensor([10, 20, 30, 40])
4
5mask = x > 20
6
7result = torch.masked_select(x, mask)
8
9print(result)

Output:

textCopy
1tensor([30, 40])

The same operation can often be written more simply as:

pythonCopy
1result = x[x > 20]

Matrix Masking with masked_select()

pythonCopy
1A = torch.tensor([
2    [5, 10],
3    [15, 20]
4])
5
6result = torch.masked_select(A, A > 10)
7
8print(result)

Output:

textCopy
1tensor([15, 20])

An important detail is that masked_select() returns a 1D tensor.

It does not preserve the original matrix structure.

For example:

textCopy
1Input:
2
35   10
415  20
5
6Mask:
7
8False False
9True  True
10
11Result:
12
1315 20

The actual PyTorch result is:

textCopy
1tensor([15, 20])

masked_fill() vs masked_select()

A useful rule is:

textCopy
1masked_fill()
2      ↓
3Modify selected positions
4
5masked_select()
6      ↓
7Extract selected positions

scatter()

scatter() writes values into a tensor using an index tensor.

A useful way to remember the difference is:

textCopy
1gather()  → read values using indices
2
3scatter() → write values using indices

Basic syntax:

pythonCopy
1torch.scatter(input, dim, index, src)

The method form is also commonly used:

pythonCopy
1tensor.scatter(dim, index, src)

Basic Example

pythonCopy
1import torch
2
3target = torch.zeros(3, 5)
4
5index = torch.tensor([
6    [0],
7    [2],
8    [4]
9])
10
11src = torch.tensor([
12    [10],
13    [20],
14    [30]
15])
16
17result = target.scatter(1, index, src)
18
19print(result)

Output:

textCopy
1tensor([
2    [10.,  0.,  0.,  0.,  0.],
3    [ 0.,  0., 20.,  0.,  0.],
4    [ 0.,  0.,  0.,  0., 30.]
5])

Here:

textCopy
1dim = 1

means that the indices identify positions along the column dimension.

The operation can be understood as:

textCopy
1Row 0 → column 0 = 10
2
3Row 1 → column 2 = 20
4
5Row 2 → column 4 = 30

Understanding scatter() Step by Step

Start with:

textCopy
10 0 0 0 0
20 0 0 0 0
30 0 0 0 0

Indices:

textCopy
10
22
34

Values:

textCopy
110
220
330

After scattering:

textCopy
110 0  0 0  0
20  0 20 0  0
30  0  0 0 30

This makes scatter() extremely useful when values must be placed at specific tensor positions.

In-place scatter_()

PyTorch also provides:

pythonCopy
1scatter_()

The underscore means that the operation modifies the destination tensor in-place.

Example:

pythonCopy
1target = torch.zeros(3, 5)
2
3index = torch.tensor([
4    [0],
5    [2],
6    [4]
7])
8
9src = torch.tensor([
10    [10],
11    [20],
12    [30]
13])
14
15target.scatter_(1, index, src)
16
17print(target)

One-Hot Encoding with scatter_()

One-hot encoding converts class labels into binary class vectors.

Suppose we have three classes:

textCopy
1Class 0
2Class 1
3Class 2

And the labels are:

pythonCopy
1labels = torch.tensor([0, 2, 1])

We can create one-hot vectors using scatter_().

pythonCopy
1import torch
2
3labels = torch.tensor([0, 2, 1])
4
5one_hot = torch.zeros(3, 3)
6
7one_hot.scatter_(1, labels.unsqueeze(1), 1)
8
9print(one_hot)

Output:

textCopy
1tensor([
2    [1., 0., 0.],
3    [0., 0., 1.],
4    [0., 1., 0.]
5])

The important part is:

pythonCopy
1labels.unsqueeze(1)

The labels originally have shape:

textCopy
1(3,)

After unsqueeze(1):

textCopy
1(3, 1)

This gives the index tensor the appropriate shape for the scatter operation.

scatter_add()

scatter_add() is similar to scatter(), but instead of replacing values, it adds values at the specified indices.

This becomes especially important when multiple source values target the same destination position.

Example

pythonCopy
1import torch
2
3target = torch.zeros(3)
4
5index = torch.tensor([0, 1, 1])
6
7src = torch.tensor([5, 10, 20])
8
9result = target.scatter_add(0, index, src)
10
11print(result)

Output:

textCopy
1tensor([ 5., 30.,  0.])

Why does index 1 contain 30?

Because both 10 and 20 were assigned to index 1:

textCopy
110 + 20 = 30

The operation is therefore:

textCopy
1Index 0 → 5
2
3Index 1 → 10 + 20 = 30
4
5Index 2 → 0

In-place scatter_add_()

The in-place version modifies the original tensor:

pythonCopy
1target = torch.zeros(3)
2
3index = torch.tensor([0, 1, 1])
4
5src = torch.tensor([5, 10, 20])
6
7target.scatter_add_(0, index, src)
8
9print(target)

Output:

textCopy
1tensor([ 5., 30.,  0.])

scatter() vs scatter_add()

A useful mental model is:

textCopy
1scatter
2index → position
3value → write
4
5scatter_add
6index → position
7value → accumulate

torch.take()

torch.take() selects values from a tensor using flattened indices.

Syntax:

pythonCopy
1torch.take(input, indices)

Example

pythonCopy
1import torch
2
3A = torch.tensor([
4    [10, 20],
5    [30, 40]
6])
7
8indices = torch.tensor([0, 3])
9
10result = torch.take(A, indices)
11
12print(result)

Output:

textCopy
1tensor([10, 40])

Conceptually, PyTorch treats the matrix as flattened:

textCopy
110 20 30 40

The flattened indices are:

textCopy
10 → 10
21 → 20
32 → 30
43 → 40

Therefore:

textCopy
1[0, 3]

selects:

textCopy
1[10, 40]

Extracting the Main Diagonal

torch.diagonal() extracts diagonal elements from a tensor.

For a two-dimensional matrix:

pythonCopy
1import torch
2
3A = torch.tensor([
4    [1, 2, 3],
5    [4, 5, 6],
6    [7, 8, 9]
7])
8
9diagonal = torch.diagonal(A)
10
11print(diagonal)

Output:

textCopy
1tensor([1, 5, 9])

The selected positions are:

textCopy
11   2   3
2    ↘
34   5   6
4        ↘
57   8   9

More precisely, the main diagonal contains:

textCopy
1A[0, 0] = 1
2A[1, 1] = 5
3A[2, 2] = 9

Diagonal Offset

You can select diagonals above or below the main diagonal using the offset parameter.

Example:

pythonCopy
1A = torch.tensor([
2    [1, 2, 3],
3    [4, 5, 6],
4    [7, 8, 9]
5])
6
7print(torch.diagonal(A, offset=1))

Output:

textCopy
1tensor([2, 6])

The offset=1 selects the diagonal immediately above the main diagonal.

For the diagonal below the main diagonal:

pythonCopy
1print(torch.diagonal(A, offset=-1))

Output:

textCopy
1tensor([4, 8])

This is useful in matrix analysis and attention-related computations.

Upper Triangular Tensor with torch.triu()

torch.triu() keeps the upper triangular portion of a tensor and sets values below the selected diagonal to zero.

Example:

pythonCopy
1import torch
2
3A = torch.arange(1, 10).reshape(3, 3)
4
5print(torch.triu(A))

Output:

textCopy
1tensor([
2    [1, 2, 3],
3    [0, 5, 6],
4    [0, 0, 9]
5])

The main diagonal and everything above it are preserved.

Upper Triangle with an Offset

You can control which diagonal is considered the boundary.

pythonCopy
1A = torch.arange(1, 10).reshape(3, 3)
2
3print(torch.triu(A, diagonal=1))

Output:

textCopy
1tensor([
2    [0, 2, 3],
3    [0, 0, 6],
4    [0, 0, 0]
5])

This is particularly useful when constructing causal masks.

Lower Triangular Tensor with torch.tril()

torch.tril() keeps the lower triangular portion of a tensor.

pythonCopy
1A = torch.arange(1, 10).reshape(3, 3)
2
3print(torch.tril(A))

Output:

textCopy
1tensor([
2    [1, 0, 0],
3    [4, 5, 0],
4    [7, 8, 9]
5])

The main diagonal and values below it are preserved.

Lower Triangle with an Offset

pythonCopy
1print(torch.tril(A, diagonal=-1))

Output:

textCopy
1tensor([
2    [0, 0, 0],
3    [4, 0, 0],
4    [7, 8, 0]
5])

This excludes the main diagonal.

triu() vs tril()

Visual representation:

textCopy
1triu()
2
3✓ ✓ ✓
40 ✓ ✓
50 0 ✓

textCopy
1tril()
2
3✓ 0 0
4✓ ✓ 0
5✓ ✓ ✓

Transformer Causal Attention Mask

One of the most important applications of triangular tensors is causal attention masking in Transformer models.

During autoregressive text generation, a token should not be allowed to attend to future tokens.

For a sequence of four tokens:

textCopy
1Token 1
2Token 2
3Token 3
4Token 4

the allowed attention pattern is:

textCopy
1✓ ✗ ✗ ✗
2✓ ✓ ✗ ✗
3✓ ✓ ✓ ✗
4✓ ✓ ✓ ✓

Each row can attend to the current token and previous tokens, but not future tokens.

Creating a Causal Mask

A convenient way to create the future-token mask is:

pythonCopy
1import torch
2
3seq_len = 4
4
5mask = torch.triu(
6    torch.ones(seq_len, seq_len, dtype=torch.bool),
7    diagonal=1
8)
9
10print(mask)

Output:

textCopy
1tensor([
2    [False,  True,  True,  True],
3    [False, False,  True,  True],
4    [False, False, False,  True],
5    [False, False, False, False]
6])

Here:

textCopy
1True

means that the corresponding future position should be masked.

Applying the Causal Mask to Attention Scores

Suppose an attention mechanism produces:

pythonCopy
1scores = torch.randn(4, 4)

We can mask future positions:

pythonCopy
1mask = torch.triu(
2    torch.ones(4, 4, dtype=torch.bool),
3    diagonal=1
4)
5
6masked_scores = scores.masked_fill(mask, float("-inf"))

Why use negative infinity?

Because after applying softmax:

textCopy
1softmax(-∞) ≈ 0

Therefore, future tokens receive approximately zero attention probability.

This pattern is fundamental to causal self-attention, autoregressive Transformers, and language models.

Why masked_fill() and triu() Work Well Together

These two operations solve different parts of the problem:

textCopy
1torch.triu()
2      ↓
3Creates the triangular Boolean mask
4
5masked_fill()
6      ↓
7Replaces forbidden attention scores

Complete example:

pythonCopy
1import torch
2
3scores = torch.randn(4, 4)
4
5causal_mask = torch.triu(
6    torch.ones(4, 4, dtype=torch.bool),
7    diagonal=1
8)
9
10scores = scores.masked_fill(
11    causal_mask,
12    float("-inf")
13)
14
15print(scores)

This is a simplified representation of an important operation performed in Transformer attention.

Practical Deep Learning Applications

One-Hot Encoding

Scatter operations can convert class labels into one-hot representations.

pythonCopy
1labels = torch.tensor([0, 2, 1, 2])
2
3one_hot = torch.zeros(4, 3)
4
5one_hot.scatter_(1, labels.unsqueeze(1), 1)
6
7print(one_hot)

Output:

textCopy
1tensor([
2    [1., 0., 0.],
3    [0., 0., 1.],
4    [0., 1., 0.],
5    [0., 0., 1.]
6])

Image Tensor Masking

Suppose invalid pixel values are represented by negative numbers.

pythonCopy
1image = torch.tensor([
2    [10., -1., 30.],
3    [40., 50., -1.]
4])
5
6valid_image = image.masked_fill(image < 0, 0)
7
8print(valid_image)

Output:

textCopy
1tensor([
2    [10.,  0., 30.],
3    [40., 50.,  0.]
4])

Selecting Positive Values

pythonCopy
1x = torch.tensor([-3, 5, -1, 8, 10])
2
3positive = torch.masked_select(x, x > 0)
4
5print(positive)

Output:

textCopy
1tensor([ 5,  8, 10])

Extracting Matrix Diagonals

pythonCopy
1attention_matrix = torch.randn(4, 4)
2
3self_attention_scores = torch.diagonal(attention_matrix)
4
5print(self_attention_scores)

This can be useful when analyzing diagonal/self-position values in matrix-based computations.

Creating a Lower-Triangular Mask

pythonCopy
1mask = torch.tril(torch.ones(4, 4))
2
3print(mask)

Output:

textCopy
1tensor([
2    [1., 0., 0., 0.],
3    [1., 1., 0., 0.],
4    [1., 1., 1., 0.],
5    [1., 1., 1., 1.]
6])

Comparing Important Indexing Operations

Several PyTorch operations can appear similar but serve different purposes.

A useful mental model is:

textCopy
1READ
2 ├── Boolean indexing
3 ├── masked_select()
4 ├── gather()
5 └── take()
6
7WRITE
8 ├── Tensor assignment
9 ├── masked_fill()
10 └── scatter()
11
12ACCUMULATE
13 └── scatter_add()
14
15STRUCTURE
16 ├── diagonal()
17 ├── triu()
18 └── tril()

Common Mistakes

Using and Instead of &

Incorrect:

pythonCopy
1mask = (x > 5) and (x < 20)

For element-wise Boolean conditions, use:

pythonCopy
1mask = (x > 5) & (x < 20)

Similarly, use:

pythonCopy
1|

for element-wise OR and:

pythonCopy
1~

for element-wise NOT.

Forgetting Parentheses

Incorrect:

pythonCopy
1x > 5 & x < 20

Correct:

pythonCopy
1(x > 5) & (x < 20)

Parentheses make the intended element-wise comparisons explicit and avoid Python operator-precedence problems.

Confusing scatter() and gather()

A simple rule:

textCopy
1gather()
2    ↓
3Read from positions
4
5scatter()
6    ↓
7Write to positions

If you need to retrieve values using indices, consider gather().

If you need to place values at indexed positions, consider scatter().

Incorrect scatter() Shapes

For example:

pythonCopy
1target = torch.zeros(3, 5)
2
3index = torch.tensor([
4    [0],
5    [2],
6    [4]
7])
8
9src = torch.tensor([
10    [10],
11    [20],
12    [30]
13])

The shapes of index and src need to be compatible with the selected dimension.

When debugging scatter operations, inspect:

pythonCopy
1print(target.shape)
2print(index.shape)
3print(src.shape)

Forgetting unsqueeze() for One-Hot Encoding

This is often incorrect for a two-dimensional scatter:

pythonCopy
1labels = torch.tensor([0, 2, 1])
2
3one_hot.scatter_(1, labels, 1)

Instead, reshape the labels into a column:

pythonCopy
1labels = labels.unsqueeze(1)

Then:

pythonCopy
1one_hot.scatter_(1, labels, 1)

Assuming masked_fill() Is In-Place

This:

pythonCopy
1y = x.masked_fill(mask, 0)

creates a result without modifying x.

For an in-place operation:

pythonCopy
1x.masked_fill_(mask, 0)

Use in-place operations carefully when autograd is involved.

Best Practices for PyTorch Tensor Indexing

• Check tensor shapes with .shape before advanced indexing operations.
• Use Boolean masks for clear conditional filtering.
• Use masked_fill() when selected positions need replacement.
• Use masked_select() when selected values need to be extracted.
• Use gather() for indexed reads.
• Use scatter() for indexed writes.
• Use scatter_add() when multiple values need to be accumulated at the same positions.
• Use torch.take() when flattened indexing is specifically required.
• Use torch.diagonal() instead of manually constructing diagonal indices.
• Use torch.triu() and torch.tril() to construct triangular tensors and masks.
• Be careful with in-place operations when tensors participate in gradient computation.
• Prefer descriptive variable names such as mask, indices, scores, and labels to make tensor operations easier to understand.

Practice Exercises

Beginner: Tensor Assignment

Create:

pythonCopy
1x = torch.tensor([5, 10, 15, 20, 25])

Perform:

1. Replace the value at index 2 with 100.
2. Replace the first two elements with 0.
3. Replace every value greater than 15 with -1.
4. Create the same result using masked_fill().

Intermediate: Mask Operations

Create:

pythonCopy
1x = torch.tensor([-5, 10, -2, 20, 30])

Perform:

1. Extract positive values using Boolean indexing.
2. Extract positive values using masked_select().
3. Replace negative values with 0.
4. Replace values greater than 20 with 100.
5. Compare the results of masked_fill() and Boolean assignment.

Intermediate: Scatter and One-Hot Encoding

Create:

pythonCopy
1labels = torch.tensor([2, 0, 1, 2])

Generate a 4 × 3 one-hot tensor using:

pythonCopy
1scatter_()

Then verify that every row contains exactly one 1.

Hint:

pythonCopy
1print(one_hot.sum(dim=1))

Advanced: Scatter Addition

Create:

pythonCopy
1index = torch.tensor([0, 1, 1, 2, 2, 2])
2src = torch.tensor([5, 10, 20, 1, 2, 3])

Use scatter_add() to produce:

textCopy
1[5, 30, 6]

Explain why repeated indices are accumulated.

Advanced: Matrix Operations

Create:

pythonCopy
1A = torch.arange(1, 17).reshape(4, 4)

Perform:

1. Extract the main diagonal.
2. Extract the diagonal above the main diagonal.
3. Extract the diagonal below the main diagonal.
4. Create the upper triangular matrix.
5. Create the lower triangular matrix.
6. Use torch.take() with flattened indices [0, 5, 10, 15].

Expert: Transformer Causal Mask

Create a sequence length of 6.

Build a Boolean causal mask using:

pythonCopy
1torch.triu()

with:

pythonCopy
1diagonal=1

Then create random attention scores:

pythonCopy
1scores = torch.randn(6, 6)

Use:

pythonCopy
1masked_fill()

to prevent attention to future tokens.

Verify that all future-token positions contain:

textCopy
1-inf

Mini Project: Attention Mask Generator

Build a reusable function that creates a causal attention mask.

pythonCopy
1import torch
2
3def create_causal_mask(seq_len):
4    return torch.triu(
5        torch.ones(seq_len, seq_len, dtype=torch.bool),
6        diagonal=1
7    )
8
9mask = create_causal_mask(5)
10
11print(mask)

Output:

textCopy
1tensor([
2    [False,  True,  True,  True,  True],
3    [False, False,  True,  True,  True],
4    [False, False, False,  True,  True],
5    [False, False, False, False,  True],
6    [False, False, False, False, False]
7])

This is a useful building block for understanding causal self-attention in Transformer models.

Mini Project: One-Hot Encoder

We can also turn the scatter operation into a reusable function.

pythonCopy
1import torch
2
3def one_hot_encode(labels, num_classes):
4    result = torch.zeros(
5        labels.numel(),
6        num_classes,
7        dtype=torch.float32
8    )
9
10    result.scatter_(1, labels.reshape(-1, 1), 1.0)
11
12    return result
13
14
15labels = torch.tensor([0, 2, 1, 2])
16
17encoded = one_hot_encode(labels, 3)
18
19print(encoded)

Output:

textCopy
1tensor([
2    [1., 0., 0.],
3    [0., 0., 1.],
4    [0., 1., 0.],
5    [0., 0., 1.]
6])

This project demonstrates how tensor indexing, reshaping, and scatter operations can work together.

Module Summary

In this chapter, you learned how to perform advanced PyTorch tensor indexing and slicing operations that are commonly used in machine learning and deep learning.

You learned:

• Tensor assignment using indices and slices.
• Conditional tensor updates with Boolean masks.
• masked_fill() for replacing selected values.
• masked_select() for extracting values.
• scatter() for writing values at indexed positions.
• scatter_add() for accumulating values at repeated indices.
• torch.take() for flattened tensor indexing.
• torch.diagonal() for extracting matrix diagonals.
• torch.triu() for creating upper triangular tensors.
• torch.tril() for creating lower triangular tensors.
• How triangular masks are used for Transformer causal attention.
• How scatter_() can be used to implement one-hot encoding.
• How advanced indexing techniques appear in computer vision, NLP, and Transformer architectures.

These operations are an important part of PyTorch tensor manipulation, deep learning tensor operations, and Transformer model implementation. Once you understand how to read, modify, mask, gather, scatter, and reshape tensors, you can work much more confidently with real neural network architectures.