Norm: A Brief Introduction to the "Size" of Vectors in Machine Learning
A quantitative measure of the length or size of a vector is often necessary in vector and matrix operations in machine learning, and we often just refer to the length of a vector as Vector's Norm.
In mathematics, a norm is a function from a vector space over the real or complex numbers to the nonnegative real numbers, that satisfies certain properties pertaining to scalability and additivity and takes the value zero only if the input vector is zero. A pseudonorm or seminorm satisfies the same properties, except that it may have a zero value for some nonzero vectors. **1
Recently, during our research on adversarial attacks, we needed to quantitatively measure the "perturbation size" between the adversarial and their corresponding benign images. In fact, in machine learning, whether "adversarial samples" or other images can essentially be represented as vectors, stored and computed as Numpy matrices. This article briefly describes some of the norm calculations and implementations.
What is adversarial examples?
Adversarial examples is a type of vulnerability in neural network models. For an image classification model, adversarial examples are produced by adding imperceptible perturbations onto the input images, in order that the model may misclassify the contents of the images.
The norm is actually a "set of norms" in a vector space 2. In my line of research, norm are also often used to measure the magnitude of the "perturbation" of an adversarial sample. We define all the norm as,
in which can be , , and . They are naturally called norm、 norm and norm.
Technically, norm is not norm (because, by definition, in cannot be 0). Still, this norm represents the number of non-zero elements in the vector. Then in the context of adversarial attack, it represents the number of non-zero elements in the "perturbation" vector.
The norm represents the sum of the lengths of all the vectors in a vector space. A better description would be that in a vector space, you need to walk from the start of one vector to the end of another, so the distance you travel (the total length through the vector) is the norm of the vector.
As shown above, can be calculated according to the following equation,
much the way a New York taxicab travels along its route. Therefore, the norm is also known as the Taxicab norm or Manhattan norm. Generally, it is formulated as,
The norm is one of the more commonly used measures of vector size in the field of machine learning. norm, also known as the Euclidean norm, represents the shortest distance required to travel from one point to another.
In the example shown above, the paradigm is calculated according to the following equation,
More generally, it is formulated as,
norm is the easiest to understand, i.e. the length (size) of the element with the largest absolute value inside the vector element.
For example, given a vector , the norm of the vector is .
In my research, I tend to use the norm to measure the perturbation size of adversarial examples. Unfortunately, instead of relying on any framework, we implement out attack from scratch, which means no automatically output direct distance value for all paradigm calculations, so I need to Numpy to calculate them instead.
For an image
img, and its adversary example
adv, we can easily compute the perturbation
# perturb is a numpy array perturb = adv - img
Then, we can use Numpy to compute the norm of the perturbation
# import numpy and relevant libraries import numpy as np from numpy.linalg import norm # L0 _l0 = norm(perturb, 0) # L1 _l1 = norm(perturb, 1) # L2 _l2 = norm(perturb) # L∞ _linf = norm(perturb, np.inf)
In fact, the implementation of
numpy.linalg.norm is just a vector operation that uses the definition of . For example, is just,
_l1_x = np.sum(np.abs(x))
and is just,
_linf_x = np.max(np.abs(x))
Howver during production, without special cases, we should directly use
numpy.linalg.norm. On top of that, Numpy's official documentation also gives methods for calculating norms in different cases for matrices and vectors.