2.3  Advanced: Inner Products to Obtain the Transform Coefficients

In the general case, calculating the forward or inverse transforms corresponds to performing a matrix multiplication. However, most linear transforms are designed such that “transforming” the original signal corresponds to the calculation of inner products. Hence, this section will elaborate on inner products and their use in transforms.

The main motivation is that we can calculate an inner product between a pair of vectors, but also between a pair of functions, and a pair of signals. We start by discussing the inner product between vectors, to make useful analogies with the inner product between signals in the context of linear transforms.

An inner product is a generalization of the dot product $\mathbf{\text{x}}\cdot \mathbf{\text{y}}$ between two vectors $\mathbf{\text{x}}$ and $\mathbf{\text{y}}$ of $N$ elements or, equivalently, equal-length sequences (with $N$ samples). The dot product is defined for vectors with real-valued elements as (the notation of an inner product $⟨\mathbf{\text{x}},\mathbf{\text{y}}⟩$ will be used hereafter instead of $\mathbf{\text{x}}\cdot \mathbf{\text{y}}$):

where $𝜃$ is the angle between $\mathbf{\text{y}}$ and $\mathbf{\text{x}}$, which is restricted to the range $0\le 𝜃\le 180$ degrees. Alternatively, this inner product can also be calculated as

where $x_i$ and $y_i$ are the $i$-th elements of $\mathbf{\text{x}}$ and $\mathbf{\text{y}}$, respectively.

Another aspect of the inner product $⟨\mathbf{\text{x}},\mathbf{\text{y}}⟩$ is that it is proportional to the norm of the projection of $\mathbf{\text{x}}$ on $\mathbf{\text{y}}$ and vice versa. Repeating Eq. (A.36) below, for convenience

and interpreting $\mathbf{\text{y}}$ as a basis vector of norm equals to 1 ($\parallel \mathbf{\text{y}}\parallel =1$, one has $\parallel {\mathbf{\text{p}}}_{\mathit{xy}}\parallel =|⟨\mathbf{\text{x}},\mathbf{\text{y}}⟩|$. A large magnitude of $⟨\mathbf{\text{x}},\mathbf{\text{y}}⟩$ corresponds to a basis vector $\mathbf{\text{y}}$ that represents well (a reasonable part of the energy) of the vector $\mathbf{\text{x}}$ (later interpreted as signal $x[n]$). On the other hand, $⟨\mathbf{\text{x}},\mathbf{\text{y}}⟩=0$ means that the vectors are orthogonal.

Example 2.3. Calculating the angle between vectors using their inner product. For example, consider $\mathbf{\text{x}}=[1,1,1]$ and $\mathbf{\text{y}}=[1,3,2]$. Their inner product is calculated from Eq. (2.4) as $⟨\mathbf{\text{x}},\mathbf{\text{y}}⟩=6$, such that $\text{cos}<!--\; FUNCTION\; APPLICATION\; -->(𝜃)=⟨\mathbf{\text{x}},\mathbf{\text{y}}⟩∕(\parallel \mathbf{\text{x}}\parallel \parallel \mathbf{\text{y}}\parallel )\approx 6∕(1.732\times 3.742)=0.926$ and consequently, $𝜃=0.387$ radians.

As another example, if $\mathbf{\text{x}}=[3,-1,0]$ and $\mathbf{\text{y}}=[1,3,2]$, their inner product is $⟨\mathbf{\text{x}},\mathbf{\text{y}}⟩=0$. In this case, from Eq. (2.3) and observing that the norms of both vectors are non-zero, the inner product is zero necessarily because $\text{cos}<!--\; FUNCTION\; APPLICATION\; -->(𝜃)=0$ and consequently, $𝜃=\pi ∕2$. Hence, when vectors are perpendicular ($𝜃=\pi ∕2$), their inner product is zero.   $\text{□}$

Orthogonal is a generalization of perpendicular. In general, when two vectors are orthogonal their inner product is zero.

There are many distinct inner products. But an operation must obey specific properties such as linearity to be “valid” as an inner product and, consequently, define an inner product space, where concepts such as norm and orthogonality are natural extensions of the ones with geometric interpretations provided by Eq. (2.3). This geometric interpretation is highly beneficial when interpreting the inner products used in transforms. Table 2.2 illustrates alternative definitions of inner products that are discussed in the sequel.

As indicated in Table 2.2, it is also possible to define an inner product for infinite duration signals. For example, consider the complex-valued signal $y[n]=e^{j\mathrm{\Omega }n}$, where $n=-\infty ,\dots ,-1,0,1,\dots ,\infty$. It is a common operation in Fourier transform to calculate inner products among such signals using the definition

Note that when applied to complex-valued signals, the inner product is defined using a complex conjugation.

When dealing with continuous-time signals, a convenient definition is obtained by changing the summation by an integral:

The inner product of finite-duration signals with support $T$ is simplified to

where $⟨T⟩$ denotes a range of $T$ such as $[0,T]$ or $[-T∕2,T∕2]$.

When using inner products between continuous-time signals, it is possible to make useful analogies to vectors in the Euclidean space and benefit from geometrical interpretations. For example, similar to vectors, the squared norm $⟨x(t),x(t)⟩=\parallel x(t){\parallel }^2$ is the signal energy.³ But the most important lesson in this context is that orthogonal signals share similar properties with orthogonal vectors.⁴

Example 2.4. Example of a pair of orthogonal signals. The signals $x(t)=1$ and $y(t)=2u(t)-1$ can be interpreted as orthogonal because $⟨x(t),y(t)⟩=0$. This inner product can be obtained by noting that $x(t)y(t)$ is $-1$ for $t<0$ and $1$ for $t\ge 0$, leading to a zero integral when using Eq. (2.7).   $\text{□}$

At this point, the reader may eventually benefit from Appendices A.14.2 and A.14.3, which provide a review of linear algebra applied to transforms. The goal of these appendices is to interpret, for example, the Fourier transform

as the inner product $⟨x(t),e^{\mathit{j\omega t}}⟩$ between the signal $x(t)$ and the basis function $e^{\mathit{j\omega t}}$.

As discussed in Appendix A.14.3, if a signal $x(t)$ can be represented as a linear combination $x(t)={\sum <!--\; FUNCTION\; APPLICATION\; -->}_{d=1}^D{m}_d{\phi }_d(t)$, where the $D$ functions ${\phi }_d(t)$ compose a set $\{{\phi }_j(t)\}$ of orthonormal basis functions, then the values (or coefficients) $m_d$ can be recovered using inner products:

This result is discussed in Appendix A.14.3 via examples with vectors.

Before dealing with infinite duration signals, the next section discusses transforms that operate on blocks of samples.