Editing Outer product (section)

==Definition==
Given two vectors of size <math>m \times 1</math> and <math>n \times 1</math> respectively
:<math display="block">\mathbf{u} = \begin{bmatrix} u_1 \\ u_2 \\ \vdots \\ u_m \end{bmatrix},
\quad
\mathbf{v} = \begin{bmatrix} v_1 \\ v_2 \\ \vdots \\ v_n \end{bmatrix}</math>
their outer product, denoted <math>\mathbf{u} \otimes \mathbf{v},</math> is defined as the <math>m \times n</math> matrix <math>\mathbf{A}</math> obtained by multiplying each element of <math>\mathbf{u}</math> by each element of {{nowrap|<math>\mathbf{v}</math>:}}<ref>{{cite book |title=Encyclopaedia of Physics |edition=2nd |first1=R. G. |last1=Lerner |author1-link=Rita G. Lerner |first2=G. L. |last2=Trigg |publisher=VHC |year=1991 |isbn=0-89573-752-3 |url-access=registration |url=https://archive.org/details/encyclopediaofph00lern}}</ref>
:<math display="block">
  \mathbf{u} \otimes \mathbf{v} = \mathbf{A} =
  \begin{bmatrix}
    u_1v_1 & u_1v_2 & \dots & u_1v_n \\
    u_2v_1 & u_2v_2 & \dots & u_2v_n \\
    \vdots & \vdots & \ddots & \vdots \\
    u_mv_1 & u_mv_2 & \dots & u_mv_n
  \end{bmatrix}
</math>

Or, in index notation:
:<math display="block">(\mathbf{u} \otimes \mathbf{v})_{ij} = u_i v_j</math>

Denoting the [[dot product]] by <math>\,\cdot,\,</math> if given an <math>n \times 1</math> vector <math>\mathbf{w},</math> then <math>(\mathbf{u} \otimes \mathbf{v}) \mathbf{w} = (\mathbf{v} \cdot \mathbf{w}) \mathbf{u}.</math> If given a <math>1 \times m</math> vector <math>\mathbf{x},</math> then <math>\mathbf{x} (\mathbf{u} \otimes \mathbf{v}) = (\mathbf{x} \cdot \mathbf{u}) \mathbf{v}^{\operatorname{T}}.</math>

If <math>\mathbf{u}</math> and <math>\mathbf{v}</math> are vectors of the same dimension bigger than 1, then <math>\det (\mathbf{u} \otimes\mathbf{v}) = 0</math>.

The outer product <math>\mathbf{u} \otimes \mathbf{v}</math> is equivalent to a [[matrix multiplication]] <math>\mathbf{u} \mathbf{v}^{\operatorname{T}},</math> provided that <math>\mathbf{u}</math> is represented as a <math>m \times 1</math> [[column vector]] and <math>\mathbf{v}</math> as a <math>n \times 1</math> column vector (which makes <math>\mathbf{v}^{\operatorname{T}}</math> a row vector).<ref>{{cite book |title=Linear Algebra |edition=4th |first1=S. |last1=Lipschutz |first2=M. |last2=Lipson |series=Schaum’s Outlines |publisher=McGraw-Hill |year=2009 |isbn=978-0-07-154352-1}}</ref><ref name=":0">{{cite web |last=Keller |first=Frank |date=February 23, 2020 |title=Algebraic Properties of Matrices; Transpose; Inner and Outer Product |url=https://www.inf.ed.ac.uk/teaching/courses/cfcs1/lectures/cfcs_l10.pdf |archive-url=https://web.archive.org/web/20171215061654/http://www.inf.ed.ac.uk/teaching/courses/cfcs1/lectures/cfcs_l10.pdf |archive-date=2017-12-15 |url-status=live |access-date=September 6, 2020 |website=inf.ed.ac.uk}}</ref> For instance, if <math>m = 4</math> and <math>n = 3,</math> then<ref>James M. Ortega (1987) ''Matrix Theory: A Second Course'', page 7, [[Plenum Press]] {{ISBN|0-306-42433-9}}</ref>
:<math display="block">
  \mathbf{u} \otimes \mathbf{v} = \mathbf{u}\mathbf{v}^\textsf{T} =
  \begin{bmatrix}u_1 \\ u_2 \\ u_3 \\ u_4\end{bmatrix}
    \begin{bmatrix}v_1 & v_2 & v_3\end{bmatrix} =
  \begin{bmatrix}
    u_1 v_1 & u_1 v_2 & u_1 v_3 \\
    u_2 v_1 & u_2 v_2 & u_2 v_3 \\
    u_3 v_1 & u_3 v_2 & u_3 v_3 \\
    u_4 v_1 & u_4 v_2 & u_4 v_3
  \end{bmatrix}.
</math>

For [[complex numbers|complex]] vectors, it is often useful to take the [[conjugate transpose]] of <math>\mathbf{v},</math> denoted <math>\mathbf{v}^\dagger</math> or <math>\left(\mathbf{v}^\textsf{T}\right)^*</math>:
:<math display="block">\mathbf{u} \otimes \mathbf{v} = \mathbf{u} \mathbf{v}^\dagger = \mathbf{u} \left(\mathbf{v}^\textsf{T}\right)^*.</math>

===Contrast with Euclidean inner product===
If <math>m = n,</math> then one can take the matrix product the other way, yielding a scalar (or <math>1 \times 1</math> matrix):
:<math display="block">\left\langle\mathbf{u}, \mathbf{v}\right\rangle = \mathbf{u}^\textsf{T} \mathbf{v}</math>
which is the standard [[inner product]] for [[Euclidean vector space]]s,<ref name=":0"/> better known as the [[dot product]]. The dot product is the [[trace (linear algebra)|trace]] of the outer product.<ref>{{cite book |first=Robert F. |last=Stengel |title=Optimal Control and Estimation |location=New York |publisher=Dover Publications |year=1994 |page=26 |isbn=0-486-68200-5 |url=https://books.google.com/books?id=jDjPxqm7Lw0C&pg=PA26}}</ref> Unlike the dot product, the outer product is not commutative.

Multiplication of a vector <math>\mathbf{w}</math> by the matrix <math>\mathbf{u} \otimes \mathbf{v}</math> can be written in terms of the inner product, using the relation <math>\left(\mathbf{u} \otimes \mathbf{v}\right)\mathbf{w} = \mathbf{u}\left\langle\mathbf{v}, \mathbf{w}\right\rangle</math>.

===The outer product of tensors===
Given two tensors <math>\mathbf{u}, \mathbf{v}</math> with dimensions <math>(k_1, k_2, \dots, k_m)</math> and <math>(l_1, l_2, \dots, l_n)</math>, their outer product <math>\mathbf{u} \otimes \mathbf{v}</math> is a tensor with dimensions <math>(k_1, k_2, \dots, k_m, l_1, l_2, \dots, l_n)</math> and entries
:<math display="block">(\mathbf{u} \otimes \mathbf{v})_{i_1, i_2, \dots i_m, j_1, j_2, \dots, j_n} = u_{i_1, i_2, \dots, i_m} v_{j_1, j_2, \dots, j_n}</math>

For example, if <math>\mathbf{A}</math> is of order 3 with dimensions <math>(3, 5, 7)</math> and <math>\mathbf{B}</math> is of order 2 with dimensions <math>(10, 100),</math> then their outer product <math>\mathbf{C}</math> is of order 5 with dimensions <math>(3, 5, 7, 10, 100).</math> If <math>\mathbf{A}</math> has a component {{math|1=''A''<sub>[2, 2, 4]</sub> = 11}} and <math>\mathbf{B}</math> has a component {{math|1=''B''<sub>[8, 88]</sub> = 13}}, then the component of <math>\mathbf{C}</math> formed by the outer product is {{math|1=''C''<sub>[2, 2, 4, 8, 88]</sub> = 143}}.

===Connection with the Kronecker product===
The outer product and Kronecker product are closely related; in fact the same symbol is commonly used to denote both operations.

If <math>\mathbf{u} = \begin{bmatrix}1 & 2 & 3\end{bmatrix}^\textsf{T}</math> and <math>\mathbf{v} = \begin{bmatrix}4 & 5\end{bmatrix}^\textsf{T}</math>, we have:
:<math display="block">\begin{align}
  \mathbf{u} \otimes_\text{Kron} \mathbf{v} &= \begin{bmatrix} 4 \\ 5 \\ 8 \\ 10 \\ 12 \\ 15\end{bmatrix}, &
  \mathbf{u} \otimes_\text{outer} \mathbf{v} &= \begin{bmatrix} 4 & 5 \\ 8 & 10 \\ 12 & 15\end{bmatrix}
\end{align}</math>

In the case of column vectors, the Kronecker product can be viewed as a form of [[vectorization (mathematics)|vectorization]] (or flattening) of the outer product. In particular, for two column vectors <math>\mathbf{u}</math> and <math>\mathbf{v}</math>, we can write:
:<math display="block">\mathbf{u} \otimes_{\text{Kron}} \mathbf{v} = \operatorname{vec}(\mathbf{v} \otimes_\text{outer} \mathbf{u})</math>

(The order of the vectors is reversed on the right side of the equation.)

Another similar identity that further highlights the similarity between the operations is
:<math display="block">\mathbf{u} \otimes_{\text{Kron}} \mathbf{v}^\textsf{T} = \mathbf u \mathbf{v}^\textsf{T} = \mathbf{u} \otimes_{\text{outer}} \mathbf{v}</math>

where the order of vectors needs not be flipped. The middle expression uses matrix multiplication, where the vectors are considered as column/row matrices.

===Connection with the matrix product===
Given a pair of matrices <math>\mathbf{A}</math> of size <math>m\times p</math> and <math>\mathbf{B}</math> of size <math>p\times n</math>, consider the [[matrix multiplication|matrix product]] <math>\mathbf{C} = \mathbf{A}\,\mathbf{B}</math> defined as usual as a matrix of size <math>m\times n</math>.

Now let <math>\mathbf a^\text{col}_k</math> be the <math>k</math>-th column vector of <math>\mathbf A</math> and let <math>\mathbf b^\text{row}_k</math> be the <math>k</math>-th row vector of <math>\mathbf B</math>. Then <math>\mathbf{C}</math> can be expressed as a sum of column-by-row outer products:
:<math display="block">\mathbf{C} = \mathbf{A}\, \mathbf{B} =
\left(
      \sum_{k=1}^p {A}_{ik}\, {B}_{kj}
\right)_{
    \begin{matrix} 1\le i \le m \\[-20pt] 1 \le j\le n \end{matrix}
} =
\begin{bmatrix} & & \\ \mathbf a^\text{col}_{1} & \cdots & \mathbf a^\text{col}_{p} \\ & & \end{bmatrix}
\begin{bmatrix} & \mathbf b^\text{row}_{1} & \\ & \vdots & \\ & \mathbf b^\text{row}_{p} & \end{bmatrix}
= \sum_{k=1}^p \mathbf a^\text{col}_k \mathbf b^\text{row}_k</math>
This expression has duality with the more common one as a matrix built with row-by-column [[inner product space|inner product]] entries (or [[dot product]]): <math>C_{ij} = \langle{\mathbf a^\text{row}_i,\,\mathbf b_j^\text{col}}\rangle</math>

This relation is relevant<ref>{{cite book |last2=Bau III |first2=David |last1=Trefethen |first1=Lloyd N. |author1-link=Lloyd N. Trefethen |title=Numerical linear algebra |publisher=Society for Industrial and Applied Mathematics |location=Philadelphia |isbn=978-0-89871-361-9 |year=1997}}</ref> in the application of the [[singular value decomposition#Separable models|Singular Value Decomposition (SVD)]] (and [[eigendecomposition of a matrix|Spectral Decomposition]] as a special case). In particular, the decomposition can be interpreted as the sum of outer products of each left (<math>\mathbf{u}_k</math>) and right (<math>\mathbf{v}_k</math>) singular vectors, scaled by the corresponding nonzero singular value <math>\sigma_k</math>:
:<math display="block">\mathbf{A} = \mathbf{U \Sigma V^T} = \sum_{k=1}^{\operatorname{rank}(A)}(\mathbf{u}_k \otimes \mathbf{v}_k) \, \sigma_k</math>

This result implies that <math>\mathbf{A}</math> can be expressed as a sum of rank-1 matrices with [[matrix norm#Norms induced by p-norms|spectral norm]] <math>\sigma_k</math> in decreasing order. This explains the fact why, in general, the last terms contribute less, which motivates the use of the [[singular value decomposition#Truncated SVD|truncated SVD]] as an approximation. The first term is the [[least squares]] fit of a matrix to an outer product of vectors.