Dynamic Programming - Matrix-chain Multiplication

Chained Matrix Multiplication

• Problem: Given a series of $n$ arrays (of appropriate sizes) to multiply: $\color{WildStrawberry}{A_1\times A_2\times\dots\times A_n}$
• Determine where to place parentheses to minimize the number of multiplications.

• Multiplying an $i\times j$ array with a $j\times k$ array takes $i\times j\times k$ array

• Matrix multiplication is associative, so all placements give same result

Matrix Chain Multiplication: Introduction

• Problem: Given a sequence of matrices $A_1, A_2, \dots, A_n$, insert parentheses so that the product of the matrices, in order, is unambiguous and needs the minimal number of multiplication

• Assume that the matrix dimensions allow multiplication, in order

• Matrix multiplication is associative: $A_1(A_2 A_3) = (A_1 A_2)A_3$

• A product is unambiguous if no factor is multiplied on both the left and the right and all factors are either a single matrix or an unambiguous product (in parentheses)

Number of Multiplications

• Multiplying an $i\times j$ and a $j\times k$ matrix requires $ijk$ multiplications

• Each element of the product requires $j$ multiplications, and there are $ik$ elements

Number of Parenthesizations

• Example:
• Given the matrices $A_1, A_2, A_3, A_4$
• Assume the dimensions of $A_1=d_0\times d_1$, etc

• Below are the five possible parenthesizations of these arrays, along with the number of multiplications:
1. $(A_1A_2) (A_3 A_4): d_0d_1d_2 + d_2d_3d_4 + d_0d_2d_4$
2. $((A_1A_2) A_3) A_4: d_0d_1d_2 + d_0d_2d_3 + d_0d_3d_4$
3. $(A_1(A_2 A_3)) A_4: d_1d_2d_3 + d_0d_1d_3 + d_0d_3d_4$
4. $A_1((A_2 A_3) A_4): d_1d_2d_3 + d_1d_3d_4 + d_0d_1d_4$
5. $A_1(A_2(A_3 A_4)): d_2d_3d_4 + d_1d_2d_4 + d_0d_1d_4$

• The number of parenthesizations is at least $T(n) ≥ T(n-1) + T(n-1)$
• Since the number with the first element removed is $T(n-1)$, which is also the number with the last removed

• Thus the number of parenthesizations is $\Omega(2^n)$

• The number is actually $T(n) = \displaystyle \sum_{k=1}^{n-1} T(k)T(n-k)$ which is related to the Catalan numbers.
• This is because the original product can be split into 2 subproducts in $k$ places. Each split is to be parenthesized optimally.
• This recurrence is related to the Catalan numbers.

Characterizing the Optimal Parenthesization

• An optimal parenthesization of $A_1\dots A_n$ must break the product into two expressions, each of which is parenthesized or is a single array

• Assume the break occurs at position $k$

• In the optimal solution, the solution to the product $A_1\dots A_k$ must be optimal
• Otherwise, we could improve $A_1\dots A_n$ by improving $A_1\dots A_k$
• But the solution to $A_1\dots A_n$ is known to be optimal
• Thus the solution to $A_1\dots An$ is known to be optimal

Principle of Optimality

• This problem exhibits the Principle of Optimality:
• The optimal solution to product $A_1\dots A_n$ contains the optimal solution to two subproducts

• Thus we can use Dynamic Programming
• Consider a recursive solution
• Then improve it's performance with memoization or by rewriting bottom up

Matrix Dimensions

• Consider matrix prodcut $A_1\times \dots\times A_n$

• Let the dimensions of matrix $A_i$ be $d_{i-1}\times d_i$

• Thus the dimensions of matrix product $A_i \times \dots\times A_j$ is $d_{i-1}\times d_j$

Recursive Solution

• Let $M[i,j]$ represent the number of multiplications required for matrix product $A_i\times\dots \times A_j$
• For $1 ≤ i ≤ j < n$

• $M[i,i] = 0$ since no product is required

• The optimal solution of $A_i\times A_j$ must break at some point, $k$, with $i ≤ k < j$
• Thus, $M[i,j] = M[i,k] + M[k+1,j] + d_{i-1}d_k d_j$

• Thus, $\displaystyle M[i,j] = \begin{cases} 0 & \text{if } i = j \\ \min_{i ≤ k < j} \{M[i,k] + M[k+1,j] + d_{i-1}d_k d_j\} & \text{if } i < j \end{cases}$

• This is easily expressed as a recursive function (with exponential complexity)
• What would this look like?

• Complexity:
• P(1) = 1
• P(n) = Sum (k=1, n-1) P(k)P(n-k)
• P(n) = $\Omega(f^n / n^{\frac{3}{2}})$

Efficient Computation

• We must find a way to calculate this bottom up

• Which values does $M[i,j]$ depend on?
• Consider a $n \times n$ matrix of values M[i,j]
• Diagonal is 0
• $1 \le i \le j \le n$ is the upper right triangle

• Consider some element $M[i,j]$

• Where are $M[i,k]$ (for $i ≤ k ≤ j$):

• Where are $M[k+1,j]$ (for $i ≤ k ≤ j$):

• This tells us the order in which to build the table: By diagonals
• By diagonal
• Moving up and right from the diagonal that goes top-left to bottom-right

• Diagonal indices:
• On diagonal $0, j=i$
• On diagonal $1, j = i+1$
• On diagonal $q, j = i+q$
• On diagonal $n-1, j = i+n-1$

Algorithm for Minimum Value

• Bottom Up Algorithm to Calculate Minimum Number of Multiplications
• n -- Number of arrays
• d -- array of dimensions of arrays 1 .. n
    minimum_multiplication(n: integer; d: array(0..n))
M: array(1..n, 1..n) := (others => 0);

for diagonal in 1 .. n-1 loop
for i in 1 .. n - diagonal loop
j := i + diagonal;

min := integer'last;
for k in i .. j-1 loop
current := M[i, k] + M[k+1, j] + d(i-1)*d(k)*d(j)
if current < min then
min := current
end if
end loop
M[i, j] := min
end loop
end loop


• Complexity?

Algorithm for Location of Minimum Value

• Bottom Up Algorithm to Calculate Minimum Number of Multiplications
• n -- Number of arrays
• d -- array of dimensions of arrays 1 .. n
• P -- 2D array of locations of M[i, j]
•     minimum_multiplication(n: integer; d: array(0..n))
M: array(1..n, 1..n) := (others => 0);

for diagonal in 1 .. n-1 loop
for i in 1 .. n-diagonal loop
j := i + diagonal;

min_value := integer'last;
for k in i .. j-1 loop
current := M[i, k] + M[k+1, j] + d(i-1)*d(k)*d(j)
if current < min_value then
min_value := current
mink := k
end if
end loop
M[i, j] := mi_nvalue
P(i, j) := mink
end loop
end loop

printbest(P, i, j):
if i=j then
print "A"_i
else
print '('
printbest(P, i, M[i, j])
printbest(P, M[i, j] + 1, j)
print ')'


• Note: the algorithm covered in class was incorrect: the two for loops were swapped. The algorithm above is correct.

Example Showing Tables and Calculations

• Array dimensions:
• A1: 2 x 3
• A2: 3 x 5
• A3: 5 x 2
• A4: 2 x 4
• A5: 4 x 3

Array sizes: Ai = di-1 by di : $\begin{array}{c|ccccc} j & 0 & 1 & 2 & 3& 4 & 5 \\ \hline d_j & 2 & \color{magenta}{3} & \color{blue}{5} & \color{orange}{2} & \color{red}{4} & \color{magenta}{3} \end{array}$
• Tables of $M_{ij}$ and $d_j$:

• Mij = number of multiplies in best way to parenthesize arrays Ai .. Aj: $\begin{array}{c|ccccc} i,j & 1 & 2 & 3 & 4 & 5 \\ \hline 1 & 0 & 30 & 42 & 58 & 78 \\ 2 & & \color{blue}{0} & \color{orange}{30} & \color{red}{ 54} & \color{cyan}{72} \\ 3 & & & 0 & 40 & \color{blue} {54} \\ 4 & & & & 0 &\color{orange}{24} \\ 5 & & & & &\color{red }{0} \end{array}$

• Calculating $M_{25}$ = number of multiplies in the optimal way to parenthesize $A_2A_3A_4A_5$:

• $\color{cyan}{M_{25}} = \min \begin{cases} \color{blue }{M_{22} + M_{35}} + \color{magenta}{d_1}\color{blue }{d_2}\color{magenta}{d_5}, \\ \color{orange}{M_{23} + M_{45}} +\color{magenta}{d_1}\color{orange}{d_3}\color{magenta}{d_5}, \\ \color{red}{M_{24} + M_{55}} + \color{magenta}{d_1}\color{red}{d_4}\color{magenta}{d_5} \end{cases}$ $= \min \begin{cases} 0 + 54 + 45 = 99 \\ 30 + 24 + 18 = 72 \\ 54 + 0 + 36 = 90 \end{cases}$ $= \min \begin{cases} (A_2)(A_3A_4A_5)\\ (A_2A_3)(A_4A_5)\\ (A_2A_3A_4)(A_5) \end{cases}$

• Optimal locations for parentheses:
• $\begin{array}{c|ccccc} i,j & 1 & 2 & 3 & 4 & 5 \\ \hline 1 & & 1 & 1 & 3 & 3 \\ 2 & & & 2 & 3 & \color{orange}{3} \\ 3 & & & & 3 & 3 \\ 4 & & & & & 4 \\ 5 & & & & & \end{array}$