Dynamic Programming - Matrix-chain Multiplication

Overview

Chained Matrix Multiplication

Problem: Given a series of $n$ arrays (of appropriate sizes) to multiply: $\color{WildStrawberry}{A_1\times A_2\times\dots\times A_n}$
Determine where to place parentheses to minimize the number of multiplications.

Multiplying an $i\times j$ array with a $j\times k$ array takes $i\times j\times k$ array

Matrix multiplication is associative, so all placements give same result

Matrix Chain Multiplication: Introduction

Problem: Given a sequence of matrices $A_1, A_2, \dots, A_n$, insert parentheses so that the product of the matrices, in order, is unambiguous and needs the minimal number of multiplication

Assume that the matrix dimensions allow multiplication, in order

Matrix multiplication is associative: $A_1(A_2 A_3) = (A_1 A_2)A_3$

A product is unambiguous if no factor is multiplied on both the left and the right and all factors are either a single matrix or an unambiguous product (in parentheses)

Number of Multiplications

Multiplying an $i\times j$ and a $j\times k$ matrix requires $ijk$ multiplications

Each element of the product requires $j$ multiplications, and there are $ik$ elements

Number of Parenthesizations

Example:

Given the matrices $A_1, A_2, A_3, A_4$

Assume the dimensions of $A_1=d_0\times d_1$, etc

Below are the five possible parenthesizations of these arrays, along with the number of multiplications:

$(A_1A_2) (A_3 A_4): d_0d_1d_2 + d_2d_3d_4 + d_0d_2d_4$
$((A_1A_2) A_3) A_4: d_0d_1d_2 + d_0d_2d_3 + d_0d_3d_4 $
$(A_1(A_2 A_3)) A_4: d_1d_2d_3 + d_0d_1d_3 + d_0d_3d_4 $
$A_1((A_2 A_3) A_4): d_1d_2d_3 + d_1d_3d_4 + d_0d_1d_4 $
$A_1(A_2(A_3 A_4)): d_2d_3d_4 + d_1d_2d_4 + d_0d_1d_4 $

The number of parenthesizations is at least $T(n) ≥ T(n-1) + T(n-1)$

Since the number with the first element removed is $T(n-1)$, which is also the number with the last removed

Thus the number of parenthesizations is $\Omega(2^n)$

The number is actually $T(n) = \displaystyle \sum_{k=1}^{n-1} T(k)T(n-k)$ which is related to the Catalan numbers.

This is because the original product can be split into 2 subproducts in $k$ places. Each split is to be parenthesized optimally.
This recurrence is related to the Catalan numbers.

Characterizing the Optimal Parenthesization

An optimal parenthesization of $A_1\dots A_n$ must break the product into two expressions, each of which is parenthesized or is a single array

Assume the break occurs at position $k$

In the optimal solution, the solution to the product $A_1\dots A_k$ must be optimal

Otherwise, we could improve $A_1\dots A_n$ by improving $A_1\dots A_k$
But the solution to $A_1\dots A_n$ is known to be optimal
This is a contradiction
Thus the solution to $A_1\dots An$ is known to be optimal

Principle of Optimality

This problem exhibits the Principle of Optimality:

The optimal solution to product $A_1\dots A_n$ contains the optimal solution to two subproducts

Thus we can use Dynamic Programming

Consider a recursive solution
Then improve it's performance with memoization or by rewriting bottom up

Matrix Dimensions

Consider matrix prodcut $A_1\times \dots\times A_n$

Let the dimensions of matrix $A_i$ be $d_{i-1}\times d_i$

Thus the dimensions of matrix product $A_i \times \dots\times A_j$ is $d_{i-1}\times d_j$

Recursive Solution

Let $M[i,j]$ represent the number of multiplications required for matrix product $A_i\times\dots \times A_j$

For $1 ≤ i ≤ j < n$

$M[i,i] = 0$ since no product is required

The optimal solution of $A_i\times A_j$ must break at some point, $k$, with $i ≤ k < j$

Thus, $M[i,j] = M[i,k] + M[k+1,j] + d_{i-1}d_k d_j$

Thus, $ \displaystyle M[i,j] = \begin{cases} 0 & \text{if } i = j \\ \min_{i ≤ k < j} \{M[i,k] + M[k+1,j] + d_{i-1}d_k d_j\} & \text{if } i < j \end{cases} $

This is easily expressed as a recursive function (with exponential complexity)

What would this look like?

Complexity:

P(1) = 1
P(n) = Sum (k=1, n-1) P(k)P(n-k)
P(n) = $\Omega(f^n / n^{\frac{3}{2}})$

Efficient Computation

We must find a way to calculate this bottom up

Which values does $M[i,j]$ depend on?

Consider a $n \times n$ matrix of values M[i,j]

Diagonal is 0
$1 \le i \le j \le n$ is the upper right triangle

Consider some element $M[i,j]$

Where are $M[i,k]$ (for $i ≤ k ≤ j$):

Where are $M[k+1,j]$ (for $i ≤ k ≤ j$):

This tells us the order in which to build the table: By diagonals

By diagonal
Moving up and right from the diagonal that goes top-left to bottom-right

Diagonal indices:

On diagonal $0, j=i$
On diagonal $1, j = i+1$
On diagonal $q, j = i+q$
On diagonal $n-1, j = i+n-1$

Algorithm for Minimum Value

Bottom Up Algorithm to Calculate Minimum Number of Multiplications

n -- Number of arrays
d -- array of dimensions of arrays 1 .. n

    minimum_multiplication(n: integer; d: array(0..n))
        M: array(1..n, 1..n) := (others => 0);

        for diagonal in 1 .. n-1 loop 
            for i in 1 .. n - diagonal loop 
                j := i + diagonal;

                min := integer'last;
                for k in i .. j-1 loop
                    current := M[i, k] + M[k+1, j] + d(i-1)*d(k)*d(j) 
                    if current < min then
                        min := current
                    end if
                end loop
                M[i, j] := min
            end loop
        end loop

Complexity?

Algorithm for Location of Minimum Value

Bottom Up Algorithm to Calculate Minimum Number of Multiplications

n -- Number of arrays
d -- array of dimensions of arrays 1 .. n
P -- 2D array of locations of M[i, j]

    minimum_multiplication(n: integer; d: array(0..n)) 
        M: array(1..n, 1..n) := (others => 0);

        for diagonal in 1 .. n-1 loop 
            for i in 1 .. n-diagonal loop 
                j := i + diagonal;

                min_value := integer'last;
                for k in i .. j-1 loop
                    current := M[i, k] + M[k+1, j] + d(i-1)*d(k)*d(j) 
                    if current < min_value then
                        min_value := current
                        mink := k
                    end if
                end loop
                M[i, j] := mi_nvalue
                P(i, j) := mink
            end loop
        end loop

        printbest(P, i, j):
            if i=j then 
                print "A"_i
            else
                print '('
                printbest(P, i, M[i, j])
                printbest(P, M[i, j] + 1, j)
                print ')'

Note: the algorithm covered in class was incorrect: the two for loops were swapped. The algorithm above is correct.

Example Showing Tables and Calculations

Array dimensions:

A₁: 2 x 3
A₂: 3 x 5
A₃: 5 x 2
A₄: 2 x 4
A₅: 4 x 3

_i-1

Tables of $M_{ij}$ and $d_j$:

_ij

Calculating $M_{25}$ = number of multiplies in the optimal way to parenthesize $A_2A_3A_4A_5$:

Optimal locations for parentheses: