Divide and Conquer Algorithms

Student Outcomes

• Define divide and conquer approach to algorithm design


• Describe and answer questions about example divide and conquer algorithms
• Binary Search
• Quick Sort
• Merge Sort
• Integer Multiplication
• Matrix Multiplication (Strassen's algorithm)
• Maximal Subsequence


• Apply the divide and conquer approach to algorithm design


• Analyze performance of a divide and conquer algorithm


• Compare a divide and conquer algorithm to another algorithm

Essence of Divide and Conquer

1. Divide problem into several smaller subproblems
• Normally, the subproblems are similar to the original


2. Conquer the subproblems by solving them recursively
• Base case: solve small enough problems by brute force


3. Combine the solutions to get a solution to the subproblems
• And finally a solution to the orginal problem


4. Divide and Conquer algorithms are normally recursive

Binary Search

Recursive Binary Search

• A Divide and Conquer Algorithm to find a key in an array:

    -- Precondition: S is a sorted list
    index binsearch(number n, index low, index high, 
            const keytype S[], keytype x)
        if low ≤ high then
            mid = (low + high) / 2
            if x = S[mid] then
                return mid
            elsif x < s[mid] then
                return binsearch(n, low, mid-1, S, x)
            else
                return binsearch(n, mid+1, high, S, x)
        else
            return 0
    end binsearch
    

• Divide: search lower or upper half
• Divide: select lower or upper half
• Conquer: search selected half
• Combine: None
• Performance:
• $T(n) = T(n/2) + \Theta(1) $
• $T(n) = \Theta(\lg n)$ (proved earlier)

Merge Sort

Recursive Merge Sort

• A Divide and Conquer Algorithm to sort an array:

    void mergesort(S: array of keytype)
        len = S'length
        if len > 1 then
            -- Divide: Copy the arrays
            mid: constant int := len / 2
            rest: int len - mid
            U: array(1..mid) of keytype := S(1..mid)
            V: array(1..rest) of keytype := S(mid+1..n)

            -- Conquire: Recursively sort
            mergesort(U)
            mergesort(V)

            -- Combine: merge sorted arrays
            merge(U, V, S)
    end mergesort
    void merge(L, R, S: array(<>) of keytype)
        i, j, k: int := 1
        lenL: constant int := L'length
        lenR: constant int := R'length
        while i <= lenL and j <= lenR 
            if L(i) < R(j) then
                S(k) := L(i);  i := i + 1;
            else -- R(i) ≤ L(j) then
                S(k) := R(j);  j := j + 1;
            k := k + 1
        if i > lenL then S(k..lenL+lenR) := R(j..lenR)
        else                S(k..lenL+lenR) := L(i..lenL)
    

• Time Performance:
• $T(n) = 2T(n/2) + \Theta(n) $
• $T(n) = \Theta(n\lg n)$ (by induction or master method)


• Space Performance:
• Requires $\theta(k/2)$ new space for each recursive call
• Number of recursive calls until size 1: $\lg n$
• Space used for call with size $n$ array: $\displaystyle n\sum_{i=0}^{\lg n} \frac{1}{2^i} = n(1+1/2 + 1/4+\dots) = n\Theta(2) = \Theta(2n)$
• This extra space can be reused for next recursive call (think about the diagram)
• Total space: $\Theta(3n)$
• Can we do better?

Recursive Merge Sort - Version 2

• Uses global variable S

    void mergesort2(low, high: int)
        if low  < high then
            mid: constant int := (low + high) / 2
            mergesort(low, mid)
            mergesort(mid+1, high)

            merge2(low, mid, high)
    end mergesort2
    -- Merges from S(LOW..MID) and S(MID+1..HIGH) to U(LOW..HIGH)
    -- Then copies U(LOW..HIGH) back to S(LOW..HIGH)
    void merge2(LOW, MID, HIGH: int)
        U: array(LOW..HIGH) of keytype
        i: int := LOW
        j: int := MID
        k: int := HIGH
        while i <= MID and j <= HIGH
            if S(i) < S(j) then
                U(k) := S(i);  i := i + 1;
            else -- S(j) < S(i) then
                U(k) := S(j);  j := j + 1;
            k := k + 1

        if i > MID then U(k..HIGH) := S(j..HIGH)
        else               U(k..HIGH) := S(i..MID)

        S(LOW..HIGH) := U(LOW..HIGH)
    

• Time Performance:
• $T(n) = 2T(n/2) + \Theta(n) $
• $T(n) = \Theta(n\lg n)$ (by induction or master method)


• Space Performance:
• Never uses more than $\theta(n)$ extra space for call to merge
• Extra space is reused for next call
• Total space: $\Theta(2n)$


• But, how much space is required for the recursion?
• Answer: ???


• Another improvement: make S size 2n, and merge from one side to the other

Quick Sort

Quick Sort - Algorithm

• Quicksort is another example of divide and conquer
• Recursive Algorithm:

    -- Assume global array S
    void quicksort(low, high: int)
        if low  < high then
            mid: constant int := (low + high) / 2
            partition(low, high, pivot)
            quicksort(low, pivot-1)
            quicksort(pivot+1, high)
    end quicksort

    void partition(LOW, HIGH: int; PIVOTPOINT: out int)
        j: int := LOW
        pivot_item: keytype := S(j)
        for i in LOW+1 .. HIGH loop
            if S(i) < pivot_item then
                j := j + 1
                swap(S(i), S(j))
            end if
        end loop
        PIVOTPOINT := j
        swap(S(LOW), S(PIVOTPOINT))
    end partition
    

Quick Sort - Worst Case Time Performance

• Time Performance - Worst Case:
• $T(n) = T(n-1) + T(0) + \Theta(n) $
• $T(n) = \Theta(n^2)$ (by induction)
• Assume $T(k) = k^2 + \Theta(k)$ for $k < n$
$\begin{align*} T(n) & = T(n-1) + \Theta(n) \\ & = (n-1)^2 + \Theta(n) \\ & = n^2 - 2n + 1 + \Theta(n) \\ & = n^2 + \Theta(n) + \Theta(n) \\ & = n^2 + \Theta(n) \\ & = \Theta(n^2) \end{align*} $
• Actually: how do we know that this is worst case?
• It could be proved, but we won't do it

Quick Sort - Expected Case Time Performance

• Time Performance - Expected Case:
• Assume all problem instances are equally likely
• This means that any element is equally likely to be the pivot (ie returned by partition
• Thus the expected value of $T(n)$ is
$\displaystyle \begin{align*} T(n) & = \frac{1}{n} \sum_{p=1}^n [ T(p-1) + T(n-p)] + n-1 \\ & = \sum_{p=1}^n \frac{1}{n} [ T(p-1) + T(n-p)] + n-1 \\ & = \Theta(n \lg n) \end{align*} $
• See text for solution

Quick Sort - Best Case Time Performance

• Time Performance - Best Case:
• $T(n) = 2T(n/2) + \Theta(n) $
• $T(n) = \Theta(n\lg n)$ (by induction or master method)


• Why is this the best case: think about a recursion tree

Quick Sort - Constant Performance Time Performance

• Time Performance: Constant proportion
• What if pivot always divides by a constant proportion: say 10:90
• $T(n) = T(9n/10) + T(n/10) + \Theta(n) = O(n\lg n)$
• Think about the tree:
• The shallow side of the tree terminates in $\log_{10}n$ levels
• Each of these levels has weight $cn$
• The deep side of the tree terminates in $\log_{10/9}n$ levels
• Each of these levels has weight $≤ cn$
• The entire tree has weight $O(cn\log_{10})n + cn\log_{10/9}n = O(n\lg n)$

Quick Sort - Space Performance

• Space Performance:
• No extra space needed
• Total space: $\Theta(n)$
• But, ... what about the space for ...?

Iterative Quick Sort

• Can we write quicksort iteratively?
• Push and pop the bounds of the half not currently sorted
• Requires an explicit stack (or array to simulate a stack)
• Explicit stack replaces recursion
• Should you stack smaller or larger?

Quick Sort - Improving Time Performance

• Partition from both ends


• How can we avoid the worst case?


• Better choice of pivot


• Randomized: choose a random element as the pivot
• Swap random element with first and use same algorithm
• Each element has equally likely chance of being pivot
• Particular characteristics of the data won't cause worst case


• Median of 3: choose 3 elements at random and take their median
• Or median of some other k


• Use insertion sort when list gets small


• Don't sort small lists and use insertion sort on entire list

One-Way, Stackless Quicksort!

• Repeat pivoting until positions 1 .. L are filled with pivots
• Store pivots as negative
• Works for positive only

Matrix Multiplication

Divide and Conquer Example 2 - Matrix Multiplication

• Input: A, B: Array(1 .. $n$) of number
• Output: C: Array(1 .. $n$) := A x B
• Remember how to multiply matrices?


• Obvious algorithm: C has $n^2$ entries, each of which multiples $n$ pairs
• $c_{ij} = \displaystyle \sum_{k=1}^n a_{ik}b_{kj}$
• Nested loop algorithm
• Performance:

Simple Divide and Conquer Method

• Break A into A11, A12, A21, A22
• Break B into B11, B12, B21, B22
• Break C into C11, C12, C21, C22


• Let C = product of 2 n/2 by n/2 arrays
$C_{11} = A_{11}\times B_{11} + A_{12}\times B_{21}$
$C_{12} = A_{11}\times B_{12} + A_{12}\times B_{22}$
$C_{21} = A_{21}\times B_{11} + A_{22}\times B_{21}$
$C_{22} = A_{21}\times B_{12} + A_{22}\times B_{22}$


• $T(n) = \Theta(1) + 8T(n/2) + \Theta(n^2)$
• $\Theta(1)$ time to partition matrices
• 8 Multiplications of arrays of size $n/2$
• $\Theta(n^2)$ time to add $n\times n$ matrices


• $T(n) = \Theta(n^3)$ by master method

Strassen's Algorithm

• Uses 7 rather than 8 multiplications of n/2 by n/2 arrays


• Has more additions and subtractions of n/2 by n/2 arrays


• Algorithm:
• Create 10 $n/2$ by $n/2$ sum arrays: S1 .. S10
• Recursively compute 7 product arrays: $m_1$ .. $m_7$
• Compute $c_{11}, c_{12}, c_{21}, c_{22}$ by adding and subtracting combinations of $m_i$

$ T(n) = \begin{cases} \Theta(1), & \text{if} n = 1 \\ 7T(n/2) + \Theta(n^2), & \text{if} n > 1 \end{cases} $

• Solution (by Master Method): $T(n) = \Theta(n^{\lg 7}) = \Theta(n^{2.81}) $
  
• Notice: Has higher constants


• Faster solutions are known: $T(n) = O(n^{2.38})$

Large Integer Multiplication

Large Integer Multiplication

• Assume integers with $n$ digits
• Implement with array with one digit per element [Ada bignumpkg]


• Grade school algorithm: $\Theta(n^2)$


• Naive divide and conquer:
• $1234 \times 5678 = (12\times 56)\times 10^4 + [(12\times 78) + (34\times 56)]\times 10^2 + (34 \times 78)\times 10^0$
• Performance: $T(n) = 4T(n/2) + \Theta(n) = \Theta(n^2) $
• More generally: $xy \times wz = xw\times 10^{2k} + (xz+yw)\times 10^k + yz$
• Assume $xy$ has $2k$ digits
• This requires 4 multiplications


• Improved divide and conquer - calculate three products to find result:
• $r = (x+y) \times (w+z) = xw + (xz + yw) + yz$
• $p = xw$
• $q = yz$
• $xy\times wz = p\times 10^{2k} + (r-p-q) \times 10^k + q$
• Performance: $T(n) = 3T(n/2) + \Theta(n) = \Theta(n^{\lg 3})$ by master method

Maximal Subarray

Example Divide and Conquer: Maximal-subarray Problem

• Problem:
• Input: A: Array(1 .. n) of numbers
• Output: Indices i and j such that sum(A(i .. j)) has the maximum value


• Assume some are negative
• Otherwise problem is trivial


• Example: A := (1, -100, 10, 20, -1, -5, 16, -23, 5)
• Solution: i = ??, j = ??

Maximal Subarray: Example Scenario

• Stock prices over n days (in the past)
• Maximize profits
• Will maximum solution sell/buy at global max/min? Not necessarily

Maximal Subarray: Brute Force Solution

• How to solve by brute force?
• Performance?

Maximal Subarray: Divide and Conquer Solution

• Try splitting in half and solve each half
• How to use partial solution in entire solution
• What complications occur?
• How to solve complications

Maximal Subarray Algorithm

• Divide and Conquer Algorithm

    maxsub(int[] S; low, high: int) return (lowIndex, highIndex, sum)
        if low = high then
            return (low, high, S(low))
        else
            mid = (low + high) / 2
            (llow, lhigh, lsum) = maxsub(S, low, mid)
            (rlow, rhigh, rsum) = maxsub(S, mid+1, high)
            (mlow, mhigh, msum) = middlemaxsub(S, low, mid, high)
        end if;
        return triple with highest sum
    end maxsub

    middlemaxsub(int[] S; low, high, int) return (lowIndex, highIndex, sum)
        start at mid and find bestleft and leftsum
        start at mid and find bestright and rightsum

        return (bestleft, bestright, rightsum+leftsum)
    end middlemaxsub
    

Performance of Divide and Conquer Solution

• Performance:
• Base Case: $T(1) = \Theta(1)$
• Divide: $ \Theta(1)$
• Conquer: $2T(n/2)$ and $\Theta(n)$
• Combine: $\Theta(1)$


• $T(n) = 2T(n/2) + \Theta(n)$


• Closed form: $T(n) = \dots$


• Can we do better?
• Yes! A $\Theta(n)$ algorithm exists! (Dynamic programming!)