Linear Sorts

Overview

Lower bounds on sorting

Decision trees

Linear sorts

Count Sort
Radix Sort
Bucket Sort

Overview of sorting algorithms

Lower Bounds on Sorting

Question: what are the lower bounds on sorting performance

What is the best best case performance that an algorithm can have?
What is the best worst case performance that an algorithm can have?

Consider the sorting algorithms that we know:

What is the best best case performance?
What is the best worst case performance?

Are there algorithms with better best and worst case performance?

We can prove that no algorithms do better!
This gives us lower bounds on sorting performance

Lower Bound of Best Case Sorting Performance

What is a lower bound for best case?

Must look at each element at least once, so ...

Lower bound for best case is Ω(n)

The best case for sorting can be no better than Ω(n)

Lower Bound of Worst Case Sorting Performance

What is a lower bound for worst case

Answer: Ω(n lg n)

The worst case for sorting can be no worse than Ω(n lg n)

How do we know? Analyze using decision trees

Decision Trees

A decision tree shows the comparisons that occur when sorting any ordering of n elements

Interior nodes represent a comparison of 2 elements
Leaves represent orderings of all elements

Example: Insertion sort on 3 elements

                       (1:2)
                    /         \
                 <=             >
                /                  \
           (2:3)                    (1:3)
           /   \                   /      \
        <=       >               <=        >
        /           \            /          \
     [1,2,3]       (1:3)     [2,1,3]       (2:3)
                  /     \                 /     \
               <=         >             <=       >
               /           \            /         \
           [1,3,2]      [3,1,2]      [2,3,1]    [3,2,1]

(i:j) means compare compare Ai and Aj, using their original numbering!
[i,j,k] is the sorted ordering, using the original numbering!

Decision Trees Continued

Any execution of a sorting algorithm can be represented as a DT

Each ordering appears as a leaf at least once
At least n! leaves

The comparisons needed to sort any size 3 array is a path from root to leaf

Length of path = number of comparisons
What if there were 4 elements?

For a given algorithm:

One tree for each n
Each order of values corresponds ton one path from root to a leaf

One DT represents the comparisons needed by a given sorting algorithm to sort all possible all possible orderings of n elements

What is longest path for quicksort? mergesort?

How many leaves on a tree for sorting n elements?
We focus on comparisons and ignore swaps and control

Decision Tree Analysis

Ultimate Result: Height of decision tree that sorts n elements is Ω(n lg n)

Proof below
For any algorithm, the worst case has Ω(n lg n) comparisons

Proof:

Let l be the number of leaves in the tree and h the height of the tree
2^h ≥ l

Prove by induction: true for height 0, assume for height h-1, at height h, each leaf generates 1 or 2 new leaves, no more than doubling l

l ≥ n! (One or more leaves per ordering, n! orderings)
2^h ≥ n!

h ≥ lg(n!)

≥ lg(n/e)ⁿ [by Stirlings approximation]

= n lg(n/e)
= n lg n - n lg e

= Ω(n lg n)

Linear Sorts!

How is this possible to have a linear sort? A: not a comparison sort

Linear sorts:

Counting Sort: n numbers in range 0..k = Θ(n + k)
Radix Sort: Θ(d(n + k)) for d digits, each in range 0..k
Bucket Sort: Θ(n) for n numbers uniformly distributed over [0,1)

Counting Sort: Algorithm Design Technique

Design technique: Transform and Conquer

Specifically: Input enhancement

Enhancement: Additional table of value counts, and thus locations

Counting Sort

Input: A(1..n) of [0 .. k]
Output: B(1..n) sorted
Auxilliary Storage: C[0..k]

    CountingSort(A, B, n, k)
        Allocate C(0..k) of Natural := (others => 0)
        for j in 1 .. n loop
            C[A[j]] := C[A[j]] + 1
        for i in 1 .. k loop
            C[i] := C[i] + C[i-1]
        for j in reverse 1 .. n loop
            B[C[A[j]]] := A[j]
            C[A[j]] := C[A[j]] - 1

Example: 2, 5, 3, 0, 2', 3', 0', 3''

Why copy A in reverse, and what does value in array C specify?

CountingSort Continued

Analysis: Θ(n + k), which is Θ(n) if k = O(n)
Practical values of k?

32 bit: Too big
16 bit: Too big??
8 bit: maybe, depending on n

Stable
Trades time for space
Counting Sort is used in Radix sort

Radix Sort

Sort by digits!
Sort least significant first

    RadixSort(A, d)  -- To sort d digits
    for i in 1 .. d loop
        use a stable sort to sort A on digit i

Example - sort: 347 343 944 928 232 466 524 426 245 334

How do we prove that it works? ...

Why does it work: Sort must be stable: values with equal digits in position i must maintain their order as sorted by digit i+1

What sort do we use on each digit?

Performance:

n numbers, with digits in range 0 .. k
Cost per pass: Θ(n + k)
each number has d digits, thus d passes
Total: Θ(d(n + k))

If k = O(n), then time = Θ(d*n))

IBM Card Sorter

Breaking a Word into Digits

Radix sort does not require using decimal digits

Instead use r-bit digits
Range of each digit 0 .. 2^r-1 (not 0..9)
Break each word into groups of r digits

Consider words that are b-bits long

Break into the b-bits words into r-bit digits
Gives d = ⌈(b/r)⌉ digits

For counting sort, gives range: 0 .. k = 0 .. 2^r-1

Example:

b = 32 bit words
r = 8 bit digits
d = ⌈(b/r)⌉ = 32/8 = 4 digits
k = 2^r-1 = 2^8-1 = 255 digit range

Choosing r ≈ lg n gives performance of

d = ⌈(b/r)⌉ = ⌈(b/lg n)⌉
Performance: Θ(n*d) = Θ(n*b/r) = Θ(n*b/lg n)

Breaking a Word into Digits: Example and Comparison

Example - to sort 2^16 numbers, each of 32 bits:

Radix Sort:

n = 2^16 numbers
r = lg n = lg 2^16 = 16 bit digits
k = 2^r-1 = 2^16-1 ≈ 64,000 digit range

b = 32 bit words
d = ⌈(b/r)⌉ = 32/16 = 2 digits
2 digits requires 2 calls to count sort

Requires 4 passes since each count sort call requires to passes: one for census, one to move
Thus, Radix Sort performance: 4n

Merge and quick sort: n lg n = 16 n

Example 2 - 1,000,000 numbers, each 32 bits :

Radix sort :

2^20 numbers, each of 32 bits
n = 2^20 numbers
r = lg n = lg 2^20 = 20 bit digits
k = 2^r-1 = 2^20-1 ≈ 1,000,000 digit range

b = 32 bit words
d = ⌈(b/r)⌉ = ⌈ 32/20⌉ = 2 digits
2 digits requires 4 passes

Thus, Radix Sort performance: 4n

Merge and quick sort: n lg n = 20 n

Bucket Sort

Sort randomly distributed data from range [0,1)

Intuition:

Divide [0,1) into n equal sized buckets
Distribute the n values into these buckets
Sort each bucket
Process buckets in order, listing elements

Algorithm:

    BucketSort(A, n)
        Allocate B[0..n-1] := (others => emptyList)

        for i in 1 .. n loop
            insert A[i] into list B[floor(n * A[i])]

        for i in 0 .. n-1 loop
            sort list B[i] with insertion sort

        Concatenate lists B[0..n-1], in order

        Return concatenated lists

Bucket Sort Performance

All lines together except insertion sort are Θ(n)

Average number of elements per bucket = ??
If all lists are small, takes O(1) time to sort

Total = Θ(n)

Formal analysis ... we will skip

What if points are not uniformly distributed?

Overview of Sorting Algorithms

Exchange Sorting

Bubble sort: n best, n^2 worst and expected
Improved: Quicksort: n lg n best and expected, n^2 worst

Selection Sorting

Selection sort: n^2 best, n^2 worst
Improved: Heapsort: n lg n best and worst

Insertion Sorting

Insertion sort: n best, n^2 worst and expected
Improved: Shellsort: n best, n^2 worst, around n lg² n or n^1.2 expected
Improved: Mergesort: n lg n best and worst
Improved: Treesort: n lg n best and expected; n^2 worst

Linear:

Counting Sort: n numbers in range 0..k = Θ(n + k)
Radix Sort: Θ(d(n + k)) for d digits, each in range 0..k
Bucket Sort: Θ(n) for n numbers uniformly distributed over [0,1)
Linear sorts don't use comparisons to gain information about value

Sorting Out Sorting