Order and Asymptotic Analysis

Topics

Topics include

Order: $O$, $\Theta$, $\Omega$ - sets of functions

Complexity Functions

Let f(n) represent the number of times that the BOP is executed

We call $f(n)$ a complexity function

Book states complexity functions are functions from positive integers to non-negative reals

We simplify by assuming non-negative integers to non-negative reals
Usually the distinction is not important

Order and Big Oh: Informal Definitions

Informally: we say function $g(n)$ is $O(f(n))$ and have $f(n)$ be $g(n)$ with constants and lower order terms removed

Example: $3n^4+10n^2+4n+11$ is $O(n^4)$

We also write: $3n^4+10n^2+4n+11$ = $O(n^4)$

We say that $g(n)$ is of order $f(n)$

Big Oh is an Upper Bound

When we formally define Big Oh, we will see $O$ is actually an upper bound

$g(n)$ is $O(f(n))$ if $g(n)$ is any function that grows at the same rate or more slowly than $f(n)$

Examples:

$3n^4+10n^2+4n+11$ = $O(n^4)$
$3n^4+10n^2+4n+11$ = $O(n^5)$
$3n^4+10n^2+4n+11$ = $O(n^6)$

$10n^2+4n+11$ = $O(n^4)$

$4n+11$ = $O(n^4)$
$4n+11$ = $O(n^3)$
$4n+11$ = $O(n^2)$
$4n+11$ = $O(n)$

$11$ = $O(n)$

$11$ = $O(1)$ [This means constant]

We call $O$ an Asymptotic Upper Bound:
Later we will see asymptotic lower and tight bounds (ie $\Omega$ and $\Theta$)

Formal Definition of $O$ : A Set of Functions

Example: $O(n^2)$ is a set of functions

What functions are in the set?

Answer: Any function that is bounded above by (a form of) $n^2$

Big Oh: Formal Definition

$O(f(n))$ is the set of complexity functions $g(n)$ for which
there exists some positive real constant $c$
and some nonnegative integer $N$
such that for all $n ≥ N$,
$g(n) ≤ c f(n)$

Big Oh: Examples Using Formal Definition

Let's look at graphs of $f$, $g$, and $cf(n)$, and calculate values for $c$ and $N$ for the examples below

Notice that we draw the graphs as continuous
Notice that $c$ and $N$ are NOT unique

$g(n) = n/2 = O(n) $
$g(n) = 1.5n = O(n) $
$g(n) = 1.5n + 1 = O(n) $

$g(n) = 4n = O(n^2) $

$g(n) = 0.5n^2 = O(n^2) $

$g(n) = n^2 + 1 = O(n^2)$

More Examples Choosing $c$ and $N$

Show that $n^2 + 10n = O(n^2)$

Choose $c=2$ and $N=10$, $(2n^2 ≥ n^2 + 10n\textrm{, for } n ≥ 10)$

Or choose $c=11$ and $N=1$, $(n^2 + 10n ≤ n^2 + 10n^2 = 11n^2$ for all $n≥ 1)$

Show $\frac{n(n-1)}{2} = O(n^2)$:

For $n ≥ 0$, $\frac{n(n-1)}{2} ≤ \frac{n^2}{2}$, so take $c=1/2$ and $N=0$

Asymptotic Lower Bounds

Asymptotic lower bound:
$\Omega(f(n))$ is the set of complexity functions $g(n)$ for which
there exists some positive real constant $c$
and some nonnegative integer $N$
such that for all $n ≥ N$,
$c f(n) ≤ g(n)$

Examples of Ω

Some functions in $\Omega(n^2):

$n^2 $
$n^2 + n$
$n^2 - n$
1000$n^2 + 1000 n$
1000$n^2 - 1000 n$

$n^3$
$n^2.00001$
$n^2 \lg \lg \lg n$
$ 2^{2^n}$

Show $\sqrt n = \Omega(\lg n)$

Choose $c = 1$ and $N = 16$ since $1\times \lg n ≤ \sqrt n$, for $n ≥ 16$
Or choose $c = 2$ and $N = 1000$ since $$\begin{align*} 2\times \lg N & = 2\times lg 1000 \\ & ≤ 2\times 10 \\ & ≤ 30 \\ & ≤ \sqrt{900} \\ & ≤ \sqrt{1000} \\ & = \sqrt N \end{align*} $$ and thus $2\times \lg N ≤ \sqrt N$ , for $N = 1000$

Use induction to show that $2\times \lg n ≤ \sqrt n$ , for $n ≥ N$.

Asymptotic Tight Bound

Asymptotic tight bound: $\Theta(f(n)) = O(f(n)) \cap \Theta(f(n))$

Thus, $\Theta(f(n))$ is the set of complexity functions $g(n)$ for which
there exists some positive real constants $c$ and $d$
and some nonnegative integer $N$
such that for all $n ≥ N$,
$c f(n) ≤ g(n) ≤ d f(n)$

Example Tight Bound

n²/2 - 2n = Θ(n²) with c₁ = 1/4, c₂ = 1/2, and n₀ = 8

Remember: Leading constants and low-order terms don't matter

Comparing $O$, $\Omega$, and $\Theta$

Figure 1.4 and 1.5 show $O$, $\Omega$, and $\Theta$

How to read:

Read $g(n) = O(f(n))$ as $g(n)$ is order at most $f(n)$
Read $g(n) = \Omega(f(n))$ as $g(n)$ is order at least $f(n)$
Read $g(n) = \Theta(f(n))$ as $g(n)$ is order exactly $f(n)$

$O$, $\Omega$, and $\Theta$ are Sets

Figure 1.6 shows relationship among sets $O$, $\Omega$, and $\Theta$

Notice that $g(n)=O(f(n))$ is misused: should be $g(n) \in O(f(n))$

Note that we write $2n^3 = O(n^3)$ but not $O(n^3) = 5n^3$
Otherwise we might end up with $2n^3 = 5 n^3$!

Properties of Asymptotic Performance

Reflexive: $f(n) = O(f(n))$ [true for all 3]

Transitive: $g(n) = \Theta(f(n)) \textrm{ and } f(n) = \Theta(h(n)) \Longrightarrow f(n) = \Theta(h(n))$ [true for all 3]

Symmetry: $g(n) = \Theta(f(n)) \Longleftrightarrow f(n) = \Theta(g(n))$ [$\Theta$ only]

$g(n) = O(f(n)) \Longleftrightarrow f(n) = \Omega(g(n))$

Constant: $g(n) = O(f(n)) \Longrightarrow dg(n) = O(f(n))$, for positive constant $d$

Logs: $\log_a n = \Theta(\log_b n)), a, b > 0$ [Thus all logs are the same complexity and we can use $\lg n$ at any time]

Powers of $n$: $n^a = o(a^{n+\epsilon}), 1 < a, 0 < \epsilon$ [larger powers dominate smaller powers]

Exponential Bases: $a^n = o(b^n), 0 < a < b$

Factorial: $a^n = o(n!), 0 < a$

Sum: $e(n) = O(f(n))$ and $g(n) = O(h(n))\Longrightarrow e(n) + g(n) = O(f(n) + (h(n))$

Product: $e(n) = O(f(n))$ and $g(n) = O(h(n))\Longrightarrow e(n)\times g(n) = O(f(n)\times (h(n))$

Motivating Asymptotic Analysis

Order ignores lots of details of performance.

Is this reasonable?

Yes:

Difficult to determine performance exactly
Order of growth makes huge difference as $n$ gets larger
In many cases, what matters is the proportional growth rate

What happens to performance as input size doubles?
Don't worry about constants: $n^2$ and $4n^2$ both increase by the square
As $n$ gets large, high terms dominate growth rate
Simplify analysis by taking n large enough that only largest term matters

No:

Sometimes constants do matter
Example: Matrix multiplication

Naive algorithm: $\Theta(n^3 - n^2)$
Strassen's algorithm: $\Theta(7n^{lg 7} - 6n^2) $
Strassens is better for $n > 654$ [Wolfram MathWorld]

Synthetic example:

$10^{100}$ better than $10n\lg n$, but only for VERY large $n$

Little Oh

$o(f(n))$ is the set of complexity functions $g(n)$ for which
there for every positive real constant $c$
and some nonnegative integer $N$
such that for all $n ≥ N$,
$g(n) ≤ c \times f(n)$

Little Oh Example and Intuition

$n \in o(n^2)$
For any $c$, choose $n ≥ \frac{1}{c}$ [see book]

No matter how small $c$ is, $f(n)$ can be large enough that $cf(n)$ is larger then $g(n)$

Notice that $f(n)$ gets large faster than $g(n)$

$o(f(n))$ is the larger members of $O(f(n))$

That is, the ones that are NOT $\Theta(f(n))$

Equivalent definition: $\displaystyle\lim_{n\to\infty}\frac{g(n)}{f(n)} = 0$

Growth Rate Comparisons

Growth rate of various functions
Each function is little oh of the function to its right

lg n	n	sqrt(n)	n	nlg n	n²	n³	2ⁿ
1	2	1.4	2	2	4	8	4
2	4	2	4	8	16	64	16
3	8	2.8	8	24	64	512	256
4	16	4	16	64	256	4096	65536
5	32	5.7	32	160	1024	32,768	4,294,967,296
6	64	8	64	384	4096	262,144	1.8 * 10^19
7	128	11.3	128	896	16,384	2,097,152	3.4 * 10^38
8	256	16	256	2,048	65,536	16,777,216	1.2 * 10^77
9	512	22.6	512	4,608	262,144	134,217,728	1.3 * 10^154
10	1024	32	1024	10,240	1,048,576	1,073,741,824	1.8 * 10^308

Problem size for a second, minute, hour (assuming 1 MIPS)

f(n)	second	minute	hour
n	1,000,000	60,000,000	3,600,000,000
n²	1000	7,746	60,000
n⁴	32	88	245
2ⁿ	20	26	32

Increase in problem size with 1024 times faster machine

n	1024.0x
n²	32.0x
n⁴	5.7x
2ⁿ	Original + 10

Common Performance Classes

1: Constant - each instruction is executed about once

lg n: solves problem by cutting it down in size

n: solves problem by visiting each element once

n lg n: solves problem by cutting it down in size, solving each part, and combining parts

n²: solves problem by processing each pair once, typically with a nested loop

n³: solves problem by processing each triple once, typically with three nested loops

2ⁿ: solves problem by brute force

Behavior of Common Performance Classes

f(n)	n for f(n) = 10^6	change in f(n) as n doubles
1		No change
lg n	10^300,000	small constant
n	1,000,000	doubles
n lg n	64,000	double +c
n²	1,000	4 times
n³	100	8 times
2ⁿ	20	squares!

Asymptotic Notation in Equations

Remember, Θ(n) is a set

Usually we describe the asymptotic performance of f(n) with notation that looks like an equation: f(n) = Θ(n²)

But remember, this is not an equation; instead it means f(n) ∈ Θ(n²

We extend this notation to more complex equations involving asymptotic notation (AN):

AN is in an expression on the RHS of an equation and not on the LHS
AN is in an expression on both LHS and the RHS of an equation
We ignore the case where AN is in an expression on the LHS of an equation and not on the LHS

Asymptotic Notation alone on the Right Side of Equations

Interpretation: Asymptotic Notation (AN) on RHS denotes the function from the AN set that makes the equation true

Example: 2n² + 3n + 1 = 2n² + &Theta(n) means 2n² + 3n + 1 = 2n² + f(n) for some for some f(n) ∈ Θ(n)

In particular, f(n) = 3n + 1

Asymptotic Notation on the Left and Right Side of an Equation

Interpretation: For every function in AN on LHS, the AN on the RHS denotes the function in the RHS AN that makes the equation true

Example: 2n² + Θ(n) = Θ(n²) means for all f(n) ∈ Θ(n) there is a g(n) ∈ Θ(n²) that makes the equation true

That is, such that 2n² + f(n) = g(n)

Now, how do we interpret this:

2n² + 3n + 1 = 2n² + Θ(n) = Θ(n²)

AN in equations is common as we deal with performance