Additional Points on Asymptotic Complexity

Topics

Terminology
Upper, lower, and exact bounds
Sets
Formal definitions and examples using them
Little O and Growth rates comparison and behavior
Omit constants and lower order terms
Additional properties
History

TERMINOLOGY

Terminology

Complexity classes are sets of functions

We use all of these notations:

$f(n)$ is $O(g(n))$

$f(n) = O(g(n))$

$f(n) \in O(g(n))$

More Terminology

We can emphasize uppper, lower, and exact bounds using this terminology:

For $f(n) \in O(g(n))$ we say $f(n)$ is order at most $g(n)$

For $f(n) \in \Omega(g(n))$ we say $f(n)$ is order at least $g(n)$

For $f(n) \in \Theta(g(n))$ we say $f(n)$ is order exactly $g(n)$

Example: $10n^2$ is order exactly $n^2$

Sometimes informally omit "order".

UPPER, LOWER, TIGHT BOUNDS

$O$ is an Upper Bound

Remember: $f(n) = O(g(n))$ means that $f(n)$ is at most $g(n)$

Thus, $10n$ is order $O(n)$, and $O(n^2)$, and $O(n^3)$

What else could we say about the example $3n^2 + 10n + 4$?

$\Omega$ is a Lower Bound

Remember: $f(n) = \Omega(g(n))$ means that $f(n)$ is at least $g(n)$

Thus, $10n^3$ is order $\Omega(n)$, and $\Omega(n^2)$, and $\Omega(n^3)$

$\Theta$ is a Tight Bound

Remember: $f(n) = \Theta(g(n))$ means that $f(n)$ is exactly $g(n)$

Thus, $10n^3$ is order exactly $\Theta(n^3)$, but not $\Omega(n^2)$, and $O(n^5)$

SETS and FORMAL DEFINITIONS

Thinking About Sets of Functions

Consider a Venn diagrams for complexity classes at least, at most, and exactly n^2:
What are some functions in each
How do the sets relate to each other

Formal Definitions of $O, \Omega$, and $\Theta$

$f(n) \in O(g(n))$ iff there exists positive constants c and $n_0$ such that $f(n) \le c\ g(n)$ for every $n \ge n_0$.

$f(n) \in \Omega(g(n))$ iff there exists positive constants c and $n_0$ such that $f(n) \ge c\ g(n)$ for every $n \ge n_0$.

$f(n) \in \Theta(g(n))$ iff there exists positive constants $c_1$ and $c_2$ and $n_0$ such that $c_1 g(n) \le f(n) \le c_2\ g(n)$ for every $n \ge n_0$.

Examples Using Formal Definitions

Definition: $f(n) \in O(g(n))$ iff there exists positive constants c and $n_0$ such that $f(n) \le c\ g(n)$ for every $n \ge n_0$.

What are $c$ and $n_0$ in each of these:

$f(n) = n/2 = O(n) $

$f(n) = 1.5n = O(n) $

$f(n) = 1.5n + 1 = O(n) $

$f(n) = 0.5n^2 = O(n^2) $

$f(n) = n^2 + 4 = O(n^2)$

Notice that larger values for $n_0$ always work.

More Examples Using Formal Definitions

Definition: $f(n) \in O(g(n))$ iff there exists positive constants c and $n_0$ such that $f(n) \le c\ g(n)$ for every $n \ge n_0$.

What are $c$ and $n_0$ in each of these:

$f(n) = \frac{1}{2}n^2 - 3n \in \Omega(n^2)$

$0 \lt c_1n^2 \le \frac{1}{2}n^2 - 3n $ for any $c_1\le\frac{1}{2} - \frac{3}{7}=\frac{1}{14},n\gtn_0, n_0\gt7$

$f(n) = \frac{1}{2}n^2 - 3n \in O(n^2)$

$\frac{1}{2}n^2 - 3n \le c_2n^2$ for any $c_2\ge\frac{1}{w},n\gtn_0, n_0\gt1$

$f(n) = 3n^2 + 7n +15 \in O(n^2)$

$= \lt c_2n^2 \le 3n^2 + 7n +15$ for any $c_2\le\frac{1}{w},n\gtn_0, n_0\gt1$

Notice that larger values for $n_0$ always work.

Showing $O$ is an Upper Bound Using Definition

Remember: $f(n) = O(g(n))$ means that $f(n)$ is at least $g(n)$

Thus, $10n$ is order $O(n)$, and $O(n^2)$, and $O(n^3)$

Show that $10n = O(n^2)$

Let $f(n) = 10n$ and $g(n) = n^2$.
Then for $n \ge 1, 10n \le 10n^2$.
Thus, $f(n) \le cg(n)$ for $n\ge 1$ and with $c=10$ and $n_0=1$

Alternate solution:

If we let $c=1$ and $n_0=10$,
then we have $10n \le 1 n^2$ for all $n\ge 10$

LITTLE O, little $\omega$, and GROWTH RATES

Little O

$\displaystyle f(n) \textrm{ is } o(g(n)) \textrm { iff } \lim_{n\to\infty}\frac{f(n)}{g(n)} = 0$

This means that g grows faster than, and not equal to, f

Thus, n^2 is o(n^3) but n^2 is NOT o(n^2)

Remember, n^2 is O(n^2) and O(n^3)

Little $\omega$

$\displaystyle f(n) \textrm{ is } \omega(g(n)) \textrm { iff } \lim_{n\to\infty}\frac{f(n)}{g(n)} = \infty$

This means that g grows slower than, and not equal to, f

Thus, $n^3$ is $\omega(n^2)$ but $n^3$ is NOT $\omega(n^3)$

Remember, n^3 is Ω(n^2) and Ω(n^3)

Growth Rate Comparisons

Growth rate of various functions
Each function is little oh of the function to its right

lg n	n	sqrt(n)	n	nlg n	n²	n³	2ⁿ
1	2	1.4	2	2	4	8	4
2	4	2	4	8	16	64	16
3	8	2.8	8	24	64	512	256
4	16	4	16	64	256	4096	65536
5	32	5.7	32	160	1024	32,768	4,294,967,296
6	64	8	64	384	4096	262,144	1.8 * 10^19
7	128	11.3	128	896	16,384	2,097,152	3.4 * 10^38
8	256	16	256	2,048	65,536	16,777,216	1.2 * 10^77
9	512	22.6	512	4,608	262,144	134,217,728	1.3 * 10^154
10	1024	32	1024	10,240	1,048,576	1,073,741,824	1.8 * 10^308

Problem size for a second, minute, hour

assume f(n) microseconds to solve problem of size n and 1 MIPS machine

f(n)	second	minute	hour
n	1,000,000	60,000,000	3,600,000,000
n²	1000	7,746	60,000
n⁴	32	88	245
2ⁿ	20	26	32

Increase in problem size with 1024 times faster machine

n	1024.0x
n²	32.0x
n⁴	5.7x
2ⁿ	Original + 10

Calculating Problem Size Increase

Assume T(n) = e C(n) , where e is the execution time for a basic operation and C(n) is the number of times the basic op is executed
Assume C(n) = k n, for some constant k

Now consider doubling problem size, $n_0$ to $n_1 = 2n_0$:
Doubling problem size causes time to double

Now assume C(n) = k n², for some constant k

Now consider doubling problem size, $n_0$ to $n_2 = 2n_0$:
Doubling problem size causes time to quadruple

Behavior of Common Growth Rates

*f(n)*	*n for f(n)* = 10^6**	*change in f(n)* as n doubles**
1		No change
lg n	10^300,000	small constant
n	1,000,000	doubles
n lg n	64,000	double +c
n²	1,000	4 times
n³	100	8 times
2ⁿ	20	squares!

OMIT CONSTANTS and LOWER TERMS

Omit Constants and Lower Order Terms

When considering asymptotic complexity, omit any constants and lower order terms, if they exist

Example: $10n^2 + 3^n + 4$ is order exactly $n^2$

Why:

As n gets large, the lower order terms become less significant

Consider the example when n is 1, 1_000, 1_000_000

The growth rate of the highest term is more important than its constant

n^3, 1000 n^3, and 0.001 n^3 are all cubics and grow faster then n^2, for example, when n is large

ADDITIONAL PROPERTIES and HISTORY

Additional Properties of Asymptotic Performance

Reflexive: $f(n) = O(f(n))$ [true for all 3]

Transitive: $g(n) = \Theta(f(n)) \textrm{ and } f(n) = \Theta(h(n)) \Longrightarrow f(n) = \Theta(h(n))$ [true for all 3]

Symmetry: $g(n) = \Theta(f(n)) \Longleftrightarrow f(n) = \Theta(g(n))$ [$\Theta$ only]

$g(n) = O(f(n)) \Longleftrightarrow f(n) = \Omega(g(n))$

Constant: $g(n) = O(f(n)) \Longrightarrow dg(n) = O(f(n))$, for positive constant $d$

Logs: $\log_a n = \Theta(\log_b n)), a, b > 0$ [Thus all logs are the same complexity and we can use $\lg n$ at any time]

Powers of $n$: $n^a = o(n^{a+\epsilon}), a > 1, \epsilon \gt 0$ [larger powers dominate smaller powers]

Exponentials: $n^k = o(a^n), a > 1$ [exponentials dominate polynomials]

Exponential Bases: $a^n = o(b^n), 0 < a < b$ [Thus, all exponentials functions are not the same.]

Factorial: $a^n = o(n!), 0 < a$ [Thus, factorial is worse than exponential]

Sum: $e(n) = O(f(n))$ and $g(n) = O(h(n))\Longrightarrow e(n) + g(n) = O(f(n) + (h(n)) = O(\textrm{max}(f(n) + h(n)))$

Product: $e(n) = O(f(n))$ and $g(n) = O(h(n))\Longrightarrow e(n)\times g(n) = O(f(n)\times (h(n))$

History

Big O, little o, and Big Omega originated in math in late 19th and early 20th century, with somewhat different definitions

In CS, Big O was originally used somewhat loosely

Terminology (particularly Ω and Θ) formalized and formal definitions (eg c and n0) given by Knuth in 1976

See Selected Papers on Analysis of Algorithms