Sorting
Agenda
We will now discuss two additional sorting algorithms. For your convenience this notebook also contains all other sorting methods we have discussed so far.
- We will introduce a divide-and-conquer based sorting algorithm called quick sort that has average case \(O(n \log n)\) runtime, but can degrade to \(O(n^2)\) runtime.
- We will then discuss merge sort, a sorting algorithm with guaranteed \(O(n \log n)\) worst-case runtime. This algorithm is also based on a divide-and-conquer paradigm.
Bubble sort
def bubble_sort(lst): # n = len(lst) => O(n^2)
for i in range(1,len(lst)): # n - 1
for j in range(0,len(lst) - i): # sum(i)
if lst[j] > lst[j+1]: # sum(i)
lst[j], lst[j+1] = lst[j+1], lst[j] # sum(i)
l = [ 1, 6, 5, 2, 4 ]
bubble_sort(l)
l
[1, 2, 4, 5, 6]
Insertion sort
-
Task: to sort the values in a given list (array) in ascending order.
import random lst = list(range(1000)) random.shuffle(lst)def insertion_sort(lst): for i in range(1,len(lst)): # number of times? n-1 for j in range(i,0,-1): # number 1, 2, 3, 4, ..., n-1 if lst[j] <= lst[j-1]: lst[j-1], lst[j] = lst[j], lst[j-1] else: breakinsertion_sort(lst)
Heap sort
def swap(lst,l,r):
lst[l], lst[r] = lst[r], lst[l]
def heapsort_inplace(lst):
heapify(lst)
print(f"\nfinal heap: {lst}\n")
for i in range(len(lst) -1, -1, -1):
swap(lst,i,0)
sift_down(lst,0,i-1)
print(f"pop and insert at {i}: heap: {lst[0:i]} sorted suffix {lst[i:len(lst)]}")
def heapify(lst):
for i in range(len(lst) -1,-1,-1):
sift_down(lst,i,len(lst) - 1)
print(f"heapified {i} to {len(lst) - 1}: {lst[0:i]} * {lst[i:len(lst)]}")
def sift_down(lst,start,end):
root = start
while Heap.left_child(root) <= end:
child = Heap.left_child(root)
swp = root
if lst[swp] < lst[child]: # left child larger
swp = child
if child+1 <= end and lst[swp] < lst[child+1]: # right child larger than left or root
swp = child + 1
if root == swp:
return
swap(lst,root,swp)
root = swp
Quick sort
Algorithm
Partitioning a list on a pivot element
Consider an operations called partition that takes an element of a list lst called a pivot and divides the list into two parts: all elements that are smaller than the pivot and all elements that are larger than or equal to pivot. We can implement partition in-place like this:
def partition_somewhere(lst):
pivotpos = -1
pivot = lst[pivotpos]
i = 0
high = len(lst) - 1
for j in range(high):
if lst[j] < pivot:
lst[i], lst[j] = lst[j], lst[i]
i = i + 1
lst[i], lst[high] = lst[high], lst[i] # make pivot middle element
return i
lst = [1,4,2,15,3,9,12]
print(f"pivot element is {lst[-1]}")
split = partition_somewhere(lst) # pivot on 9
strlst = "["
for i in range(len(lst)):
if i == split:
strlst += f"] {lst[i]} ["
else:
strlst += f"{lst[i]}"
strlst += ", " if i < len(lst) - 1 and i != split -1 else ""
strlst += "]"
print(strlst)
print(lst)
pivot element is 12 [1, 4, 2, 3, 9] 12 [15] [1, 4, 2, 3, 9, 12, 15]
Divide-and-conquer strategy of quick sort
Once we have partitioned a list on a pivot, then we can recursively partition the parts of the list. Note that any list of size one is trivially sorted.
Implementation
def quicksort(lst):
qsort(lst,0,len(lst) - 1)
def qsort(lst,low,high):
if low < high:
p = partition(lst,low,high)
qsort(lst,low,p-1)
qsort(lst,p+1,high)
def partition(lst,low,high):
pivot = lst[high]
i = low
for j in range(low,high):
if lst[j] < pivot:
lst[i], lst[j] = lst[j], lst[i]
i = i + 1
lst[i], lst[high] = lst[high], lst[i]
return i
lst = [4,3,5,10,2,9,8,7]
quicksort(lst)
print(lst)
[2, 3, 4, 5, 7, 8, 9, 10]
Merge sort
Algorithm
Merging sorted lists
Given two sorted lists l1 and l2, we can merge them into a longer sorted list by iterating through both lists in parallel keeping an index i for l1 and an index j for l2. If l1[i] < l2[j], then l1[i] is appended to the result and i is increased by one. Otherwise, l2[j] is appended to the result and j is increased by one. Once we reach the end of one of the lists, then remaining elements of the other list have to be appended to the result.
def merge(l,r):
result = []
i = 0
j = 0
while i < len(l) and j < len(r):
if l[i] < r[j]:
result.append(l[i])
i += 1
else:
result.append(r[j])
j += 1
while i < len(l):
result.append(l[i])
i += 1
while j < len(r):
result.append(r[j])
j += 1
return result
l1 = [1, 4, 5, 10, 12]
l2 = [2, 6, 8, 9, 16, 22]
merge(l1,l2)
[1, 2, 4, 5, 6, 8, 9, 10, 12, 16, 22]
Note that in each loop iteration either i or j is increased by one. Thus, if we use n to denote the length of the resulting list, then this algorithm is in \(O(n \log n)\).
The divide-and-conquer strategy of merge sort
To sort any list of length n, we can split it in the middle to get two lists of length n/2. Then we recursively sort these two lists and merge them into a sorted list of length n. Since any list of length 1 is trivially sorted, the recursion will stop once the two lists are of length 1.
Implementation
def merge(l,r):
result = []
i = 0
j = 0
while i < len(l) and j < len(r):
if l[i] < r[j]:
result.append(l[i])
i += 1
else:
result.append(r[j])
j += 1
while i < len(l):
result.append(l[i])
i += 1
while j < len(r):
result.append(r[j])
j += 1
return result
def mergesort(lst):
def msort(lst,low,high):
if low < high:
mid = (high + low) // 2
left = msort(lst,low,mid)
right = msort(lst,mid+1,high)
return merge(left,right)
else:
return [lst[low]]
return msort(lst,0,len(lst)-1)
lst = [4,3,5,10,2,9,8,7]
result = mergesort(lst)
print(result)
[2, 3, 4, 5, 7, 8, 9, 10]
Runtime analysis
Merging of 2 lists of length n takes 2n time. In each recursive call we divide the length of the list to be sorted by a factor of two.
- to sort a list of length
nwe sort two lists of lengthn/2 - for each of these we sort two lists of length
n/4, and so on
After \(O(\log n)\) steps, the length of the list to be are sorted is \(1\) and any \(1\) element list is trivially sorted. Let us reorganize the subproblems by the lengths of lists they have to merge:
- \(2 * n/2 = n\)
- \(4 * n/4 = n\)
- \(8 * n/8 = n\)
…
- \(n * 1 = n\)
\[ \sum_{i=1}^{\log n} 2^{i} * \frac{n}{2^{i}} = \sum_{i=1}^{\log n} n = n \log n \]
Counting sort
If the domain of elements that appear in the input list is small (m elements), then we can just maintain a count to record in an array of size m how often each possible element appears in the list. This can be done by one pass over the list as shown below. Given, the counts we just iterate over the array and output a number of copies of the element at the current position that is equal to the count.
print(ord('a'))
print(chr(ord('a')))
97 a
def counting_sort(lst):
h = { }
for i in range(256):
h[i] = 0
for i in s:
h[i] += 1
result = []
for i in range(0,255):
for j in range(h[i]):
result.append(i)
return result
s = [ i for i in b"asdubhiabusaaaasdkjasdksjnqdadsfjhabdsfadhjsfb" ]
result = counting_sort(s)
''.join([ chr(i) for i in result ])
'aaaaaaaaaabbbbdddddddfffhhhijjjjkknqssssssssuu'
Runtime comparison of sorting algorithms
| algorithm | worst-case runtime | average case runtime |
|---|---|---|
| Bubble sort | \(O(n^2)\) | \(O(n^2)\) |
| Insertion sort | \(O(n^2)\) | \(O(n^2)\) |
| Heap sort | \(O(n \log n)\) | \(O(n \log n)\) |
| Quick sort | \(O(n^2)\) | \(O(n \log n)\) |
| Merge sort | \(O(n^2)\) | \(O(n^2)\) |
| Counting sort | \(O(n)\) | \(O(n)\) |
Note that, however, counting sort needs \(O(m)\) memory where \(m\) is the size of the domain of elements, e.g., for 64-bit integers there are \(2^{64} = 18,446,744,073,709,551,616\) elements in the domain. Also to address these elements in an array we need \(O(\log m)\) time.