Example. Merge Sort

[comp1110-29]
  • Liam O'Connor, Fabian Muehlboeck

Merge sort is both an efficient sorting algorithm (although it is not in-place) and a classic example of Divide and Conquer techniques. The algorithm works by dividing the input array in half, recursively sorting each half, and then merging the results. We can write an efficient merge function that combines two sorted lists into a single combined sorted lists. Note that we just loop through both arrays once, so this merge function is more efficient than simply appending the two arrays and then sorting them by other means.

/** * Returns the sorted combination of two * sorted arrays * @param left a sorted array of numbers * @param right a sorted array of numbers * @return a sorted array that contains * exactly the elements of left and right */ int[] merge(int[] left, int[] right) { int leftPos = 0; int rightPos = 0; int[] result = new int[left.length + right.length]; while(leftPos < left.length && rightPos < right.length) { int leftElement = left[leftPos]; int rightElement = right[rightPos]; if(leftElement < rightElement) { result[leftPos + rightPos] = leftElement; leftPos = leftPos + 1; } else { result[leftPos + rightPos] = rightElement; rightPos = rightPos + 1; } } while(leftPos < left.length) { result[leftPos + rightPos] = left[leftPos]; leftPos = leftPos + 1; } while(rightPos < right.length) { result[leftPos + rightPos] = right[rightPos]; rightPos = rightPos + 1; } return result; }

This function consistutes the "combine" step in the divide and conquer strategy. Now all that is left is the "dividing" and "conquering":

/** * Returns a sorted version of the part of the * given array between lower and upper * @param arr an array of integers, PRE: not null * @param lower the lower index of the part of the * array we want to sort, PRE: lower <= upper, * PRE: 0 <= lower < arr.length * @param upper the upper index of the part of the * array we want to sort, PRE: lower <= upper, * PRE: 0 <= upper < arr.length * @return a sorted version of the numbers in the * arr between lower and uppper */ int[] mergeSort(int[] arr, int lower, int upper) { if(arr.length==0) { return arr; } if(upper == lower) { return new int[]{arr[lower]}; } int middle = (lower + upper) / 2; int[] left = mergeSort(arr, lower, middle); int[] right = mergeSort(arr, middle+1, upper); return merge(left, right); }

Note: This functions efficiency can be slightly improved by only allocating one output array and passing it into each function, however this obscures the functionality of the algorithm.

Context

    More general kinds of recursion

    [comp1110-7L]
    • Fabian Muehlboeck

    The recursive functions we have seen so far directly matched the structure of a data type definition. Sometimes, such a structure is less obvious, and sometimes, we don't follow the structure of a data type that closely. But as long as you can argue that a recursive call is on "smaller" input (in the sense that we get close to a base case), recursion is a good option.

    Example. Factorial

    [comp1110-24]
    • Liam O'Connor

    You may have seen this self-referential (recursive) definition of factorial before:

    \[\begin {array}{lcl} 0! &=& 1 \\ n! &=& (n-1)! \cdot n \end {array}\] int factorial(int n) { if (n == 0) { //the base case return 1; } else { // the recursive case (inductive case) return factorial(n-1) * n; } }

    Try it out in the Java visualiser.

    Warning. Factorial of Zero

    [comp1110-7M]
    • Fabian Muehlboeck

    In the lecture, we used the following definition of factorial:

    /** * @param n a natural number (>= 0) */ int factorial(int n) { if(n==0) { return 0; } else if(n==1) { return 1; } else { return n * factorial(n-1); } }

    This is not the standard definition of factorial, which usually holds that 0! == 1. Our definition above still matches the definition of factorial for every other valid input. The main point here is that a function may have multiple base cases, some of which may never be hit coming from a recursive case.

    Definition. Divide and Conquer

    [comp1110-26]
    • Liam O'Connor

    Divide and conquer is a recursive problem solving strategy that solves a problem by following three distinct steps: Divide the problem into one or more subproblems that are smaller instances of the same problem; Conquer the subproblems by solving them recursively or directly if they are small enough; and Combine the subsolutions into the solution for the original problem. This approach is defined by reducing the problem size by a significant factor, such as halving the input in binary search, rather than just a constant amount (like in Factorial).

    Example. Recursive binary search

    [comp1110-27]
    • Liam O'Connor, Fabian Muehlboeck

    A recursive version of our iterative Binary Search. Note that we had to add additional parameters lower and upper to give us enough flexibility to use this function recursively. Such generalisation is often required when working with recursive functions. Where needed, you can define another function without those parameters that simply calls the main one with suitable initial values.

    /** Searches for a needle in a sorted haystack @param haystack a sorted (in ascending order) array of integers @param needle an integer to be searched for @param lower an integer expressing the lower bound of where needle could be in haystack; PRE: if exists i, haystack[i] == needle, then i >= lower @param upper an integer expressing the upper bound of where needle could be in haystack; PRE: if exists i, haystack[i] == needle, then i <= upper @return true iff there exists i, haystack[i] == needle */ boolean binarySearch(int[] haystack, int needle, int lower, int upper) { if(lower>upper) { return false; } int middle = (lower + upper) / 2; if(haystack[middle] == needle) { return true; } if(needle > haystack[middle]) { return binarySearch(haystack, needle, middle + 1, upper); } else { return binarySearch(haystack, needle, lower, middle - 1); } }

    Try it out in the Java visualiser.

    Recursion and Loop Invariants

    [comp1110-7N]
    • Fabian Muehlboeck

    Consider the iterative version of binary search, and specifically its loop invariant:

    Example. Binary Search

    [comp1110-1J]
    • Liam O'Connor

    If we add a precondition to the spec of contains, that the input array is sorted, we can use a trick similar to our solution algorithm for our guessing game, which as we saw, can reduce the amount of iterations of the program from linear in the size of the input to logarithmic.

    The idea is that we consistently pick the middle index from the range of possible indices. If the value at this index is our target, we have found what we're looking for, and we're done. If the value at this index is larger than our target, that means our desired index is to the left, and reduces the range of possible indices by half. If it is smaller, then our desired index is to the other half, on the right of the selected midpoint.

    This narrows the range of possible indices, which permits us to repeat this technique until we reach single elements.

    /** Searches for a needle in a sorted haystack @param haystack a sorted (in ascending order) array of integers @param needle an integer to be searched for @return true iff there exists i, haystack[i] == needle */ boolean binarySearch(int[] haystack, int needle) { int lower = 0; int upper = haystack.length - 1; while(lower <= upper) { // INV: if there exists i, haystack[i] == needle, // then lower <= i <= upper int middle = (lower + upper) / 2; if(needle == haystack[middle]) { return true; } else if(needle > haystack[middle]) { lower = middle + 1; } else { upper = middle - 1; } } return false; }

    Contrast this with Recursive binary search. Notice how the loop invariant is essentially encoded in the preconditions for lower and upper. The line of reasoning is the same - the loop invariant must be true at the end of each loop, just as the preconditions of the recursive function call have to be true at each such recursive function call. The precondition must be true at the first call of the recursive function, just as the loop invariant has to be true when entering the loop.

    In exchange, we can assume that the loop invariant is true at the start of the loop and after it, just as we can assume the precondition holds at the start of the function, and (typically) in the base cases.

    Example. Merge Sort

    [comp1110-29]
    • Liam O'Connor, Fabian Muehlboeck

    Merge sort is both an efficient sorting algorithm (although it is not in-place) and a classic example of Divide and Conquer techniques. The algorithm works by dividing the input array in half, recursively sorting each half, and then merging the results. We can write an efficient merge function that combines two sorted lists into a single combined sorted lists. Note that we just loop through both arrays once, so this merge function is more efficient than simply appending the two arrays and then sorting them by other means.

    /** * Returns the sorted combination of two * sorted arrays * @param left a sorted array of numbers * @param right a sorted array of numbers * @return a sorted array that contains * exactly the elements of left and right */ int[] merge(int[] left, int[] right) { int leftPos = 0; int rightPos = 0; int[] result = new int[left.length + right.length]; while(leftPos < left.length && rightPos < right.length) { int leftElement = left[leftPos]; int rightElement = right[rightPos]; if(leftElement < rightElement) { result[leftPos + rightPos] = leftElement; leftPos = leftPos + 1; } else { result[leftPos + rightPos] = rightElement; rightPos = rightPos + 1; } } while(leftPos < left.length) { result[leftPos + rightPos] = left[leftPos]; leftPos = leftPos + 1; } while(rightPos < right.length) { result[leftPos + rightPos] = right[rightPos]; rightPos = rightPos + 1; } return result; }

    This function consistutes the "combine" step in the divide and conquer strategy. Now all that is left is the "dividing" and "conquering":

    /** * Returns a sorted version of the part of the * given array between lower and upper * @param arr an array of integers, PRE: not null * @param lower the lower index of the part of the * array we want to sort, PRE: lower <= upper, * PRE: 0 <= lower < arr.length * @param upper the upper index of the part of the * array we want to sort, PRE: lower <= upper, * PRE: 0 <= upper < arr.length * @return a sorted version of the numbers in the * arr between lower and uppper */ int[] mergeSort(int[] arr, int lower, int upper) { if(arr.length==0) { return arr; } if(upper == lower) { return new int[]{arr[lower]}; } int middle = (lower + upper) / 2; int[] left = mergeSort(arr, lower, middle); int[] right = mergeSort(arr, middle+1, upper); return merge(left, right); }

    Note: This functions efficiency can be slightly improved by only allocating one output array and passing it into each function, however this obscures the functionality of the algorithm.

    Backlinks

      Space complexity

      [comp1110-2E]
      • Liam O'Connor

      We can also analyse how much memory an algorithm uses. This is called space complexity. Algorithms like insertion sort and selection sort operate directly on the array and use only a few extra variables, so their space complexity is \(\mathrm {O}(1)\). Merge sort, however, needs an additional temporary array during the merge step, so it requires \(\mathrm {O}(n)\) extra space.

      Example. Time complexity of searching and sorting

      [comp1110-2F]
      • Liam O'Connor
      • Linear search through an array of \(n\) elements: If the target happens to be the first element, the algorithm stops immediately; this is the best case and takes constant time, \(\mathrm {O}(1)\). In the worst case the element is last or not present, so the algorithm must check all \(n\) elements. The worst-case running time is therefore \(\mathrm {O}(n)\).
      • Binary search on a sorted array. Each comparison eliminates half of the remaining elements. After one step at most \(\frac {n}{2}\) elements remain, then \(\frac {n}{4}\), then \(\frac {n}{8}\), and so on. After about \(\log _2 n\) steps only one element remains. Thus binary search runs in \(\mathrm {O}(\log n)\) time in the worst case.
      • Selection sort repeatedly scans the remaining part of the array to find the smallest element. Even if the array is already sorted, it still performs the same number of comparisons, roughly \(\frac {n(n-1)}{2}\). Its running time therefore grows like \(n^2\), so it is \(\mathrm {O}(n^2)\) in both the best and worst cases.
      • Insertion sort behaves differently. If the array is already sorted, each new element is compared once and left in place, so the running time grows linearly: \(\mathrm {O}(n)\). But if the array is in reverse order, each insertion shifts many elements and the total work grows like \(n^2\). Thus the best case is \(\mathrm {O}(n)\) and the worst case is \(\mathrm {O}(n^2)\).
      • Bubble sort with early exit is similar. If the list is already sorted, it finishes after one pass, which is \(\mathrm {O}(n)\). In the worst case — where one small element appears out of order towards the end of the list — it performs many passes and becomes \(\mathrm {O}(n^2)\).
      • Merge sort works differently. It divides the list in half, sorts each half, and then merges them. The list can be halved only about \(\log _2 n\) times, and each level of this process requires work proportional to \(n\) to perform the merges. The total work therefore grows like \(n \log n\). Unlike some previous sorts, this running time does not depend on the initial order of the input: both the best and worst cases are \(\mathrm {O}(n \log n)\).

      Related

        Definition. Divide and Conquer

        [comp1110-26]
        • Liam O'Connor

        Divide and conquer is a recursive problem solving strategy that solves a problem by following three distinct steps: Divide the problem into one or more subproblems that are smaller instances of the same problem; Conquer the subproblems by solving them recursively or directly if they are small enough; and Combine the subsolutions into the solution for the original problem. This approach is defined by reducing the problem size by a significant factor, such as halving the input in binary search, rather than just a constant amount (like in Factorial).

        Definition. In-place

        [comp1110-20]

          An algorithm is in-place if only modifies existing structures, rather than create new ones. For example, for sorting an in-place function returns void, as they don't produce a new array, they just modify an old one. All our sorting algorithms in this section are in-place.

          bars magnifying-glass caret-down plus minus arrow-right xmark