Much ado about testing

# Much ado about testing

Nicolas Rinaudo • [@NicolasRinaudo]

---

# Software testing

---

## Testing

---

## Testing

---

## Testing

---

## Testing

---

## Testing

---

## System Under Test

> Write a program that, given a number, prints it.
>
> But for multiples of three print `Fizz` instead of the number and for the multiples of  five print `Buzz`.

```scala
def mult3(i: Int) = !(i % 3 > 0)
def mult5(i: Int) = !(i % 5 > 0)

def fizzBuzz(i: Int) =
  if mult3(i) then      "Fizz"
  else if mult5(i) then "Buzz"
  else                  i.toString

```

---

## System Under Test

> Write a program that, .highlight[given a number], prints it.
>
> But for multiples of three print `Fizz` instead of the number and for the multiples of five print `Buzz`.

```scala
def mult3(i: Int) = !(i % 3 > 0)
def mult5(i: Int) = !(i % 5 > 0)

def fizzBuzz(`i: Int`) =
  if mult3(i) then      "Fizz"
  else if mult5(i) then "Buzz"
  else                  i.toString
```

---

## System Under Test

> Write a program that, given a number, .highlight[prints it].
>
> But for multiples of three print `Fizz` instead of the number and for the multiples of five print `Buzz`.

```scala
def mult3(i: Int) = !(i % 3 > 0)
def mult5(i: Int) = !(i % 5 > 0)

def fizzBuzz(i: Int) =
  if mult3(i) then      "Fizz"
  else if mult5(i) then "Buzz"
  else                  `i.toString`
```

---

## System Under Test

> Write a program that, given a number, prints it.
>
> But for .highlight[multiples of three] print `Fizz` instead of the number and for the multiples of five print `Buzz`.

```scala
def mult3(i: Int) = !(i % 3 > 0)
def mult5(i: Int) = !(i % 5 > 0)

def fizzBuzz(i: Int) =
  if `mult3(i)` then      "Fizz"
  else if mult5(i) then "Buzz"
  else                  i.toString
```

---

## System Under Test

> Write a program that, given a number, prints it.
>
> But for multiples of three print .highlight[`Fizz`] instead of the number and for the multiples of five print `Buzz`.

```scala
def mult3(i: Int) = !(i % 3 > 0)
def mult5(i: Int) = !(i % 5 > 0)

def fizzBuzz(i: Int) =
  if mult3(i) then      `"Fizz"`
  else if mult5(i) then "Buzz"
  else                  i.toString
```

---

## System Under Test

> Write a program that, given a number, prints it.
>
> But for multiples of three print `Fizz` instead of the number and for the .highlight[multiples of five] print `Buzz`.

```scala
def mult3(i: Int) = !(i % 3 > 0)
def mult5(i: Int) = !(i % 5 > 0)

def fizzBuzz(i: Int) =
  if mult3(i) then      "Fizz"
  else if `mult5(i)` then "Buzz"
  else                  i.toString
```

---

## System Under Test

> Write a program that, given a number, prints it.
>
> But for multiples of three print `Fizz` instead of the number and for the multiples of five print .highlight[`Buzz`].

```scala
def mult3(i: Int) = !(i % 3 > 0)
def mult5(i: Int) = !(i % 5 > 0)

def fizzBuzz(i: Int) =
  if mult3(i) then      "Fizz"
  else if mult5(i) then `"Buzz"`
  else                  i.toString
```

---

# Example-based testing

---

## Overview

---

## Overview

---

## Overview

---

## Overview

---

## Overview

---

## Overview

---

## Overview

```scala
class ListTests extends AnyFunSuite:
  test("List(3, 2, 1) should sort to List(1, 2, 3)") {
    assert(`List(3, 2, 1)`.sorted == List(1, 2, 3))
  }
```

---

## Overview

```scala
class ListTests extends AnyFunSuite:
  test("List(3, 2, 1) should sort to List(1, 2, 3)") {
    assert(List(3, 2, 1)`.sorted` == List(1, 2, 3))
  }
```

---

## Overview

```scala
class ListTests extends AnyFunSuite:
  test("List(3, 2, 1) should sort to List(1, 2, 3)") {
    assert(List(3, 2, 1).sorted == `List(1, 2, 3)`)
  }
```

---

## FizzBuzz test suite

```scala
class FizzBuzzSuite extends AnyFunSuite:
  test("n multiple of 3 outputs Fizz") {
    assert(fizzBuzz(3) == "Fizz")
  }

test("n multiple of 5 outputs Buzz") {
    assert(fizzBuzz(5) == "Buzz")
  }

test("n multiple of neither 3 or 5 outputs n") {
    assert(fizzBuzz(7) == "7")
  }
```

---

## FizzBuzz test suite

```scala
class FizzBuzzSuite extends AnyFunSuite:
  test(`"n multiple of 3 outputs Fizz"`) {
    assert(fizzBuzz(3) == "Fizz")
  }

test(`"n multiple of 5 outputs Buzz"`) {
    assert(fizzBuzz(5) == "Buzz")
  }

test(`"n multiple of neither 3 or 5 outputs n"`) {
    assert(fizzBuzz(7) == "7")
  }
```

---

## FizzBuzz test suite

```scala
class FizzBuzzSuite extends AnyFunSuite:
  test("n multiple of 3 outputs Fizz") {
    assert(fizzBuzz(`3`) == "Fizz")
  }

test("n multiple of 5 outputs Buzz") {
    assert(fizzBuzz(`5`) == "Buzz")
  }

test("n multiple of neither 3 or 5 outputs n") {
    assert(fizzBuzz(`7`) == "7")
  }
```

---

## FizzBuzz test suite

```scala
class FizzBuzzSuite extends AnyFunSuite:
  test("n multiple of 3 outputs Fizz") {
    assert(`fizzBuzz`(3) == "Fizz")
  }

test("n multiple of 5 outputs Buzz") {
    assert(`fizzBuzz`(5) == "Buzz")
  }

test("n multiple of neither 3 or 5 outputs n") {
    assert(`fizzBuzz`(7) == "7")
  }
```

---

## FizzBuzz test suite

```scala
class FizzBuzzSuite extends AnyFunSuite:
  test("n multiple of 3 outputs Fizz") {
    assert(fizzBuzz(3) == `"Fizz"`)
  }

test("n multiple of 5 outputs Buzz") {
    assert(fizzBuzz(5) == `"Buzz"`)
  }

test("n multiple of neither 3 or 5 outputs n") {
    assert(fizzBuzz(7) == `"7"`)
  }
```

---

## FizzBuzz test suite

```scala
class FizzBuzzSuite extends AnyFunSuite:
  test(`"n multiple of 3 outputs Fizz"`) {
    assert(`fizzBuzz(3) == "Fizz"`)
  }

test("n multiple of 5 outputs Buzz") {
    assert(fizzBuzz(5) == "Buzz")
  }

test("n multiple of neither 3 or 5 outputs n") {
    assert(fizzBuzz(7) == "7")
  }
```

---

## FizzBuzz test suite

```scala
class FizzBuzzSuite extends AnyFunSuite:
  test("n multiple of 3 outputs Fizz") {
    assert(fizzBuzz(3) == "Fizz")
  }

test(`"n multiple of 5 outputs Buzz"`) {
    assert(`fizzBuzz(5) == "Buzz"`)
  }

test("n multiple of neither 3 or 5 outputs n") {
    assert(fizzBuzz(7) == "7")
  }
```

---

## FizzBuzz test suite

```scala
class FizzBuzzSuite extends AnyFunSuite:
  test("n multiple of 3 outputs Fizz") {
    assert(fizzBuzz(3) == "Fizz")
  }

test("n multiple of 5 outputs Buzz") {
    assert(fizzBuzz(5) == "Buzz")
  }

test(`"n multiple of neither 3 or 5 outputs n"`) {
    assert(`fizzBuzz(7) == "7"`)
  }
```

---

## FizzBuzz test suite

```scala
new FizzBuzzSuite().execute()
```

---

## FizzBuzz test suite

```scala
new FizzBuzzSuite().execute()
// rs$line$3$FizzBuzzSuite:
// - n multiple of 3 outputs Fizz
// - n multiple of 5 outputs Buzz
// - n multiple of neither 3 or 5 outputs n
```

---

## Client Demo

```scala
fizzBuzz(15)
```

---

## Client Demo

```scala
fizzBuzz(15)
// val res1: String = Fizz
```

---

## Clarifying specifications

n % 3 | n % 5 || Output
-------|-------||----------
 T     | T     || FizzBuzz
 T     | F     || Fizz
 F     | T     || Buzz
 F     | F     || n

---

## Clarifying specifications

.highlight[n % 3] | n % 5 || Output
-------|-------||----------
 T     | T     || FizzBuzz
 T     | F     || Fizz
 F     | T     || Buzz
 F     | F     || n

---

## Clarifying specifications

n % 3 | .highlight[n % 5] || Output
-------|-------||----------
 T     | T     || FizzBuzz
 T     | F     || Fizz
 F     | T     || Buzz
 F     | F     || n

---

## Clarifying specifications

n % 3 | n % 5 || .highlight[Output]
-------|-------||----------
 T     | T     || FizzBuzz
 T     | F     || Fizz
 F     | T     || Buzz
 F     | F     || n

---

## Clarifying specifications

n % 3 | n % 5 || Output
-------|-------||----------
 .highlight[T]     | T     || FizzBuzz
 T     | F     || Fizz
 F     | T     || Buzz
 F     | F     || n

---

## Clarifying specifications

n % 3 | n % 5 || Output
-------|-------||----------
 T     | .highlight[T]     || FizzBuzz
 T     | F     || Fizz
 F     | T     || Buzz
 F     | F     || n

---

## Clarifying specifications

n % 3 | n % 5 || Output
-------|-------||----------
 T     | T     || .highlight[FizzBuzz]
 T     | F     || Fizz
 F     | T     || Buzz
 F     | F     || n

---

## Fixing FizzBuzz

def fizzBuzz(i: Int) =
* if mult3(i) then      `"Fizz"`
  else if mult5(i) then "Buzz"
  else                  i.toString
```
]

---

## Fixing FizzBuzz

def fizzBuzz(i: Int) =
  if mult3(i) then
* `  if mult5(i) then    "FizzBuzz"`
* `  else                "Fizz"`
  else if mult5(i) then "Buzz"
  else                  i.toString
```
]

---

## Fixing FizzBuzz

```scala
def mult3(i: Int) = !(i % 3 > 0)
def mult5(i: Int) = !(i % 5 > 0)

def fizzBuzz(i: Int) =
  if `mult3(i)` then
    if `mult5(i)` then    `"FizzBuzz"`
    else                "Fizz"
  else if mult5(i) then "Buzz"
  else                  i.toString
```

---

## Fixing FizzBuzz

```scala
def mult3(i: Int) = !(i % 3 > 0)
def mult5(i: Int) = !(i % 5 > 0)

def fizzBuzz(i: Int) =
  if `mult3(i)` then
    if mult5(i) then    "FizzBuzz"
    else                `"Fizz"`
  else if mult5(i) then "Buzz"
  else                  i.toString
```

---

## Fixing tests

.diff-add[
```scala
class FizzBuzzSuite extends AnyFunSuite:
  test("n multiple of 3 outputs Fizz") {
    assert(fizzBuzz(3) == "Fizz")
  }

test("n multiple of 5 outputs Buzz") {
    assert(fizzBuzz(5) == "Buzz")
  }

test("n multiple of neither 3 or 5 outputs n") {
    assert(fizzBuzz(7) == "7")
  }
*
* `test("n multiple of 3 and 5 outputs FizzBuzz") {`
* `  assert(fizzBuzz(15) == "FizzBuzz")`
* `}`
```
]

---

## Fixing tests

```scala
class FizzBuzzSuite extends AnyFunSuite:
  test("n multiple of 3 outputs Fizz") {
    assert(fizzBuzz(3) == "Fizz")
  }

test("n multiple of 5 outputs Buzz") {
    assert(fizzBuzz(5) == "Buzz")
  }

test("n multiple of neither 3 or 5 outputs n") {
    assert(fizzBuzz(7) == "7")
  }

test("n multiple of 3 and 5 outputs FizzBuzz") {
    assert(fizzBuzz(`15`) == "FizzBuzz")
  }
```

---

## Fixing tests

```scala
class FizzBuzzSuite extends AnyFunSuite:
  test("n multiple of 3 outputs Fizz") {
    assert(fizzBuzz(3) == "Fizz")
  }

test("n multiple of 5 outputs Buzz") {
    assert(fizzBuzz(5) == "Buzz")
  }

test("n multiple of neither 3 or 5 outputs n") {
    assert(fizzBuzz(7) == "7")
  }

test("n multiple of 3 and 5 outputs FizzBuzz") {
    assert(`fizzBuzz`(15) == "FizzBuzz")
  }
```

---

## Fixing tests

```scala
class FizzBuzzSuite extends AnyFunSuite:
  test("n multiple of 3 outputs Fizz") {
    assert(fizzBuzz(3) == "Fizz")
  }

test("n multiple of 5 outputs Buzz") {
    assert(fizzBuzz(5) == "Buzz")
  }

test("n multiple of neither 3 or 5 outputs n") {
    assert(fizzBuzz(7) == "7")
  }

test("n multiple of 3 and 5 outputs FizzBuzz") {
    assert(fizzBuzz(15) == `"FizzBuzz"`)
  }
```

---

## Fixing tests

```scala
new FizzBuzzSuite().execute()
```

---

## Fixing tests

```scala
new FizzBuzzSuite().execute()
// rs$line$7$FizzBuzzSuite:
// - n multiple of 3 outputs Fizz
// - n multiple of 5 outputs Buzz
// - n multiple of neither 3 or 5 outputs n
// - n multiple of 3 and 5 outputs FizzBuzz
```

---

## Client Demo

```scala
fizzBuzz(-3)
```

---

## Client Demo

```scala
fizzBuzz(-3)
// val res3: String = FizzBuzz
```

---

## Key takeaways

Example-based testing is:

* very good at the _oracle problem_.

* very bad at the _test case generation problem_.

* a little bit discouraging...

---

# Generative Testing

---

## Overview

---

## Overview

---

## Overview

---

## Overview

---

## Overview

---

## Overview

---

## Property-Based Testing

---

## Property-Based Testing

```scala
forAll { `(is: List[Int])` =>
  is.sorted.diff(is).isEmpty
}
```

---

## Property-Based Testing

```scala
forAll { (is: List[Int]) =>
  `is.sorted`.diff(is).isEmpty
}
```

---

## Property-Based Testing

```scala
forAll { (is: List[Int]) =>
  is.sorted.`diff(is).isEmpty`
}
```

---

## Property-Based Testing

```scala
val propFizzBuzz = forAll { `(i: Int)` =>
  if mult3(i) then
    if mult5(i) then    fizzBuzz(i) == "FizzBuzz"
    else                fizzBuzz(i) == "Fizz"
  else if mult5(i) then fizzBuzz(i) == "Buzz"
  else                  fizzBuzz(i) == i.toString
}
```

---

## Property-Based Testing

```scala
val propFizzBuzz = forAll { (i: Int) =>
  if `mult3(i)` then
    if `mult5(i)` then    fizzBuzz(i) == `"FizzBuzz"`
    else                fizzBuzz(i) == "Fizz"
  else if mult5(i) then fizzBuzz(i) == "Buzz"
  else                  fizzBuzz(i) == i.toString
}
```

---

## Property-Based Testing

```scala
val propFizzBuzz = forAll { (i: Int) =>
  if `mult3(i)` then
    if mult5(i) then    fizzBuzz(i) == "FizzBuzz"
    else                fizzBuzz(i) == `"Fizz"`
  else if mult5(i) then fizzBuzz(i) == "Buzz"
  else                  fizzBuzz(i) == i.toString
}
```

---

## Property-Based Testing

```scala
val propFizzBuzz = forAll { (i: Int) =>
  if mult3(i) then
    if mult5(i) then    fizzBuzz(i) == "FizzBuzz"
    else                fizzBuzz(i) == "Fizz"
  else if `mult5(i)` then fizzBuzz(i) == `"Buzz"`
  else                  fizzBuzz(i) == i.toString
}
```

---

## Property-Based Testing

```scala
val propFizzBuzz = forAll { (i: Int) =>
  if mult3(i) then
    if mult5(i) then    fizzBuzz(i) == "FizzBuzz"
    else                fizzBuzz(i) == "Fizz"
  else if mult5(i) then fizzBuzz(i) == "Buzz"
  else                  fizzBuzz(i) == `i.toString`
}
```

---

## Property-Based Testing

```scala
propFizzBuzz.check()
```

---

## Property-Based Testing

```scala
propFizzBuzz.check()
// + OK, passed 100 tests.
```

---

## Property-Based Testing

```scala
val propFizzBuzz = forAll { (i: Int) =>
* if mult3(i) then
*   if mult5(i) then    fizzBuzz(i) == "FizzBuzz"
*   else                fizzBuzz(i) == "Fizz"
* else if mult5(i) then fizzBuzz(i) == "Buzz"
* else                  fizzBuzz(i) == `i.toString`
}
```

---

## Test oracle

---

## Test oracle

---

## Test oracle

---

## Test oracle

---

## Test oracle

---

## Test oracle

---

## Test oracle

```scala
forAll { `(is: List[Int])` =>
  fastSort(is) == is.sorted
}
```

---

## Test oracle

```scala
forAll { (is: List[Int]) =>
  `fastSort(is)` == is.sorted
}
```

---

## Test oracle

```scala
forAll { (is: List[Int]) =>
  fastSort(is) == `is.sorted`
}
```

---

## Test oracle

```scala
forAll { (is: List[Int]) =>
  fastSort(is) `==` is.sorted
}
```

---

## Test oracle: use case

---

## Test oracle: use case

---

## Test oracle: use case

---

## Test oracle: use case

---

## Test oracle: use case

---

## Test oracle: use case

---

## Test oracle: use case

---

## Test oracle: fizzBuzz

---

## Validity

---

## Validity

---

## Validity

---

## Validity

```scala
forAll { `(i: Int)` =>
  math.abs(i) >= 0
}
```

---

## Validity

```scala
forAll { (i: Int) =>
  `math.abs(i)` >= 0
}
```

---

## Validity

```scala
forAll { (i: Int) =>
  math.abs(i) `>= 0`
}
```

---

## Validity: use case

---

## Validity: use case

---

## Validity: use case

---

## Validity: use case

---

## Validity: fizzBuzz

```scala
val propValidFizzBuzz = forAll { `(i: Int)` =>
  fizzBuzz(i) in Set(i.toString, "Fizz", "Buzz", "FizzBuzz")
}
```

---

## Validity: fizzBuzz

```scala
val propValidFizzBuzz = forAll { (i: Int) =>
  `fizzBuzz(i)` in Set(i.toString, "Fizz", "Buzz", "FizzBuzz")
}
```

---

## Validity: fizzBuzz

```scala
val propValidFizzBuzz = forAll { (i: Int) =>
  fizzBuzz(i) `in Set(i.toString, "Fizz", "Buzz", "FizzBuzz")`
}
```

---

## Validity: fizzBuzz

```scala
propValidFizzBuzz.check()
```

---

## Validity: fizzBuzz

```scala
propValidFizzBuzz.check()
// + OK, passed 100 tests.
```

---

## Involutivity

---

## Involutivity

---

## Involutivity

---

## Involutivity

---

## Involutivity

---

## Involutivity

---

## Involutivity

---

## Involutivity

```scala
forAll { `(i: Int)` =>
  i.toString.toInt == i
}
```

---

## Involutivity

```scala
forAll { (i: Int) =>
  i.`toString`.toInt == i
}
```

---

## Involutivity

```scala
forAll { (i: Int) =>
  i.toString.`toInt` == i
}
```

---

## Involutivity

```scala
forAll { (i: Int) =>
  i.toString.toInt `== i`
}
```

---

## Involutivity: use case

```scala
Track(
   artist  = "Iron Maiden",
   year    = 1982,
   album   = "The Number of the Beast",
   name    = "Children of the Damned"
)
```

---

## Involutivity: use case

```scala
Track(
   artist  = "Iron Maiden",
   year    = `1982`,
   album   = "The Number of the Beast",
   name    = "Children of the Damned"
)
```

---

## Involutivity: use case

```json
{
  "artist" : "Iron Maiden",
  "year"   : `1982`,
  "album"  : "The Number of the Beast",
  "name"   : "Children of the Damned"
}
```

---

## Involutivity: use case

```scala
Track(
   artist  = "Iron Maiden",
   year    = `1970`,
   album   = "The Number of the Beast",
   name    = "Children of the Damned"
)
```

---

## Involutivity: fizzBuzz

```scala
val propInvolutiveFizzBuzz = forAll { `(i: Int)` =>
  !(mult3(i) || mult5(i)) ==> (fizzBuzz(i).toInt == i)
}
```

---

## Involutivity: fizzBuzz

```scala
val propInvolutiveFizzBuzz = forAll { (i: Int) =>
  `!(mult3(i) || mult5(i))` ==> (fizzBuzz(i).toInt == i)
}
```

---

## Involutivity: fizzBuzz

```scala
val propInvolutiveFizzBuzz = forAll { (i: Int) =>
  !(mult3(i) || mult5(i)) `==>` (fizzBuzz(i).toInt == i)
}
```

---

## Involutivity: fizzBuzz

```scala
val propInvolutiveFizzBuzz = forAll { (i: Int) =>
  !(mult3(i) || mult5(i)) ==> (`fizzBuzz(i)`.toInt == i)
}
```
---

## Involutivity: fizzBuzz

```scala
val propInvolutiveFizzBuzz = forAll { (i: Int) =>
  !(mult3(i) || mult5(i)) ==> (fizzBuzz(i).`toInt` == i)
}
```

---

## Involutivity: fizzBuzz

```scala
val propInvolutiveFizzBuzz = forAll { (i: Int) =>
  !(mult3(i) || mult5(i)) ==> (fizzBuzz(i).toInt `== i`)
}
```

---

## Involutivity: fizzBuzz

```scala
propInvolutiveFizzBuzz.check()
```

---

## Involutivity: fizzBuzz

```scala
propInvolutiveFizzBuzz.check()
// + OK, passed 100 tests.
```

---

## Idempotence

---

## Idempotence

---

## Idempotence

---

## Idempotence

---

## Idempotence

---

## Idempotence

```scala
forAll { `(is: List[Int])` =>
  is.sorted == is.sorted.sorted
}
```

---

## Idempotence

```scala
forAll { (is: List[Int]) =>
  `is.sorted` == is.sorted.sorted
}
```

---

## Idempotence

```scala
forAll { (is: List[Int]) =>
  is.sorted == `is.sorted.sorted`
}
```

---

## Idempotence

```scala
forAll { (is: List[Int]) =>
  is.sorted `==` is.sorted.sorted
}
```

---

## Idempotence: use case

`artist`     | `year`     | `album`                 | `name`
--------------|-------------|--------------------------------------------------
Iron Maiden   | 1982        | The Number of the Beast | Children of the Damned
Iron Maiden   | 1982        | The Number of the Beast | Invaders
Iron Maiden   | 1982        | The Number of the Beast | Gangland

```scala
db.delete(
  artist  = "Iron Maiden",
  name    = "Gangland"
)
```

---

## Idempotence: use case

```scala
db.delete(
  artist  = "Iron Maiden",
  name    = "Gangland"
)
```

---

## Idempotence: use case

`artist`     | `year`     | `album`                 | `name`
--------------|-------------|--------------------------------------------------
Iron Maiden   | 1982        | The Number of the Beast | Children of the Damned
       |          |                    |  
       |          |                    |

```scala
db.delete(
  artist  = "Iron Maiden",
  name    = "Gangland"
)
```

---

## Idempotence: fizzBuzz

---

## Invariance

---

## Invariance

---

## Invariance

---

## Invariance

```scala
forAll { `(is: List[Int])` =>
  is.sorted.diff(is).isEmpty
}
```

---

## Invariance

```scala
forAll { (is: List[Int]) =>
  `is.sorted`.diff(is).isEmpty
}
```

---

## Invariance

```scala
forAll { (is: List[Int]) =>
  is.sorted.`diff(is).isEmpty`
}
```

---

## Invariance: use case

---

## Invariance: use case

`artist`     | `year`     | `album`                 | `name`
--------------|-------------|--------------------------------------------------
İ̵̺͝ṛ̸̣͕̐͜o̴̢̎̿̓̌n̸̥͙̬̣̾́͛̾̌͘ͅ ̶̻͂M̷̗̣̠̠̎̀ȧ̸̛͔͇̐͘į̵̗̞̩̯̫̇̔̾̋̕͝d̵̟͓͆̅̇̋̐̚ͅe̷̯̪̙̼̩̩͐n̴̛̗̟͈͇̖̎̈̕   | 1̶͍͓̣̔̍́̒̒̅͘͜9̷̬̖̹̗̻͍̋͗̈́͠8̶͓̫̹̹̍2̸̨̲̝̏͌        | T̸̜̟͕̩̘̃̇͂͠h̶̦̓͋̇͝e̴̢̝̱̦͎͗͂̅̓͜ͅ ̶̝̬͠͝Ǹ̵̛̘̺̯̃̈́̍ù̷̱̖m̷̗͙̱̥͇̹͑̎̂͊̈́́̌b̸̢̦̪̽̅̂̑e̴̤̫̝̎̈́̇r̵͇̥̃̃̈͋̎̋͗͜ ̴͇̖̼͙̥̲̹̄̀ò̷͉̠̒f̶̲͇̳͛̔̃̽ ̸͍̉̚͝t̶̼̗̬̞̣̂͗͠͠͝h̴̰͒̾̑͐́͝ě̷̠̤͎̳͍̿͆́̕͝ ̸̭̬̊̐̾̍B̷̨̑é̴̡̟͔͖̏̃̔̓͝͠ą̷͉̫͇̫̈́̍̄́̃s̵͍̀̅͂͂͑̌ẗ̷̡̪̙͉͔́͑͗̇̋̑ | C̵͚̣̬̠̞̝̰̐̍̃h̷̥͛͂͋̚̚ï̴̤l̴͉̳̘̝͖̤͋͝d̴̢̲̫̯͙̺̮́̈́̕r̸̢̗̥͇̻͓͗̋̆́ȇ̸̲̉̆n̵̞̙͎̠̲͕̭̅̃̉̂ ̵͔̦͙̯̎͂̑̚͝õ̷̳̱͈̻̤̠̫̽̓̑̊͒f̸̦͋ͅ ̴̹̖̌ṫ̷̻̼͖͗̉̀̑h̴̹͎̰̬̟͊̍ͅe̸̤̱͛́͑ ̸̞̾͊͠D̵̢̜̬̳̍̑͛̄͆̀̅a̴͙̼̤̤͓͇̟̒̈́͋m̷̡̞̬̞̬͆̔̿n̴̨̢̖͉͓̥̂̄͘ë̴̬͚͎̭̭ͅd̸̢͕̼̗̠̂
Ì̷̧̺͓͕̘͓̯̗̬͈̤̗̝̰͜ȓ̵̨̜̟̱̘̟̬̙̯̱͍̩̭̈̆̅̍̌̿́̈́́̚͜͝ͅo̴̡͗͗͛̑̿̾͒͂̎̆n̶͖̳͖͉̗͕̱͔̱̖̫̽̎͜ͅ ̸̢̨̛͇͚͔̘̦̘̱̮̖̿͑̒̉̽̎́̑̚̚̕M̷̜͖̊͗̅̆̏̈́͛̍̂̉́̃̚̕͝ả̷̡̛̫͔̳̞̻͔͉̜̍̅̈́̆͛͒̋̑͠i̵̢̡͎̯̙͇͍̇͊̃̓̚͜ͅd̸̟̤̼̲̈́̎̑͜ͅȩ̸͓̝̘̙͉̼̺̖̳͓̩̽̕ͅn̴͈̈̃̿͝   | 1̴̛͈̣̲̹͗̾͂͆̒̓͛̋͝9̵̨̢̨̻̲̬͕̲͔͖̘̀̿͝ͅ8̷̩̹̣̦̙͖͚̞̮́̉̑̓̉͑̈̾́͊̀̽͛͘̚2̵̡̘̠̼͙̅̇͛̽͒̀̄̾́̋̎͐͛͝        | T̷̢̟̙̥̬̰͎͚͈͕̬͆̋͊͛̂̓̈́͊̌̀͆̅̐͘͝ȟ̴̛̹̝͕͒͗è̵̛̩͇͎̤̪͍͎̳̺͔̖̝͖͊̏̀̽͝ ̵͕͇͌͌́͛̑̏͊̈̔̿̚̕͠͝N̴̤̥̖̠͍̻̤̣͈̩̞̾̍̉͐͋̇̈̈́̓̈́̚ͅu̵̲̹̙̦̞̎̏͗͋̌͐̈́͘͜͝͝m̶̡̙̖̳̖͇̻̰̼̝̖̎̈̈̄́̀̽̈́͘͜͝b̸̡̙͚̙̮̀̀̅́̔́̿̑͆̂͒̕è̷̫͕̗̰̌̈́̏͗͆͂r̶̡̢̨̛̛̙̖̲͔̥̲͕̈́̀͆̀̅̀̔̃̿̉̽͝͝ ̵̨͍͎͈̹̺̟̠̪̼̟̟͔̜̆̈́̈́o̶̡̧̯̠̜̤͊̒̀͆̒̀̋̅̽̕͜͝ͅf̸͓̥̻͚̘̹͈͉̱̆̅̊̿͑͛́͆̽̋͘͜͜ ̶̣̭͙̯͇̦͂̋͆͆̕ţ̴̡̨̭̭̺̣̲̤͎̩̮̮̥̿́̔́̂͂h̵͉̤͛͊͛͐͒͒ė̶̜̰͓̤̼̖̜̣͎̫̑̍̌́̂̈̌͆̍ ̴̘̞̖̳͕̳͑̀̆̑͜B̷̨͈̹̣̘̰͕͉̰̹͗̔ȩ̷̡̠̙̖̱̜̲̤͚̼͓̠͛̾̓͆̅̒͂̅̀̿͘͘ạ̶̢̨̮̩͇͚͓̀̄̐̈́̇̇͊̄̾̕͝͝s̵̡̛͚̭̞͌̋̐̑͑̓̒̋͊͊͌̈̎̚ṱ̴̫̤̩͂͒̅̋̓̓̿̅̈́̎ | I̴̢̦̠͈̊̆͌͝͝ͅn̶̛͈̺̜͇̲̭͓͉̎̽̓̊̈̈́̾̔́̽̇͘v̷͈̘̞̬̼̲̍͊͊̋̓̕ą̷̰͙͙̖̖̤̬͙̃d̷̠̀́̓͆̽̽̽̓͘ȩ̵̤̱͓̫̻̫̯̪̎͌̇̆̄̓ͅr̴̗̹̪̬̘̪̝̰͆̒͆̋̃̃̏̈͑̅͝ş̴͎̭̻͔̘͈͉̟̻͈̯̩͓̔̐

---

## Invariance: use case

```scala
forAll { `(cmds: List[Command])` =>
  cmds.forall { cmd =>
    db.execute(cmd)
    notCorrupt(db)
  }
}
```

---

## Invariance: use case

```scala
forAll { (cmds: List[Command]) =>
  cmds.forall { cmd =>
    `db.execute(cmd)`
    notCorrupt(db)
  }
}
```

---

## Invariance: use case

```scala
forAll { (cmds: List[Command]) =>
  cmds.forall { cmd =>
    db.execute(cmd)
    `notCorrupt(db)`
  }
}
```

---

## Invariance: use case

---

## Invariance: fizzBuzz

```scala
val propSafeFizzBuzz = forAll { `(i: Int)` =>
  Try(fizzBuzz(i)).isSuccess
}
```

---

## Invariance: fizzBuzz

```scala
val propSafeFizzBuzz = forAll { (i: Int) =>
  Try(`fizzBuzz(i)`).isSuccess
}
```

---

## Invariance: fizzBuzz

```scala
val propSafeFizzBuzz = forAll { (i: Int) =>
  `Try`(fizzBuzz(i))`.isSuccess`
}
```

---

## Invariance: fizzBuzz

```scala
propSafeFizzBuzz.check()
```

---

## Invariance: fizzBuzz

```scala
propSafeFizzBuzz.check()
// + OK, passed 100 tests.
```

---

## Metamorphic relation

---

## Metamorphic relation

---

## Metamorphic relation

---

## Metamorphic relation

---

## Metamorphic relation

---

## Metamorphic relation

---

## Metamorphic relation

```scala
forAll { `(i: Int)` =>
  mult3(i) == mult3(-i)
}
```

---

## Metamorphic relation

```scala
forAll { (i: Int) =>
  `mult3(i)` == mult3(-i)
}
```

---

## Metamorphic relation

```scala
forAll { (i: Int) =>
  mult3(i) == mult3(`-i`)
}
```

---

## Metamorphic relation

```scala
forAll { (i: Int) =>
  mult3(i) == `mult3`(-i)
}
```

---

## Metamorphic relation

```scala
forAll { (i: Int) =>
  mult3(i) `==` mult3(-i)
}
```

---

## Metamorphic relation: use case

---

## Metamorphic relation: use case

---

## Metamorphic relation: use case

---

## Metamorphic relation: use case

---

## Metamorphic relation: use case

---

## Metamorphic relation: use case

---

## Metamorphic relation: use case

_[Metamorphic testing of RESTful web apis](http://www.lsi.us.es/~segura/files/papers/segura17-tse.pdf)_:

* 3 new unique Spotify bugs.

* 8 new unique YouTube bugs.

---

## Metamorphic relation: fizzBuzz

```scala
val propStableFizz = forAll { `(i: Int)` =>
  fizzBuzz(i).contains("Fizz") ==> fizzBuzz(-i).contains("Fizz")
}
```

---

## Metamorphic relation: fizzBuzz

```scala
val propStableFizz = forAll { (i: Int) =>
  `fizzBuzz(i)`.contains("Fizz") ==> fizzBuzz(-i).contains("Fizz")
}
```

---

## Metamorphic relation: fizzBuzz

```scala
val propStableFizz = forAll { (i: Int) =>
  fizzBuzz(i).contains("Fizz") ==> fizzBuzz(`-i`).contains("Fizz")
}
```

---

## Metamorphic relation: fizzBuzz

```scala
val propStableFizz = forAll { (i: Int) =>
  fizzBuzz(i).contains("Fizz") ==> `fizzBuzz(-i)`.contains("Fizz")
}
```

---

## Metamorphic relation: fizzBuzz

```scala
val propStableFizz = forAll { (i: Int) =>
  fizzBuzz(i)`.contains("Fizz")` ==> fizzBuzz(-i).contains("Fizz")
}
```

---

## Metamorphic relation: fizzBuzz

```scala
val propStableFizz = forAll { (i: Int) =>
  fizzBuzz(i).contains("Fizz") `==>` fizzBuzz(-i).contains("Fizz")
}
```

---

## Metamorphic relation: fizzBuzz

```scala
val propStableFizz = forAll { (i: Int) =>
  fizzBuzz(i).contains("Fizz") ==> fizzBuzz(-i)`.contains("Fizz")`
}
```

---

## Metamorphic relation: fizzBuzz

```scala
propStableFizz.check()
```

---

## Metamorphic relation: fizzBuzz

```scala
propStableFizz.check()
// ! Falsified after 1 passed tests.
// > ARG_0: -1
// > ARG_0_ORIGINAL: -67
```

---

## Fixing `fizzBuzz`

def fizzBuzz(i: Int) =
  if mult3(i) then
    if mult5(i) then    "FizzBuzz"
    else                "Fizz"
  else if mult5(i) then "Buzz"
  else                  i.toString
```
]

---

## Fixing `fizzBuzz`

def fizzBuzz(i: Int) =
  if mult3(i) then
    if mult5(i) then    "FizzBuzz"
    else                "Fizz"
  else if mult5(i) then "Buzz"
  else                  i.toString
```
]

---

## Fixing `fizzBuzz`

```scala
propStableFizz.check()
```

---

## Fixing `fizzBuzz`

```scala
propStableFizz.check()
// + OK, passed 100 tests.
```

---

## Challenging properties

```scala
class FizzBuzzProps extends Properties("FizzBuzz"):

property("validity") = forAll { (i: Int) =>
    fizzBuzz(i) in Set(i.toString, "Fizz", "Buzz", "FizzBuzz")
  }

property("involutivity") = forAll { (i: Int) =>
    !(mult3(i) || mult5(i)) ==> (fizzBuzz(i).toInt == i)
  }

property("metamorphic") = forAll { (i: Int) =>
    fizzBuzz(i).contains("Fizz") ==> fizzBuzz(-i).contains("Fizz")
  }

property("invariant") = forAll { (i: Int) =>
    Try(fizzBuzz(i)).isSuccess
  }
```

---

## Challenging properties

```scala
class FizzBuzzProps extends Properties("FizzBuzz"):

* property("validity") = forAll { (i: Int) =>
*   fizzBuzz(i) in Set(i.toString, "Fizz", "Buzz", "FizzBuzz")
* }

property("involutivity") = forAll { (i: Int) =>
    !(mult3(i) || mult5(i)) ==> (fizzBuzz(i).toInt == i)
  }

property("metamorphic") = forAll { (i: Int) =>
    fizzBuzz(i).contains("Fizz") ==> fizzBuzz(-i).contains("Fizz")
  }

property("invariant") = forAll { (i: Int) =>
    Try(fizzBuzz(i)).isSuccess
  }
```

---

## Challenging properties

```scala
class FizzBuzzProps extends Properties("FizzBuzz"):

property("validity") = forAll { (i: Int) =>
    fizzBuzz(i) in Set(i.toString, "Fizz", "Buzz", "FizzBuzz")
  }

* property("involutivity") = forAll { (i: Int) =>
*   !(mult3(i) || mult5(i)) ==> (fizzBuzz(i).toInt == i)
* }

property("metamorphic") = forAll { (i: Int) =>
    fizzBuzz(i).contains("Fizz") ==> fizzBuzz(-i).contains("Fizz")
  }

property("invariant") = forAll { (i: Int) =>
    Try(fizzBuzz(i)).isSuccess
  }
```

---

## Challenging properties

```scala
class FizzBuzzProps extends Properties("FizzBuzz"):

property("validity") = forAll { (i: Int) =>
    fizzBuzz(i) in Set(i.toString, "Fizz", "Buzz", "FizzBuzz")
  }

property("involutivity") = forAll { (i: Int) =>
    !(mult3(i) || mult5(i)) ==> (fizzBuzz(i).toInt == i)
  }

* property("metamorphic") = forAll { (i: Int) =>
*   fizzBuzz(i).contains("Fizz") ==> fizzBuzz(-i).contains("Fizz")
* }

property("invariant") = forAll { (i: Int) =>
    Try(fizzBuzz(i)).isSuccess
  }
```

---

## Challenging properties

```scala
class FizzBuzzProps extends Properties("FizzBuzz"):

property("validity") = forAll { (i: Int) =>
    fizzBuzz(i) in Set(i.toString, "Fizz", "Buzz", "FizzBuzz")
  }

property("involutivity") = forAll { (i: Int) =>
    !(mult3(i) || mult5(i)) ==> (fizzBuzz(i).toInt == i)
  }

property("metamorphic") = forAll { (i: Int) =>
    fizzBuzz(i).contains("Fizz") ==> fizzBuzz(-i).contains("Fizz")
  }

* property("invariant") = forAll { (i: Int) =>
*   Try(fizzBuzz(i)).isSuccess
* }
```

---

## Challenging properties

```scala
new FizzBuzzProps().check()
```

---

## Challenging properties

```scala
new FizzBuzzProps().check()
// + FizzBuzz.validity: OK, passed 100 tests.
// + FizzBuzz.involutivity: OK, passed 100 tests.
// + FizzBuzz.metamorphic: OK, passed 100 tests.
// + FizzBuzz.invariant: OK, passed 100 tests.
```

---

## Challenging properties

.diff-rm[
```scala
def fizzBuzz(i: Int) =
  if mult3(i) then
    if mult5(i) then    "FizzBuzz"
*   else                `"Fizz"`
* else if mult5(i) then `"Buzz"`
  else                  i.toString
```
]

---

## Challenging properties

.diff-add[
```scala
def fizzBuzz(i: Int) =
  if mult3(i) then
    if mult5(i) then    "FizzBuzz"
*   else                `"Buzz"`
* else if mult5(i) then `"Fizz"`
  else                  i.toString
```
]

---

## Challenging properties

```scala
new FizzBuzzProps().check()
```

---

## Challenging properties

---

## Challenging properties

```scala
val propFizz = forAll { (i: Int) =>
  fizzBuzz(i * 3).contains("Fizz")
}
```

---

## Challenging properties

```scala
val propFizz = forAll { `(i: Int)` =>
  fizzBuzz(i * 3).contains("Fizz")
}
```

---

## Challenging properties

```scala
val propFizz = forAll { (i: Int) =>
  fizzBuzz(`i * 3`).contains("Fizz")
}
```

---

## Challenging properties

```scala
val propFizz = forAll { (i: Int) =>
  `fizzBuzz`(i * 3).contains("Fizz")
}
```

---

## Challenging properties

```scala
val propFizz = forAll { (i: Int) =>
  fizzBuzz(i * 3).`contains("Fizz")`
}
```

---

## Challenging properties

```scala
propFizz.check()
```

---

## Challenging properties

```scala
propFizz.check()
// ! Falsified after 0 passed tests.
// > ARG_0: 1
// > ARG_0_ORIGINAL: 34
```

---

## Key takeaways

Property-based testing is:

* very good at the _test case generation problem_.

* good at the _oracle problem_.

* hard to get the hang of.

---

# Generative Example-Based Tests

---

## Example-Based Tests

```scala
class FizzBuzzSuite extends AnyFunSuite:
  test("n multiple of 3 outputs Fizz") {
    assert(fizzBuzz(3) == "Fizz")
  }

test("n multiple of 5 outputs Buzz") {
    assert(fizzBuzz(5) == "Buzz")
  }

test("n multiple of neither 3 or 5 outputs n") {
    assert(fizzBuzz(7) == "7")
  }

test("n multiple of 3 and 5 outputs FizzBuzz") {
    assert(fizzBuzz(15) == "FizzBuzz")
  }
```

---

## Example-Based Tests

```scala
class FizzBuzzSuite extends AnyFunSuite:
* test("n multiple of 3 outputs Fizz") {
*   assert(fizzBuzz(3) == "Fizz")
* }

test("n multiple of 5 outputs Buzz") {
    assert(fizzBuzz(5) == "Buzz")
  }

test("n multiple of neither 3 or 5 outputs n") {
    assert(fizzBuzz(7) == "7")
  }

test("n multiple of 3 and 5 outputs FizzBuzz") {
    assert(fizzBuzz(15) == "FizzBuzz")
  }
```

---

## Example-Based Tests

```scala
class FizzBuzzSuite extends AnyFunSuite:
  test("n multiple of 3 outputs Fizz") {
    assert(`fizzBuzz(3) == "Fizz"`)
  }

test("n multiple of 5 outputs Buzz") {
    assert(fizzBuzz(5) == "Buzz")
  }

test("n multiple of neither 3 or 5 outputs n") {
    assert(fizzBuzz(7) == "7")
  }

test("n multiple of 3 and 5 outputs FizzBuzz") {
    assert(fizzBuzz(15) == "FizzBuzz")
  }
```

---

## Naive property

```scala
val propFizzBuzz3 =
  fizzBuzz(3) == "Fizz"
```

---

## Naive property

```scala
val propFizzBuzz3 =
  fizzBuzz(`3`) == "Fizz"
```

---

## Naive property

.diff-add[
```scala
val propFizzBuzz3 =
* `forAll(const(3)) { i =>`
* `  `fizzBuzz(3) == "Fizz"
* `}`
```
]

---

## Naive property

```scala
val propFizzBuzz3 =
  forAll(const(3)) { `i` =>
    fizzBuzz(3) == "Fizz"
  }
```

---

## Naive property

```scala
val propFizzBuzz3 =
  forAll(`const(3)`) { i =>
    fizzBuzz(3) == "Fizz"
  }
```

---

## Naive property

.diff-rm[
```scala
val propFizzBuzz3 =
  forAll(const(3)) { i =>
*   fizzBuzz(`3`) == "Fizz"
  }
```
]

---

## Naive property

.diff-add[
```scala
val propFizzBuzz3 =
  forAll(const(3)) { i =>
*   fizzBuzz(`i`) == "Fizz"
  }
```
]

---

## Naive property

```scala
val propFizzBuzz3 =
  forAll(`const(3)`) { i =>
    fizzBuzz(i) == "Fizz"
  }
```

---

## Multiples of 3

```scala
val genMult3: Gen[Int] =
  `arbitrary[Int]`
    .map(_ * 3)
    .filter(_ % 5 != 0)
```

---

## Multiples of 3

```scala
val genMult3: Gen[Int] =
  arbitrary[Int]
    .`map(_ * 3)`
    .filter(_ % 5 != 0)
```

---

## Multiples of 3

```scala
val genMult3: Gen[Int] =
  arbitrary[Int]
    .map(_ * 3)
    .`filter(_ % 5 != 0)`
```

---

## Multiples of 3

.diff-rm[
```scala
val propFizzBuzz3 =
* forAll(`const(3)`) { i =>
    fizzBuzz(i) == "Fizz"
  }
```
]

---

## Multiples of 3

.diff-add[
```scala
val propFizzBuzz3 =
* forAll(`genMult3`) { i =>
    fizzBuzz(i) == "Fizz"
  }
```
]

---

## Generic property

```scala
val propFizzBuzz3 =
  forAll(`genMult3`) { i =>
    fizzBuzz(i) == "Fizz"
  }
```

---

## Generic property

```scala
val propFizzBuzz3 =
  forAll(genMult3) { i =>
    fizzBuzz(i) == `"Fizz"`
  }
```

---

## Generic property

.diff-rm[
```scala
*`val propFizzBuzz3` =
  forAll(genMult3) { i =>
    fizzBuzz(i) == "Fizz"
  }
```
]

---

## Generic property

.diff-add[
```scala
*`def propFizzBuzz(gen: Gen[Int])` =
  forAll(genMult3) { i =>
    fizzBuzz(i) == "Fizz"
  }
```
]

---

## Generic property

.diff-rm[
```scala
def propFizzBuzz(gen: Gen[Int]) =
* forAll(`genMult3`) { i =>
    fizzBuzz(i) == "Fizz"
  }
```
]

---

## Generic property

.diff-add[
```scala
def propFizzBuzz(gen: Gen[Int]) =
* forAll(`gen`) { i =>
    fizzBuzz(i) == "Fizz"
  }
```
]

---

## Generic property

.diff-add[
```scala
*def propFizzBuzz(gen: Gen[Int]`, expected: String`) =
  forAll(gen) { i =>
    fizzBuzz(i) == "Fizz"
  }
```
]

---

## Generic property

.diff-rm[
```scala
def propFizzBuzz(gen: Gen[Int], expected: String) =
  forAll(gen) { i =>
*   fizzBuzz(i) == `"Fizz"`
  }
```
]

---

## Generic property

.diff-add[
```scala
def propFizzBuzz(gen: Gen[Int], expected: String) =
  forAll(gen) { i =>
*   fizzBuzz(i) == `expected`
  }
```
]

---

## Generic property

.diff-rm[
```scala
val propFizzBuzz3 =
*`  forAll(genMult3) { i =>`
*`    fizzBuzz(i) == "Fizz"`
*`  }`
```
]

---

## Generic property

.diff-add[
```scala
val propFizzBuzz3 =
*`  propFizzBuzz(`
*`    gen      = genMult3,`
*`    expected = "Fizz"`
*`  )`
```
]

---

## Generic property

```scala
val propFizzBuzz3 =
  propFizzBuzz(
    `gen      = genMult3`,
    expected = "Fizz"
  )
```

---

## Generic property

```scala
val propFizzBuzz3 =
  propFizzBuzz(
    gen      = genMult3,
    `expected = "Fizz"`
  )
```

---

## Multiples of 5

```scala
class FizzBuzzSuite extends AnyFunSuite:
  test("n multiple of 3 outputs Fizz") {
    assert(fizzBuzz(3) == "Fizz")
  }

* test("n multiple of 5 outputs Buzz") {
*   assert(fizzBuzz(5) == "Buzz")
* }

test("n multiple of neither 3 or 5 outputs n") {
    assert(fizzBuzz(7) == "7")
  }

test("n multiple of 3 and 5 outputs FizzBuzz") {
    assert(fizzBuzz(15) == "FizzBuzz")
  }
```

---

## Multiples of 5

```scala
class FizzBuzzSuite extends AnyFunSuite:
  test("n multiple of 3 outputs Fizz") {
    assert(fizzBuzz(3) == "Fizz")
  }

test("n multiple of 5 outputs Buzz") {
    assert(fizzBuzz(`5`) == "Buzz")
  }

test("n multiple of neither 3 or 5 outputs n") {
    assert(fizzBuzz(7) == "7")
  }

test("n multiple of 3 and 5 outputs FizzBuzz") {
    assert(fizzBuzz(15) == "FizzBuzz")
  }
```

---

## Multiples of 5

```scala
class FizzBuzzSuite extends AnyFunSuite:
  test("n multiple of 3 outputs Fizz") {
    assert(fizzBuzz(3) == "Fizz")
  }

test("n multiple of 5 outputs Buzz") {
    assert(fizzBuzz(5) == `"Buzz"`)
  }

test("n multiple of neither 3 or 5 outputs n") {
    assert(fizzBuzz(7) == "7")
  }

test("n multiple of 3 and 5 outputs FizzBuzz") {
    assert(fizzBuzz(15) == "FizzBuzz")
  }
```

---

## Multiples of 5

```scala
val genMult5: Gen[Int] =
  `arbitrary[Int]`
    .map(_ * 5)
    .filter(_ % 3 != 0)
```

---

## Multiples of 5

```scala
val genMult5: Gen[Int] =
  arbitrary[Int]
    .`map(_ * 5)`
    .filter(_ % 3 != 0)
```

---

## Multiples of 5

```scala
val genMult5: Gen[Int] =
  arbitrary[Int]
    .map(_ * 5)
    .`filter(_ % 3 != 0)`
```

---

## Multiples of 5

```scala
val propFizzBuzz5 =
  propFizzBuzz(
    `gen      = genMult5`,
    expected = "Buzz"
  )
```

---

## Multiples of 5

```scala
val propFizzBuzz5 =
  propFizzBuzz(
    gen      = genMult5,
    `expected = "Buzz"`
  )
```

---

## Multiples of neither 3 nor 5

```scala
class FizzBuzzSuite extends AnyFunSuite:
  test("n multiple of 3 outputs Fizz") {
    assert(fizzBuzz(3) == "Fizz")
  }

test("n multiple of 5 outputs Buzz") {
    assert(fizzBuzz(5) == "Buzz")
  }

* test("n multiple of neither 3 or 5 outputs n") {
*   assert(fizzBuzz(7) == "7")
* }

test("n multiple of 3 and 5 outputs FizzBuzz") {
    assert(fizzBuzz(15) == "FizzBuzz")
  }
```

---

## Multiples of neither 3 nor 5

```scala
class FizzBuzzSuite extends AnyFunSuite:
  test("n multiple of 3 outputs Fizz") {
    assert(fizzBuzz(3) == "Fizz")
  }

test("n multiple of 5 outputs Buzz") {
    assert(fizzBuzz(5) == "Buzz")
  }

test("n multiple of neither 3 or 5 outputs n") {
    assert(fizzBuzz(`7`) == "7")
  }

test("n multiple of 3 and 5 outputs FizzBuzz") {
    assert(fizzBuzz(15) == "FizzBuzz")
  }
```

---

## Multiples of neither 3 nor 5

```scala
class FizzBuzzSuite extends AnyFunSuite:
  test("n multiple of 3 outputs Fizz") {
    assert(fizzBuzz(3) == "Fizz")
  }

test("n multiple of 5 outputs Buzz") {
    assert(fizzBuzz(5) == "Buzz")
  }

test("n multiple of neither 3 or 5 outputs n") {
    assert(fizzBuzz(7) == `"7"`)
  }

test("n multiple of 3 and 5 outputs FizzBuzz") {
    assert(fizzBuzz(15) == "FizzBuzz")
  }
```

---

## Multiples of neither 3 nor 5

```scala
val genMultNone: Gen[Int] =
  `arbitrary[Int]`
    .filter(_ % 5 != 0)
    .filter(_ % 3 != 0)
```

---

## Multiples of neither 3 nor 5

```scala
val genMultNone: Gen[Int] =
  arbitrary[Int]
    .`filter(_ % 5 != 0)`
    .filter(_ % 3 != 0)
```

---

## Multiples of neither 3 nor 5

```scala
val genMultNone: Gen[Int] =
  arbitrary[Int]
    .filter(_ % 5 != 0)
    .`filter(_ % 3 != 0)`
```

---

## Multiples of neither 3 nor 5

```scala
val propFizzBuzzNone =
  propFizzBuzz(
    `gen      = genMultNone`,
    expected = ???
  )
```

---

## Multiples of neither 3 nor 5

```scala
val propFizzBuzzNone =
  propFizzBuzz(
    gen      = genMultNone,
    `expected = ???`
  )
```

---

## Multiples of neither 3 nor 5

.diff-rm[
```scala
*def propFizzBuzz(gen: Gen[Int], `expected: String`) =
  forAll(gen) { i =>
*   fizzBuzz(i) == `expected`
  }
```
]

---

## Multiples of neither 3 nor 5

.diff-add[
```scala
*def propFizzBuzz(gen: Gen[Int], `oracle: Int => String`) =
  forAll(gen) { i =>
*   fizzBuzz(i) == `oracle(i)`
  }
```
]

---

## Multiples of neither 3 nor 5

.diff-rm[
```scala
val propFizzBuzzNone =
  propFizzBuzz(
    gen      = genMultNone,
*   `expected = ???`
  )
```
]

---

## Multiples of neither 3 nor 5

.diff-add[
```scala
val propFizzBuzzNone =
  propFizzBuzz(
    gen      = genMultNone,
*   `oracle   = _.toString`
  )
```
]

---

## Multiples of 3 and 5

```scala
class FizzBuzzSuite extends AnyFunSuite:
  test("n multiple of 3 outputs Fizz") {
    assert(fizzBuzz(3) == "Fizz")
  }

test("n multiple of 5 outputs Buzz") {
    assert(fizzBuzz(5) == "Buzz")
  }

test("n multiple of neither 3 or 5 outputs n") {
    assert(fizzBuzz(7) == "7")
  }

* test("n multiple of 3 and 5 outputs FizzBuzz") {
*   assert(fizzBuzz(15) == "FizzBuzz")
* }
```

---

## Multiples of 3 and 5

```scala
class FizzBuzzSuite extends AnyFunSuite:
  test("n multiple of 3 outputs Fizz") {
    assert(fizzBuzz(3) == "Fizz")
  }

test("n multiple of 5 outputs Buzz") {
    assert(fizzBuzz(5) == "Buzz")
  }

test("n multiple of neither 3 or 5 outputs n") {
    assert(fizzBuzz(7) == "7")
  }

test("n multiple of 3 and 5 outputs FizzBuzz") {
    assert(fizzBuzz(`15`) == "FizzBuzz")
  }
```

---

## Multiples of 3 and 5

```scala
class FizzBuzzSuite extends AnyFunSuite:
  test("n multiple of 3 outputs Fizz") {
    assert(fizzBuzz(3) == "Fizz")
  }

test("n multiple of 5 outputs Buzz") {
    assert(fizzBuzz(5) == "Buzz")
  }

test("n multiple of neither 3 or 5 outputs n") {
    assert(fizzBuzz(7) == "7")
  }

test("n multiple of 3 and 5 outputs FizzBuzz") {
    assert(fizzBuzz(15) == `"FizzBuzz"`)
  }
```

---

## Multiples of 3 and 5

```scala
val genMult15: Gen[Int] =
  `arbitrary[Int]`.
    map(_ * 3 * 5)
```

---

## Multiples of 3 and 5

```scala
val genMult15: Gen[Int] =
  arbitrary[Int].
    `map(_ * 3 * 5)`
```

---

## Multiples of 3 and 5

```scala
val propFizzBuzz15 =
  propFizzBuzz(
    `gen    = genMult15`,
    oracle = _ => "FizzBuzz"
  )
```

---

## Multiples of 3 and 5

```scala
val propFizzBuzz15 =
  propFizzBuzz(
    gen    = genMult15,
    `oracle = _ => "FizzBuzz"`
  )
```

---

## Key takeaways

* you don't need to learn a new way to write tests (right away).

* you can adapt existing tests with little work.

* you will acquire the skills you need as you do so.

---

# In closing

---

## If you only remember 1 slide...

Property-based testing:

* finds more flaws than example-based testing.

* doesn't need to be harder than example-based testing.

* importantly, is a lot of fun!

---

# Questions?

Nicolas Rinaudo • [@NicolasRinaudo]

---

# Metamorphic testing

---

## Profile moderation

---

## Profile moderation

---

## Profile moderation

---

## Profile moderation

---

## Profile moderation

---

## Profile moderation

---

## Metamorphic testing

---

## Metamorphic testing

---

## Metamorphic testing

---

## Metamorphic testing

---

## Metamorphic testing

---

## Compiler validation

```scala
def fizzBuzz(i: Int) =
  if mult3(i) then
    if mult5(i) then    "FizzBuzz"
    else                "Fizz"
  else if mult5(i) then "Buzz"
  else                  i.toString
```

---

## Compiler validation

```scala
println(fizzBuzz(5))
```

---

## Compiler validation

```scala
println(fizzBuzz(`5`))
```

---

## Compiler validation

```scala
println(fizzBuzz(5))
// Buzz
```

---

## Compiler validation

```scala
println(fizzBuzz(5))
// `Buzz`
```

---

## Compiler validation

.coverage-on[
```scala
def fizzBuzz(i: Int) =
* if mult3(i) then
    if mult5(i) then    "FizzBuzz"
    else                "Fizz"
* else if mult5(i) then "Buzz"
  else                  i.toString
```
]

---

## Compiler validation

.coverage-off[
```scala
def fizzBuzz(i: Int) =
  if mult3(i) then
*   if mult5(i) then    "FizzBuzz"
*   else                "Fizz"
  else if mult5(i) then "Buzz"
* else                  i.toString
```
]

---

## Compiler validation

.diff-rm[
```scala
def fizzBuzz(i: Int) =
  if mult3(i) then
*   `if mult5(i) then    "FizzBuzz"`
*   `else                "Fizz"`
  else if mult5(i) then "Buzz"
  else                  i.toString
```
]

---

## Compiler validation

.diff-add[
```scala
def fizzBuzz(i: Int) =
* if mult3(i) then      `"Fizz"`
  else if mult5(i) then "Buzz"
  else                  i.toString
```
]
---

## Compiler validation

.diff-rm[
```scala
def fizzBuzz(i: Int) =
* if mult3(i) then      `"Fizz"`
  else if mult5(i) then "Buzz"
  else                  i.toString
```
]

---

## Compiler validation

.diff-add[
```scala
def fizzBuzz(i: Int) =
* if mult3(i) then      `"Zzif"`
  else if mult5(i) then "Buzz"
  else                  i.toString
```
]

---

## Compiler validation

```scala
println(fizzBuzz(5))
```

---

## Compiler validation

```scala
println(fizzBuzz(`5`))
```

---

## Compiler validation

```scala
println(fizzBuzz(5))
// Buzz
```

---

## Compiler validation

```scala
println(fizzBuzz(5))
// `Buzz`
```

---

## Compiler validation

```scala
println(fizzBuzz(5))
// `Zzif`
```

---

## Compiler validation

_[Compiler Validation via Equivalence Modulo Inputs](http://vuminhle.com/pdf/pldi14-emi.pdf)_:

* 79 new unique gcc bugs.

* 68 new unique LLVM bugs.

* 42 new unique scalac bugs.

---

## Key takeaways

Metamorphic Testing:

* helps generating test cases when it's too expensive.

* helps with the oracle problem.

* a good word to use when you want to sound smart.

---

# Questions?

Nicolas Rinaudo • [@NicolasRinaudo]

[@NicolasRinaudo]:https://twitter.com/NicolasRinaudo