Much ado about testing

1 / 330

Software testing2 / 330

Testing

3 / 330

Testing

4 / 330

Testing

5 / 330

Testing

6 / 330

Testing

7 / 330

System Under Test

Write a program that, given a number, prints it.

But for multiples of three print Fizz instead of the number and for the multiples of five print Buzz.

def mult3(i: Int) = !(i % 3 > 0)
def mult5(i: Int) = !(i % 5 > 0)
def fizzBuzz(i: Int) =
  if mult3(i) then      "Fizz"
  else if mult5(i) then "Buzz"
  else                  i.toString

8 / 330

System Under Test

Write a program that, given a number, prints it.

But for multiples of three print Fizz instead of the number and for the multiples of five print Buzz.

def mult3(i: Int) = !(i % 3 > 0)
def mult5(i: Int) = !(i % 5 > 0)
def fizzBuzz(i: Int) =
  if mult3(i) then      "Fizz"
  else if mult5(i) then "Buzz"
  else                  i.toString

9 / 330

System Under Test

Write a program that, given a number, prints it.

But for multiples of three print Fizz instead of the number and for the multiples of five print Buzz.

def mult3(i: Int) = !(i % 3 > 0)
def mult5(i: Int) = !(i % 5 > 0)
def fizzBuzz(i: Int) =
  if mult3(i) then      "Fizz"
  else if mult5(i) then "Buzz"
  else                  i.toString

10 / 330

System Under Test

Write a program that, given a number, prints it.

But for multiples of three print Fizz instead of the number and for the multiples of five print Buzz.

def mult3(i: Int) = !(i % 3 > 0)
def mult5(i: Int) = !(i % 5 > 0)
def fizzBuzz(i: Int) =
  if mult3(i) then      "Fizz"
  else if mult5(i) then "Buzz"
  else                  i.toString

11 / 330

System Under Test

Write a program that, given a number, prints it.

But for multiples of three print Fizz instead of the number and for the multiples of five print Buzz.

def mult3(i: Int) = !(i % 3 > 0)
def mult5(i: Int) = !(i % 5 > 0)
def fizzBuzz(i: Int) =
  if mult3(i) then      "Fizz"
  else if mult5(i) then "Buzz"
  else                  i.toString

12 / 330

System Under Test

Write a program that, given a number, prints it.

But for multiples of three print Fizz instead of the number and for the multiples of five print Buzz.

def mult3(i: Int) = !(i % 3 > 0)
def mult5(i: Int) = !(i % 5 > 0)
def fizzBuzz(i: Int) =
  if mult3(i) then      "Fizz"
  else if mult5(i) then "Buzz"
  else                  i.toString

13 / 330

System Under Test

Write a program that, given a number, prints it.

But for multiples of three print Fizz instead of the number and for the multiples of five print Buzz.

def mult3(i: Int) = !(i % 3 > 0)
def mult5(i: Int) = !(i % 5 > 0)
def fizzBuzz(i: Int) =
  if mult3(i) then      "Fizz"
  else if mult5(i) then "Buzz"
  else                  i.toString

14 / 330

Example-based testing15 / 330

Overview

Testing

16 / 330

Overview

Testing

17 / 330

Overview

Testing

18 / 330

Overview

Testing

19 / 330

Overview

Testing

20 / 330

Overview

Testing

21 / 330

Overview

Testing

class ListTests extends AnyFunSuite:
  test("List(3, 2, 1) should sort to List(1, 2, 3)") {
    assert(List(3, 2, 1).sorted == List(1, 2, 3))
  }

22 / 330

Overview

Testing

class ListTests extends AnyFunSuite:
  test("List(3, 2, 1) should sort to List(1, 2, 3)") {
    assert(List(3, 2, 1).sorted == List(1, 2, 3))
  }

23 / 330

Overview

Testing

class ListTests extends AnyFunSuite:
  test("List(3, 2, 1) should sort to List(1, 2, 3)") {
    assert(List(3, 2, 1).sorted == List(1, 2, 3))
  }

24 / 330

FizzBuzz test suite

class FizzBuzzSuite extends AnyFunSuite:
  test("n multiple of 3 outputs Fizz") {
    assert(fizzBuzz(3) == "Fizz")
  }
  test("n multiple of 5 outputs Buzz") {
    assert(fizzBuzz(5) == "Buzz")
  }
  test("n multiple of neither 3 or 5 outputs n") {
    assert(fizzBuzz(7) == "7")
  }

25 / 330

FizzBuzz test suite

class FizzBuzzSuite extends AnyFunSuite:
  test("n multiple of 3 outputs Fizz") {
    assert(fizzBuzz(3) == "Fizz")
  }
  test("n multiple of 5 outputs Buzz") {
    assert(fizzBuzz(5) == "Buzz")
  }
  test("n multiple of neither 3 or 5 outputs n") {
    assert(fizzBuzz(7) == "7")
  }

26 / 330

FizzBuzz test suite

class FizzBuzzSuite extends AnyFunSuite:
  test("n multiple of 3 outputs Fizz") {
    assert(fizzBuzz(3) == "Fizz")
  }
  test("n multiple of 5 outputs Buzz") {
    assert(fizzBuzz(5) == "Buzz")
  }
  test("n multiple of neither 3 or 5 outputs n") {
    assert(fizzBuzz(7) == "7")
  }

27 / 330

FizzBuzz test suite

class FizzBuzzSuite extends AnyFunSuite:
  test("n multiple of 3 outputs Fizz") {
    assert(fizzBuzz(3) == "Fizz")
  }
  test("n multiple of 5 outputs Buzz") {
    assert(fizzBuzz(5) == "Buzz")
  }
  test("n multiple of neither 3 or 5 outputs n") {
    assert(fizzBuzz(7) == "7")
  }

28 / 330

FizzBuzz test suite

class FizzBuzzSuite extends AnyFunSuite:
  test("n multiple of 3 outputs Fizz") {
    assert(fizzBuzz(3) == "Fizz")
  }
  test("n multiple of 5 outputs Buzz") {
    assert(fizzBuzz(5) == "Buzz")
  }
  test("n multiple of neither 3 or 5 outputs n") {
    assert(fizzBuzz(7) == "7")
  }

29 / 330

FizzBuzz test suite

class FizzBuzzSuite extends AnyFunSuite:
  test("n multiple of 3 outputs Fizz") {
    assert(fizzBuzz(3) == "Fizz")
  }
  test("n multiple of 5 outputs Buzz") {
    assert(fizzBuzz(5) == "Buzz")
  }
  test("n multiple of neither 3 or 5 outputs n") {
    assert(fizzBuzz(7) == "7")
  }

30 / 330

FizzBuzz test suite

class FizzBuzzSuite extends AnyFunSuite:
  test("n multiple of 3 outputs Fizz") {
    assert(fizzBuzz(3) == "Fizz")
  }
  test("n multiple of 5 outputs Buzz") {
    assert(fizzBuzz(5) == "Buzz")
  }
  test("n multiple of neither 3 or 5 outputs n") {
    assert(fizzBuzz(7) == "7")
  }

31 / 330

FizzBuzz test suite

class FizzBuzzSuite extends AnyFunSuite:
  test("n multiple of 3 outputs Fizz") {
    assert(fizzBuzz(3) == "Fizz")
  }
  test("n multiple of 5 outputs Buzz") {
    assert(fizzBuzz(5) == "Buzz")
  }
  test("n multiple of neither 3 or 5 outputs n") {
    assert(fizzBuzz(7) == "7")
  }

32 / 330

FizzBuzz test suite

new FizzBuzzSuite().execute()

33 / 330

FizzBuzz test suite

new FizzBuzzSuite().execute()
// rs$line$3$FizzBuzzSuite:
// - n multiple of 3 outputs Fizz
// - n multiple of 5 outputs Buzz
// - n multiple of neither 3 or 5 outputs n

34 / 330

Client Demo

fizzBuzz(15)

35 / 330

Client Demo

fizzBuzz(15)
// val res1: String = Fizz

36 / 330

Client Demo

fizzBuzz(15)
// val res1: String = Fizz

Fail

37 / 330

Clarifying specifications

n % 3
n % 5

Output

 T
T

FizzBuzz

 T
F

Fizz

 F
T

Buzz

 F
F

n

38 / 330

n % 3	n % 5	Output
T	T	FizzBuzz
T	F	Fizz
F	T	Buzz
F	F	n

Clarifying specifications

n % 3
n % 5

Output

 T
T

FizzBuzz

 T
F

Fizz

 F
T

Buzz

 F
F

n

39 / 330

n % 3	n % 5	Output
T	T	FizzBuzz
T	F	Fizz
F	T	Buzz
F	F	n

Clarifying specifications

n % 3
n % 5

Output

 T
T

FizzBuzz

 T
F

Fizz

 F
T

Buzz

 F
F

n

40 / 330

n % 3	n % 5	Output
T	T	FizzBuzz
T	F	Fizz
F	T	Buzz
F	F	n

Clarifying specifications

n % 3
n % 5

Output

 T
T

FizzBuzz

 T
F

Fizz

 F
T

Buzz

 F
F

n

41 / 330

n % 3	n % 5	Output
T	T	FizzBuzz
T	F	Fizz
F	T	Buzz
F	F	n

Clarifying specifications

n % 3
n % 5

Output

 T
T

FizzBuzz

 T
F

Fizz

 F
T

Buzz

 F
F

n

42 / 330

n % 3	n % 5	Output
T	T	FizzBuzz
T	F	Fizz
F	T	Buzz
F	F	n

Clarifying specifications

n % 3
n % 5

Output

 T
T

FizzBuzz

 T
F

Fizz

 F
T

Buzz

 F
F

n

43 / 330

n % 3	n % 5	Output
T	T	FizzBuzz
T	F	Fizz
F	T	Buzz
F	F	n

Clarifying specifications

n % 3
n % 5

Output

 T
T

FizzBuzz

 T
F

Fizz

 F
T

Buzz

 F
F

n

44 / 330

n % 3	n % 5	Output
T	T	FizzBuzz
T	F	Fizz
F	T	Buzz
F	F	n

Fixing FizzBuzzdef mult3(i: Int) = !(i % 3 > 0)
def mult5(i: Int) = !(i % 5 > 0)
def fizzBuzz(i: Int) =
  if mult3(i) then      "Fizz"
  else if mult5(i) then "Buzz"
  else                  i.toString

45 / 330

Fixing FizzBuzzdef mult3(i: Int) = !(i % 3 > 0)
def mult5(i: Int) = !(i % 5 > 0)
def fizzBuzz(i: Int) =
  if mult3(i) then
    if mult5(i) then    "FizzBuzz"
    else                "Fizz"
  else if mult5(i) then "Buzz"
  else                  i.toString

46 / 330

Fixing FizzBuzz

def mult3(i: Int) = !(i % 3 > 0)
def mult5(i: Int) = !(i % 5 > 0)
def fizzBuzz(i: Int) =
  if mult3(i) then
    if mult5(i) then    "FizzBuzz"
    else                "Fizz"
  else if mult5(i) then "Buzz"
  else                  i.toString

47 / 330

Fixing FizzBuzz

def mult3(i: Int) = !(i % 3 > 0)
def mult5(i: Int) = !(i % 5 > 0)
def fizzBuzz(i: Int) =
  if mult3(i) then
    if mult5(i) then    "FizzBuzz"
    else                "Fizz"
  else if mult5(i) then "Buzz"
  else                  i.toString

48 / 330

Fixing testsclass FizzBuzzSuite extends AnyFunSuite:
  test("n multiple of 3 outputs Fizz") {
    assert(fizzBuzz(3) == "Fizz")
  }
  test("n multiple of 5 outputs Buzz") {
    assert(fizzBuzz(5) == "Buzz")
  }
  test("n multiple of neither 3 or 5 outputs n") {
    assert(fizzBuzz(7) == "7")
  }
  test("n multiple of 3 and 5 outputs FizzBuzz") {
    assert(fizzBuzz(15) == "FizzBuzz")
  }

49 / 330

Fixing tests

class FizzBuzzSuite extends AnyFunSuite:
  test("n multiple of 3 outputs Fizz") {
    assert(fizzBuzz(3) == "Fizz")
  }
  test("n multiple of 5 outputs Buzz") {
    assert(fizzBuzz(5) == "Buzz")
  }
  test("n multiple of neither 3 or 5 outputs n") {
    assert(fizzBuzz(7) == "7")
  }
  test("n multiple of 3 and 5 outputs FizzBuzz") {
    assert(fizzBuzz(15) == "FizzBuzz")
  }

50 / 330

Fixing tests

class FizzBuzzSuite extends AnyFunSuite:
  test("n multiple of 3 outputs Fizz") {
    assert(fizzBuzz(3) == "Fizz")
  }
  test("n multiple of 5 outputs Buzz") {
    assert(fizzBuzz(5) == "Buzz")
  }
  test("n multiple of neither 3 or 5 outputs n") {
    assert(fizzBuzz(7) == "7")
  }
  test("n multiple of 3 and 5 outputs FizzBuzz") {
    assert(fizzBuzz(15) == "FizzBuzz")
  }

51 / 330

Fixing tests

class FizzBuzzSuite extends AnyFunSuite:
  test("n multiple of 3 outputs Fizz") {
    assert(fizzBuzz(3) == "Fizz")
  }
  test("n multiple of 5 outputs Buzz") {
    assert(fizzBuzz(5) == "Buzz")
  }
  test("n multiple of neither 3 or 5 outputs n") {
    assert(fizzBuzz(7) == "7")
  }
  test("n multiple of 3 and 5 outputs FizzBuzz") {
    assert(fizzBuzz(15) == "FizzBuzz")
  }

52 / 330

Fixing tests

new FizzBuzzSuite().execute()

53 / 330

Fixing tests

new FizzBuzzSuite().execute()
// rs$line$7$FizzBuzzSuite:
// - n multiple of 3 outputs Fizz
// - n multiple of 5 outputs Buzz
// - n multiple of neither 3 or 5 outputs n
// - n multiple of 3 and 5 outputs FizzBuzz

54 / 330

Client Demo

fizzBuzz(-3)

55 / 330

Client Demo

fizzBuzz(-3)
// val res3: String = FizzBuzz

56 / 330

Client Demo

fizzBuzz(-3)
// val res3: String = FizzBuzz

Fail

57 / 330

Key takeaways

Example-based testing is:

58 / 330

Key takeaways

Example-based testing is:

very good at the oracle problem.

59 / 330

Key takeaways

Example-based testing is:

very good at the oracle problem.
very bad at the test case generation problem.

60 / 330

Key takeaways

Example-based testing is:

very good at the oracle problem.
very bad at the test case generation problem.
a little bit discouraging...

61 / 330

Generative Testing62 / 330

Overview

Testing

63 / 330

Overview

Testing

64 / 330

Overview

Testing

65 / 330

Overview

Testing

66 / 330

Overview

Testing

67 / 330

Overview

Testing

68 / 330

Property-Based Testing

Testing

69 / 330

Property-Based Testing

Testing

forAll { (is: List[Int]) =>
  is.sorted.diff(is).isEmpty
}

70 / 330

Property-Based Testing

Testing

forAll { (is: List[Int]) =>
  is.sorted.diff(is).isEmpty
}

71 / 330

Property-Based Testing

Testing

forAll { (is: List[Int]) =>
  is.sorted.diff(is).isEmpty
}

72 / 330

Property-Based Testing

val propFizzBuzz = forAll { (i: Int) =>
  if mult3(i) then
    if mult5(i) then    fizzBuzz(i) == "FizzBuzz"
    else                fizzBuzz(i) == "Fizz"
  else if mult5(i) then fizzBuzz(i) == "Buzz"
  else                  fizzBuzz(i) == i.toString
}

73 / 330

Property-Based Testing

val propFizzBuzz = forAll { (i: Int) =>
  if mult3(i) then
    if mult5(i) then    fizzBuzz(i) == "FizzBuzz"
    else                fizzBuzz(i) == "Fizz"
  else if mult5(i) then fizzBuzz(i) == "Buzz"
  else                  fizzBuzz(i) == i.toString
}

74 / 330

Property-Based Testing

val propFizzBuzz = forAll { (i: Int) =>
  if mult3(i) then
    if mult5(i) then    fizzBuzz(i) == "FizzBuzz"
    else                fizzBuzz(i) == "Fizz"
  else if mult5(i) then fizzBuzz(i) == "Buzz"
  else                  fizzBuzz(i) == i.toString
}

75 / 330

Property-Based Testing

val propFizzBuzz = forAll { (i: Int) =>
  if mult3(i) then
    if mult5(i) then    fizzBuzz(i) == "FizzBuzz"
    else                fizzBuzz(i) == "Fizz"
  else if mult5(i) then fizzBuzz(i) == "Buzz"
  else                  fizzBuzz(i) == i.toString
}

76 / 330

Property-Based Testing

val propFizzBuzz = forAll { (i: Int) =>
  if mult3(i) then
    if mult5(i) then    fizzBuzz(i) == "FizzBuzz"
    else                fizzBuzz(i) == "Fizz"
  else if mult5(i) then fizzBuzz(i) == "Buzz"
  else                  fizzBuzz(i) == i.toString
}

77 / 330

Property-Based Testing

propFizzBuzz.check()

78 / 330

Property-Based Testing

propFizzBuzz.check()
// + OK, passed 100 tests.

79 / 330

Property-Based Testing

val propFizzBuzz = forAll { (i: Int) =>
  if mult3(i) then
    if mult5(i) then    fizzBuzz(i) == "FizzBuzz"
    else                fizzBuzz(i) == "Fizz"
  else if mult5(i) then fizzBuzz(i) == "Buzz"
  else                  fizzBuzz(i) == i.toString
}

80 / 330

Test oracle

Oracle

81 / 330

Test oracle

Oracle

82 / 330

Test oracle

Oracle

83 / 330

Test oracle

Oracle

84 / 330

Test oracle

Oracle

85 / 330

Test oracle

Oracle

86 / 330

Test oracle

Oracle

forAll { (is: List[Int]) =>
  fastSort(is) == is.sorted
}

87 / 330

Test oracle

Oracle

forAll { (is: List[Int]) =>
  fastSort(is) == is.sorted
}

88 / 330

Test oracle

Oracle

forAll { (is: List[Int]) =>
  fastSort(is) == is.sorted
}

89 / 330

Test oracle

Oracle

forAll { (is: List[Int]) =>
  fastSort(is) == is.sorted
}

90 / 330

Test oracle: use case

Oracle

91 / 330

Test oracle: use case

Oracle

92 / 330

Test oracle: use case

Oracle

93 / 330

Test oracle: use case

Oracle

94 / 330

Test oracle: use case

Oracle

95 / 330

Test oracle: use case

Oracle

96 / 330

Test oracle: use case

Oracle

97 / 330

Test oracle: use case

Oracle

Fail

98 / 330

Test oracle: fizzBuzz

Fail

99 / 330

Validity

100 / 330

Validity

101 / 330

Validity

102 / 330

Validity

forAll { (i: Int) =>
  math.abs(i) >= 0
}

103 / 330

Validity

forAll { (i: Int) =>
  math.abs(i) >= 0
}

104 / 330

Validity

forAll { (i: Int) =>
  math.abs(i) >= 0
}

105 / 330

Validity: use case

Validity

106 / 330

Validity: use case

Validity

107 / 330

Validity: use case

Validity

108 / 330

Validity: use case

Validity

109 / 330

Validity: use case

Validity

Fail

110 / 330

Validity: fizzBuzz

val propValidFizzBuzz = forAll { (i: Int) =>
  fizzBuzz(i) in Set(i.toString, "Fizz", "Buzz", "FizzBuzz")
}

111 / 330

Validity: fizzBuzz

val propValidFizzBuzz = forAll { (i: Int) =>
  fizzBuzz(i) in Set(i.toString, "Fizz", "Buzz", "FizzBuzz")
}

112 / 330

Validity: fizzBuzz

val propValidFizzBuzz = forAll { (i: Int) =>
  fizzBuzz(i) in Set(i.toString, "Fizz", "Buzz", "FizzBuzz")
}

113 / 330

Validity: fizzBuzz

propValidFizzBuzz.check()

114 / 330

Validity: fizzBuzz

propValidFizzBuzz.check()
// + OK, passed 100 tests.

115 / 330

Involutivity

116 / 330

Involutivity

117 / 330

Involutivity

118 / 330

Involutivity

119 / 330

Involutivity

120 / 330

Involutivity

121 / 330

Involutivity

122 / 330

Involutivity

forAll { (i: Int) =>
  i.toString.toInt == i
}

123 / 330

Involutivity

forAll { (i: Int) =>
  i.toString.toInt == i
}

124 / 330

Involutivity

forAll { (i: Int) =>
  i.toString.toInt == i
}

125 / 330

Involutivity

forAll { (i: Int) =>
  i.toString.toInt == i
}

126 / 330

Involutivity: use case

Track(
   artist  = "Iron Maiden",
   year    = 1982,
   album   = "The Number of the Beast",
   name    = "Children of the Damned"
)

127 / 330

Involutivity: use case

Track(
   artist  = "Iron Maiden",
   year    = 1982,
   album   = "The Number of the Beast",
   name    = "Children of the Damned"
)

128 / 330

Involutivity: use case

{
  "artist" : "Iron Maiden",
  "year"   : 1982,
  "album"  : "The Number of the Beast",
  "name"   : "Children of the Damned"
}

129 / 330

Involutivity: use case

Track(
   artist  = "Iron Maiden",
   year    = 1970,
   album   = "The Number of the Beast",
   name    = "Children of the Damned"
)

130 / 330

Involutivity: use case

Track(
   artist  = "Iron Maiden",
   year    = 1970,
   album   = "The Number of the Beast",
   name    = "Children of the Damned"
)

Fail

131 / 330

Involutivity: fizzBuzz

val propInvolutiveFizzBuzz = forAll { (i: Int) =>
  !(mult3(i) || mult5(i)) ==> (fizzBuzz(i).toInt == i)
}

132 / 330

Involutivity: fizzBuzz

val propInvolutiveFizzBuzz = forAll { (i: Int) =>
  !(mult3(i) || mult5(i)) ==> (fizzBuzz(i).toInt == i)
}

133 / 330

Involutivity: fizzBuzz

val propInvolutiveFizzBuzz = forAll { (i: Int) =>
  !(mult3(i) || mult5(i)) ==> (fizzBuzz(i).toInt == i)
}

134 / 330

Involutivity: fizzBuzz

val propInvolutiveFizzBuzz = forAll { (i: Int) =>
  !(mult3(i) || mult5(i)) ==> (fizzBuzz(i).toInt == i)
}

135 / 330

Involutivity: fizzBuzz

val propInvolutiveFizzBuzz = forAll { (i: Int) =>
  !(mult3(i) || mult5(i)) ==> (fizzBuzz(i).toInt == i)
}

136 / 330

Involutivity: fizzBuzz

val propInvolutiveFizzBuzz = forAll { (i: Int) =>
  !(mult3(i) || mult5(i)) ==> (fizzBuzz(i).toInt == i)
}

137 / 330

Involutivity: fizzBuzz

propInvolutiveFizzBuzz.check()

138 / 330

Involutivity: fizzBuzz

propInvolutiveFizzBuzz.check()
// + OK, passed 100 tests.

139 / 330

Idempotence

140 / 330

Idempotence

141 / 330

Idempotence

142 / 330

Idempotence

143 / 330

Idempotence

144 / 330

Idempotence

forAll { (is: List[Int]) =>
  is.sorted == is.sorted.sorted
}

145 / 330

Idempotence

forAll { (is: List[Int]) =>
  is.sorted == is.sorted.sorted
}

146 / 330

Idempotence

forAll { (is: List[Int]) =>
  is.sorted == is.sorted.sorted
}

147 / 330

Idempotence

forAll { (is: List[Int]) =>
  is.sorted == is.sorted.sorted
}

148 / 330

Idempotence: use case

`artist`	`year`	`album`	`name`
Iron Maiden	1982	The Number of the Beast	Children of the Damned
Iron Maiden	1982	The Number of the Beast	Invaders
Iron Maiden	1982	The Number of the Beast	Gangland

db.delete(
  artist  = "Iron Maiden",
  name    = "Gangland"
)

149 / 330

Idempotence: use case

`artist`	`year`	`album`	`name`
Iron Maiden	1982	The Number of the Beast	Children of the Damned
Iron Maiden	1982	The Number of the Beast	Invaders

db.delete(
  artist  = "Iron Maiden",
  name    = "Gangland"
)

150 / 330

Idempotence: use case

`artist`	`year`	`album`	`name`
Iron Maiden	1982	The Number of the Beast	Children of the Damned

db.delete(
  artist  = "Iron Maiden",
  name    = "Gangland"
)

151 / 330

Idempotence: use case

`artist`	`year`	`album`	`name`
Iron Maiden	1982	The Number of the Beast	Children of the Damned

db.delete(
  artist  = "Iron Maiden",
  name    = "Gangland"
)

Fail

152 / 330

Idempotence: fizzBuzz

Fail

153 / 330

Invariance

154 / 330

Invariance

155 / 330

Invariance

156 / 330

Invariance

forAll { (is: List[Int]) =>
  is.sorted.diff(is).isEmpty
}

157 / 330

Invariance

forAll { (is: List[Int]) =>
  is.sorted.diff(is).isEmpty
}

158 / 330

Invariance

forAll { (is: List[Int]) =>
  is.sorted.diff(is).isEmpty
}

159 / 330

Invariance: use case

artist
year
album
name


Iron Maiden
1982
The Number of the Beast
Children of the Damned

Iron Maiden
1982
The Number of the Beast
Invaders

160 / 330

Invariance: use case

artist
year
album
name


İ̵̺͝ṛ̸̣͕̐͜o̴̢̎̿̓̌n̸̥͙̬̣̾́͛̾̌͘ͅ ̶̻͂M̷̗̣̠̠̎̀ȧ̸̛͔͇̐͘į̵̗̞̩̯̫̇̔̾̋̕͝d̵̟͓͆̅̇̋̐̚ͅe̷̯̪̙̼̩̩͐n̴̛̗̟͈͇̖̎̈̕
1̶͍͓̣̔̍́̒̒̅͘͜9̷̬̖̹̗̻͍̋͗̈́͠8̶͓̫̹̹̍2̸̨̲̝̏͌
T̸̜̟͕̩̘̃̇͂͠h̶̦̓͋̇͝e̴̢̝̱̦͎͗͂̅̓͜ͅ ̶̝̬͠͝Ǹ̵̛̘̺̯̃̈́̍ù̷̱̖m̷̗͙̱̥͇̹͑̎̂͊̈́́̌b̸̢̦̪̽̅̂̑e̴̤̫̝̎̈́̇r̵͇̥̃̃̈͋̎̋͗͜ ̴͇̖̼͙̥̲̹̄̀ò̷͉̠̒f̶̲͇̳͛̔̃̽ ̸͍̉̚͝t̶̼̗̬̞̣̂͗͠͠͝h̴̰͒̾̑͐́͝ě̷̠̤͎̳͍̿͆́̕͝ ̸̭̬̊̐̾̍B̷̨̑é̴̡̟͔͖̏̃̔̓͝͠ą̷͉̫͇̫̈́̍̄́̃s̵͍̀̅͂͂͑̌ẗ̷̡̪̙͉͔́͑͗̇̋̑
C̵͚̣̬̠̞̝̰̐̍̃h̷̥͛͂͋̚̚ï̴̤l̴͉̳̘̝͖̤͋͝d̴̢̲̫̯͙̺̮́̈́̕r̸̢̗̥͇̻͓͗̋̆́ȇ̸̲̉̆n̵̞̙͎̠̲͕̭̅̃̉̂ ̵͔̦͙̯̎͂̑̚͝õ̷̳̱͈̻̤̠̫̽̓̑̊͒f̸̦͋ͅ ̴̹̖̌ṫ̷̻̼͖͗̉̀̑h̴̹͎̰̬̟͊̍ͅe̸̤̱͛́͑ ̸̞̾͊͠D̵̢̜̬̳̍̑͛̄͆̀̅a̴͙̼̤̤͓͇̟̒̈́͋m̷̡̞̬̞̬͆̔̿n̴̨̢̖͉͓̥̂̄͘ë̴̬͚͎̭̭ͅd̸̢͕̼̗̠̂

Ì̷̧̺͓͕̘͓̯̗̬͈̤̗̝̰͜ȓ̵̨̜̟̱̘̟̬̙̯̱͍̩̭̈̆̅̍̌̿́̈́́̚͜͝ͅo̴̡͗͗͛̑̿̾͒͂̎̆n̶͖̳͖͉̗͕̱͔̱̖̫̽̎͜ͅ ̸̢̨̛͇͚͔̘̦̘̱̮̖̿͑̒̉̽̎́̑̚̚̕M̷̜͖̊͗̅̆̏̈́͛̍̂̉́̃̚̕͝ả̷̡̛̫͔̳̞̻͔͉̜̍̅̈́̆͛͒̋̑͠i̵̢̡͎̯̙͇͍̇͊̃̓̚͜ͅd̸̟̤̼̲̈́̎̑͜ͅȩ̸͓̝̘̙͉̼̺̖̳͓̩̽̕ͅn̴͈̈̃̿͝
1̴̛͈̣̲̹͗̾͂͆̒̓͛̋͝9̵̨̢̨̻̲̬͕̲͔͖̘̀̿͝ͅ8̷̩̹̣̦̙͖͚̞̮́̉̑̓̉͑̈̾́͊̀̽͛͘̚2̵̡̘̠̼͙̅̇͛̽͒̀̄̾́̋̎͐͛͝
T̷̢̟̙̥̬̰͎͚͈͕̬͆̋͊͛̂̓̈́͊̌̀͆̅̐͘͝ȟ̴̛̹̝͕͒͗è̵̛̩͇͎̤̪͍͎̳̺͔̖̝͖͊̏̀̽͝ ̵͕͇͌͌́͛̑̏͊̈̔̿̚̕͠͝N̴̤̥̖̠͍̻̤̣͈̩̞̾̍̉͐͋̇̈̈́̓̈́̚ͅu̵̲̹̙̦̞̎̏͗͋̌͐̈́͘͜͝͝m̶̡̙̖̳̖͇̻̰̼̝̖̎̈̈̄́̀̽̈́͘͜͝b̸̡̙͚̙̮̀̀̅́̔́̿̑͆̂͒̕è̷̫͕̗̰̌̈́̏͗͆͂r̶̡̢̨̛̛̙̖̲͔̥̲͕̈́̀͆̀̅̀̔̃̿̉̽͝͝ ̵̨͍͎͈̹̺̟̠̪̼̟̟͔̜̆̈́̈́o̶̡̧̯̠̜̤͊̒̀͆̒̀̋̅̽̕͜͝ͅf̸͓̥̻͚̘̹͈͉̱̆̅̊̿͑͛́͆̽̋͘͜͜ ̶̣̭͙̯͇̦͂̋͆͆̕ţ̴̡̨̭̭̺̣̲̤͎̩̮̮̥̿́̔́̂͂h̵͉̤͛͊͛͐͒͒ė̶̜̰͓̤̼̖̜̣͎̫̑̍̌́̂̈̌͆̍ ̴̘̞̖̳͕̳͑̀̆̑͜B̷̨͈̹̣̘̰͕͉̰̹͗̔ȩ̷̡̠̙̖̱̜̲̤͚̼͓̠͛̾̓͆̅̒͂̅̀̿͘͘ạ̶̢̨̮̩͇͚͓̀̄̐̈́̇̇͊̄̾̕͝͝s̵̡̛͚̭̞͌̋̐̑͑̓̒̋͊͊͌̈̎̚ṱ̴̫̤̩͂͒̅̋̓̓̿̅̈́̎
I̴̢̦̠͈̊̆͌͝͝ͅn̶̛͈̺̜͇̲̭͓͉̎̽̓̊̈̈́̾̔́̽̇͘v̷͈̘̞̬̼̲̍͊͊̋̓̕ą̷̰͙͙̖̖̤̬͙̃d̷̠̀́̓͆̽̽̽̓͘ȩ̵̤̱͓̫̻̫̯̪̎͌̇̆̄̓ͅr̴̗̹̪̬̘̪̝̰͆̒͆̋̃̃̏̈͑̅͝ş̴͎̭̻͔̘͈͉̟̻͈̯̩͓̔̐

161 / 330

Invariance: use case

`artist`	`year`	`album`	`name`
Iron Maiden	1982	The Number of the Beast	Children of the Damned
Iron Maiden	1982	The Number of the Beast	Invaders

forAll { (cmds: List[Command]) =>
  cmds.forall { cmd =>
    db.execute(cmd)
    notCorrupt(db)
  }
}

162 / 330

Invariance: use case

`artist`	`year`	`album`	`name`
Iron Maiden	1982	The Number of the Beast	Children of the Damned
Iron Maiden	1982	The Number of the Beast	Invaders

forAll { (cmds: List[Command]) =>
  cmds.forall { cmd =>
    db.execute(cmd)
    notCorrupt(db)
  }
}

163 / 330

Invariance: use case

`artist`	`year`	`album`	`name`
Iron Maiden	1982	The Number of the Beast	Children of the Damned
Iron Maiden	1982	The Number of the Beast	Invaders

forAll { (cmds: List[Command]) =>
  cmds.forall { cmd =>
    db.execute(cmd)
    notCorrupt(db)
  }
}

164 / 330

Invariance: use case

`artist`	`year`	`album`	`name`
İ̵̺͝ṛ̸̣͕̐͜o̴̢̎̿̓̌n̸̥͙̬̣̾́͛̾̌͘ͅ ̶̻͂M̷̗̣̠̠̎̀ȧ̸̛͔͇̐͘į̵̗̞̩̯̫̇̔̾̋̕͝d̵̟͓͆̅̇̋̐̚ͅe̷̯̪̙̼̩̩͐n̴̛̗̟͈͇̖̎̈̕	1̶͍͓̣̔̍́̒̒̅͘͜9̷̬̖̹̗̻͍̋͗̈́͠8̶͓̫̹̹̍2̸̨̲̝̏͌	T̸̜̟͕̩̘̃̇͂͠h̶̦̓͋̇͝e̴̢̝̱̦͎͗͂̅̓͜ͅ ̶̝̬͠͝Ǹ̵̛̘̺̯̃̈́̍ù̷̱̖m̷̗͙̱̥͇̹͑̎̂͊̈́́̌b̸̢̦̪̽̅̂̑e̴̤̫̝̎̈́̇r̵͇̥̃̃̈͋̎̋͗͜ ̴͇̖̼͙̥̲̹̄̀ò̷͉̠̒f̶̲͇̳͛̔̃̽ ̸͍̉̚͝t̶̼̗̬̞̣̂͗͠͠͝h̴̰͒̾̑͐́͝ě̷̠̤͎̳͍̿͆́̕͝ ̸̭̬̊̐̾̍B̷̨̑é̴̡̟͔͖̏̃̔̓͝͠ą̷͉̫͇̫̈́̍̄́̃s̵͍̀̅͂͂͑̌ẗ̷̡̪̙͉͔́͑͗̇̋̑	C̵͚̣̬̠̞̝̰̐̍̃h̷̥͛͂͋̚̚ï̴̤l̴͉̳̘̝͖̤͋͝d̴̢̲̫̯͙̺̮́̈́̕r̸̢̗̥͇̻͓͗̋̆́ȇ̸̲̉̆n̵̞̙͎̠̲͕̭̅̃̉̂ ̵͔̦͙̯̎͂̑̚͝õ̷̳̱͈̻̤̠̫̽̓̑̊͒f̸̦͋ͅ ̴̹̖̌ṫ̷̻̼͖͗̉̀̑h̴̹͎̰̬̟͊̍ͅe̸̤̱͛́͑ ̸̞̾͊͠D̵̢̜̬̳̍̑͛̄͆̀̅a̴͙̼̤̤͓͇̟̒̈́͋m̷̡̞̬̞̬͆̔̿n̴̨̢̖͉͓̥̂̄͘ë̴̬͚͎̭̭ͅd̸̢͕̼̗̠̂
Ì̷̧̺͓͕̘͓̯̗̬͈̤̗̝̰͜ȓ̵̨̜̟̱̘̟̬̙̯̱͍̩̭̈̆̅̍̌̿́̈́́̚͜͝ͅo̴̡͗͗͛̑̿̾͒͂̎̆n̶͖̳͖͉̗͕̱͔̱̖̫̽̎͜ͅ ̸̢̨̛͇͚͔̘̦̘̱̮̖̿͑̒̉̽̎́̑̚̚̕M̷̜͖̊͗̅̆̏̈́͛̍̂̉́̃̚̕͝ả̷̡̛̫͔̳̞̻͔͉̜̍̅̈́̆͛͒̋̑͠i̵̢̡͎̯̙͇͍̇͊̃̓̚͜ͅd̸̟̤̼̲̈́̎̑͜ͅȩ̸͓̝̘̙͉̼̺̖̳͓̩̽̕ͅn̴͈̈̃̿͝	1̴̛͈̣̲̹͗̾͂͆̒̓͛̋͝9̵̨̢̨̻̲̬͕̲͔͖̘̀̿͝ͅ8̷̩̹̣̦̙͖͚̞̮́̉̑̓̉͑̈̾́͊̀̽͛͘̚2̵̡̘̠̼͙̅̇͛̽͒̀̄̾́̋̎͐͛͝	T̷̢̟̙̥̬̰͎͚͈͕̬͆̋͊͛̂̓̈́͊̌̀͆̅̐͘͝ȟ̴̛̹̝͕͒͗è̵̛̩͇͎̤̪͍͎̳̺͔̖̝͖͊̏̀̽͝ ̵͕͇͌͌́͛̑̏͊̈̔̿̚̕͠͝N̴̤̥̖̠͍̻̤̣͈̩̞̾̍̉͐͋̇̈̈́̓̈́̚ͅu̵̲̹̙̦̞̎̏͗͋̌͐̈́͘͜͝͝m̶̡̙̖̳̖͇̻̰̼̝̖̎̈̈̄́̀̽̈́͘͜͝b̸̡̙͚̙̮̀̀̅́̔́̿̑͆̂͒̕è̷̫͕̗̰̌̈́̏͗͆͂r̶̡̢̨̛̛̙̖̲͔̥̲͕̈́̀͆̀̅̀̔̃̿̉̽͝͝ ̵̨͍͎͈̹̺̟̠̪̼̟̟͔̜̆̈́̈́o̶̡̧̯̠̜̤͊̒̀͆̒̀̋̅̽̕͜͝ͅf̸͓̥̻͚̘̹͈͉̱̆̅̊̿͑͛́͆̽̋͘͜͜ ̶̣̭͙̯͇̦͂̋͆͆̕ţ̴̡̨̭̭̺̣̲̤͎̩̮̮̥̿́̔́̂͂h̵͉̤͛͊͛͐͒͒ė̶̜̰͓̤̼̖̜̣͎̫̑̍̌́̂̈̌͆̍ ̴̘̞̖̳͕̳͑̀̆̑͜B̷̨͈̹̣̘̰͕͉̰̹͗̔ȩ̷̡̠̙̖̱̜̲̤͚̼͓̠͛̾̓͆̅̒͂̅̀̿͘͘ạ̶̢̨̮̩͇͚͓̀̄̐̈́̇̇͊̄̾̕͝͝s̵̡̛͚̭̞͌̋̐̑͑̓̒̋͊͊͌̈̎̚ṱ̴̫̤̩͂͒̅̋̓̓̿̅̈́̎	I̴̢̦̠͈̊̆͌͝͝ͅn̶̛͈̺̜͇̲̭͓͉̎̽̓̊̈̈́̾̔́̽̇͘v̷͈̘̞̬̼̲̍͊͊̋̓̕ą̷̰͙͙̖̖̤̬͙̃d̷̠̀́̓͆̽̽̽̓͘ȩ̵̤̱͓̫̻̫̯̪̎͌̇̆̄̓ͅr̴̗̹̪̬̘̪̝̰͆̒͆̋̃̃̏̈͑̅͝ş̴͎̭̻͔̘͈͉̟̻͈̯̩͓̔̐

forAll { (cmds: List[Command]) =>
  cmds.exists { cmd =>
    db.execute(cmd)
    notCorrupt(db)
  }
}

165 / 330

Invariance: use case

`artist`	`year`	`album`	`name`
İ̵̺͝ṛ̸̣͕̐͜o̴̢̎̿̓̌n̸̥͙̬̣̾́͛̾̌͘ͅ ̶̻͂M̷̗̣̠̠̎̀ȧ̸̛͔͇̐͘į̵̗̞̩̯̫̇̔̾̋̕͝d̵̟͓͆̅̇̋̐̚ͅe̷̯̪̙̼̩̩͐n̴̛̗̟͈͇̖̎̈̕	1̶͍͓̣̔̍́̒̒̅͘͜9̷̬̖̹̗̻͍̋͗̈́͠8̶͓̫̹̹̍2̸̨̲̝̏͌	T̸̜̟͕̩̘̃̇͂͠h̶̦̓͋̇͝e̴̢̝̱̦͎͗͂̅̓͜ͅ ̶̝̬͠͝Ǹ̵̛̘̺̯̃̈́̍ù̷̱̖m̷̗͙̱̥͇̹͑̎̂͊̈́́̌b̸̢̦̪̽̅̂̑e̴̤̫̝̎̈́̇r̵͇̥̃̃̈͋̎̋͗͜ ̴͇̖̼͙̥̲̹̄̀ò̷͉̠̒f̶̲͇̳͛̔̃̽ ̸͍̉̚͝t̶̼̗̬̞̣̂͗͠͠͝h̴̰͒̾̑͐́͝ě̷̠̤͎̳͍̿͆́̕͝ ̸̭̬̊̐̾̍B̷̨̑é̴̡̟͔͖̏̃̔̓͝͠ą̷͉̫͇̫̈́̍̄́̃s̵͍̀̅͂͂͑̌ẗ̷̡̪̙͉͔́͑͗̇̋̑	C̵͚̣̬̠̞̝̰̐̍̃h̷̥͛͂͋̚̚ï̴̤l̴͉̳̘̝͖̤͋͝d̴̢̲̫̯͙̺̮́̈́̕r̸̢̗̥͇̻͓͗̋̆́ȇ̸̲̉̆n̵̞̙͎̠̲͕̭̅̃̉̂ ̵͔̦͙̯̎͂̑̚͝õ̷̳̱͈̻̤̠̫̽̓̑̊͒f̸̦͋ͅ ̴̹̖̌ṫ̷̻̼͖͗̉̀̑h̴̹͎̰̬̟͊̍ͅe̸̤̱͛́͑ ̸̞̾͊͠D̵̢̜̬̳̍̑͛̄͆̀̅a̴͙̼̤̤͓͇̟̒̈́͋m̷̡̞̬̞̬͆̔̿n̴̨̢̖͉͓̥̂̄͘ë̴̬͚͎̭̭ͅd̸̢͕̼̗̠̂
Ì̷̧̺͓͕̘͓̯̗̬͈̤̗̝̰͜ȓ̵̨̜̟̱̘̟̬̙̯̱͍̩̭̈̆̅̍̌̿́̈́́̚͜͝ͅo̴̡͗͗͛̑̿̾͒͂̎̆n̶͖̳͖͉̗͕̱͔̱̖̫̽̎͜ͅ ̸̢̨̛͇͚͔̘̦̘̱̮̖̿͑̒̉̽̎́̑̚̚̕M̷̜͖̊͗̅̆̏̈́͛̍̂̉́̃̚̕͝ả̷̡̛̫͔̳̞̻͔͉̜̍̅̈́̆͛͒̋̑͠i̵̢̡͎̯̙͇͍̇͊̃̓̚͜ͅd̸̟̤̼̲̈́̎̑͜ͅȩ̸͓̝̘̙͉̼̺̖̳͓̩̽̕ͅn̴͈̈̃̿͝	1̴̛͈̣̲̹͗̾͂͆̒̓͛̋͝9̵̨̢̨̻̲̬͕̲͔͖̘̀̿͝ͅ8̷̩̹̣̦̙͖͚̞̮́̉̑̓̉͑̈̾́͊̀̽͛͘̚2̵̡̘̠̼͙̅̇͛̽͒̀̄̾́̋̎͐͛͝	T̷̢̟̙̥̬̰͎͚͈͕̬͆̋͊͛̂̓̈́͊̌̀͆̅̐͘͝ȟ̴̛̹̝͕͒͗è̵̛̩͇͎̤̪͍͎̳̺͔̖̝͖͊̏̀̽͝ ̵͕͇͌͌́͛̑̏͊̈̔̿̚̕͠͝N̴̤̥̖̠͍̻̤̣͈̩̞̾̍̉͐͋̇̈̈́̓̈́̚ͅu̵̲̹̙̦̞̎̏͗͋̌͐̈́͘͜͝͝m̶̡̙̖̳̖͇̻̰̼̝̖̎̈̈̄́̀̽̈́͘͜͝b̸̡̙͚̙̮̀̀̅́̔́̿̑͆̂͒̕è̷̫͕̗̰̌̈́̏͗͆͂r̶̡̢̨̛̛̙̖̲͔̥̲͕̈́̀͆̀̅̀̔̃̿̉̽͝͝ ̵̨͍͎͈̹̺̟̠̪̼̟̟͔̜̆̈́̈́o̶̡̧̯̠̜̤͊̒̀͆̒̀̋̅̽̕͜͝ͅf̸͓̥̻͚̘̹͈͉̱̆̅̊̿͑͛́͆̽̋͘͜͜ ̶̣̭͙̯͇̦͂̋͆͆̕ţ̴̡̨̭̭̺̣̲̤͎̩̮̮̥̿́̔́̂͂h̵͉̤͛͊͛͐͒͒ė̶̜̰͓̤̼̖̜̣͎̫̑̍̌́̂̈̌͆̍ ̴̘̞̖̳͕̳͑̀̆̑͜B̷̨͈̹̣̘̰͕͉̰̹͗̔ȩ̷̡̠̙̖̱̜̲̤͚̼͓̠͛̾̓͆̅̒͂̅̀̿͘͘ạ̶̢̨̮̩͇͚͓̀̄̐̈́̇̇͊̄̾̕͝͝s̵̡̛͚̭̞͌̋̐̑͑̓̒̋͊͊͌̈̎̚ṱ̴̫̤̩͂͒̅̋̓̓̿̅̈́̎	I̴̢̦̠͈̊̆͌͝͝ͅn̶̛͈̺̜͇̲̭͓͉̎̽̓̊̈̈́̾̔́̽̇͘v̷͈̘̞̬̼̲̍͊͊̋̓̕ą̷̰͙͙̖̖̤̬͙̃d̷̠̀́̓͆̽̽̽̓͘ȩ̵̤̱͓̫̻̫̯̪̎͌̇̆̄̓ͅr̴̗̹̪̬̘̪̝̰͆̒͆̋̃̃̏̈͑̅͝ş̴͎̭̻͔̘͈͉̟̻͈̯̩͓̔̐

forAll { (cmds: List[Command]) =>
  cmds.exists { cmd =>
    db.execute(cmd)
    notCorrupt(db)
  }
}

Fail

166 / 330

Invariance: fizzBuzz

val propSafeFizzBuzz = forAll { (i: Int) =>
  Try(fizzBuzz(i)).isSuccess
}

167 / 330

Invariance: fizzBuzz

val propSafeFizzBuzz = forAll { (i: Int) =>
  Try(fizzBuzz(i)).isSuccess
}

168 / 330

Invariance: fizzBuzz

val propSafeFizzBuzz = forAll { (i: Int) =>
  Try(fizzBuzz(i)).isSuccess
}

169 / 330

Invariance: fizzBuzz

propSafeFizzBuzz.check()

170 / 330

Invariance: fizzBuzz

propSafeFizzBuzz.check()
// + OK, passed 100 tests.

171 / 330

Metamorphic relation

Metamorphic Relation

172 / 330

Metamorphic relation

Metamorphic Relation

173 / 330

Metamorphic relation

Metamorphic Relation

174 / 330

Metamorphic relation

Metamorphic Relation

175 / 330

Metamorphic relation

Metamorphic Relation

176 / 330

Metamorphic relation

Metamorphic Relation

177 / 330

Metamorphic relation

Metamorphic Relation

forAll { (i: Int) =>
  mult3(i) == mult3(-i)
}

178 / 330

Metamorphic relation

Metamorphic Relation

forAll { (i: Int) =>
  mult3(i) == mult3(-i)
}

179 / 330

Metamorphic relation

Metamorphic Relation

forAll { (i: Int) =>
  mult3(i) == mult3(-i)
}

180 / 330

Metamorphic relation

Metamorphic Relation

forAll { (i: Int) =>
  mult3(i) == mult3(-i)
}

181 / 330

Metamorphic relation

Metamorphic Relation

forAll { (i: Int) =>
  mult3(i) == mult3(-i)
}

182 / 330

Metamorphic relation: use case

Metamorphic Relation

183 / 330

Metamorphic relation: use case

Metamorphic Relation

184 / 330

Metamorphic relation: use case

Metamorphic Relation

185 / 330

Metamorphic relation: use case

Metamorphic Relation

186 / 330

Metamorphic relation: use case

Metamorphic Relation

187 / 330

Metamorphic relation: use case

Metamorphic Relation

188 / 330

Metamorphic relation: use case

Metamorphic Relation

Fail

189 / 330

Metamorphic relation: use case

Metamorphic testing of RESTful web apis:

3 new unique Spotify bugs.
8 new unique YouTube bugs.

190 / 330

Metamorphic relation: fizzBuzz

val propStableFizz = forAll { (i: Int) =>
  fizzBuzz(i).contains("Fizz") ==> fizzBuzz(-i).contains("Fizz")
}

191 / 330

Metamorphic relation: fizzBuzz

val propStableFizz = forAll { (i: Int) =>
  fizzBuzz(i).contains("Fizz") ==> fizzBuzz(-i).contains("Fizz")
}

192 / 330

Metamorphic relation: fizzBuzz

val propStableFizz = forAll { (i: Int) =>
  fizzBuzz(i).contains("Fizz") ==> fizzBuzz(-i).contains("Fizz")
}

193 / 330

Metamorphic relation: fizzBuzz

val propStableFizz = forAll { (i: Int) =>
  fizzBuzz(i).contains("Fizz") ==> fizzBuzz(-i).contains("Fizz")
}

194 / 330

Metamorphic relation: fizzBuzz

val propStableFizz = forAll { (i: Int) =>
  fizzBuzz(i).contains("Fizz") ==> fizzBuzz(-i).contains("Fizz")
}

195 / 330

Metamorphic relation: fizzBuzz

val propStableFizz = forAll { (i: Int) =>
  fizzBuzz(i).contains("Fizz") ==> fizzBuzz(-i).contains("Fizz")
}

196 / 330

Metamorphic relation: fizzBuzz

val propStableFizz = forAll { (i: Int) =>
  fizzBuzz(i).contains("Fizz") ==> fizzBuzz(-i).contains("Fizz")
}

197 / 330

Metamorphic relation: fizzBuzz

propStableFizz.check()

198 / 330

Metamorphic relation: fizzBuzz

propStableFizz.check()
// ! Falsified after 1 passed tests.
// > ARG_0: -1
// > ARG_0_ORIGINAL: -67

199 / 330

Fixing fizzBuzzdef mult3(i: Int) = !(i % 3 > 0)
def mult5(i: Int) = !(i % 5 > 0)
def fizzBuzz(i: Int) =
  if mult3(i) then
    if mult5(i) then    "FizzBuzz"
    else                "Fizz"
  else if mult5(i) then "Buzz"
  else                  i.toString

200 / 330

Fixing fizzBuzzdef mult3(i: Int) = i % 3 == 0
def mult5(i: Int) = i % 5 == 0
def fizzBuzz(i: Int) =
  if mult3(i) then
    if mult5(i) then    "FizzBuzz"
    else                "Fizz"
  else if mult5(i) then "Buzz"
  else                  i.toString

201 / 330

Fixing `fizzBuzz`

propStableFizz.check()

202 / 330

Fixing `fizzBuzz`

propStableFizz.check()
// + OK, passed 100 tests.

203 / 330

Challenging properties

class FizzBuzzProps extends Properties("FizzBuzz"):
  property("validity") = forAll { (i: Int) =>
    fizzBuzz(i) in Set(i.toString, "Fizz", "Buzz", "FizzBuzz")
  }
  property("involutivity") = forAll { (i: Int) =>
    !(mult3(i) || mult5(i)) ==> (fizzBuzz(i).toInt == i)
  }
  property("metamorphic") = forAll { (i: Int) =>
    fizzBuzz(i).contains("Fizz") ==> fizzBuzz(-i).contains("Fizz")
  }
  property("invariant") = forAll { (i: Int) =>
    Try(fizzBuzz(i)).isSuccess
  }

204 / 330

Challenging properties

class FizzBuzzProps extends Properties("FizzBuzz"):
  property("validity") = forAll { (i: Int) =>
    fizzBuzz(i) in Set(i.toString, "Fizz", "Buzz", "FizzBuzz")
  }
  property("involutivity") = forAll { (i: Int) =>
    !(mult3(i) || mult5(i)) ==> (fizzBuzz(i).toInt == i)
  }
  property("metamorphic") = forAll { (i: Int) =>
    fizzBuzz(i).contains("Fizz") ==> fizzBuzz(-i).contains("Fizz")
  }
  property("invariant") = forAll { (i: Int) =>
    Try(fizzBuzz(i)).isSuccess
  }

205 / 330

Challenging properties

class FizzBuzzProps extends Properties("FizzBuzz"):
  property("validity") = forAll { (i: Int) =>
    fizzBuzz(i) in Set(i.toString, "Fizz", "Buzz", "FizzBuzz")
  }
  property("involutivity") = forAll { (i: Int) =>
    !(mult3(i) || mult5(i)) ==> (fizzBuzz(i).toInt == i)
  }
  property("metamorphic") = forAll { (i: Int) =>
    fizzBuzz(i).contains("Fizz") ==> fizzBuzz(-i).contains("Fizz")
  }
  property("invariant") = forAll { (i: Int) =>
    Try(fizzBuzz(i)).isSuccess
  }

206 / 330

Challenging properties

class FizzBuzzProps extends Properties("FizzBuzz"):
  property("validity") = forAll { (i: Int) =>
    fizzBuzz(i) in Set(i.toString, "Fizz", "Buzz", "FizzBuzz")
  }
  property("involutivity") = forAll { (i: Int) =>
    !(mult3(i) || mult5(i)) ==> (fizzBuzz(i).toInt == i)
  }
  property("metamorphic") = forAll { (i: Int) =>
    fizzBuzz(i).contains("Fizz") ==> fizzBuzz(-i).contains("Fizz")
  }
  property("invariant") = forAll { (i: Int) =>
    Try(fizzBuzz(i)).isSuccess
  }

207 / 330

Challenging properties

class FizzBuzzProps extends Properties("FizzBuzz"):
  property("validity") = forAll { (i: Int) =>
    fizzBuzz(i) in Set(i.toString, "Fizz", "Buzz", "FizzBuzz")
  }
  property("involutivity") = forAll { (i: Int) =>
    !(mult3(i) || mult5(i)) ==> (fizzBuzz(i).toInt == i)
  }
  property("metamorphic") = forAll { (i: Int) =>
    fizzBuzz(i).contains("Fizz") ==> fizzBuzz(-i).contains("Fizz")
  }
  property("invariant") = forAll { (i: Int) =>
    Try(fizzBuzz(i)).isSuccess
  }

208 / 330

Challenging properties

new FizzBuzzProps().check()

209 / 330

Challenging properties

new FizzBuzzProps().check()
// + FizzBuzz.validity: OK, passed 100 tests.
// + FizzBuzz.involutivity: OK, passed 100 tests.
// + FizzBuzz.metamorphic: OK, passed 100 tests.
// + FizzBuzz.invariant: OK, passed 100 tests.

210 / 330

Challenging propertiesdef fizzBuzz(i: Int) =
  if mult3(i) then
    if mult5(i) then    "FizzBuzz"
    else                "Fizz"
  else if mult5(i) then "Buzz"
  else                  i.toString

211 / 330

Challenging propertiesdef fizzBuzz(i: Int) =
  if mult3(i) then
    if mult5(i) then    "FizzBuzz"
    else                "Buzz"
  else if mult5(i) then "Fizz"
  else                  i.toString

212 / 330

Challenging properties

new FizzBuzzProps().check()

213 / 330

Challenging properties

new FizzBuzzProps().check()
// + FizzBuzz.validity: OK, passed 100 tests.
// + FizzBuzz.involutivity: OK, passed 100 tests.
// + FizzBuzz.metamorphic: OK, passed 100 tests.
// + FizzBuzz.invariant: OK, passed 100 tests.

214 / 330

Challenging properties

new FizzBuzzProps().check()
// + FizzBuzz.validity: OK, passed 100 tests.
// + FizzBuzz.involutivity: OK, passed 100 tests.
// + FizzBuzz.metamorphic: OK, passed 100 tests.
// + FizzBuzz.invariant: OK, passed 100 tests.

Fail

215 / 330

Challenging properties

val propFizz = forAll { (i: Int) =>
  fizzBuzz(i * 3).contains("Fizz")
}

216 / 330

Challenging properties

val propFizz = forAll { (i: Int) =>
  fizzBuzz(i * 3).contains("Fizz")
}

217 / 330

Challenging properties

val propFizz = forAll { (i: Int) =>
  fizzBuzz(i * 3).contains("Fizz")
}

218 / 330

Challenging properties

val propFizz = forAll { (i: Int) =>
  fizzBuzz(i * 3).contains("Fizz")
}

219 / 330

Challenging properties

val propFizz = forAll { (i: Int) =>
  fizzBuzz(i * 3).contains("Fizz")
}

220 / 330

Challenging properties

propFizz.check()

221 / 330

Challenging properties

propFizz.check()
// ! Falsified after 0 passed tests.
// > ARG_0: 1
// > ARG_0_ORIGINAL: 34

222 / 330

Key takeaways

Property-based testing is:

223 / 330

Key takeaways

Property-based testing is:

very good at the test case generation problem.

224 / 330

Key takeaways

Property-based testing is:

very good at the test case generation problem.
good at the oracle problem.

225 / 330

Key takeaways

Property-based testing is:

very good at the test case generation problem.
good at the oracle problem.
hard to get the hang of.

226 / 330

Generative Example-Based Tests227 / 330

Example-Based Tests

class FizzBuzzSuite extends AnyFunSuite:
  test("n multiple of 3 outputs Fizz") {
    assert(fizzBuzz(3) == "Fizz")
  }
  test("n multiple of 5 outputs Buzz") {
    assert(fizzBuzz(5) == "Buzz")
  }
  test("n multiple of neither 3 or 5 outputs n") {
    assert(fizzBuzz(7) == "7")
  }
  test("n multiple of 3 and 5 outputs FizzBuzz") {
    assert(fizzBuzz(15) == "FizzBuzz")
  }

228 / 330

Example-Based Tests

class FizzBuzzSuite extends AnyFunSuite:
  test("n multiple of 3 outputs Fizz") {
    assert(fizzBuzz(3) == "Fizz")
  }
  test("n multiple of 5 outputs Buzz") {
    assert(fizzBuzz(5) == "Buzz")
  }
  test("n multiple of neither 3 or 5 outputs n") {
    assert(fizzBuzz(7) == "7")
  }
  test("n multiple of 3 and 5 outputs FizzBuzz") {
    assert(fizzBuzz(15) == "FizzBuzz")
  }

229 / 330

Example-Based Tests

class FizzBuzzSuite extends AnyFunSuite:
  test("n multiple of 3 outputs Fizz") {
    assert(fizzBuzz(3) == "Fizz")
  }
  test("n multiple of 5 outputs Buzz") {
    assert(fizzBuzz(5) == "Buzz")
  }
  test("n multiple of neither 3 or 5 outputs n") {
    assert(fizzBuzz(7) == "7")
  }
  test("n multiple of 3 and 5 outputs FizzBuzz") {
    assert(fizzBuzz(15) == "FizzBuzz")
  }

230 / 330

Naive property

val propFizzBuzz3 =
  fizzBuzz(3) == "Fizz"

231 / 330

Naive property

val propFizzBuzz3 =
  fizzBuzz(3) == "Fizz"

232 / 330

Naive propertyval propFizzBuzz3 =
  forAll(const(3)) { i =>
    fizzBuzz(3) == "Fizz"
  }

233 / 330

Naive property

val propFizzBuzz3 =
  forAll(const(3)) { i =>
    fizzBuzz(3) == "Fizz"
  }

234 / 330

Naive property

val propFizzBuzz3 =
  forAll(const(3)) { i =>
    fizzBuzz(3) == "Fizz"
  }

235 / 330

Naive propertyval propFizzBuzz3 =
  forAll(const(3)) { i =>
    fizzBuzz(3) == "Fizz"
  }

236 / 330

Naive propertyval propFizzBuzz3 =
  forAll(const(3)) { i =>
    fizzBuzz(i) == "Fizz"
  }

237 / 330

Naive property

val propFizzBuzz3 =
  forAll(const(3)) { i =>
    fizzBuzz(i) == "Fizz"
  }

238 / 330

Multiples of 3

val genMult3: Gen[Int] =
  arbitrary[Int]
    .map(_ * 3)
    .filter(_ % 5 != 0)

239 / 330

Multiples of 3

val genMult3: Gen[Int] =
  arbitrary[Int]
    .map(_ * 3)
    .filter(_ % 5 != 0)

240 / 330

Multiples of 3

val genMult3: Gen[Int] =
  arbitrary[Int]
    .map(_ * 3)
    .filter(_ % 5 != 0)

241 / 330

Multiples of 3val propFizzBuzz3 =
  forAll(const(3)) { i =>
    fizzBuzz(i) == "Fizz"
  }

242 / 330

Multiples of 3val propFizzBuzz3 =
  forAll(genMult3) { i =>
    fizzBuzz(i) == "Fizz"
  }

243 / 330

Generic property

val propFizzBuzz3 =
  forAll(genMult3) { i =>
    fizzBuzz(i) == "Fizz"
  }

244 / 330

Generic property

val propFizzBuzz3 =
  forAll(genMult3) { i =>
    fizzBuzz(i) == "Fizz"
  }

245 / 330

Generic property​val propFizzBuzz3 =
  forAll(genMult3) { i =>
    fizzBuzz(i) == "Fizz"
  }

246 / 330

Generic property​def propFizzBuzz(gen: Gen[Int]) =
  forAll(genMult3) { i =>
    fizzBuzz(i) == "Fizz"
  }

247 / 330

Generic propertydef propFizzBuzz(gen: Gen[Int]) =
  forAll(genMult3) { i =>
    fizzBuzz(i) == "Fizz"
  }

248 / 330

Generic propertydef propFizzBuzz(gen: Gen[Int]) =
  forAll(gen) { i =>
    fizzBuzz(i) == "Fizz"
  }

249 / 330

Generic propertydef propFizzBuzz(gen: Gen[Int], expected: String) =
  forAll(gen) { i =>
    fizzBuzz(i) == "Fizz"
  }

250 / 330

Generic propertydef propFizzBuzz(gen: Gen[Int], expected: String) =
  forAll(gen) { i =>
    fizzBuzz(i) == "Fizz"
  }

251 / 330

Generic propertydef propFizzBuzz(gen: Gen[Int], expected: String) =
  forAll(gen) { i =>
    fizzBuzz(i) == expected
  }

252 / 330

Generic propertyval propFizzBuzz3 =
​  forAll(genMult3) { i =>
​    fizzBuzz(i) == "Fizz"
​  }

253 / 330

Generic propertyval propFizzBuzz3 =
​  propFizzBuzz(
​    gen      = genMult3,
​    expected = "Fizz"
​  )

254 / 330

Generic property

val propFizzBuzz3 =
  propFizzBuzz(
    gen      = genMult3,
    expected = "Fizz"
  )

255 / 330

Generic property

val propFizzBuzz3 =
  propFizzBuzz(
    gen      = genMult3,
    expected = "Fizz"
  )

256 / 330

Multiples of 5

class FizzBuzzSuite extends AnyFunSuite:
  test("n multiple of 3 outputs Fizz") {
    assert(fizzBuzz(3) == "Fizz")
  }
  test("n multiple of 5 outputs Buzz") {
    assert(fizzBuzz(5) == "Buzz")
  }
  test("n multiple of neither 3 or 5 outputs n") {
    assert(fizzBuzz(7) == "7")
  }
  test("n multiple of 3 and 5 outputs FizzBuzz") {
    assert(fizzBuzz(15) == "FizzBuzz")
  }

257 / 330

Multiples of 5

class FizzBuzzSuite extends AnyFunSuite:
  test("n multiple of 3 outputs Fizz") {
    assert(fizzBuzz(3) == "Fizz")
  }
  test("n multiple of 5 outputs Buzz") {
    assert(fizzBuzz(5) == "Buzz")
  }
  test("n multiple of neither 3 or 5 outputs n") {
    assert(fizzBuzz(7) == "7")
  }
  test("n multiple of 3 and 5 outputs FizzBuzz") {
    assert(fizzBuzz(15) == "FizzBuzz")
  }

258 / 330

Multiples of 5

class FizzBuzzSuite extends AnyFunSuite:
  test("n multiple of 3 outputs Fizz") {
    assert(fizzBuzz(3) == "Fizz")
  }
  test("n multiple of 5 outputs Buzz") {
    assert(fizzBuzz(5) == "Buzz")
  }
  test("n multiple of neither 3 or 5 outputs n") {
    assert(fizzBuzz(7) == "7")
  }
  test("n multiple of 3 and 5 outputs FizzBuzz") {
    assert(fizzBuzz(15) == "FizzBuzz")
  }

259 / 330

Multiples of 5

val genMult5: Gen[Int] =
  arbitrary[Int]
    .map(_ * 5)
    .filter(_ % 3 != 0)

260 / 330

Multiples of 5

val genMult5: Gen[Int] =
  arbitrary[Int]
    .map(_ * 5)
    .filter(_ % 3 != 0)

261 / 330

Multiples of 5

val genMult5: Gen[Int] =
  arbitrary[Int]
    .map(_ * 5)
    .filter(_ % 3 != 0)

262 / 330

Multiples of 5

val propFizzBuzz5 =
  propFizzBuzz(
    gen      = genMult5,
    expected = "Buzz"
  )

263 / 330

Multiples of 5

val propFizzBuzz5 =
  propFizzBuzz(
    gen      = genMult5,
    expected = "Buzz"
  )

264 / 330

Multiples of neither 3 nor 5

class FizzBuzzSuite extends AnyFunSuite:
  test("n multiple of 3 outputs Fizz") {
    assert(fizzBuzz(3) == "Fizz")
  }
  test("n multiple of 5 outputs Buzz") {
    assert(fizzBuzz(5) == "Buzz")
  }
  test("n multiple of neither 3 or 5 outputs n") {
    assert(fizzBuzz(7) == "7")
  }
  test("n multiple of 3 and 5 outputs FizzBuzz") {
    assert(fizzBuzz(15) == "FizzBuzz")
  }

265 / 330

Multiples of neither 3 nor 5

class FizzBuzzSuite extends AnyFunSuite:
  test("n multiple of 3 outputs Fizz") {
    assert(fizzBuzz(3) == "Fizz")
  }
  test("n multiple of 5 outputs Buzz") {
    assert(fizzBuzz(5) == "Buzz")
  }
  test("n multiple of neither 3 or 5 outputs n") {
    assert(fizzBuzz(7) == "7")
  }
  test("n multiple of 3 and 5 outputs FizzBuzz") {
    assert(fizzBuzz(15) == "FizzBuzz")
  }

266 / 330

Multiples of neither 3 nor 5

class FizzBuzzSuite extends AnyFunSuite:
  test("n multiple of 3 outputs Fizz") {
    assert(fizzBuzz(3) == "Fizz")
  }
  test("n multiple of 5 outputs Buzz") {
    assert(fizzBuzz(5) == "Buzz")
  }
  test("n multiple of neither 3 or 5 outputs n") {
    assert(fizzBuzz(7) == "7")
  }
  test("n multiple of 3 and 5 outputs FizzBuzz") {
    assert(fizzBuzz(15) == "FizzBuzz")
  }

267 / 330

Multiples of neither 3 nor 5

val genMultNone: Gen[Int] =
  arbitrary[Int]
    .filter(_ % 5 != 0)
    .filter(_ % 3 != 0)

268 / 330

Multiples of neither 3 nor 5

val genMultNone: Gen[Int] =
  arbitrary[Int]
    .filter(_ % 5 != 0)
    .filter(_ % 3 != 0)

269 / 330

Multiples of neither 3 nor 5

val genMultNone: Gen[Int] =
  arbitrary[Int]
    .filter(_ % 5 != 0)
    .filter(_ % 3 != 0)

270 / 330

Multiples of neither 3 nor 5

val propFizzBuzzNone =
  propFizzBuzz(
    gen      = genMultNone,
    expected = ???
  )

271 / 330

Multiples of neither 3 nor 5

val propFizzBuzzNone =
  propFizzBuzz(
    gen      = genMultNone,
    expected = ???
  )

272 / 330

Multiples of neither 3 nor 5def propFizzBuzz(gen: Gen[Int], expected: String) =
  forAll(gen) { i =>
    fizzBuzz(i) == expected
  }

273 / 330

Multiples of neither 3 nor 5def propFizzBuzz(gen: Gen[Int], oracle: Int => String) =
  forAll(gen) { i =>
    fizzBuzz(i) == oracle(i)
  }

274 / 330

Multiples of neither 3 nor 5val propFizzBuzzNone =
  propFizzBuzz(
    gen      = genMultNone,
    expected = ???
  )

275 / 330

Multiples of neither 3 nor 5val propFizzBuzzNone =
  propFizzBuzz(
    gen      = genMultNone,
    oracle   = _.toString
  )

276 / 330

Multiples of 3 and 5

class FizzBuzzSuite extends AnyFunSuite:
  test("n multiple of 3 outputs Fizz") {
    assert(fizzBuzz(3) == "Fizz")
  }
  test("n multiple of 5 outputs Buzz") {
    assert(fizzBuzz(5) == "Buzz")
  }
  test("n multiple of neither 3 or 5 outputs n") {
    assert(fizzBuzz(7) == "7")
  }
  test("n multiple of 3 and 5 outputs FizzBuzz") {
    assert(fizzBuzz(15) == "FizzBuzz")
  }

277 / 330

Multiples of 3 and 5

class FizzBuzzSuite extends AnyFunSuite:
  test("n multiple of 3 outputs Fizz") {
    assert(fizzBuzz(3) == "Fizz")
  }
  test("n multiple of 5 outputs Buzz") {
    assert(fizzBuzz(5) == "Buzz")
  }
  test("n multiple of neither 3 or 5 outputs n") {
    assert(fizzBuzz(7) == "7")
  }
  test("n multiple of 3 and 5 outputs FizzBuzz") {
    assert(fizzBuzz(15) == "FizzBuzz")
  }

278 / 330

Multiples of 3 and 5

class FizzBuzzSuite extends AnyFunSuite:
  test("n multiple of 3 outputs Fizz") {
    assert(fizzBuzz(3) == "Fizz")
  }
  test("n multiple of 5 outputs Buzz") {
    assert(fizzBuzz(5) == "Buzz")
  }
  test("n multiple of neither 3 or 5 outputs n") {
    assert(fizzBuzz(7) == "7")
  }
  test("n multiple of 3 and 5 outputs FizzBuzz") {
    assert(fizzBuzz(15) == "FizzBuzz")
  }

279 / 330

Multiples of 3 and 5

val genMult15: Gen[Int] =
  arbitrary[Int].
    map(_ * 3 * 5)

280 / 330

Multiples of 3 and 5

val genMult15: Gen[Int] =
  arbitrary[Int].
    map(_ * 3 * 5)

281 / 330

Multiples of 3 and 5

val propFizzBuzz15 =
  propFizzBuzz(
    gen    = genMult15,
    oracle = _ => "FizzBuzz"
  )

282 / 330

Multiples of 3 and 5

val propFizzBuzz15 =
  propFizzBuzz(
    gen    = genMult15,
    oracle = _ => "FizzBuzz"
  )

283 / 330

Key takeaways284 / 330

Key takeawaysyou don't need to learn a new way to write tests (right away).
285 / 330

Key takeaways

you don't need to learn a new way to write tests (right away).
you can adapt existing tests with little work.

286 / 330

Key takeaways

you don't need to learn a new way to write tests (right away).
you can adapt existing tests with little work.
you will acquire the skills you need as you do so.

287 / 330

In closing288 / 330

If you only remember 1 slide...289 / 330

If you only remember 1 slide...

Property-based testing:

finds more flaws than example-based testing.

290 / 330

If you only remember 1 slide...

Property-based testing:

finds more flaws than example-based testing.
doesn't need to be harder than example-based testing.

291 / 330

If you only remember 1 slide...

Property-based testing:

finds more flaws than example-based testing.
doesn't need to be harder than example-based testing.
importantly, is a lot of fun!

292 / 330

Questions?

Nicolas Rinaudo • @NicolasRinaudo

293 / 330

Metamorphic testing294 / 330

Profile moderation

Self

295 / 330

Profile moderation

Self

296 / 330

Profile moderation

Self

297 / 330

Profile moderation

Self

298 / 330

Profile moderation

Self

299 / 330

Profile moderation

Self

300 / 330

Profile moderation

Self

Fail

301 / 330

Metamorphic testing

Metamorphic Testing

302 / 330

Metamorphic testing

Metamorphic Testing

303 / 330

Metamorphic testing

Metamorphic Testing

304 / 330

Metamorphic testing

Metamorphic Testing

305 / 330

Metamorphic testing

Metamorphic Testing

306 / 330

Compiler validation

def fizzBuzz(i: Int) =
  if mult3(i) then
    if mult5(i) then    "FizzBuzz"
    else                "Fizz"
  else if mult5(i) then "Buzz"
  else                  i.toString

307 / 330

Compiler validation

println(fizzBuzz(5))

308 / 330

Compiler validation

println(fizzBuzz(5))

309 / 330

Compiler validation

println(fizzBuzz(5))
// Buzz

310 / 330

Compiler validation

println(fizzBuzz(5))
// Buzz

311 / 330

Compiler validationdef fizzBuzz(i: Int) =
  if mult3(i) then
    if mult5(i) then    "FizzBuzz"
    else                "Fizz"
  else if mult5(i) then "Buzz"
  else                  i.toString

312 / 330

Compiler validationdef fizzBuzz(i: Int) =
  if mult3(i) then
    if mult5(i) then    "FizzBuzz"
    else                "Fizz"
  else if mult5(i) then "Buzz"
  else                  i.toString

313 / 330

Compiler validationdef fizzBuzz(i: Int) =
  if mult3(i) then
    if mult5(i) then    "FizzBuzz"
    else                "Fizz"
  else if mult5(i) then "Buzz"
  else                  i.toString

314 / 330

Compiler validationdef fizzBuzz(i: Int) =
  if mult3(i) then      "Fizz"
  else if mult5(i) then "Buzz"
  else                  i.toString

315 / 330

Compiler validationdef fizzBuzz(i: Int) =
  if mult3(i) then      "Fizz"
  else if mult5(i) then "Buzz"
  else                  i.toString

316 / 330

Compiler validationdef fizzBuzz(i: Int) =
  if mult3(i) then      "Zzif"
  else if mult5(i) then "Buzz"
  else                  i.toString

317 / 330

Compiler validation

println(fizzBuzz(5))

318 / 330

Compiler validation

println(fizzBuzz(5))

319 / 330

Compiler validation

println(fizzBuzz(5))
// Buzz

320 / 330

Compiler validation

println(fizzBuzz(5))
// Buzz

321 / 330

Compiler validation

println(fizzBuzz(5))
// Zzif

322 / 330

Compiler validation

println(fizzBuzz(5))
// Zzif

Fail

323 / 330

Compiler validation

Compiler Validation via Equivalence Modulo Inputs:

79 new unique gcc bugs.
68 new unique LLVM bugs.

324 / 330

Compiler validation

Compiler Validation via Equivalence Modulo Inputs:

79 new unique gcc bugs.
68 new unique LLVM bugs.
42 new unique scalac bugs.

325 / 330

Key takeaways

Metamorphic Testing:

326 / 330

Key takeaways

Metamorphic Testing:

helps generating test cases when it's too expensive.

327 / 330

Key takeaways

Metamorphic Testing:

helps generating test cases when it's too expensive.
helps with the oracle problem.

328 / 330

Key takeaways

Metamorphic Testing:

helps generating test cases when it's too expensive.
helps with the oracle problem.
a good word to use when you want to sound smart.

329 / 330

Questions?

Nicolas Rinaudo • @NicolasRinaudo

330 / 330

Help

Keyboard shortcuts

↑, ←, Pg Up, k

Go to previous slide

↓, →, Pg Dn, Space, j

Go to next slide

Home

Go to first slide

End

Go to last slide

Number + Return

Go to specific slide

b / m / f

Toggle blackout / mirrored / fullscreen mode

Clone slideshow

Toggle presenter mode

Restart the presentation timer

?, h

Toggle this help