Topic: TDD – Thwarting Database Defects It’s bad enough to spend hours finding and fixing database defects, explaining to the rest of the team what went wrong and trying to clean up the mess. It’s even worse when a defect [...]]]> When: Saturday, November 21st 18:00 – 21:00 GMT

Presenters: Sebastian Meine and Dennis Lloyd

Topic: TDD – Thwarting Database Defects
It’s bad enough to spend hours finding and fixing database defects, explaining to the rest of the team what went wrong and trying to clean up the mess. It’s even worse when a defect causes the end-user to make a bad decision. Database defects are far too costly and most testing practices do not adequately detect or prevent them. This presentation introduces tSQLt, a framework for automated database unit testing. You’ll learn techniques to write SQL code that is resilient to defects and is easier to change and maintain.

Location:
Red Gate Software Ltd.
Newnham House
Cambridge Business Park
CB4 0WZ Cambridge
United Kingdom

More Details and Registration: Cambridgeshire SQL Server User Group

Resources:

tSQLt Website: http://tSQLt.org

tSQLt Mailing List: tsqlt@googlegroups.com

sqlity.net Webstie: http://sqlity.net

Curiously Correct Website: http://curiouslycorrect.com

 
Download course content here: Thwarting Database Defects Class Distribution – Includes slides and examples
 

Topic: TDD – Thwarting Database Defects It’s bad enough to spend hours finding and fixing database defects, explaining to the rest of the team what went wrong and trying to clean up the mess. It’s even worse when a defect causes [...]]]> When: Saturday, November 5th 12:00pm – 1:00pm

Presenters: Sebastian Meine and Dennis Lloyd

Topic: TDD – Thwarting Database Defects
It’s bad enough to spend hours finding and fixing database defects, explaining to the rest of the team what went wrong and trying to clean up the mess. It’s even worse when a defect causes the end-user to make a bad decision. Database defects are far too costly and most testing practices do not adequately detect or prevent them. This presentation introduces tSQLt, a framework for automated database unit testing. You’ll learn techniques to write SQL code that is resilient to defects and is easier to change and maintain.

Location:
The Microsoft Technology Center
5404 Wisconsin Ave, Suite 700
Chevy Chase, MD, 20815

More Details and Registration: SQLSaturday #96 Washington DC

Resources:

tSQLt Website: http://tSQLt.org

tSQLt Mailing List: tsqlt@googlegroups.com

sqlity.net Webstie: http://sqlity.net

Curiously Correct Website: http://curiouslycorrect.com

 
Download course content here: Thwarting Database Defects Class Distribution – Includes slides and examples
 

Checkered garden at Jardín Japonés in Buenos Aires Argentina

The boundary test heuristic applies to requirements which specify different behavior [...]]]> This is the first post on a series of test case heuristics. Test case heuristics patterns used to help decide the next test to write and ensure test coverage of requirements.

Definition:

Checkered garden at Jardín Japonés in Buenos Aires Argentina

The boundary test heuristic applies to requirements which specify different behavior on either side of a particular value (called a boundary value) within a range of values.

For example, a requirement specifies that more inventory should be ordered if the number of units in the warehouse drops below 50. In this case, 50 is a boundary value.

A requirement often has several boundary values (see example below). The range of values on either side of a boundary value (or in between two boundary values) is called a partition.

The boundary test heuristic says that at least one test case should exist for each boundary value and each partition. It is useful to pick values very close to the boundary value to ensure that the code specifies the proper condition.

Purpose:

Boundary testing helps:

• Clarify and identify gaps in the requirements
• Provide coverage for conditional statements
• Prevent mistakes arising from using the wrong comparison operator (e.g. coding “<”, when “<=” was intended).

Example:

The business sets the following discount structure to encourage higher sales:

• If the order total is between $50 and $100 then a 5% discount is applied.
• If the order total is more than $100 and less than $200 then a 10% discount is applied.
• If the order total is more than $200 than a 15% discount is applied.

This requirement specifies three boundaries: $50, $100, and $200. There are four partitions.

<--------------|-----------------|---------------|-------------->
    par 1      50     par 2     100   part 3    200    par 4

As we choose certain test cases, we are forced to ask certain clarifying questions, such as:

• $49.99: What should happen when the order total is less than $50. The requirement didn’t say.
• $50.00: When the requirement says “between”, did the they mean to include or exclude $50 and $100.00. The word “between” can mean different things in regards to including the end points.
• $200.00: The requirement doesn’t say what happens if the order total is $200.

Notice that the number of decimal places is important. The value $49.99 was chosen above because we’re dealing with a monetary value. However, in another context, a different number of decimal places may be relevant.

Special Cases:

Null: While null is not truly a boundary value, it is a special value in most languages (particularly SQL).

Relative boundaries: Many times a boundary is specified in terms of another input. For example, return (2 * x) if x > y. In this case, y is a boundary value.

Dates and times: Boundary testing for dates and times must take into account the precision of the value (e.g. Is it a date only? Are hours and minutes relevant? How about seconds or milliseconds?). Dates and times are often used with relative boundaries. For example, a citizen is allowed to vote if they are at least 18 years old; the boundary (18 years) is relative to the citizen’s date of birth.

Boundaries on multiple inputs: Sometimes a requirement will specify boundary values on a variety of inputs. For example, a patient should be admitted if their respiratory rate is >= 30 and their age is >= 65. Combining boundary testing and all-pairs testing can be extremely valuable in coming up with appropriate tests when there are many inputs.

Output boundaries: Sometimes boundaries will be specified as part of a required output. This is often the case for a non-deterministic process. For example, a player will receive a random prize between $50 and $100.

Topic: TDD – Thwarting Database Defects It’s bad enough to spend hours finding and fixing database defects, explaining to the rest of the team what went wrong and trying to clean up the mess. It’s even worse when a defect causes [...]]]> When: Tuesday, October 18th 6:00pm – 8:30pm

Presenters: Sebastian Meine and Dennis Lloyd

Topic: TDD – Thwarting Database Defects
It’s bad enough to spend hours finding and fixing database defects, explaining to the rest of the team what went wrong and trying to clean up the mess. It’s even worse when a defect causes the end-user to make a bad decision. Database defects are far too costly and most testing practices do not adequately detect or prevent them. This presentation introduces tSQLt, a framework for automated database unit testing. You’ll learn techniques to write SQL code that is resilient to defects and is easier to change and maintain.

Location:
SetFocus
4 Century Drive
Parsippany , NJ

More Details and Registration: njsql.org

Resources:

tSQLt Website: http://tSQLt.org

tSQLt Mailing List: tsqlt@googlegroups.com

sqlity.net Webstie: http://sqlity.net

Curiously Correct Website: http://curiouslycorrect.com

 
Download course content here: Thwarting Database Defects Class Distribution – Includes slides and examples
 

Concern: If we fake the table, we’re not testing the [...]]]> In the first post in this series, I presented several reasons to not us a database pre-populated with data for unit testing. The second post, presented an alternative approach to using a pre-populated database. This post will address the common concerns with the approach.

Concern: If we fake the table, we’re not testing the real thing. The reality: “Not testing the real thing” is a common argument against any mock framework. Remember, we’re talking about unit testing – pinpointing a small, specific piece of functionality to exercise. Other forms of testing will cover the entire system as it is integrated together. By focusing on small units, you can achieve higher test coverage. You will also establish a safety-net for refactoring.

Concern: Constraints need to be tested too. The reality: I completely agree. While unit testing, you can write test cases against the constraints. After faking a table, you can re-apply the constraints which you’d like to test. This actually improves your ability to test the constraints, as you can do so in isolation of other functionality.

Concern: We need realistic data that we got from the customer. The reality: There are times and places to use such data if you have it. Unit testing is not it. This type of data is better utilized during integration, usability, and performance testing. Using customer data during unit testing will prevent you from thinking about scenarios which don’t occur now in the data, but may occur later. Defects will hide among those untested scenarios.

As you become experienced with this approach to unit testing, you will also find the following benefits:

1. Code is less coupled.
2. It is easier to test date and time based scenarios.
3. Your tests are not slowed down by having a large amount of data in the database.
4. It is easier to version control everything needed for unit testing.
5. You’ll prevent more defects before integrating the code.

Other Posts in this Series:

Pre-populated Test Databases – Part 1

Pre-populated Test Databases – Part 2

The test data should be inside the test case. This keeps everything together. You know exactly what data is important to the [...]]]> In the previous post on this topic, I gave you some very good reasons to not pre-populate your unit test database. So, the burning question is, where should test data come from?

The test data should be inside the test case. This keeps everything together. You know exactly what data is important to the test case by reading the test case. The first step of the test case generally inserts the data it needs into the database (part of the Assemble-Act-Assert pattern).

Only data needed for the test should be included in the test case. By inserting only the data you need in the test case, you create focus. There is much less chance that unrelated schema changes will break your tests and that the test will actually validate the small piece of code you are targeting.

Tables should be faked. The biggest problem with inserting test data inside the test case is that there are constraints on the tables. To solve this problem, we can borrow a unit testing practice from object-oriented languages, that is creating a fake version of a dependency. For example, if we’re testing a stored procedure which reads data from a table, the stored procedure is dependent upon the table. We can replace the real table with a fake table which does not have the constraints. This allows us to easily insert data in the test case.

In tSQLt, tables are faked using the tSQLt.FakeTable procedure. This procedure renames the original table and puts a new one in its place. The new table contains the same columns with the same data-types, but does not have the constraints including primary keys, foreign keys, checks or null restrictions. Let’s look at an example of a test case that fakes a table:

EXEC tSQLt.NewTestClass 'OrderReportTests';
GO

CREATE PROCEDURE OrderReportTests.[test: OrderReport works for several customers]
AS
BEGIN
    EXEC tSQLt.FakeTable 'dbo', 'Orders';


    INSERT INTO dbo.Orders(OrderId, CustomerId) 
    VALUES (1,13),(2,13),(3,42),(4,73),(5,73),(6,73);
   
    SELECT CustomerId, Cnt 
      INTO #actual
      FROM dbo.CustomerReport;
 
    SELECT TOP(0) * INTO #expected FROM #actual;
 
    INSERT INTO #expected(CustomerId,Cnt) VALUES (42,1),(13,2),(73,3);
 
    EXEC tSQLt.AssertEqualsTable #expected,#actual;
END
GO

EXEC tSQLt.Run 'OrderReportTests';

In this example, we fake the Orders table. The CustomerReport view depends on only the OrderId and CustomerId columns from the Orders table. We only need to insert those two columns into the Orders table, even though it has several other ‘not null’ columns. The rest of the test validates that the records returned by CustomerReport view are the expected values. The documentation of the AssertEqualsTable feature in tSQLt discusses this technique in more detail.

Tests should be executed in transactions which are rolled-back after each test. In order for test cases to be independent, anything that happens during the test case must be undone. The easiest way of doing this in a database is to create a transaction and roll it back when the test is complete. SQL Server conveniently rolls back schema changes also, so a faked table is rolled back.

The tests should be written in parallel with the code. The best way to achieve test coverage of your code is to only write enough code to make a failing test pass. This is a critical component of test driven development and will be discussed in great detail on this website. Writing test cases this way will improve the ability to make changes down stream because you will be confident that there is adequate test coverage. Furthermore, knowledge of test case identification heuristics will help you discover the correct tests to write and which test to write next.

Pre-populated databases are still useful in other areas of application development. They are especially useful for performance and scalability testing. Usability testing and prototyping also benefit from pre-populated databases. They’re also necessary for data-mining and discover tasks, and development of data dependent certain algorithms. However, they get in your way when you’re writing unit tests.

In the final post in this series, I’ll counter some of the criticisms of this approach to database unit testing.

Other Posts in this Series:

Pre-populated Test Databases – Part 1

Pre-populated Test Databases – Part 3

This post discusses pre-populating a database for database unit tests.

Your tests need data to work with. There are several approaches to supplying test data for unit tests. One approach is to use a pre-populated test database. The data in this database could be obtained from an existing customer or generated in some [...]]]>  

This post discusses pre-populating a database for database unit tests.

Your tests need data to work with. There are several approaches to supplying test data for unit tests. One approach is to use a pre-populated test database. The data in this database could be obtained from an existing customer or generated in some way. This series of posts discuss why you should NOT use a pre-populated test database for unit testing and how you should actually go about creating test data.

Why Not?

There are several reasons to not pre-populate test data for unit testing. Primarily these are: it makes test cases harder to read, more expensive to maintain, and limits the types of tests you can create. Let’s look at these claims in more detail:

Obscurity

First, the test data is in a separate place than the test case. By this, I don’t simply mean that the test data lives in a separate folder or in another database. The test data helps express the intent of the test. For example, if a test is validating a calculation based on the data, then the specific values used are an important part of the test. When reading the test case, if you have to look somewhere else to find out the intent of the test case, then you’re going to slow yourself down considerably.

As a consequence, tests often contain “magic” values which refer to special records in the test database. Trying to keep track of the meaning of special test case records is particularly difficult. For example, you’re likely to end up with a record for “Joe Smith” and everyone on the team needs to remember that Joe Smith was the customer who bought 50 widgets at a total cost of $500 but doesn’t have a credit limit (as opposed to the record for “Johnny Wilkenson” who has the same data, but has a sufficient credit limit, and so on).

Interdependency

If all (or even some) of the test cases share the same data set, then you will find that test cases become more difficult to maintain.

Let’s look at an example: Suppose you’re system contains several reports which provide different ways of considering the same data. The first report you write displays the name of each customer who ordered your most expensive product and the total order value. Your pre-populated test database has the following records in it:

So you’re first test expects to see the following results:

Now, it’s a month later and the next report must be written. In the meantime, the business has started selling more expensive items. This report will display all the items in the catalog costing more than $100. If you change the price of an existing item or add a new order with a new item, you’ll break the tests for the first report.

In this simple example, you might not be concerned with the amount of work to fix the first test. However, in a large system with many tests, continually fixing your existing tests just because you’ve added new functionality is painful. What normally happens when developers face this pain is they stop writing tests.

Impossible Tests

If your test database is already populated, the data in there may prevent you from writing certain tests. Consider if, for example, the code has special functionality if one of the tables is empty. You won’t be able to test this functionality if the table is populated. An alternative example is when the code is supposed to do something if data in the database meets some condition and do something else if the data does not meet that condition. Two mutually exclusive conditions are impossible to test when using a pre-populated test database.

A commonly used solution to this is to have some of the test cases modify the database for their special need and then undo the modification after the test. However, this leads to some tests containing the test data and the rest of the test data somewhere else. This means that you need to go to multiple places to update the data when schema changes are made or to get a clear picture of what is happening.

Limited Thinking

The final point I’d like to expose about using pre-populated test data is that it often leads to limited thinking about test cases. If there’s a lot of data already populated in the database, it is easy to assume that every important case is covered by the existing data. This is almost never true. The data in the database now is not necessarily a good reflection of the data that will be put in the database in the future from the perspective of unit testing. Writing unit tests requires you consider exception scenarios, inputs that don’t exist yet and varying the combinations of inputs.

Furthermore, it is more difficult to see if the important edge cases have been considered if the data is heaped into one large data set. Modifying the test data to support a new test may inadvertently remove an important condition needed for an existing test.

The Results

Unfortunately, when faced with these problems, many developers will either give up on database unit testing, or instead miss important test cases. Both of these outcomes lead to reduced code quality and makes me unhappy.

The next post in this series will propose an approach that solves these problems, and the final post will address the common concerns of the proposed approach.

Other Posts in this Series:

Pre-populated Test Databases – Part 2

Pre-populated Test Databases – Part 3