SQL Server

SQL Server: Dynamic column lists with UNPIVOT — Give your support

Stephen Horsfield — Tue, 11 Mar 2008 12:36:56 GMT

Overview

Here's another product suggestion I've logged with Microsoft along the same lines as the previous one. This one is to allow you to dynamically specify the list of columns you want to unpivot. This might be from a stored procedure parameter or a subquery, either way you can't do it easily at the moment. Register your support here:

https://connect.microsoft.com/SQLServer/feedback/ViewFeedback.aspx?FeedbackID=332512

Detail

Current behaviour

UNPIVOT (value FOR column_name IN (Col1, Col2, Col3, Col4))

All columns must be known in advance and specified in full.

Suggested behaviour with subquery using a table-valued parameter

UNPIVOT (value FOR column_name IN (SELECT colname FROM @tblParam))

Columns can be determined at runtime.

Versions

SQL Server 2008
SQL Server 2005

Metadata

Categories: IT Management, Software Development, SQL Server
Additional keywords: UNPIVOT, dynamic column lists
Technorati Tags: SQL, SQL Server, UNPIVOT

SQL Server: Inclusion of NULLs with UNPIVOT — Give your support

Stephen Horsfield — Tue, 11 Mar 2008 10:54:54 GMT

Overview

I've logged a suggestion with Microsoft for support for the inclusion of NULLs when you use the UNPIVOT operator. Add your support if you think this would be useful:

https://connect.microsoft.com/SQLServer/feedback/ViewFeedback.aspx?FeedbackID=332325

Example

Source data

ID	Col1	Col2	Col3	Col4
1	Value	NULL	Value	NULL

Current behaviour

UNPIVOT (value FOR column_name IN (Col1, Col2, Col3, Col4))

ID	column_name	value
1	Col1	Value
1	Col3	Value

Suggested behaviour with new INCLUDE NULL optional clause

UNPIVOT (value FOR column_name IN (Col1, Col2, Col3, Col4) INCLUDE NULL)

ID	column_name	value
1	Col1	Value
1	Col2	NULL
1	Col3	Value
1	Col4	NULL

This feature is available in Oracle 11g, for example.

Versions

SQL Server 2008
SQL Server 2005

Metadata

Categories: IT Management, Software Development, SQL Server
Additional keywords: UNPIVOT, NULL
Technorati Tags: SQL, SQL Server, UNPIVOT, NULL

SQL Server: Fun with column-level auditing

Stephen Horsfield — Mon, 10 Mar 2008 12:35:31 GMT

Overview

Auditing the actual changes made to a table can be quite complex, but is often required. In SQL Server 2008, you can use the new feature Change Data Capture, but in 2005 you have fewer options.

Here's an approach using common table expressions, table variables and a nice, big and hopefully fast query!

Context

click here to view this online and see the code formatted for viewing

Suppose you have a data table and an audit table as follows:

   1: CREATE TABLE tblToAudit (

   2:     pkid int identity(1,1) primary key,

   3:     col0 nvarchar(20),

   4:     col1 nvarchar(20),

   5:     col2 nvarchar(20),

   6:     col3 nvarchar(20),

   7:     col4 nvarchar(20),

   8:     col5 nvarchar(20),

   9:     col6 nvarchar(20),

  10:     col7 nvarchar(20),

  11:     col8 nvarchar(20),

  12:     col9 nvarchar(20),

  13:     lastModified datetime,

  14:     lastModifiedBy nvarchar(20)

  15:     )

  16: GO

  17: CREATE TABLE tblAudit (

  18:   object_name nvarchar(20) NOT NULL,

  19:   pkid int NOT NULL,

  20:   column_name nvarchar(20) NOT NULL,

  21:   prior_value nvarchar(20),

  22:   new_value nvarchar(20),

  23:   modified datetime,

  24:   modifiedBy nvarchar(20),

  25:   msg nvarchar(255)

  26: )

  27: GO

And now suppose you have some stored procedures that make changes to one or more rows, possibly also adding new rows (although this isn't particularly well suited to identity columns). Each column value is passed to the stored procedure using a separate parameter. How do you audit which columns have changed, and how have they changed?

One approach is to use successive IF ... THEN clauses, but this is slow and cannot be combined for multiple changes at once.

Solution

Here's my solution:

   1: BEGIN

2:

   3: SET NOCOUNT ON

4:

   5: -- Declare variables to represent the input data

   6: DECLARE @c0 nvarchar(20) = N'B', @c1 nvarchar(20) = N'C', @c2 nvarchar(20) = N'B',

   7:     @c3 nvarchar(20) = NULL, @c4 nvarchar(20) = N'D', @c5 nvarchar(20) = N'A',

   8:     @c6 nvarchar(20) = NULL, @c7 nvarchar(20) = NULL, @c8 nvarchar(20) = NULL,

   9:     @c9 nvarchar(20) = NULL, @user nvarchar(20) = N'Test',

  10:     @pkid int = 10

11:

  12: -- Create a table variable that is equivalent to the source table

  13: -- table variables are stored in tempdb so make sure we use this

  14: -- database's collation

  15: DECLARE @tgtData TABLE (

  16:     pkid int,

  17:     col0 nvarchar(20) COLLATE DATABASE_DEFAULT,

  18:     col1 nvarchar(20) COLLATE DATABASE_DEFAULT,

  19:     col2 nvarchar(20) COLLATE DATABASE_DEFAULT,

  20:     col3 nvarchar(20) COLLATE DATABASE_DEFAULT,

  21:     col4 nvarchar(20) COLLATE DATABASE_DEFAULT,

  22:     col5 nvarchar(20) COLLATE DATABASE_DEFAULT,

  23:     col6 nvarchar(20) COLLATE DATABASE_DEFAULT,

  24:     col7 nvarchar(20) COLLATE DATABASE_DEFAULT,

  25:     col8 nvarchar(20) COLLATE DATABASE_DEFAULT,

  26:     col9 nvarchar(20) COLLATE DATABASE_DEFAULT,

  27:     modified datetime,

  28:     modifiedBy nvarchar(20) COLLATE DATABASE_DEFAULT)

29:

  30: -- Insert new values into the table variable, can repeat multiple times

  31: INSERT INTO @tgtData VALUES (@pkid, @c0, @c1, @c2, @c3, @c4, @c5, @c6,

                                   @c7, @c8, @c9, GETDATE(), @user)

32:

  33: -- This query eliminates NULLs as part of an aggregate so turn ANSI WARNINGS off

  34: SET ANSI_WARNINGS OFF;

35:

  36: -- Audit and change are one atomic operation, so need a transaction

  37: BEGIN TRAN;

38:

  39: -- Previous statement is completed with the required ;

  40: -- THIS IS THE AUDIT statement

  41: WITH

  42: -- mergelist is a cartesian join of columns and affected row identifiers

  43: mergelist (pkid, colname) AS (

  44:     SELECT pkid, colname

  45:     FROM

  46:         -- Get a list of row identifiers which are being updated

  47:         (SELECT DISTINCT pkid FROM @tgtData) AS ids,

  48:         -- Get a full list of columns

  49:         (    SELECT 'col0' AS colname UNION ALL

  50:             SELECT 'col1' AS colname UNION ALL

  51:             SELECT 'col2' AS colname UNION ALL

  52:             SELECT 'col3' AS colname UNION ALL

  53:             SELECT 'col4' AS colname UNION ALL

  54:             SELECT 'col5' AS colname UNION ALL

  55:             SELECT 'col6' AS colname UNION ALL

  56:             SELECT 'col7' AS colname UNION ALL

  57:             SELECT 'col8' AS colname UNION ALL

  58:             SELECT 'col9' AS colname

  59:         ) AS cols

  60: ),

  61: -- srcData is an unpivoted set of column values from the source table with NULLs missing

  62: srcData (pkid, colname, value) AS (

  63:     SELECT t.pkid, t.colname, t.value

  64:     FROM tblToAudit UNPIVOT (value FOR colname IN (col0, col1, col2, col3, col4, col5,

                                                         col6, col7, col8, col9)) AS t

  65: ),

  66: -- tgtData is an unpivoted set of column values from the new data with NULLs missing

  67: tgtData (pkid, colname, value, modified, modifiedBy) AS (

  68:     SELECT t.pkid, t.colname, t.value, t.modified, t.modifiedBy

  69:     FROM @tgtData UNPIVOT (value FOR colname IN (col0, col1, col2, col3, col4, col5,

                                                       col6, col7, col8, col9)) AS t

  70: ),

  71: -- changeList is the expanded set of srcData and tgtData including NULLs and with

  72: -- change indicating which values are NULL

  73: changeList (pkid, colname, old_value, new_value, change, modified, modifiedBy) AS (

  74:     SELECT mergelist.pkid, mergelist.colname, srcData.value, tgtData.value,

  75:         (2*COUNT(srcData.value) OVER (PARTITION BY mergelist.pkid, mergelist.colname)

  76:          + COUNT(tgtData.value) OVER (PARTITION BY mergelist.pkid, mergelist.colname))

  77:         AS change,

  78:         tgtData.modified, tgtData.modifiedBy

  79:     FROM mergeList

  80:         LEFT JOIN srcData

  81:             ON mergelist.pkid = srcData.pkid AND mergelist.colname = srcData.colname

  82:         LEFT JOIN tgtData

  83:             ON mergelist.pkid = tgtData.pkid AND mergelist.colname = tgtData.colname

  84: )

  85: -- inserting into the audit table

  86: INSERT INTO tblAudit ([object_name], pkid, column_name, prior_value, new_value,

                            modified, modifiedBy, msg)

  87: SELECT N'tblToAudit', pkid, colname,

  88:     old_value, new_value,

  89:     modified, modifiedBy,

  90:     -- use the change column to determine the action being taken

  91:     CASE change

  92:         WHEN 1 THEN N'set'

  93:         WHEN 2 THEN N'reset'

  94:         WHEN 3 THEN N'modify'

  95:     END AS msg

  96: FROM changeList

  97: -- change = 0 means both source and target values are NULL

  98: WHERE changeList.change > 0 AND

  99: -- change = 3 means both source and target are NOT NULL, so need to compare them

 100:     (changeList.change < 3 OR (old_value <> new_value))

 101:

 102: -- UPDATE existing rows

 103: UPDATE tblToAudit

 104: SET

 105:     tblToAudit.col0 = t.col0,

 106:     tblToAudit.col1 = t.col1,

 107:     tblToAudit.col2 = t.col2,

 108:     tblToAudit.col3 = t.col3,

 109:     tblToAudit.col4 = t.col4,

 110:     tblToAudit.col5 = t.col5,

 111:     tblToAudit.col6 = t.col6,

 112:     tblToAudit.col7 = t.col7,

 113:     tblToAudit.col8 = t.col8,

 114:     tblToAudit.col9 = t.col9,

 115:     tblToAudit.lastModified = t.modified,

 116:     tblToAudit.lastModifiedBy = t.modifiedBy

 117: FROM tblToAudit INNER JOIN @tgtData AS t ON tblToAudit.pkid = t.pkid

 118:

 119: -- inserting identity column

 120: SET IDENTITY_INSERT tblToAudit ON

 121:

 122: -- INSERT new rows (not inserting identity column)

 123: INSERT INTO tblToAudit (pkid, col0, col1, col2, col3, col4, col5, col6,

                              col7, col8, col9, lastModified, lastModifiedBy)

 124: SELECT t.pkid, t.col0, t.col1, t.col2, t.col3, t.col4, t.col5, t.col6,

             t.col7, t.col8, t.col9, t.modified, t.modifiedBy

 125: FROM @tgtData AS t LEFT JOIN tblToAudit ON t.pkid = tblToAudit.pkid

 126: WHERE tblToAudit.pkid IS NULL

 127:

 128: -- reset identity insertion option

 129: SET IDENTITY_INSERT tblToAudit OFF

 130:

 131: -- Commit the transaction

 132: COMMIT

 133:

 134: END

 135: GO

I hope it is of some use to you. If you want clarification of the code, or if you want to make a suggestion, please do send me a comment.

Versions

SQL Server 2008, SQL Server 2005

Metadata

Categories: IT Management, Software Development, SQL Server
Additional keywords: common table expressions, table variables, window functions, auditing, change tracking, column-level tracking
Technorati Tags: SQL, SQL Server, auditing, change tracking, column%20auditing

SQL Server: Generating random data

Stephen Horsfield — Mon, 03 Mar 2008 17:08:35 GMT

Overview

I've been looking at the performance of SQL Server 2008, following the recent SQL Bits conference (see http://www.sqlbits.com), and have been looking at creating some test data for my analysis. This is probably old stuff to most of you, but I hit an interesting issue...

Context

Specifically, I've been looking at the performance of the new filtered indexes support in 2008. If you haven't come across these, I recommend that you have a look but I'll probably post about it in the next few days (no promises, though!). To do the analysis I wanted a table with heavily weighted data so I thought I'd just generate it using the RAND() function. Only, the RAND() function didn't seem to give particularly random data.

Here's my original code:

SELECT TOP(1000)
    RAND() AS [value]
    FROM master.sys.objects a, master.sys.objects b
GO

What results do you expect? This is what I get, and I've checked the same occurs in SQL Server 2005:

0.863657983742771
0.863657983742771
0.863657983742771
0.863657983742771
0.863657983742771
0.863657983742771
0.863657983742771
...

So, the RAND() function was only executed once for the query. Not quite the random data I'd hoped for :(

Solution

The most efficient solution when generating many rows is to use a table-valued CLR function to generate the row data. For testing, I had two options:

a temporary table
a table variable

I cannot use the RAND() function within a table-valued user-defined function, so I had to use one of these approaches. I'll show the table variable solution but the temporary table solution is virtually identical. Here it is:

BEGIN
    SET ROWCOUNT 0
    DECLARE @randomNumbers TABLE (value float)
    DECLARE @i int

    SET @i = 0
    WHILE @i < 1000
        BEGIN
            INSERT INTO @randomNumbers (value) SELECT RAND()
            SET @i = @i + 1
        END
    SELECT * FROM @randomNumbers
END    
GO

Of course you need to adjust this script for it to be useful :)

Versions

SQL Server 2008
SQL Server 2005

Metadata

Categories: Software development, SQL Server, Random Data
Additional keywords: RAND(), generate random data, test data
Technorati Tags: SQL, SQL Server, random%20numbers, RAND

SQL Server: Viewing Database Permissions Using a Query

Stephen Horsfield — Thu, 17 Jan 2008 09:02:18 GMT

Overview

Have you ever wondered how to view object permissions using a query?

Simple example

Try this on a database:

select o.name, u.name, p.permission_name, p.state, p.state_desc 

  from sys.database_permissions p 

       inner join sys.all_objects o 

         on p.major_id = o.object_id 

       inner join sys.database_principals u 

         on p.grantee_principal_id = u.principal_id

References

Versions

Microsoft SQL Server 2005

Metadata

Categories: SQL Server, IT Management, Software Development, Security
Additional keywords: information schema, system views, permissions, security, query
Technorati Tags: SQL, SQL Server, software development, IT Management, security, permissions

SQL Server: Primer — Using Alias Data Types for Simpler Database Design

Stephen Horsfield — Tue, 15 Jan 2008 11:00:09 GMT

Overview

Do you have lots of tables that reference each other using ID fields? Are they of different types, such as nvarchar(10) and int? Do you sometimes find that it can be hard to remember how many characters should be in the type? And what about when you decide that you need a bigger type, such as nvarchar(30)? How can you ensure that your database design remains consistent?

Alias Data Types and the CREATE TYPE statement

An Alias Data Type is a type defined by you in SQL Server that refers to an internal data type. You can use it anywhere you would use a normal SQL type, except in table variables. For example, the following command declares adtCustomerID as a non-NULL nvarchar(10):

CREATE TYPE [dbo].[adtCustomerID] FROM nvarchar(10) NOT NULL

You can then use the type adtCustomerID in your table definitions and stored procedures, never worrying about how many characters are in the type:

CREATE TABLE [dbo].[customers] ( customer_id adtCustomerID PRIMARY KEY, customer_name nvarchar(100), ... )

CREATE TABLE [dbo].[orders] ( order_id adtOrderID PRIMARY KEY, customer_id adtCustomerID, ... )

Caveat

You cannot change a type while it is in use. This means that you will have to change stored procedures (etc.) and tables to use the underlying data type before changing it, so type definitions are most useful when they are not going to change, such as in key fields.

References

Versions

Microsoft SQL Server 2005
Microsoft SQL Server 2000

Metadata

Categories: SQL Server, IT Management, Software Development, Alias Data Types
Additional keywords: simplicity, CREATE TYPE, user defined type, best practice, primer
Technorati Tags: SQL, SQL Server, software development, architecture, database design, CREATE TYPE

SQL Server: Primer — Using Database Views

Stephen Horsfield — Mon, 14 Jan 2008 15:25:02 GMT

Overview

I'm in the process of reviewing a large number of SQL stored procedures, and occasionally I find that a table join isn't properly expressed. This can result in hard to find bugs, but how can you make your SQL code simpler? Isn't using a stored procedure best-practice anyway?

Scenario

It is quite common to have a series of related tables. Consider a stock control system. Perhaps you have tables for customers, orders, order lines and stock items. In this simple scenario, you have four tables and clearly there are joins between several of the tables.

Common solution

A common solution is to describe the table joins in SQL code in each stored procedure. This way the database is only loosely-coupled to the application code, but the stored procedure is tightly-coupled to the database schema. Suppose the stored procedure returns the number of times a customer has ordered each of the items he/she has ordered. The SQL might look like the following

SELECT items.item_description, COUNT(order_lines.item_id) FROM customers INNER JOIN orders ON customers.customer_id = orders.customer_id INNER JOIN order_lines ON orders.order_id = order_lines.order_id INNER JOIN items ON order_lines.item_id = items.item_id WHERE customers.customer_name = @customer_name GROUP BY order_lines.item_id, items.description

This isn't wrong, but it is easy to forget a join field, particularly where a join is on multiple fields, and you have to remember the joins every time. Essentially, the stored procedure encompasses both schema knowledge and business logic. Wouldn't it be better to separate the two?

A better solution

Key Point: Views allow you to express table relationships without affecting server performance. They allow for reusable table expressions.

Let's build some useful views, getting closer to our original SQL as we go along:

Note: I've omitted several columns in the resulting view for brevity, but there is no reason to do so unless you are also using a clustered index on the view (in which case the columns determine the size of the index)

CREATE VIEW all_customer_orders AS SELECT customers.customer_name, orders.* FROM customers INNER JOIN orders ON customers.customer_id = orders.customer_id

CREATE VIEW all_customer_lines AS SELECT customer_id, customer_name, order_lines.* FROM all_customer_orders INNER JOIN order_lines ON all_customer_orders.order_id = order_lines.order_id

CREATE VIEW all_line_item_details AS SELECT order_lines.order_line_id, order_lines.item_id, items.item_description FROM order_lines INNER JOIN items ON order_lines.item_id = items.item_id

CREATE VIEW all_customer_line_item_details AS SELECT all_customer_lines.*, all_line_item_details.item_description FROM all_customer_lines INNER JOIN all_line_item_details ON all_customer_lines.order_line_id = all_line_item_details.order_line_id

Finally, the stored procedure code can be expressed as

SELECT item_description, COUNT(item_id) 

  FROM all_customer_line_item_details 

GROUP BY item_id, item_description 

WHERE customer_name = @customer_name

This SQL no longer assumes any underlying knowledge of the table structure, it simply refers to a single view. Also, each view only has a single join and is therefore much easier to both test and understand.

Which views to create is a matter of experience, but whenever you are joining several tables, consider using a view. The view is easier to maintain and easier to test. It is also reusable. If you need to parameterise the view, consider using a table-valued function to represent it.

Remember as well that views can be indexed. This can introduce significant performance benefits without the penalties of denormalised data. The indexes aren't free from a performance perspective, but judicious use can result in optimal and simple queries.

If you are using SQL Server 2005 Enterprise edition you also get the benefit of having the SQL Server optimiser consider your views (and the associated indices) automatically.

References

Versions

Microsoft SQL Server 2005
Microsoft SQL Server 2000

Metadata

Categories: SQL Server, IT Management, Software Development, Views, Stored Procedures
Additional keywords: simplicity, queries, clustered index, views, best practice, primer, user-defined functions, UDF
Technorati Tags: SQL, SQL Server, software development, architecture, database design, database views, best practice

SQL Server: Missing Indices and a Helpful Database

Stephen Horsfield — Fri, 11 Jan 2008 17:39:38 GMT

Overview

I've just found this by chance while in the process of reviewing around 100 stored procedures to check for performance efficiency. It doesn't help with badly written queries, but if the query is reasonable then it can help you find those elusive missing indices. Thought I'd have to find them myself, but thankfully help is at hand...

Details

Normally, you would use the SQL Server Profiler to help track performance issues, but what if you are not in a live environment? SQL Server 2005 helpfully collects some data that you can use.

Warning: the data is only kept for while the server is running. If it restarts, you'll have to wait for it to recalculate the information!

The database automatically collects information on indices that it would have used had they existed. You can use this to investigate whether creating the index is warranted. SQL Server also lets you know its potential value. This feature is on by default.

Have a look at the references section for more details on the feature, but here is an example script to show you some of the information that is available:

USE [master] 

GO 



SELECT statement, avg_total_user_cost, avg_user_impact, statement, 

equality_columns, inequality_columns, included_columns 

FROM sys.dm_db_missing_index_group_stats a INNER JOIN sys.dm_db_missing_index_groups b ON a.group_handle = b.index_group_handle INNER JOIN 

sys.dm_db_missing_index_details c ON b.index_handle = c.index_handle 

ORDER BY avg_user_impact DESC 

GO

References

Versions

Microsoft SQL Server 2005

Metadata

Categories: SQL Server, IT Management, Software Development, Performance Analysis
Additional keywords: query optimisation, missing indexes, statistics, tuning
Technorati Tags: SQL, SQL Server, performance, database indexing, IT Management

SQL Server: Primer — Foreign Key Constraints

Stephen Horsfield — Fri, 11 Jan 2008 15:31:07 GMT

Overview

You may know that it is best practice to create FOREIGN KEY constraints on related tables in a database. Why is this?

Details

A foreign key constraint can be used for:

Ensuring referential integrity: a foreign key constraint can ensure database consistency
Self-documenting databases: the foreign key clause can be viewed in the database and describes a relationship
Efficiency: even without an index, the SQL Server engine recognises the constraint and uses it to optimise table joins automatically

Generally, you will also want to index the foreign key columns in the table, but it is worth adding the FOREIGN KEY constraint even if you don't create the index because it helps the database to optimise the execution plan.

References

Versions

Microsoft SQL Server 2005
Microsoft SQL Server 2000

Metadata

Categories: SQL Server, IT Management, Software Development, Performance Analysis
Additional keywords: query optimisation, table join, foreign key, primary key, primer
Technorati Tags: SQL, SQL Server, performance, database indexing, architecture, database design, IT Management, best practice, software development, foreign keys, referential integrity

SQL Server: Which tables are storing the most physical data?

Stephen Horsfield — Wed, 09 Jan 2008 15:16:07 GMT

Overview

Following on from my previous post, here's a quick way to find out which data tables are using the most disk space...

References

Table and Index Organization (Microsoft SQL Server 2005 Books Online (September 2007))

Solution

SELECT o.name AS table_name, SUM(au.data_pages) as pages, 

       ((SUM(au.data_pages) * 8192) / (1024 * 1024)) AS MB 

FROM sys.allocation_units AS au 

     JOIN sys.partitions AS p ON au.container_id = p.partition_id 

     JOIN sys.tables AS o ON p.object_id = o.object_id 

     JOIN sys.indexes AS i ON p.index_id = i.index_id AND i.object_id = p.object_id 

GROUP BY o.name 

ORDER BY pages desc, o.name 

GO

Versions

Microsoft SQL Server 2005

Metadata

Categories: SQL Server, Performance Analysis
Additional keywords: table statistics, index statistics, optimisation, performance
Technorati Tags: IT Management, SQL, SQL Server, storage