Database Backup

Procedure name: DatabaseBackup

Purpose: Create backup directories, backup databases, verify backups and delete old backups.

Parameters: See table below.

Name Description Supported Values
Databases Selection of databases. E.g. SYSTEM_DATABASES
E.g. USER_DATABASES
E.g. ALL_DATABASES
E.g. Database1
E.g. Database1, Database2
E.g. %Database%
E.g. %Database%, -Database1
Directory Backup root directory. It can be a local directory or a network share. If not specified the default backup directory is used. NULL (DEFAULT)
E.g. C:\Backup
E.g. \\Server1\Backup
BackupType Type of backup. FULL
DIFF
LOG
Verify Verify the backup. Y
N (DEFAULT)
CleanupTime Time in hours after which the backup files are deleted. If not specified no backup files are deleted. NULL (DEFAULT)
E.g. 24
Compress Compress the backup. If not specified the backup compression default in sys.configurations is used. NULL (DEFAULT)
Y *
N
CopyOnly Perform a copy-only backup. Y
N (DEFAULT)
ChangeBackupType Change backup type if a differential or transaction log backup cannot be performed. Y
N (DEFAULT)
BackupSoftware Third party backup software. If not specified a sql server native backup is performed. NULL (DEFAULT)
LITESPEED
SQLBACKUP
HYPERBAC
CheckSum Enable backup checksums. Y
N (DEFAULT)
BlockSize Physical blocksize in bytes. NULL (DEFAULT)
BufferCount The number of I/O buffers to be used for the backup operation. NULL (DEFAULT)
MaxTransferSize The largest unit of transfer in bytes to be used between SQL Server and the backup media. NULL (DEFAULT)
NumberOfFiles The number of backup files. 1 (DEFAULT)
CompressionLevel LiteSpeed and SQLBackup compression level. NULL (DEFAULT)
0 - 10 (LITESPEED)
0 - 4 (SQLBACKUP)
Description Backup description. NULL (DEFAULT)
E.g. Backup description
Execute Execution of commands. Default the commands are executed normally. If set to 'N' the commands are only printed. Y (DEFAULT)
N

* Backup compression is only supported in SQL Server 2008 Enterprise and Developer Edition and in SQL Server 2008 R2 Standard, Enterprise, Developer and Datacenter Edition.

Integrity Check

Procedure name: DatabaseIntegrityCheck

Purpose: Check the integrity of databases.

Parameters: See table below.

Name Description Supported Values
Databases Selection of databases. E.g. SYSTEM_DATABASES
E.g. USER_DATABASES
E.g. ALL_DATABASES
E.g. Database1
E.g. Database1, Database2
E.g. %Database%
E.g. %Database%, -Database1
PhysicalOnly Limit the checks to the physical structures of the database. Y
N (DEFAULT)
NoIndex Nonclustered indexes are not checked. Y
N (DEFAULT)
ExtendedLogicalChecks Perform extended logical checks. Y *
N (DEFAULT)
Execute Execution of commands. Default the commands are executed normally. If set to 'N' the commands are only printed. Y (DEFAULT)
N

* Extended logical checks is only supported in SQL Server 2008 and SQL Server 2008 R2.

Index Optimization

Procedure name: IndexOptimize

Purpose: Rebuild and reorganize indexes and update statistics.

Parameters: See table below.

Name Description Supported Values
Databases Selection of databases. E.g. SYSTEM_DATABASES
E.g. USER_DATABASES
E.g. ALL_DATABASES
E.g. Database1
E.g. Database1, Database2
E.g. %Database%
E.g. %Database%, -Database1
FragmentationHigh_LOB Action to be performed on a high fragmented index that does contain a LOB. INDEX_REBUILD_OFFLINE (DEFAULT)
INDEX_REORGANIZE
STATISTICS_UPDATE
INDEX_REORGANIZE_STATISTICS_UPDATE
NOTHING
FragmentationHigh_NonLOB Action to be performed on a high fragmented index that does not contain a LOB. INDEX_REBUILD_ONLINE *
INDEX_REBUILD_OFFLINE (DEFAULT)
INDEX_REORGANIZE
STATISTICS_UPDATE
INDEX_REORGANIZE_STATISTICS_UPDATE
NOTHING
FragmentationMedium_LOB Action to be performed on a medium fragmented index that does contain a LOB. INDEX_REBUILD_OFFLINE
INDEX_REORGANIZE  (DEFAULT)
STATISTICS_UPDATE
INDEX_REORGANIZE_STATISTICS_UPDATE
NOTHING
FragmentationMedium_NonLOB Action to be performed on a medium fragmented index that does not contain a LOB. INDEX_REBUILD_ONLINE *
INDEX_REBUILD_OFFLINE
INDEX_REORGANIZE (DEFAULT)
STATISTICS_UPDATE
INDEX_REORGANIZE_STATISTICS_UPDATE
NOTHING
FragmentationLow_LOB Action to be performed on a low fragmented index that does contain a LOB. INDEX_REBUILD_OFFLINE
INDEX_REORGANIZE
STATISTICS_UPDATE
INDEX_REORGANIZE_STATISTICS_UPDATE
NOTHING (DEFAULT)
FragmentationLow_NonLOB Action to be performed on a low fragmented index that does not contain a LOB. INDEX_REBUILD_ONLINE *
INDEX_REBUILD_OFFLINE
INDEX_REORGANIZE
STATISTICS_UPDATE
INDEX_REORGANIZE_STATISTICS_UPDATE
NOTHING (DEFAULT)
FragmentationLevel1 The lower limit in percent for medium fragmentation. E.g. 5 (DEFAULT)
FragmentationLevel2 The lower limit in percent for high fragmentation. E.g. 30 (DEFAULT)
PageCountLevel Indexes under this size in pages are considered to be low fragmented regardless of actual fragmentation level. E.g. 1000 (DEFAULT)
SortInTempdb Use tempdb for sort operations when rebuilding indexes. Y
N (DEFAULT)
MaxDOP The number of CPUs that are used when rebuilding indexes. If not specified the global max degree of parallelism is used. NULL (DEFAULT)
0 - 64
FillFactor Percentage that indicates how full the pages should be made when rebuilding indexes. If not specified the fill factor in sys.indexes is used. NULL (DEFAULT)
1 - 100
LOBCompaction Compact pages that contain LOB columns when reorganizing indexes. Y (DEFAULT)
N
StatisticsSample Percentage that indicates how much of a table that is gathered when updating statistics. 100 is equivalent to a full scan. If not specified SQL Server automatically computes the required sample. NULL (DEFAULT)
1 - 100
PartitionLevel Optimize partitioned indexes on the partition level. Y *
N (DEFAULT)
TimeLimit Time in seconds after which no commands are executed. NULL (DEFAULT)
E.g. 3600
Indexes Selection of indexes. If not specified all indexes are selected. NULL (DEFAULT)
Execute Execution of commands. Default the commands are executed normally. If set to 'N' the commands are only printed. Y (DEFAULT)
N

IndexOptimize places indexes into one of six groups based on their level of fragmentation (High, Medium or Low) and whether there are LOB columns (LOB or NonLOB).

It uses avg_fragmentation_in_ percent in sys.dm_db_index_physical_stats to check the percentage of fragmentation in each index. Using the threshold limits you set in the @FragmentationLevel1 (lower threshold) and @FragmentationLevel2 (upper threshold) parameters, it places each index in the appropriate group. Indexes with fragmentation levels higher than the upper threshold go into one of the high fragmentation groups. Indexes with fragmentation levels between the two thresholds go into one of the medium fragmentation groups. Indexes with fragmentation levels under the lower threshold go into one of the low fragmentation groups.

For each group, you can select one of the following actions:

Action Description
INDEX_REBUILD_ONLINE * Rebuild indexes online.
INDEX_REBUILD_OFFLINE Rebuild indexes offline.
INDEX_REORGANIZE Reorganize indexes.
INDEX_REORGANIZE_STATISTICS_UPDATE Reorganize indexes and update statistics.
STATISTICS_UPDATE Update statistics.
NOTHING Do nothing.

* Online rebuild is only supported in Enterprise, Developer and Datacenter Edition.

* Table partitioning is only supported in Enterprise, Developer and Datacenter Edition.

Selecting Databases

DatabaseBackup, DatabaseIntegrityCheck and IndexOptimize all have a parameter called @Databases that are used for selecting which databases to work with. It can be used like this.

Usage Description
@Databases = 'SYSTEM_DATABASES' All system databases (master, msdb and model).
@Databases = 'USER_DATABASES' All user databases.
@Databases = 'ALL_DATABASES' All databases.
@Databases = 'Database1' The database Database1.
@Databases = 'Database1, Database2' The databases Database1 and Database2.
@Databases = 'USER_DATABASES, -Database1' All user databases, except Database1.
@Databases = '%Database%' All databases that have Database in the name.
@Databases = '%Database%, -Database1' All databases that have Database in the name, except Database1.
@Databases = 'ALL_DATABASES, -%Database%' All databases that do not have Database in the name.

The keywords SYSTEM_DATABASES, USER_DATABASES and ALL_DATABASES are supported. The (-) character is used to exclude databases and the (%) character is used for wild-card selection. All of these things can be combined using the (,) character.

Selecting Indexes

IndexOptimize has an optional parameter called @Indexes that can be used for selecting which indexes to work with. It can be used like this.

Usage Description
@Indexes = 'ALL_INDEXES' All indexes.
@Indexes = 'Database1.Schema1.Object1.Index1' The index Index1 on the object Schema1.Object1 in the database Database1.
@Indexes = 'Database1.Schema1.Object1.Index1, Database2.Schema2.Object2.Index2' The index Index1 on the object Schema1.Object1 in the database Database1 and the index Index2 on the object Schema2.Object2 in the database Database2.
@Indexes = 'Database1.Schema1.Object1' All indexes on the object Schema1.Object1 in the database Database1.
@Indexes = 'Database1.Schema1.Object1, Database2.Schema2.Object2' All indexes on the object Schema1.Object1 in the database Database1 and all indexes on the object Schema2.Object2 in the database Database2.
@Indexes = 'Database1.Schema1.%' All indexes in the schema Schema1 in the database Database1.
@Indexes = '%.Schema1.%' All indexes in the schema Schema1 in all databases.
@Indexes = 'ALL_INDEXES, -Database1.Schema1.Object1.Index1' All indexes except the index Index1 on the object Schema1.Object1 in the database Database1.
@Indexes = 'ALL_INDEXES, -Database1.Schema1.Object1' All indexes except indexes on the object Schema1.Object1 in the database Database1.

The keyword ALL_INDEXES is supported. The (-) character is used to exclude indexes and the (%) character is used for wild-card selection. All of these things can be combined using the (,) character.

Setup

To use the solution for backup you need the objects DatabaseBackup, CommandExecute and DatabaseSelect,
to use it for integrity check you need DatabaseIntegrityCheck, CommandExecute and DatabaseSelect
and to use it for index optimization you need IndexOptimize, CommandExecute and DatabaseSelect.

You can download and install the solution in one script.

Execute the stored procedures from SQL Server Agent Jobs, through sqlcmd and the -b option. Pass the parameters to the stored procedures by name and not by position.

E.g. sqlcmd -E -S $(ESCAPE_SQUOTE(SRVR)) -d master -Q "EXECUTE [dbo].[DatabaseIntegrityCheck] @Databases = 'SYSTEM_DATABASES'" -b

Use SQL Server Agent Job Output files with tokens for the logging.

E.g. C:\Program Files\Microsoft SQL Server\MSSQL.1\MSSQL\LOG\DatabaseBackup
_$(ESCAPE_SQUOTE(JOBID))_$(ESCAPE_SQUOTE(STEPID))_$(ESCAPE_SQUOTE(STRTDT))_$(ESCAPE_SQUOTE(STRTTM)).txt

The solution is supported on SQL Server 2005, SQL Server 2008 and SQL Server 2008 R2 all Editions; Datacenter, Enterprise, Standard, Web, Workgroup, Developer and Express.

Getting Started

It's easy to get started with the maintenance solution.

1. Download MaintenanceSolution.sql.

2. Change this line in the script to your backup directory.
SET @BackupDirectory = N'C:\Backup' -- <== Change this to your backup directory.

3. Execute MaintenanceSolution.sql.
This script creates the stored procedures, the function and the jobs that you need.

4. Go into [SQL Server Agent] / [Jobs] and start the jobs that have been created. Verify that they're completing successfully.
Verify that the backup files are created. Check the output files in the error log directory.

5. Schedule the jobs.

Upgrading

It's easy to upgrade to the latest version of the maintenance solution.

1. Download MaintenanceSolution.sql.

2. Change this line in the script not to create jobs.
SET @CreateJobs = 'N' -- <== Should jobs be created, 'Y' or 'N'?

3. Execute MaintenanceSolution.sql.
The script drops the old versions the stored procedures and the function and creates the new versions.

4. All your existing jobs will continue to work with no changes. This even if you have made customizations to the jobs or if you have created your own jobs. Start the jobs and verify that they're completing successfully.

If you have made customizations to the stored procedures or the function, please contact me before upgrading to find out if the customizations are still needed in the new version.

Error Scenarios

Description Behavior
A backup command fails. The procedure skips the verify backup and delete backup steps for that database. The procedure then continues to the next database. In the end the job reports failure.
A verify backup command fails. The procedure skips the delete backup step for that database. The procedure continues to the next database. In the end the job reports failure.
An integrity check command fails. The procedure logs the error and continues to the next database. In the end the job reports failure.
An index rebuild or index reorganize command fails. The procedure logs the error and continues to the next index. In the end the job reports failure.
A table is locked and the index rebuild or index reorganize is waiting. When the index rebuild or index reorganize has been waiting on locks for one hour it logs an error and the procedure continues to the next index. In the end the job reports failure.

Frequently Asked Questions

Question Answer
I'm getting this error message when I try to create the objects.

"Msg 102, Level 15, State 1, Procedure IndexOptimize, Line 456"
"Incorrect syntax near '('."
This error is because of that the database that you are creating the objects in, is not in compatibility level 90 or 100. Change the compatibility level to 90 or 100 and try again.

EXECUTE dbo.sp_dbcmptlevel @dbname=N'DatabaseName', @new_cmptlevel=90
Does all databases have to be in compatibility level 90 or 100? It is only the database that you are creating the objects in, that has to be in compatibility level 90 or 100. All other databases can be in any compatibility level.
I'm getting these informational messages when I create the stored procedure DatabaseBackup, DatabaseIntegrityCheck or IndexOptimize.

"Cannot add rows to sys.sql_dependencies for the stored procedure because it depends on the missing table 'dbo.CommandExecute'. The stored procedure will still be created; however, it cannot be successfully executed until the table exists."

"The module 'DatabaseBackup' depends on the missing object 'dbo.CommandExecute'. The module will still be created; however, it cannot run successfully until the object exists."
DatabaseBackup, DatabaseIntegrityCheck and IndexOptimize are using the stored procedure CommandExecute and the function DatabaseSelect. Download these objects and create them in the same database.

You could also use the script MaintenanceSolution.sql. This script contains all the objects that you need.
I'm getting these error messages, when I execute the stored procedure DatabaseBackup, DatabaseIntegrityCheck or IndexOptimize.

"Msg 208, Level 16, State 1, Procedure DatabaseBackup, Line 111"
"Invalid object name 'dbo.DatabaseSelect'."

"Msg 2812, Level 16, State 62, Procedure DatabaseBackup, Line 397"
"Could not find stored procedure 'dbo.CommandExecute'."
DatabaseBackup, DatabaseIntegrityCheck and IndexOptimize are using the stored procedure CommandExecute and the function DatabaseSelect. Download these objects and create them in the same database.

You could also use the script MaintenanceSolution.sql. This script contains all the objects that you need.
Does the stored procedures have to be executed in a CmdExec job step with sqlcmd? Can I not use a T-SQL job step? The stored procedures have to be executed in a CmdExec job step with sqlcmd and the -b option. Otherwise the error handling and logging will not work as designed.

You could use the script MaintenanceSolution.sql to create the jobs.
My job stops after the first error. What could the reason be for that? This is the behavior if you're using T-SQL job steps. Then the job stops at the first error.

I recommend that you use CmdExec job steps with sqlcmd and the -b option. Then the job will continue after an error. In the end the job will report failure.

You could use the script MaintenanceSolution.sql to create the jobs.
The output files have a [SQLSTATE 01000] or [SQLSTATE 42000] in the end of each line. Why does it look like this? This is the behavior if you're using T-SQL job steps.

I recommend that you use CmdExec job steps with sqlcmd and the -b option. Then this will go away.

You could use the script MaintenanceSolution.sql to create the jobs.
Can IndexOptimize be used to update statistics? Yes, IndexOptimize can be used to update statistics.

If you use these parameters it will reorganize indexes and update the statistics, instead of only reorganize the indexes.

@FragmentationMedium_LOB = 'INDEX_REORGANIZE_STATISTICS_UPDATE', @FragmentationMedium_NonLOB = 'INDEX_REORGANIZE_STATISTICS_UPDATE'

For high fragmentation the indexes are rebuilt. Then the statistics are automatically updated.
I'm getting this error message when I'm executing IndexOptimize. Why do I get this error?

"Msg 50000, Level 16, State 1, Procedure CommandExecute, Line 103"
"Msg 1222, Lock request time out period exceeded."
This is because of that one of your indexes is locked by another connection. IndexOptimize can therefore not rebuild or reorganize the index.
Can the maintenance solution be used with SQL Server Express? The maintenance solution is supported on SQL Server Express. SQL Server Express has no SQL Server Agent. Therefore the execution of the stored procedures has to be scheduled using cmd - files and Windows Scheduled Tasks instead.
What should the fragmentation levels be for reorganization and rebuild of indexes? Microsoft has some recommendations in Books Online.

The default values in IndexOptimize are based on this.
Some of my indexes are still fragmented after I have just executed IndexOptimize. How could that be? Small indexes sometimes show a high fragmentation even after they've just been rebuilt or reorganized. This is because the pages are stored on mixed extents. You can read about this in Books Online.

"In general, fragmentation on small indexes is often not controllable. The pages of small indexes are stored on mixed extents. Mixed extents are shared by up to eight objects, so the fragmentation in a small index might not be reduced after reorganizing or rebuilding the index."

In addition fragmentation on very small tables has no impact on performance. Microsoft has a white paper about this.

"Generally, you should not be concerned with fragmentation levels of indexes with less than 1,000 pages. In the tests, indexes containing more than 10,000 pages realized performance gains, with the biggest gains on indexes with significantly more pages (greater than 50,000 pages)."

The default value in IndexOptimize is based on this.
I have a heap table (a table without a clustered index). How can I remove the fragmentation? You should not remove fragmentation on a heap table. The best practise is to create a clustered index on the table. Paul Randal has a blog post on this topic.
What do I have to think about to backup to a network share? Here are some things to think about.

1. Run the SQL Server service under a domain account.

2. Grant the SQL Server service account full control of the directory and the network share.

3. Backup to an UNC path.
E.g. @Directory = '\\Server1\Backup'
DatabaseBackup is not deleting old backup files. What could the problem be? It could be because of one of the following things.

1. Windows security. Check that the SQL Server service account has full control of the backup directory. If it's a network share, also check that the SQL Server service account has full control of the network share.

2. File system locking. It could for example be a backup or anti-virus software that is locking the file.

3. DatabaseBackup has been designed not to delete transaction log backups that are newer than the latest full or differential backup.
I'm getting this error message when I'm backing up a database with DatabaseBackup. Doesn't DatabaseBackup check if a differential backup can be done?

"Msg 3035, Level 16, State 10, Line 3"
"Cannot perform a differential backup for database "Database", because a current database backup does not exist. Perform a full database backup by reissuing BACKUP DATABASE, omitting the WITH DIFFERENTIAL option."
DatabaseBackup is checking sys.master_files.differential_base_lsn to see if a differential backup can be done. If it cannot be done it is doing one of two things. Default the database is skipped. A full backup can also be performed. This is if @ChangeBackupType = 'Y'.

Unfortunately this does not work if a database was upgraded since the full backup and the database was originally created on SQL Server 2000.
How can I tune performance of backup compression in SQL Server 2008? The Microsoft SQLCAT - team has an article about how to tune performance of backup compression in SQL Server 2008. You can tune performance by using multiple backup files and by using the options BUFFERCOUNT and MAXTRANSFERSIZE.
DatabaseBackup does backup to a directory with the SQL Server instance name, the database name and the backup type in the directory path. Could I change this? It's very easy to change, but you need to be very careful.

DatabaseBackup has been designed to only delete backup files if the backup and verify (if selected) was successful. It is then only deleting backups of the same instance, database and type. This means that you're guaranteed to always have the latest backup on disk.

If you remove any of this from the directory path this is no longer guaranteed.
I'm executing MaintenanceSolution.sql. The stored procedures and the function are created, but not the jobs. How could that be? This could be because of that you are using SQL Server Express. SQL Server Express does not have SQL Server Agent. Therefore the jobs are not created. You can use cmd - files and Windows Scheduled Tasks instead.
I'm getting the following error message when I'm executing the jobs.

"Unable to start execution of step 1 (reason: The job step contains tokens. For SQL Server 2005 Service Pack 1 or later, you must use the appropriate ESCAPE_xxx macro to update job steps containing tokens before the job can run.)"
The syntax for SQL Server Agent tokens has changed between SQL Server 2005 RTM and SQL Server 2005 SP1.

If you are upgrading from SQL Server 2005 RTM to a later service pack or version you need to update the jobs. Microsoft has a script that can help you with this.
I have read something about Service Pack requirements on SQL Server 2005. What Service Pack is required? The most recent version of the maintenance solution is supported on all versions and editions of SQL Server 2005, SQL Server 2008 and SQL Server 2008 R2. No Service Pack is required.
Is the maintenance solution using xp_cmdshell? The maintenance solution is not using xp_cmdshell. It works on an out of the box SQL Server 2005, SQL Server 2008 or SQL Server 2008 R2 installation.
Is the maintenance solution supported on SQL Server 2000? The maintenance solution is only supported and only works on SQL Server 2005, SQL Server 2008 and SQL Server 2008 R2.

I'm using a lot of things that were introduced in SQL Server 2005; catalog views, dynamic management views, TRY CATCH, ALTER INDEX and some more.
How should I schedule the jobs? This depends on your maintenance window, the size of the databases, the maximum data loss and many other things. Here are some guidelines that you can start with, but you will need to adjust it to your environment.

User databases: Full backup one day a week. Differential backup all other days of the week. Transaction log backup every hour. Integrity check one day a week. Index optimization one day a week.

System databases: Full backup every day. Integrity check one day a week.

Integrity check after index optimization. This is because index rebuilds sometimes can fix database corruption.

Full backup after index optimization. Then the differential backups will be small in the coming week.

Full backup after the integrity check. Then you know that the integrity of the backup is ok.
I would like to have a mail when a job fails. How can I achieve this? SQL Server Agent has built in support for sending mails when a job fails. You can read about SQL Server Agent Mail in Books Online. Please note that you should use Database Mail (SMTP) and that you need SQL Server 2005 SP1 or later if you're on a 64-bit version of SQL Server.
Isn't it easier just to use the SQL Server Maintenance Plans? The SQL Server Maintenance Plans are good in some situations, but it's not always what you need.

This maintenance solution is very reliable, easily deployable, has extensive logging and has the features that often are needed in an enterprise environment.
Is the maintenance solution free? The maintenance solution is free. You can use it in your database administration, in your hosting environment, in your consultancy work or distribute it with your software. You can read the license here.

If you would like to support this project you can do a donation through PayPal.
I'm a so called accidental DBA. How do I get started with your maintenance solution? It's easy to get started with my maintenance solution. I have made a Getting Started Guide that I recommend you to start with.
I'm using an old version of the maintenance solution. How do I upgrade to the latest version? It's easy to upgrade to the latest version of the maintenance solution. I have made an Upgrading Guide that I recommend you to use.

Vote for SQL Server features on Microsoft Connect

There are some SQL Server features that would be very useful in this solution if Microsoft would add them.
Vote for them on Microsoft Connect.
  • Rebuild an index partition online - 252433
  • Fill factor on the partition level - 242025
  • Statistics on the partition level - 328093
  • MAXDOP option in DBCC CHECKDB - 468694
  • CHECKCONSTRAINTS option in DBCC CHECKDB - 508837
  • DMV to check the percentage of a database that has changed since the last full backup - 511305
  • Continue SQL Server Agent T-SQL job step execution after an error - 244855
  • Include output files in SQL Server Agent Job failure notification - 293041