Friday, April 14, 2017

You cannot do an online rebuild of a clustered index if the table contains any LOB data (text, ntext, image, varchar(max), nvarchar(max), varbinary(max))



My rebuild index job (With ONLINE-Enterprise edition only) failed with error 




An online operation cannot be performed for %S_MSG '%.*ls' because the index contains column '%.*ls' 
of data type text, ntext, image or FILESTREAM. 
For a non-clustered index, the column could be an include column of the index. For a clustered index,
 the column could be any column of the table.
  If DROP_EXISTING is used, the column could be part of a new or old index. The operation must be performed offline.




select * from sys.messages 
where message_id=2725 and language_id=1033

You cannot do an online rebuild of a clustered index if the table contains any LOB data (text, ntext, image, varchar(max), nvarchar(max), varbinary(max))



Solution to search the Table with LOB data (text, ntext, image, varchar(max), nvarchar(max), varbinary(max))
and for that particular db, modify the rebuild indexes job not to online rebuild..


SELECT o.[name], o.[object_id ], c.[object_id ], c.[name], t.[name]
FROM sys.all_columns c
INNER JOIN sys.all_objects o
ON c.object_id = o.object_id
INNER JOIN sys.types t
ON c.system_type_id = t.system_type_id 
WHERE c.system_type_id IN (35, 165, 99, 34, 173)
AND o.[name] NOT LIKE 'sys%'
AND o.[name] <> 'dtproperties'
AND o.[type] = 'U'
GO

Wednesday, April 12, 2017

Will the linked server honor the NOLOCK hint?

Will the linked server honor the NOLOCK hint? -No
Ex:
INSERT INTO  CallHistory
SELECT *  FROM LINKSERVER.DATABASE.DBO.CallHistory WITH(NOLOCK)WHERE callplacedtimeUTC >= DATEADD(hh,-4,GETUTCDATE())  ORDER BY callplacedtimeUTC DESC
linked server do not  honor the NOLOCKso Create a view like below 
 Create view test
as
SELECT *  FROM  DATABASE.DBO.CallHistory WITH(NOLOCK)WHERE callplacedtimeUTC >= DATEADD(hh,-4,GETUTCDATE()) 

and then Finally:INSERT INTO  CallHistory
SELECT *  FROM LINKSERVER.DATABASE.DBO.test  --view  ORDER BY callplacedtimeUTC DESC

converting local time to UTC time With daylight Savings

I have a job to get data from a table with 4 hours data..my column is datetime and UTC default. and i prepared some logic below...
SELECT *  FROM   CallHistory WITH(NOLOCK)
WHERE callplacedtimeUTC >=  DATEADD(hh,-4,GETUTCDATE())
GO

But it dint work with daylight savings...so, my boss asked me to convert local time to UTC time which will work and gave me below logic...
Just  add below in place of -4

SELECT DATEADD(HOUR, -1 * DATEDIFF(HOUR, GETDATE(), GETUTCDATE()), GETUTCDAT


SELECT *  FROM  CallHistory WITH(NOLOCK)
WHERE callplacedtimeUTC >=  DATEADD(hh,-1 * DATEDIFF(HOUR, GETDATE(), GETUTCDATE()),GETUTCDATE())
GO


it works for daylight savings...

Thursday, March 16, 2017

Find HIGH CPU, IO, Blocking

--CPU usage
SELECT TOP 50
    [Average CPU used] = total_worker_time / qs.execution_count,
    [Total CPU used] = total_worker_time,
    [Execution count] = qs.execution_count,
    [Individual Query] = SUBSTRING(
                                      qt.text,
                                      qs.statement_start_offset / 2,
                                      (CASE
                                           WHEN qs.statement_end_offset = -1 THEN
                                               LEN(CONVERT(NVARCHAR(MAX), qt.text)) * 2
                                           ELSE
                                               qs.statement_end_offset
                                       END - qs.statement_start_offset
                                      ) / 2
                                  ),
    [Parent Query] = qt.text,
    DatabaseName = DB_NAME(qt.dbid)
FROM sys.dm_exec_query_stats qs
    CROSS APPLY sys.dm_exec_sql_text(qs.sql_handle) AS qt
ORDER BY [Average CPU used] DESC;

--IO
DECLARE @SystemIO FLOAT
SELECT @SystemIO = SUM(total_logical_reads + total_logical_writes)
FROM sys.dm_exec_query_stats;

SELECT TOP 20 [Row Number] = ROW_NUMBER() OVER (ORDER BY total_logical_reads + total_logical_writes DESC),
    [Query Text] = CASE
        WHEN [sql_handle] IS NULL THEN ' '
        ELSE (SUBSTRING(ST.TEXT,(QS.statement_start_offset + 2) / 2,
            (CASE
                WHEN QS.statement_end_offset = -1 THEN LEN(CONVERT(nvarchar(MAX),ST.text)) * 2
                ELSE QS.statement_end_offset
                END - QS.statement_start_offset) / 2))
        END,
    [Execution Count] = execution_count,
    [Total IO] = total_logical_reads + total_logical_writes,
    [Average IO] = (total_logical_reads + total_logical_writes) / (execution_count + 0.0),
    [System Percentage] = 100 * (total_logical_reads + total_logical_writes) / @SystemIO,
    [Object Name] = OBJECT_NAME(ST.objectid),
    [Total System IO] = @SystemIO,
    [SQL Handle] = [sql_handle]
FROM sys.dm_exec_query_stats QS
CROSS APPLY sys.dm_exec_sql_text ([sql_handle]) ST
WHERE total_logical_reads + total_logical_writes > 0
ORDER BY [Total IO] DESC



--Put this below in job and run for every 10 seconds
IF EXISTS (SELECT 1 FROM sys.sysprocesses WHERE blocked >0)
BEGIN

    INSERT INTO AuditDB.dbo.SYSprocess
    SELECT *
    FROM sys.sysprocesses
    WHERE blocked >0


    INSERT INTO AuditDB.dbo.SYSprocess_sp_who2
    EXEC sp_who2 'Active';

END;
ELSE
BEGIN
    PRINT 'No Blocking found' + ' at ' + CONVERT(VARCHAR(100), GETDATE());
END;






---read data

SELECT  * FROM AuditDB.dbo.SYSprocess_sp_who2  WHERE Blkby   NOT LIKE '%.%'

/****** Script for SelectTopNRows command from SSMS  ******/
SELECT
      *
  FROM [AuditDB].[dbo].[SYSprocess]
WHERE
--blocked>1
--AND lastwaittype   NOT LIKE '%LATCH_EX%'
--AND
last_batch BETWEEN '2017-03-16 10:00:05'  AND '2017-03-16 11:20:05'
ORDER BY login_time desc




 DECLARE @Handle varbinary(64);
SELECT @Handle =0x0300170020029172DA72F40033A6000001000000
FROM sys.dm_exec_requests  

SELECT * FROM sys.fn_get_sql(@Handle);

other way to caprture blocking is


SELECT * FROM sys.configurations
WHERE description LIKE '%blocked%'
GO


sp_configure 'show advanced options', 1;
GO
RECONFIGURE;
GO
sp_configure 'blocked process threshold', 2;
GO
RECONFIGURE;
GO

https://www.brentozar.com/sql/locking-and-blocking-in-sql-server/
----TRACE RUN
 sp_configure 'show advanced options', 1;
GO
RECONFIGURE;
GO
sp_configure 'blocked process threshold', 0;
GO
RECONFIGURE;
GO




Wednesday, February 1, 2017

Encryption demo updated

USE master
GO
CREATE DATABASE EncryptionDemo
GO
Use   EncryptionDemo
GO
--create a database master key (DMK) for the master database
CREATE MASTER KEY ENCRYPTION BY PASSWORD = 'F#4*KN^7$';
GO

--create a certificate for use as the database encryption key   (DEK) protector and is protected by the DMK.
CREATE CERTIFICATE CertificateforColumns
WITH SUBJECT = 'Certificate for Column Level encryption';
GO


--Create a symmetric key and encrypt it by using above created   certificate
CREATE SYMMETRIC KEY   EncryptColumn
     WITH ALGORITHM   = AES_256 --Supported encryption algorithms are AES with 128-bit,   192?bit, or 256?bit keys or 3 Key Triple DES
     ENCRYPTION BY   CERTIFICATE CertificateforColumns;
GO


 --Create test table
CREATE TABLE dbo.Customer
(
CustomerID   INT IDENTITY PRIMARY KEY,
FirstName   VARCHAR(100),
LastName   VARCHAR(100),
CreditCardNumber   Varchar(128) NULL
)
GO
 INSERT INTO dbo.Customer(FirstName,   LastName, CreditCardNumber)
VALUES
('Nathan', 'Jones','1111-1111-1111-1111'),
('Mike', 'Mitchell','2222-2222-2222-2222'),
('Dan', 'Straton', '3333-3333-3333-3333')
GO

--view data
SELECT * from Customer
go


 ----FROM HERE Actuall Step Start
ALTER TABLE Customer
ADD NewCreditCardNumber Varbinary(256)
GO

--CHECK DATA
SELECT * from Customer


 --Must open a key to update data
OPEN SYMMETRIC KEY EncryptColumn DECRYPTION BY CERTIFICATE CertificateforColumns
UPDATE Customer
SET NewCreditCardNumber = ENCRYPTBYKEY(KEY_GUID('EncryptColumn'),CreditCardNumber)
GO

--CHECK DATA
SELECT * from Customer

SELECT CustomerID,FirstName,LastName,CreditCardNumber,NewCreditCardNumber, CONVERT(VARCHAR(50),DECRYPTBYKEY(NewCreditCardNumber)) AS DecryptNewCreditCardNumber
from Customer

CLOSE SYMMETRIC KEY   EncryptColumn;
GO



 --cannot view the Card details : as we did not open the key
 SELECT CustomerID,FirstName,LastName,CreditCardNumber,CONVERT(VARCHAR(50),DECRYPTBYKEY(NewCreditCardNumber)) AS  NewCreditCardNumber
from Customer



OPEN SYMMETRIC KEY EncryptColumn DECRYPTION BY CERTIFICATE CertificateforColumns
SELECT * FROM Customer---cannot view NewCreditCardNumber even if we open, we have to use DECRYPTBYKEY for the coulumn to view the data

SELECT CustomerID,FirstName,LastName,CreditCardNumber,NewCreditCardNumber, CONVERT(VARCHAR(50),DECRYPTBYKEY(NewCreditCardNumber)) AS DecryptedNewCreditCardNumber
from Customer






--Test INSERT:1
OPEN SYMMETRIC KEY EncryptColumn DECRYPTION BY CERTIFICATE CertificateforColumns

DECLARE @CreditCardNumber VARCHAR(128)
SET @CreditCardNumber ='123456789087'
INSERT INTO dbo.Customer(FirstName,   LastName, CreditCardNumber,NewCreditCardNumber)
VALUES ('Tom', 'Hanks',  @CreditCardNumber,EncryptByKey( Key_GUID('EncryptColumn'), CONVERT(varchar, @CreditCardNumber)))

SELECT CustomerID,FirstName,LastName, CreditCardNumber,NewCreditCardNumber,CONVERT(VARCHAR(50),DECRYPTBYKEY(NewCreditCardNumber)) AS DecryptSecondCol
from Customer

CLOSE SYMMETRIC KEY   EncryptColumn;
GO

--Test INSERT:2: This will have NUll as we closed the key and inserting the data
--: WE MUST HAVE THE KEY OPEN WHEN WE UPDATE OR INSERT DATA

DECLARE @CreditCardNumber VARCHAR(128)
SET @CreditCardNumber ='098-989-5423'
INSERT INTO dbo.Customer(FirstName,   LastName, CreditCardNumber,NewCreditCardNumber)
VALUES ('Brad', 'Pitt',  @CreditCardNumber,EncryptByKey( Key_GUID('EncryptColumn'), CONVERT(varchar, @CreditCardNumber)))

OPEN SYMMETRIC KEY EncryptColumn DECRYPTION BY CERTIFICATE CertificateforColumns

SELECT CustomerID,FirstName,LastName, CreditCardNumber,NewCreditCardNumber,CONVERT(VARCHAR(50),DECRYPTBYKEY(NewCreditCardNumber)) AS DecryptSecondCol
from Customer

CLOSE SYMMETRIC KEY   EncryptColumn;
GO



--Test INSERT:2: OPEN and INSERT AGAIN
--: WE MUST HAVE THE KEY OPEN WHEN WE UPDATE OR INSERT DATA

 OPEN SYMMETRIC KEY EncryptColumn DECRYPTION BY CERTIFICATE CertificateforColumns
DECLARE @CreditCardNumber VARCHAR(128)
SET @CreditCardNumber ='098-098-0098'
INSERT INTO dbo.Customer(FirstName,   LastName, CreditCardNumber,NewCreditCardNumber)
VALUES ('Morgan', 'Freeman',  @CreditCardNumber,EncryptByKey( Key_GUID('EncryptColumn'), CONVERT(varchar, @CreditCardNumber)))



SELECT CustomerID,FirstName,LastName, CreditCardNumber,NewCreditCardNumber,CONVERT(VARCHAR(50),DECRYPTBYKEY(NewCreditCardNumber)) AS DecryptSecondCol
from Customer

CLOSE SYMMETRIC KEY   EncryptColumn;
GO




Notes:
--Application must use EncryptByKey( Key_GUID('EncryptColumn'), CONVERT(varchar, @CreditCardNumber)) TO insert / Update Data

--need to check if any triggers update Encrypted Columns, also Jobs

--Replication Handles Encryption, we must eput the keys on Every subscribed server if we ant to view the data

-- WE WILL have to discuss what to do with Existing original CreditCardNumber ???

--To decrypt the encrypted data we need to use another inbuilt function called DecryptByKey

--When we try encrypting or decrypting data without opening the symmetric key, the command will not fail-- but it will not work

--It’s very important and essential to take a backup of the keys and certificates in order to restore or attach the encrypted database on another SQL Server instance after restoring these keys\certificates there.
--Please note, enabling encryption at cell level has overhead and performance penalties (as it is a resource intensive operation and salts the data differently, which causes a table scan) as discussed above and hence it’s recommended to first evaluate the need and
--then plan for its implementation.







--ALTER TABLE Customer
--DROP COLUMN CreditCardNumber;
--GO
--SELECT CustomerID,FirstName,LastName,  CONVERT(VARCHAR(50),DECRYPTBYKEY(NewCreditCardNumber)) AS NewCreditCardNumber
--from Customer

----rename back to original
--Sp_rename 'Customer.NewCreditCardNumber', 'CreditCardNumber'

--SELECT CustomerID,FirstName,LastName,  CONVERT(VARCHAR(50),DECRYPTBYKEY(CreditCardNumber)) AS CreditCardNumber
--from Customer

--SELECT CustomerID,FirstName,LastName, CreditCardNumber AS  CreditCardNumber, CONVERT(VARCHAR(50),DECRYPTBYKEY(CreditCardNumber)) AS DecryptedCreditCardNumber
--from Customer

Wednesday, January 11, 2017

SQL Server service crashes when you run an Oracle linked server query

Check event viewer for more info



The SQL Server (MSSQLSERVER) service terminated unexpectedly. It has done this 1 time(s). 


Example of query i ran 
SELECT fname , lname 
--ssn
from oraclelinkedserver.dbname.dbo.table

Cause
This issue occurs because the special characters "--" exist in the query to the Oracle linked server. These characters are used as a comment symbol.

The SQL Server process crashes because of the third-party linked server provider which is loaded inside SQL Server process. If the third-party linked server provider is enabled together with the Allow inprocess option, the SQL Server process crashes when this third-party linked server experiences internal problems.
Workaround
To workaround this issue, use one of the following methods:
Remove the comments symbol, or 
Replace  the comments symbol with the following comments symbol: 
/* */


this works;
SELECT fname , lname  
from oraclelinkedserver.dbname.dbo.table

Wednesday, October 26, 2016

DISK FULL cannot SHRNIk mdf

I would avoid shrinking if possible. It will cause fragmentation issues with any indexes you have and thus performance issues. It can also cause fragmentation in the file system.
If your in a crunch, add a second datafile in the same filegroup to a different volume. SQL Server will start putting new data in that new file. It will do this until the files are of equal size and then basically round robin going forward.
This should allow you to focus on averting a full drive until you can get a bigger disk.

Friday, September 23, 2016

JOB to run last day minus 1 day of the month



IF OBJECT_ID('tempdb.dbo.#CalendarView', 'U') IS NOT NULL
  DROP TABLE #CalendarView;
  go

 Create Table #CalendarView(CalendarDate DATE);
 GO

  DECLARE @StartDate DATE = '20160101', @NumberOfYears INT = 30;
  DECLARE @CutoffDate DATE = DATEADD(YEAR, @NumberOfYears, @StartDate);
 INSERT INTO #CalendarView(CalendarDate)
  SELECT d
FROM
(
  SELECT d = DATEADD(DAY, rn - 1, @StartDate)
  FROM
  (
    SELECT TOP (DATEDIFF(DAY, @StartDate, @CutoffDate))
      rn = ROW_NUMBER() OVER (ORDER BY s1.[object_id])
    FROM sys.all_objects AS s1
    CROSS JOIN sys.all_objects AS s2
    ORDER BY s1.[object_id]
  ) AS x
) AS y;
SELECT * from #CalendarView
go



--Create a job and keep it not with schedule.Now, execute your job in below query:
DECLARE @Today DATE = GETDATE();
DECLARE @FirstDayNextMonth DATE = DATEADD(MONTH, 1,
                                          DATEADD(DAY, 1 - DAY(@Today), @Today));
DECLARE @LastBut1DayThisMonth DATE= DATEADD(DD, -2,
                                            DATEADD(MM,
                                                    DATEDIFF(MM, 0,
                                                             CONVERT(VARCHAR, GETDATE(), 101))
                                                    + 1, 0));

--PRINT @Today;
--PRINT @FirstDayNextMonth;
--PRINT @LastBut1DayThisMonth;
SELECT  MAX(CalendarDate)
FROM    #CalendarView
WHERE   CalendarDate = @LastBut1DayThisMonth;
IF @LastBut1DayThisMonth = @Today
    BEGIN
        EXEC msdb.dbo.sp_start_job 'BACKUPTEST';--I have to create a Job which kicks SSIS package with no schedule
    END;

  

Audit Table SP Create alter drop Statements

CREATE TABLE AuditTableChanges (EventTime DATETIME, DatabaseName Varchar(100), EventName Varchar(1000), ObjectName Varchar(100), TSQLStatement Varchar(max), LoginName Varchar(100))
go
---CREATE_PROC, ALTER_PROC, DROP_PROC,

CREATE TRIGGER AuditTableChanges
ON DATABASE
FOR  CREATE_TABLE, ALTER_TABLE, DROP_TABLE
AS

DECLARE @ed XML
SET @ed = EVENTDATA()

INSERT INTO AuditTableChanges (EventTime, DatabaseName, EventName, ObjectName, TSQLStatement, LoginName)
VALUES
(
GetDate(),
@ed.value('(/EVENT_INSTANCE/DatabaseName)[1]', 'varchar(256)'),
@ed.value('(/EVENT_INSTANCE/EventType)[1]',' nvarchar(100)'),
@ed.value('(/EVENT_INSTANCE/ObjectName)[1]', 'varchar(256)'),
@ed.value('(/EVENT_INSTANCE/TSQLCommand)[1]', 'nvarchar(2000)'),
@ed.value('(/EVENT_INSTANCE/LoginName)[1]', 'varchar(256)')
) ;


SELECT * from AuditTableChanges


to send email notifications:
USE Db
GO

/****** Object:  DdlTrigger [TableAltered]    Script Date: 9/23/2016 1:29:35 PM ******/
SET ANSI_NULLS ON
GO

SET QUOTED_IDENTIFIER ON
GO






CREATE TRIGGER [TableAltered]
ON DATABASE
FOR  ALTER_TABLE

AS

DECLARE @bd VARCHAR(MAX)
DECLARE @tsql VARCHAR(MAX)
SET @tsql = EVENTDATA().value
        ('(/EVENT_INSTANCE/TSQLCommand/CommandText)[1]','varchar(max)')
SET @bd = 'UserName: ' + UPPER(SUSER_NAME()) + '

  ServerName: ' + @@SERVERNAME + '

  Time: ' + CONVERT(VARCHAR(25),GETDATE()) + '

  HostName: ' + HOST_NAME() + '

  Database: ' + DB_NAME() + '

  T-SQL: ' +  @tsql


BEGIN
PRINT 'Table has been Altered'

EXEC msdb.dbo.sp_send_dbmail @profile_name = 'DBMail',
@recipients = 'abc@xyz.com',
@subject = 'A new Table has been created!',
@body_format = 'HTML',
@importance = 'High',
@body = @bd
END









GO

ENABLE TRIGGER [TableAltered] ON DATABASE
GO


Thursday, September 1, 2016

Finding Implicit Column Conversions in the Plan Cache

 http://sqlblog.com/blogs/jonathan_kehayias/archive/2010/01/08/finding-implicit-column-conversions-in-the-plan-cache.aspx



http://www.sqlconsulting.com/news1210.htm

Good Article on Full Text Search , Change tracking on full textsearch

http://sql-articles.com/articles/dba/sql-server-2005-full-text-search/




Populating full text index:
Creating and maintaining a full-text index involves populating the index by using a process called a population (also known as a crawl). SQL Server supports the following types of population: full population, change tracking-based automatic or manual population, and incremental timestamp-based population.
Full population:
Firstly, whenever we create a full text index there will be a full population of the index for all the rows. To avoid it we need to use the option CHANGE_TRACKING OFF, NO POPULATION. Those options will turn off change tracking and will prevent populating the full text index while FTX creation. Since the Full population consumes a lot of resources we need to do them at offproduction hours. During a full population, index entries are built for all the rows of a table or indexed view. A full population of a full-text index builds index entries for all the rows of the base table or indexed view.
Change tracking based population: There are 2 types
a.) Using Automatic whenever an insert/update or delete happens to the full text index there will be a population of those rows alone (automatically)
b.) Using Manual whenever DML operations happen it will not get propagated automatically to the FTX. Hence we need to use automatic population so that whenever a change happens in the text field it will immediately be available in the full text index i.e. it will be populated instantaneously. If we use manual population we need to run the command using a job to populate the changes alone manually.
The overhead involved here is the SQL Server will maintain a table to track the list of tables and rows modified.
Incremental population: We need to have a timestamp column in the table in order to make use of Incremental population. This will populate only the rows that have modified since the previous incremental population. This method is highly suitable for tables which change frequently.
The time delay for the full text index data to be available while searching after getting populated is close to 1 to 2 minutes maximum. Without getting populated it will not be available.

https://blog.sqlauthority.com/2009/06/27/sql-server-fix-error-17892-logon-failed-for-login-due-to-trigger-execution-changed-database-context...