Sybase: Why does changing the locking scheme take so long ?

If you’ve already changed the locking scheme of a huge table from allpages locking (which was the locking scheme available in older versions of ASE) to datapages or datarows locking, you’ve noticed that it does take a long time and keeps you CPU pretty busy.

The reason is that when switching between allpages locking and data locking basically means copying the whole table and recreating the indexes.

Here are all the steps involved in such a change:

  • All rows of the table are copied into new data pages and formatted according to the new locking scheme.
  • All indexes are dropped and the re-created.
  • The old table is removed.
  • The information regarding this table in the system tables are updated.
  • The table counter is updated to force recompilation of query plans.

Note that this is also the reason why some people use this switching back and forth as a way to defragment a table.

The first two steps are the one taking the most time. It’s difficult to estimate the time required for this. But you can get an idea by checking the size of the data and indexes for this table. This can be done using sp_spaceused:

1> sp_spaceused report
2> go
 name   rowtotal reserved   data      index_size unused
 ------ -------- ---------- --------- ---------- --------
 report 1837164  8377528 KB 687468 KB 7676644 KB 13416 KB

(1 row affected)
(return status = 0)

It doesn’t tell you how much time is needed but if you do it on different table, you could assume the time needed is almost proportional to the size of data+indexes.

Note that switching between datapages and datarows locking schemes only updates system tables and is thus pretty fast.

Sybase: Access the database with C# using ODBC

I needed to write a very short C# program to access a Sybase ASE database and extract some information.

First had to download the appropriate version of ASE. It contains a directory called \archives\odbc. There is a setup.exe. Just run it.

Now there a new driver available:

create new data sourceThere is no need to add a data source to access ASE from an ODBC connection using C#. Just went there to check whether the driver was properly installed.

Then just create a program connecting to ASE using the following connection string:

Driver={Adaptive Server Enterprise};server=THE_HOSTNAME;port=2055;db=THE_DB_NAME;uid=sa;pwd=THE_SA_PASSWORD;

If you omit the db=… part, you’ll just land in the master database.

With this connection, you can then execute statements.

Here a sample code:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Data.Odbc;

namespace ConsoleApplication1
{
    class Program
    {
        static void Main(string[] args)
        {
            String errorMsg;
            OdbcConnection con = Connect("sa", "sa.pwd", "2055", "192.168.190.200", "mydb", out errorMsg);
            Console.WriteLine(errorMsg);
            if (con != null)
            {
                Console.WriteLine("In database {0}", con.Database);
                OdbcCommand command = con.CreateCommand();
                command.CommandText = "SELECT name FROM sysobjects WHERE type='U' ORDER BY name";
                OdbcDataReader reader = command.ExecuteReader();
                int fCount = reader.FieldCount;
                for (int i = 0; i < fCount; i++)
                {
                    String fName = reader.GetName(i);
                    Console.Write(fName + ":");
                }
                Console.WriteLine();
                while (reader.Read())
                {
                    for (int i = 0; i < fCount; i++)
                    {
                        String col = reader.GetValue(i).ToString();
                        Console.Write(col + ":");
                    }
                    Console.WriteLine();
                }
                reader.Close();
                command.Dispose();
                Close(con);
                Console.WriteLine("Press any key too continue...");
                Console.ReadLine();
            }
        }

        private static OdbcConnection Connect(String strUserName, String strPassword, String strPort, String strHostName, String dbName, out String strErrorMsg)
        {
            OdbcConnection con = null;
            strErrorMsg = String.Empty;
            try
            {
                String conString = "Driver={Adaptive Server Enterprise};server=" + strHostName + ";" + "port=" + strPort + ";db=" + dbName + ";uid=" + strUserName + ";pwd=" + strPassword + ";";
                con = new OdbcConnection(conString);
                con.Open();
            }
            catch (Exception exp)
            {
                con = null;
                strErrorMsg = exp.ToString();
            }

            return con;
        }

        private static void Close(OdbcConnection con)
        {
            con.Close();
        }
    }
}

That was really easy ! Not that it’d have been more difficult with Java or PHP, but I’d have expected to waste ours making mistakes and trying to debug it…

Sybase: Disable the transaction log

In some cases, you keep getting problems with a transaction log always filling up but do not need to be able to restore all data in a disaster recovery scenario (e.g. because you’re initially filling the database and if something goes wrong, you can just repeat the process or because it’s a test or development system and you do not care about the data).

Unfortunately, it is not possible to completely disable it. But you can make sure that the transaction log will be truncate on every checkpoint. The transaction log is then still there. It still costs you resources but it will be cleared at every checkpoint which will prevent it from filling up.

In order to have it truncated on checkpoint, use the following command:

exec master..sp_dboption mydb, 'trunc log on chkpt', true

Replace mydb by the name of the database on which you want to perform it.

To reenable the transaction log, just execute the following:

exec master..sp_dboption mydb, 'trunc log on chkpt', false

Sybase ASE: Get first or last day of previous, current or next month

Important: This post is about Sybase ASE. It will not work in iAnywhere.

If you want to provide some filter possibilities in your application showing the data stored for the previous, the current or the next month. So you basically need to figure out the first and last day of the corresponding month.
In case you cannot or do not want to do it in the application code itself, you can use simple SQL statements to get this info.

First you need to figure out the current date:

declare @today datetime
select @today=getdate()

Let’s start with the current month:

Today is January, 30th. So the first day of the month is January, 1st. Unfortunately, you cannot directly set the day to 1. But you can extract the day from the date, in our case 30 and 1 is 30 – (30 – 1), so:

declare @today datetime
select @today=getdate()
select dateadd(dd,-(day(@today)-1),@today)

This will return: Jan 1 2013 1:49PM

So basically we have the right date but for comparison purposes, we’d rather have midnight than 1:49pm. In order to do it, you need to convert it to a date string and back to a datetime:

declare @today datetime
select @today=getdate()
select convert(datetime, convert(varchar(10),dateadd(dd,-(day(@today)-1),@today),101))

Now we get: Jan 1 2013 12:00AM

if you’re only interested in a string containing the date, just drop the outer convert:

declare @today datetime
select @today=getdate()
select convert(varchar(10),dateadd(dd,-(day(@today)-1),@today),101)

Use another format than 101 if needed. The complete list of date conversion formats can be found here. For example, for the German date format, use 104 (dd.mm.yyyy).

Now let’s get the last day of the current month. This is basically the day before the first day of next month.

So first let’s get the first day of next month. This is actually just 1 month later than the first day of the current month:

declare @today datetime
select @today=getdate()
select convert(varchar(10),dateadd(mm,1,dateadd(dd,-(day(@today)-1),@today)),101)

This returns: 02/01/2013

Now let’s just substract one day and we’ll get the last day of the current month:

declare @today datetime
select @today=getdate()
select convert(varchar(10),dateadd(dd,-1,dateadd(mm,1,dateadd(dd,-(day(@today)-1),@today))),101)

This returns: 01/31/2013

Since we already have the first day of next month, let’s get the last day of next month. This is basically the same again but instead of adding 1 month, you add 2 months:

declare @today datetime
select @today=getdate()
select convert(varchar(10),dateadd(mm,2,dateadd(dd,-day(@today),@today)),101)

This returns: 02/28/2013

Now let’s tackle the previous month. The first day of last month is basically the first day of the current month minus 1 month:

declare @today datetime
select @today=getdate()
select convert(varchar(10),dateadd(mm,-1,dateadd(dd,-(day(@today)-1),@today)),101)

This returns: 12/01/2012

And then the last day of previous month. It is one day before the first day of the current month:

declare @today datetime
select @today=getdate()
select convert(varchar(10),dateadd(dd,-(day(@today)),@today),101)

This returns: 12/31/2012

MySQL: Compare two similar tables to find unmatched records

Sometimes you have two tables with similar structures and want to compare them. One could be the table in an older schema on an old server and the other one the newer schema on a new server. Let’s say you move data from one server to the other and want to check the data.

If you just want to compare two tables only once, you can go for a non-generic approach. Let’s assume that you have two tables (table A and table B) with a primary key called primary_key and two other columns (column1 and column2). What you want to get are:

  1. keys present in A but not in B
  2. keys present in B but not in A
  3. keys present in both A and B but where the other columns differ

The first part can be fetched like this:

SELECT A.primary_key FROM A LEFT JOIN B ON A.primary_key=B.primary_key WHERE B.primary_key IS NULL

The second part is basically the same but switching A and B:

SELECT B.primary_key FROM B LEFT JOIN A ON B.primary_key=A.primary_key WHERE A.primary_key IS NULL

The third part just involves an inner join and checking the other columns:

SELECT A.primary_key FROM A INNER JOIN B ON A.primary_key=B.primary_key WHERE A.column1 <> B.column1 OR A.column2 <> B.column2

An alternative is to get the following:

  1. entries in A which have no entry in B exactly matching
  2. keys present in B but not in A

The second part uses the same statement as the second part above. The first part can be fetched like this:

SELECT A.primary_key FROM A WHERE A.primary_key NOT IN (SELECT B.primary_key FROM B WHERE A.primary_key=B.primary_key AND A.column1=B.column1 and A.column2=B.column2)

This step basically combines the steps 1 and 3 in the first list.

Note that you can always replace a left join with a not in e.g. the following:

SELECT A.primary_key FROM A LEFT JOIN B ON A.primary_key=B.primary_key WHERE B.primary_key IS NULL

By:

SELECT A.primary_key FROM A WHERE A.primary_key NOT IN (SELECT B.primary_key FROM B WHERE A.primary_key=B.primary_key)

You just have to check what’s best from a performance point of view in you particular case.

If you do not have a primary key, you’ll have to do it using the algorithm described further down in the generic solution.

But if you often need to compare tables or need to compare many tables at once, then the following generic approach is probably better:

First since the two tables do not have the exact same columns, we’ll need to figure out what are the columns they have in common. To do this, we’ll simply read from INFORMATION_SCHEMA.COLUMNS.

A function to get the list of columns would look like this:

DELIMITER $$
DROP FUNCTION IF EXISTS getColumnList$$
CREATE FUNCTION getColumnList(schema1 VARCHAR(64), table1 VARCHAR(64)) RETURNS text
BEGIN 
	DECLARE column_name VARCHAR(64);
	DECLARE column_list TEXT DEFAULT '';
	DECLARE done INT DEFAULT 0;
	DECLARE cur CURSOR FOR SELECT c1.COLUMN_NAME FROM INFORMATION_SCHEMA.COLUMNS c1 WHERE c1.TABLE_SCHEMA=schema1 AND c1.TABLE_NAME=table1;
	DECLARE CONTINUE HANDLER FOR NOT FOUND SET done = 1;
	OPEN cur;
	read_loop: LOOP
    	FETCH cur INTO column_name;
    	IF done THEN
			LEAVE read_loop;
		END IF;
		IF column_list <> '' THEN
			SET column_list = CONCAT(column_list, ', ');
		END IF;
		SET column_list = CONCAT(column_list, '`', column_name, '`');
	END LOOP;
	CLOSE cur;
	RETURN column_list; 
END $$
DELIMITER ;

You can all the function like this:

SELECT getColumnList('phpBugTracker', 'comment');

Now we do not only want the columns of one table but of the two tables. It’s just a matter of joining INFORMATION_SCHEMA.COLUMNS with itself:

DELIMITER $$
DROP FUNCTION IF EXISTS getCommonColumnList$$
CREATE FUNCTION getCommonColumnList(schema1 VARCHAR(64), table1 VARCHAR(64), schema2 VARCHAR(64), table2 VARCHAR(64)) RETURNS text
BEGIN 
	DECLARE column_name VARCHAR(64);
	DECLARE column_list TEXT DEFAULT '';
	DECLARE done INT DEFAULT 0;
	DECLARE cur CURSOR FOR SELECT c1.COLUMN_NAME FROM INFORMATION_SCHEMA.COLUMNS c1 INNER JOIN INFORMATION_SCHEMA.COLUMNS c2 ON c1.COLUMN_NAME=c2.COLUMN_NAME WHERE c1.TABLE_SCHEMA=schema1 AND c1.TABLE_NAME=table1 AND c2.TABLE_SCHEMA=schema1 AND c2.TABLE_NAME=table2;
	DECLARE CONTINUE HANDLER FOR NOT FOUND SET done = 1;
	OPEN cur;
	read_loop: LOOP
    	FETCH cur INTO column_name;
    	IF done THEN
			LEAVE read_loop;
		END IF;
		IF column_list <> '' THEN
			SET column_list = CONCAT(column_list, ', ');
		END IF;
		SET column_list = CONCAT(column_list, '`', column_name, '`');
	END LOOP;
	CLOSE cur;
	RETURN column_list; 
END $$
DELIMITER ;

This one takes 4 parameters instead of 2:

SELECT getCommonColumnList('phpBugTracker', 'comment', 'phpBugTracker', 'comment_copy');

Now we know the columns, we just need an algorithm to compare these columns in the two tables. We’ll use the following:

SELECT MIN (TableName) AS TableName, column_list
FROM
 (
  SELECT 'source_table ' as TableName, column_list
  FROM source_table
  UNION ALL
  SELECT 'destination_table' as TableName, column_list
  FROM destination_table
)  AS tmp
GROUP BY column_list
HAVING COUNT (*) = 1

Now we need to create a function with parameters, calling our previous function to get the relevant column names, creating this statement and executing it:

DELIMITER $$
DROP PROCEDURE IF EXISTS compareTables$$
CREATE PROCEDURE compareTables(schema1 VARCHAR(64), table1 VARCHAR(64), schema2 VARCHAR(64), table2 VARCHAR(64))
BEGIN 
	SET @columns=getCommonColumnList(schema1, table1, schema2, table2);
	SET @st = CONCAT("SELECT MIN(TableName) as TableName, ",@columns," FROM ( SELECT '", schema1,".", table1,"' as TableName, ",@columns," FROM ", schema1,".", table1," UNION ALL SELECT '", schema2,".", table2,"' as TableName, ",@columns," FROM ", schema2,".", table2," ) tmp GROUP BY ",@columns," HAVING COUNT(*) = 1");
	PREPARE stmt FROM @st;
	EXECUTE stmt;
END $$
DELIMITER ;

And you call it like this:

CALL compareTables('phpBugTracker', 'comment', 'phpBugTracker', 'comment_copy');

Of course you could also adapt it to store the results in a temporary table…

MySQLBuddy 1.0 released

Since I’m not only using Sybase at work but also MySQL, I finally took the time to create a clone of SybaseBuddy for MySQL.

This new tool is called MySQLBuddy. Since both tools mainly provide the same functionality, they share most of their code. Of course at some point in time, I’ll put my software architect hat back on and will refactor the tools so that I have one tool supporting multiple databases. Unfortunately, I currently lack the time for this and will thus for the time being maintain both tools separately.

Sybase: List all tables with a foreign key to a specific table

If you want to know which tables are all referencing a specific table using a foreign key, you can use one the following system tables:

  1. sysconstraints and sysreferences
  2. syskeys

The first way will only work if you have constraints backing the foreign keys. If you have just foreign keys without any referential constraints, you will not be able to see anything.

Here’s how you can use the syskeys system table to get information about foreign keys:

First the following columns of syskeys are relevant to us:

  • id: it’s the ID of the table having the foreign/primary/common key.
  • type: it tells you whether it’s a primary, foreign or common key. In our case we’re only interested in foreign keys, so type=2.
  • keycnt: since this table contains 8 columns with column IDs, it’s useful to know how many of them actually contain something.
  • depid: it is the ID of the table we want to find foreign keys for.
  • key1 to key8: these are the column IDs in the referencing table.
  • depkey1 to depkey8: these are the column IDs in the referenced table.

Let’s say we want to find all tables with a foreign key to a table called ‘report’. We’d then have the following WHERE clause:

select ... from	syskeys k
where k.type = 2 and k.depid = object_id('report')

Meaning: look for foreign keys (type=2) referecing the report table(depid=).

Now we want to get the name of the referencing table:

select object_name(k.id)

The names of the referencing columns:

select	object_name(k.id),
		col_name(k.depid, depkey1)
		+', '+col_name(k.depid, depkey2)
		+', '+col_name(k.depid, depkey3)
		+', '+col_name(k.depid, depkey4)
		+', '+col_name(k.depid, depkey5)
		+', '+col_name(k.depid, depkey6)
		+', '+col_name(k.depid, depkey7)
		+', '+col_name(k.depid, depkey8)
from	syskeys k
where	k.type = 2
and		k.depid = object_id('report')

You’ll notice the output looks ugly because:

  • object_name returns a huge string with many trailing blanks
  • the list of columns looks like key, , , , , , ,

In order to make it look nicer, we’ll need to trim the strings for trailing spaces, limit the length of the individual strings to 30 characters and remove the extra commas.

For the extra commas, we can use syskeys.keycnt which will tell us how many depkeyX columns are filled. And we can use the fact that substring(‘blabla’, X, Y) will return NULL if X is less than 1. This means we need a function which returns 1 if the column index is less than keycnt and returns 0 or a negative number otherwise. Luckily such a function does exist: sign. It will return -1 if the argument is negative, 0 if it’s 0 and 1 if it’s positive. So we can use sign(keycnt – columnIndex + 1). If there are 3 columns:

  • sign(3 – 1 +1) = 1
  • sign(3 – 2 +1) = 0
  • sign(3 – 3 +1) = -1
  • sign(3 – 4 +1) = -1
  • sign(3 – 5 +1) = -1
  • sign(3 – 6 +1) = -1
  • sign(3 – 7 +1) = -1
  • sign(3 – 8 +1) = -1

So we end up with something like this:

select	rtrim(substring(object_name(k.id), 1, 30)),
		rtrim(substring(col_name(k.depid, depkey1),sign(keycnt),30))
		+rtrim(substring(', '+col_name(k.depid, depkey2),sign(keycnt-1),30))
		+rtrim(substring(', '+col_name(k.depid, depkey3),sign(keycnt-2),30))
		+rtrim(substring(', '+col_name(k.depid, depkey4),sign(keycnt-3),30))
		+rtrim(substring(', '+col_name(k.depid, depkey5),sign(keycnt-4),30))
		+rtrim(substring(', '+col_name(k.depid, depkey6),sign(keycnt-5),30))
		+rtrim(substring(', '+col_name(k.depid, depkey7),sign(keycnt-6),30))
		+rtrim(substring(', '+col_name(k.depid, depkey8),sign(keycnt-7),30))
		from	syskeys k
where	k.type = 2
and		k.depid = object_id('report')

And if we also want to have the column names of the referenced table:

select	rtrim(substring(object_name(k.id), 1, 30)),
		rtrim(substring(col_name(k.depid, depkey1),sign(keycnt),30))
		+rtrim(substring(', '+col_name(k.depid, depkey2),sign(keycnt-1),30))
		+rtrim(substring(', '+col_name(k.depid, depkey3),sign(keycnt-2),30))
		+rtrim(substring(', '+col_name(k.depid, depkey4),sign(keycnt-3),30))
		+rtrim(substring(', '+col_name(k.depid, depkey5),sign(keycnt-4),30))
		+rtrim(substring(', '+col_name(k.depid, depkey6),sign(keycnt-5),30))
		+rtrim(substring(', '+col_name(k.depid, depkey7),sign(keycnt-6),30))
		+rtrim(substring(', '+col_name(k.depid, depkey8),sign(keycnt-7),30)),
		rtrim(substring(col_name(k.id, key1),sign(keycnt),30))
		+rtrim(substring(', '+col_name(k.id, key2),sign(keycnt-1),30))
		+rtrim(substring(', '+col_name(k.id, key3),sign(keycnt-2),30))
		+rtrim(substring(', '+col_name(k.id, key4),sign(keycnt-3),30))
		+rtrim(substring(', '+col_name(k.id, key5),sign(keycnt-4),30))
		+rtrim(substring(', '+col_name(k.id, key6),sign(keycnt-5),30))
		+rtrim(substring(', '+col_name(k.id, key7),sign(keycnt-6),30))
		+rtrim(substring(', '+col_name(k.id, key8),sign(keycnt-7),30))
from	syskeys k
where	k.type = 2
and		k.depid = object_id('report')

Sybase: Incorrect syntax near ‘go’

If you execute a bunch of SQL statement from a shell script (e.g. to create tables/procedures) and get the following error message:

Msg 102, Level 15, State 1:
Server ‘SYBASE’, Procedure ‘xxxxxx’, Line xx:
Incorrect syntax near ‘go’.

But it all works fine when executing it in an SQL client (like SqlDbx or ASEIsql), the problem is probably related to the end of line. If you create the SQL file executed by the script on a Windows machine, it will end the lines with Carriage Return and Line Feed (CR+LF) and on a Linux/Unix system, it should only be a Line Feed (LF). After converting the file to Unix format, it should not see the error anymore.

Under Linux, you can convert the file using:

# tr -d '\r' < inputfile > outputfile

or:

# sed -e 's/\r$//' inputfile > outputfile

or:

# dos2unix inputfile

Sybase: LOBs (Text and image columns) are less evil in ASE 15.7

Working with LOBs in ASE has always been kind of a pain. There are not only multiple restrictions:

  • They cannot be passed as parameters to stored procedures.
  • You cannot define a local variable of type image or text, so you could only work with them through temporary tables.
  • They cannot be used in group by, order by and union (this is not a problem for image columns but is for text columns).
  • You cannot use isnull with them.
  • They cannot be used in an index (although I doubt there are real scenarios where it would be needed).
  • They cannot be used in subqueries and joins (although I doubt there are real scenarios where it would be needed).
  • You can only use them in a where clause using like. This is usually not a problem for image columns but is definitely for text columns. And even for image columns being able to check whether they are null
  • You cannot append data to it, so there if you gather data and want to create a long string out of it, your only solutions are to use a varchar(16384) or to do it outside of SQL e.g. in a perl script.
  • You have to use special commands to write to them.

but it was also very inefficient from a storage point of view:

If any LOB (text/image) column on a row doesn’t contain NULL, at least one data page will be allocated per row for the LOBs. This means that if you have an empty string in in a text column for all rows, it will eat up one logical page (2K, 4K… whatever you have configured) per row although it actually doesn’t store any real data there. So with a page size of 4K and 1 million such entries, you’re using up 4 Gigabytes of disk space for nothing.

Now, we have a good and a bad news.

Let’s start with the bad news: I haven’t seen any changes to the restrictions listed above.

The good news: ASE 15.7 brings in two major improvements regarding the storage of LOBs:

  1. Compression: ASE 15.7 supports in-database LOB compression. It supports FastLZ and ZLib compression. The LOB compression can be configured at database, table or column level.
  2. In-Row LOBs: LOBs can be inlined i.e. when possible they are stored in the parent row. This is especially very useful in such scenarios where the LOBs mostly contain less than a logical page worth of data. A nice side effect is also that accessing this small LOBs is also faster since you have less I/O overhead when retrieving them. And the great part is that in-rowing and out-rowing is done seamlessly !

These two changes will not turn me into a fan of LOBs but it will at least make storing LOBs come closer to storing on disk from a storage space point of view.

Sybase: show active trace flags

If you need to know which trace flags are active, you can use the dbcc traceflags command. Before using it, you show switch on the trace flag 3604 to display the output to standard output (i.e. the console).

1> use master
2> go
1> dbcc traceon(3604)
2> go
DBCC execution completed. If DBCC printed error messages, contact a user with System Administrator (SA) role.
1> dbcc traceflags
2> go
Active traceflags: 3604, 7717

DBCC execution completed. If DBCC printed error messages, contact a user with System Administrator (SA) role.

If you do not switch on the trace flag 3604, you’ll see the following:

1> dbcc traceflags
2> go
DBCC execution completed. If DBCC printed error messages, contact a user with System Administrator (SA) role.

Instead of trace flag 3604 you can also use the flag 3605. The ouput will then be writte to the error log:

1> use master
2> go
1> dbcc traceon (3605)
2> go
DBCC execution completed. If DBCC printed error messages, contact a user with System Administrator (SA) role.
1> dbcc traceflags
2> go
DBCC execution completed. If DBCC printed error messages, contact a user with System Administrator (SA) role.

And in the error log:

00:00:00000:00027:2012/10/22 17:19:41.28 server  DBCC TRACEON 3605, SPID 27
Active traceflags: 3604, 3605, 7717

You’ll notice that it also returns 3604 although it is not active in our session (the ouput wasn’t written to the console). Actually trace 3604 is a global trace. Switching it off in another session will also disable it in this session. But switching it on in another session, will not have the output displayed on the console for this session. No clue why…

Please also note that you can switch on the trace flags only for a session using the following:

1> set switch on 302
2> go
Switch 302 ('print_plan_index_selection') is turned on.
All supplied switches are successfully turned on.

This will switch on the trace flag for this session but it will not be visible with dbcc traceflags (even in this session):

1> dbcc traceflags
2> go
Active traceflags: 3604, 7717

DBCC execution completed. If DBCC printed error messages, contact a user with System Administrator (SA) role.