Jeroen Mostert

The Perils and Pitfalls of SqlBulkCopy (part 2)

2019-03-27T00:00:00+00:00

When we last saw our (peculiarly anonymous) heroes, they had just managed to do a very basic bulk import using SqlBulkCopy, despite some initial hurdles figuring out the finer points. As a reminder, here’s the code we ended up with:

var dt = new DataTable();
dt.Columns.Add("ID");
dt.Columns.Add("Name");
dt.Columns.Add("WeightInGrams");
foreach (var widget in ReadFromFile("widgets.txt")) {
    var row = dt.NewRow();
    row["ID"] = widget.Id;
    row["Name"] = widget.Name;
    row["WeightInGrams"] = widget.WeightInGrams;
    dt.Rows.Add(row);
}

using (var connection = new SqlConnection("Data Source=.;Initial Catalog=Widgets;Integrated Security=SSPI")) {
    connection.Open();
    using (var sqlBulkCopy = new SqlBulkCopy(connection, SqlBulkCopyOptions.KeepIdentity, null)) {
        foreach (DataColumn column in dt.Columns) {
            sqlBulkCopy.ColumnMappings.Add(column.ColumnName, column.ColumnName);
        }
        sqlBulkCopy.DestinationTableName = "Widgets";
        sqlBulkCopy.WriteToServer(dt);
    }
}

This code works fine for a file containing 4 rows. What about a file with 10 million rows? That’s where it gets a little more interesting. First, let’s make a little script for generating such a file:

var r = new Random(42);
using (var f = File.OpenWrite(@"superwidgets.txt"))
using (var w = new StreamWriter(f)) {
    int id = 1;
    for (int i = 0; i != 10_000_000; ++i) {
        string name = "azultronic radiator " + i;
        string weight = "2.60";
        w.WriteLine($"{id}\t{name}\t{weight}");
        id += r.Next(5) + 1;
    }
}

That’s a lot of azultronic radiators, which may not be completely representative of our widget inventory, but it’s a good start.

So how does our little bulk insert program fare with this data set? If we add some very simple performance logging with Stopwatch and calls to GC.GetTotalMemory()/GC.GetAllocatedBytesForCurrentThread(), I get the following:

Reading rows     : 32.06 s
Memory (total)   : 7 102 941 056
Memory (current) : 3 417 498 648

Unhandled Exception: System.Data.SqlClient.SqlException: Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding. —> System.ComponentModel.Win32Exception: The wait operation timed out.

Defaults strike again! .BulkCopyTimeout starts off at 30 seconds. A reasonable value for run of the mill SQL commands, but rather arbitrary for bulk inserts, which can easily take longer, of course – we’re usually more interested in whether the copy is making sufficient progress, not how long it takes in total. No fear, we can set the timeout to 0, indicating we’re willing to wait for however long it takes. So let’s do that and try again.

Reading rows     : 33.34 s
Memory (total)   : 7 102 942 536
Memory (current) : 3 417 510 576
Inserting rows   : 47.96 s
Total time       : 86.88 s

The exact numbers shouldn’t concern us a lot, here – and I’m intentionally not mentioning my machine specs, so as to not give the impression this is a representative benchmark. Suffice it to say it has a lot of memory, the disks are all SSD and the database files have been pre-sized, so this is pretty much a best case scenario.

What’s immediately striking is that we spend 30 seconds and a whopping 7 GB in allocated memory (of which 3.5 GB still present at the end) just reading the widgets, before anything is sent to SQL Server at all. This is a direct consequence of the fact that we’re using DataTable, which, by virtue of having to be able to store rows of any format, is quite the memory hog. There isn’t a whole lot we can do, here. We can reduce the memory use by about 1 GB by specifying the types of the columns in advance:

dt.Columns.Add("ID", typeof(int));
dt.Columns.Add("Name", typeof(string));
dt.Columns.Add("WeightInGrams", typeof(int));

Result:

Reading rows     : 25.15 s
Memory (total)   : 6 123 159 104
Memory (current) : 2 567 739 576

A measurable improvement, but not an order of magnitude. For that, we’ll have to ditch DataTable altogether. Recall that .WriteToServer had more overloads. The one we should be interested in is this one:

WriteToServer(IDataReader)

IDataReader is a bit of a weird interface: it implements IDataRecord, which represents a single record of data, and adds .Read() to actually advance to the next record. SqlDataReader ultimately implements this to represent row sets read from a SQL Server.

The key insight here is that IDataReader never has to make more than one row at a time available – in effect it’s just like IEnumerable, but with the ability to address columns individually. In fact, we can just make an implementation that wraps IEnumerable:

class ObjectReader<T> : IDataReader {
    private readonly IEnumerator<T> sequence;
    private readonly PropertyInfo[] properties;
    private readonly Dictionary<string, PropertyInfo> propertiesByName;

    public ObjectReader(IEnumerable<T> sequence) {
        this.sequence = sequence.GetEnumerator();
        this.properties = typeof(T).GetProperties();
        this.propertiesByName = properties.ToDictionary(p => p.Name, p => p);
    }

    public object this[int i] => properties[i].GetValue(sequence.Current);
    public object this[string name] => propertiesByName[name].GetValue(sequence.Current);

    public int Depth => 0;
    public bool IsClosed { get; private set; }
    public int RecordsAffected => 0;
    public int FieldCount => properties.Length;

    public void Close() => IsClosed = true;

    public void Dispose() => sequence.Dispose();

    public bool GetBoolean(int i) => (bool) this[i];
    public byte GetByte(int i) => (byte) this[i];
    // and many more like this...

    public string GetDataTypeName(int i) => this[i].GetType().Name;
    public Type GetFieldType(int i) => this[i].GetType();
    public string GetName(int i) => properties[i].Name;
    public int GetOrdinal(string name) => Array.FindIndex(properties, p => p.Name == name);
    public object GetValue(int i) => this[i];

    public int GetValues(object[] values) {
        for (int i = 0; i != properties.Length; ++i) {
            values[i] = this[i];
        }
        return properties.Length;
    }

    public bool IsDBNull(int i) => this[i] == null || this[i] == DBNull.Value;
    public bool NextResult() => false;
    public bool Read() => sequence.MoveNext();
}

This is pretty obviously not the safest or most efficient implementation of this concept, but ignore that for the moment. We can now change our program to:

using (var objectReader = new ObjectReader<Widget>(ReadFromFile("superwidgets.txt")))
using (var connection = new SqlConnection("Data Source=.;Initial Catalog=Widgets;Integrated Security=SSPI")) {
    connection.Open();
    using (var sqlBulkCopy = new SqlBulkCopy(connection, SqlBulkCopyOptions.KeepIdentity, null)) {
        foreach (string name in typeof(Widget).GetProperties().Select(p => p.Name)) {
            sqlBulkCopy.ColumnMappings.Add(name, name);
        }
        sqlBulkCopy.DestinationTableName = "Widgets";
        sqlBulkCopy.BulkCopyTimeout = 0;
        sqlBulkCopy.WriteToServer(objectReader);
    }
}

And now our performance characteristics look quite different:

Memory (total)   : 4 157 254 328
Memory (current) : 160 840
Inserting rows   : 48.17
Total time       : 48.17

There is no more time for “reading rows”, since the reading is now part of the copy process, and we also make a lot less demand on memory.

So, is this it? Is this the fastest possible way to fill the table? Not by a long shot, but that wasn’t really the point – the point was to show that SqlBulkCopy could do a lot more to give you a good experience out of the box. There is still more it could be doing, and we’ll look at that in part 3. Spoilers: this is going to involve changing code in the framework itself, so it’ll be quite fun!

Stroustrup on simplifying things

2019-03-07T00:00:00+00:00

Quoth Bjarne Stroustrup, creator of C++, in a 2006 interview with MIT Technology Review, answering the implied question of whether simplifying the process of coding to enable more wide-spread participation was a good idea:

The idea of programming as a semiskilled task, practiced by people with a few months’ training, is dangerous. We wouldn’t tolerate plumbers or accountants that poorly educated. We don’t have as an aim that architecture (of buildings) and engineering (of bridges and trains) should become more accessible to people with progressively less training. Indeed, one serious problem is that currently, too many software developers are undereducated and undertrained.

Obviously, we don’t want our tools–including our programming languages–to be more complex than necessary. But one aim should be to make tools that will serve skilled professionals–not to lower the level of expressiveness to serve people who can hardly understand the problems, let alone express solutions. We can and do build tools that make simple tasks simple for more people, but let’s not let most people loose on the infrastructure of our technical civilization or force the professionals to use only tools designed for amateurs.

He is right that the idea of programming as a semiskilled task is dangerous (insofar as we’re willing to uncritically accept the output, at least), but completely wrong when he willfully glosses over the nature of what makes programming a skilled task in the first place, and indirectly encourages a notion that’s equally dangerous (if not more so): that whatever passes for being qualified in the use of “professional tools” also serves as qualification for working on “the infrastructure of our technical civilization”. It’s certainly true that commercial business is transacted as if this were actually true – but that’s more for lack of anything better.

Telling here is the tentatively implied correlation between effort required to master “tools that will serve skilled professionals” and “education”. You’ll be hard pressed to find a plumber or accountant claiming that knowledge of all the ways to hold a pipe wrench, or how to write array formulas in Excel, was at the very heart of their professional education – but with programming we feel comfortable putting a great deal of it on “tools” (which, as Stroustrup notes, is not restricted to programming languages – as he puts it, “most applications involve standards, operating systems, libraries, and tools that far exceed modern programming languages in complexity”). There is a dangerous and fatal notion embedded here that doing complicated things requires complicated tools, or at the very least that those complicated things are more effectively done with complicated tools. Stroustrup conveniently conjures up a dichotomy between “tools designed for amateurs” and “tools that will serve skilled professionals” in such a way that any amount of complexity can be hand-waved away with the remark that it’s there because the tool was intended for professionals to do their job (as often expressed in the content-free but ever popular chestnuts of “the right tool for the right job” and “any tool can be misused”).

You might say I’m being unfair to Stroustrup, here – all he did, after all, was caution against the idea that oversimplifying things was the answer, and the popular idea that programming should be simpler so that more people can do it (or even “so simple that anyone can do it”) is admittedly more naive and misguided than the opposite idea that making things more complicated will improve matters. But Stroustrup’s way of framing this shows a misconception endemic to the “professional world”, I feel, and is more worthy of a rebuttal than the easier strawman of not oversimplifying things, or the trivial observation that programming is not, in fact, something that can be done casually by anyone.

If a sign of professionalism is the quality of work, we are forced to admit that professionals who encourage the use of complex tools are not less prone to making mistakes, or even just less prone to making easy mistakes. On the contrary, the very complexity they so admire encourages mistakes that are easy to make, but hard to solve, and even harder to diagnose in the first place. But these mistakes are tolerated and seen as the cost of doing business, because this fits the narrative of programming as a highly skilled profession suited only to the most exalted minds (or, at least, a profession which ought to be like that even if it currently isn’t). Programmers in love with complexity tend to appreciate hard to fix mistakes as problems they care about solving, to the detriment of the actual problems that the programs are intended to solve. The existence of hard to understand, self-inflicted problems becomes a self-fulfilling prophecy about how programming can only be entrusted to those willing to spend a lot of time on complexity and difficulty, “as things are meant to be”, rather than an impetus for investigating how the problems came to be in the first place, and whether they are avoidable, or at least admit effective mitigations. The mere fact that a tool takes a lot of time and effort to master proves neither that the tool itself is worth mastering, nor that the person doing so possesses the skills to make them effective in their profession – nor does an unwillingness or inability to master it necessarily prove the opposite. This is understandably hard to consider, since most professional programmers (including myself) happen to be people who have invested a lot of time and effort into mastering some tools and nobody wants to think that time might possibly have been better spent, but it’s nevertheless worth considering, if only to remain intellectually honest.

Programming is simply too young and too underdeveloped as a discipline for anyone to believe that we can put it on the same level as engineering bridges and trains – even putting it on the same level as plumbing seems rather charitable. Plumbers, at least, have a higher success rate with getting things right on the first try, and certainly far higher success rates if we allow for fixing mistakes when they don’t get it right on the first try. Their profession is also mature enough that they don’t seek to justify dissatisfaction with the work of amateur plumbers by the existence of overly simple tools, or spurious claims that professionals could be “forced” to use them. Programming has a long way to go yet before such reasoning is no longer seen as a valid defense of the frankly squalid state it’s in – which is as depressingly true as it will be in 2020 as it was in 1960.

Interlude: the Perils and Pitfalls of BULK INSERT

2019-02-21T00:00:00+00:00

As I’ve been ranting against SqlBulkCopy, here’s a brief interlude about how not everything is its fault.

Consider our little table again:

CREATE TABLE Widgets(
    [ID] INT IDENTITY PRIMARY KEY, 
    [Name] NVARCHAR(100), 
    [WeightInGrams] INT
)

Every bulk insert mechanism that’s not SqlBulkCopy offers a way to specify the order of the rows you’re inserting using the ORDER hint. The purpose of this is to speed up inserting rows into a table with a clustered index, which otherwise forces an intermediate sort step – which can be very, very costly when you’re inserting big batches.

Unfortunately, there’s one crucial scenario where this ORDER hint will do nothing: if the clustered index is on a column with the IDENTITY property (which it is, in the above table, since the primary key implicitly becomes the clustered index unless we specify otherwise). In this case, SQL Server will sort the incoming rows no matter what – it will not even give an error if you’re not supplying the rows in order, as you’re supposed to.

The irony here (if you can call it that) is that this sort will be omitted if you do not supply a value for the identity column. In this case, SQL Server will insert the rows any which way, and generate identity values on its own. That’s great if you’re inserting new values; not so much if you’re copying from an existing source.

It turns out this is a very old problem, that just happens to have not been fixed yet. There is a workaround, of sorts (it can’t be universally applied in all cases): ALTER TABLE SWITCH, which does not care about IDENTITY. We can create the following table:

CREATE TABLE _BulkWidgets(
    [ID] INT PRIMARY KEY, 
    [Name] NVARCHAR(100), 
    [WeightInGrams] INT
)

We can bulk insert into that without running into this bug (I think we can call it a bug) and then issue the following statement:

ALTER TABLE _BulkWidgets SWITCH TO Widgets;

And you probably want to follow that up with DBCC CHECKIDENT(Widgets, RESEED).

For this to work Widgets must be empty, so this doesn’t help if you want to bulk insert into a table that already contains data. If your table isn’t empty, you can still use this technique, but only if you’re willing to live without your data while the bulk insert is happening: you can ALTER TABLE Widgets SWITCH TO _BulkWidgets, bulk insert, then switch back. It’s possible to stuff all these operations (switch, bulk insert, switch) into a transaction, so clients don’t see an empty table, and the whole thing would be rolled back as one. Nevertheless, this still makes your data temporarily unavailable, so we can only hope that Microsoft sees fit to fix this.

OK, so it’s not that hard.

2019-02-19T00:00:00+00:00

You just follow the instructions and have a bit of patience.

The Perils and Pitfalls of SqlBulkCopy (part 1)

2019-02-19T00:00:00+00:00

Eric Lippert has written about the concept of the “Pit of Quality” – designing a language so the rules guide developers towards writing correct code, and actively make it harder to write incorrect code.

SqlBulkCopy is a class with a lot of design decisions that seem tailor made to prevent developers from falling into the Pit of Quality, and produce decidedly worse results instead. This matters especially because, by its very nature, SqlBulkCopy is intended to expose an efficient mechanism for loading lots of data into SQL Server, and yet it seems designed to make sure it actually loads data in the slowest way possible, by virtue of its defaults – and before you even get to loading the data, there’s plenty of pits to fall in.

I should be fair and point out that some of this can be partially attributed to the defaults picked by the bulk insert mechanism of SQL Server itself. Nevertheless, it’s SqlBulkCopy which chooses to expose these defaults as-is, instead of choosing more sensible ones. This is a kind of “laziness” you often see when developers are tasked with writing classes that wrap functionality already present in another system. In some cases, there is something to say for keeping the wrapper simple and one-to-one, so developers can use the existing documentation and knowledge and map this easily onto the new way. I don’t think that applies to SqlBulkCopy, however, and there’s a better design struggling to get out.

So in which way does SqlBulkCopy fail? First, let’s set up a very simple bulk load scenario, which is admittedly simplified but not unrepresentative of what a developer would be tasked to do. Let’s say we have a table Widgets as follows:

CREATE TABLE Widgets(
    [ID] INT IDENTITY PRIMARY KEY, 
    [Name] NVARCHAR(100), 
    [WeightInGrams] INT
)

Let’s also say we’ve got a file that contains a load of widget definitions from elsewhere that looks like this:

azultronic radiator	2.60
boson catcher	0.43
nuclear accelerator	10.0
screw	0.045
...

In this file, the columns are separated by tabs, and the weight is sneakily given in kilograms, rather than grams, just so our application has something to do and you couldn’t just import this with a wizard. The ID has been given to us from another database, and even though it’s an identity, we’d like to preserve it because, for better or worse, we’ve come to depend on these values. (Isn’t that just how these things go?)

There are several ways of tackling this issue that don’t involve writing any .NET code at all:

We could import the file using bcp, BULK INSERT or OPENROWSET(BULK, ...) and write some T-SQL code to fix up the weight.
We could use an SSIS package to perform the necessary load/transformation option.
We could use some other third party ETL tool, like Pentaho’s Kettle.

These solutions all have their pros and cons, and for a simple scenario like this, it probably doesn’t matter what you do: pick whatever solution you can develop the quickest, depending on what you have the most experience with. For the sake of argument, let’s say I’ve decided to give pure .NET a go, because I’m unwilling or unable to port my existing logic to a completely new environment.

Let’s first cook up some code to read the file. I’m going to use the simplest thing that could possibly work, as that’s usually a nice place to start…

class Widget {
    public int Id { get; set; }
    public string Name { get; set; }
    public int WeightInGrams { get; set; }
}

IEnumerable<Widget> ReadFromFile(string fileName) {
    foreach (var line in File.ReadLines(fileName)) {
        string[] fields = line.Split('\t');
        yield return new Widget {
            Id = int.Parse(fields[0], CultureInfo.InvariantCulture),
            Name = fields[1],
            WeightInGrams = (int) (decimal.Parse(fields[2], CultureInfo.InvariantCulture) * 1000)
        };
    }
}

This code is quite naive, having no way to nicely handle input errors, but let’s look past that for the moment and figure out how to get it imported into our table.

Here we hit our first speed bump. The method in SqlBulkCopy that actually performs the work is WriteToServer, which has the following overloads:

WriteToServer(DataTable, DataRowState)
WriteToServer(IDataReader)
WriteToServer(DataTable)
WriteToServer(DbDataReader)
WriteToServer(DataRow[])

Since implementing IDataReader looks like quite a bit of work, and DataRow has no public constructors, it looks like DataTable is the way to go. We could re-tool our existing code to directly read things in a DataTable, but let’s say I like the current separation of concerns and would rather have this as a separate step. Writing our bulk import code is a simple affair:

var dt = new DataTable();
dt.Columns.Add("Id");
dt.Columns.Add("Name");
dt.Columns.Add("WeightInGrams");
foreach (var widget in ReadFromFile(@"widgets.txt")) {
    var row = dt.NewRow();
    row["Id"] = widget.Id;
    row["Name"] = widget.Name;
    row["WeightInGrams"] = widget.WeightInGrams;
    dt.Rows.Add(row);
}
using (var connection = new SqlConnection("Data Source=.;Initial Catalog=Widgets;Integrated Security=SSPI")) {
    connection.Open();
    using (var sqlBulkCopy = new SqlBulkCopy(connection)) {
        sqlBulkCopy.DestinationTableName = "Widgets";
        sqlBulkCopy.WriteToServer(dt);
    }
}

If we run this code, our resulting table will end up like this:

ID	Name	WeightInGrams
1	azultronic radiator	2600
2	boson catcher	430
3	nuclear accelerator	10000
4	screw	45

Wait, what happened to our ID? Well, remember that ID is an IDENTITY. By default, any values you supply will simply be ignored in favor of the generated IDENTITY.

Here I’d like to make a little aside – suppose this is exactly what you wanted, and the input file had no ID. How would you perform the bulk copy in that case? It would seem obvious to simply leave it out and do this:

var dt = new DataTable();
dt.Columns.Add("Name");
dt.Columns.Add("WeightInGrams");
foreach (var widget in ReadFromFile(@"widgets.txt")) {
    var row = dt.NewRow();
    row["Name"] = widget.Name;
    row["WeightInGrams"] = widget.WeightInGrams;
    dt.Rows.Add(row);
}

Alright, let’s try that:

ID	Name	WeightInGrams
1	2600	NULL
2	430	NULL
3	10000	NULL
4	45	NULL

Wait, what? How did we get this bizarre result? Well, unless you explicitly tell it otherwise, SqlBulkCopy will map input to output columns in the physical order they occur in your input and the table. This is true even if you supply your columns with names in a DataTable.

This is obviously the Wrong Thing to do. If the fact that we gave it column names wasn’t enough, depending on the physical order of columns in a database table is a Bad Idea – this order doesn’t logically matter, and about the only place where you can see it is if you do a SELECT *. The fix isn’t hard, all we have to do is add the following:

foreach (DataColumn column in dt.Columns) {
    sqlBulkCopy.ColumnMappings.Add(column.ColumnName, column.ColumnName);
}

But the fact that we have to add this at all seems clumsy, to say the least.

Back to our original scenario – we want our supplied ID to be respected. This requires an explicit option:

using (var sqlBulkCopy = new SqlBulkCopy(connection, SqlBulkCopyOptions.KeepIdentity, null)) {

Why is this an option you have to pass to the constructor, instead of a property, or an argument you can pass to WriteToServer? Well, um, it just is – no reason I can think of. We now also have to pass an explicit null for SqlTransaction, even though we’re not using that. So now we run that, and:

System.InvalidOperationException: The given ColumnMapping does not match up with any column in the source or destination.

What now? How did we displease the SqlBulkCopy gods this time? I’ll just come out and tell you, since you could waste quite a bit of time trying to puzzle it out otherwise: in our table, the column is spelled ID, with a capital D, but in our DataTable we’ve used Id, to match the property name. If you’re like a lot of people, your database will use a case-insensitive collation, so you don’t normally notice such mismatches. SqlBulkCopy performs an ordinal comparison, however, and does not care for sloppiness. Still, you would think it could be nice and tell us which columns, exactly, fail to map – but no such luck. (And note how it did not complain when the KeepIdentity option was not supplied, because it ignored the whole column in the first place!)

If we fix our column spelling, we finally obtain the desired result:

ID	Name	WeightInGrams
1	azultronic radiator	2600
12	boson catcher	430
15	nuclear accelerator	10000
21	screw	45

Great! Are we done? Well, as long as we’re content with importing only four rows into an empty table, yes, we’re done. That, however, is hardly what we need SqlBulkCopy for – what about inserting lots of data? Well… there will be more about that in part 2.