To New, Perchance To Throw, Part 2

This article appeared in C/C++ Users Journal, 19(5), May 2001.

In the previous column,^[1] I illustrated and justified the following coding guideline:

This time, we'll delve deeper into the question of what operator new() failures mean, and how best to detect and handle them:

The first part may be mildly surprising advice, but it's the latter that is likely to raise even more eyebrows - because on certain popular real-world platforms, memory allocation failures usually don't even manifest in the way the standard says they must.

Like last time, for simplicity, I'm not going to mention the array forms of new specifically. What's said about the single-object forms applies correspondingly to the array forms.

Exceptions, Errors, and new(nothrow)

First, a recap of what we all know: Whereas most forms of new report failure by throwing a bad_alloc exception, nothrow new reports failure the time-honored malloc() way, namely by returning a null pointer. This guarantees that nothrow new will never throw, as indeed its name implies.

The question is whether this really buys us anything. Some people have had the mistaken idea that nothrow new enhances exception safety because it prevents some exceptions from occurring. So here's the $64,000 motivating question: Does using nothrow new enhance program safety in general, or exception safety in particular?

The (possibly surprising) answer is: No, not really. Error reporting and error handling are orthogonal issues. "It's just syntax after all."^[2]

The choice between throwing bad_alloc and returning null is just a choice between two equivalent ways of reporting an error. Therefore, detecting and handling the failure is just a choice between checking for an exception and checking for the null.

To a calling program that checks for the error, the difference is just syntactic. That means that it can be written with exactly equivalent program safety, and exception safety, in either case - the syntax by which the error happens to be detected isn't relevant to safety, because it is just syntax, and leads to only minor variations in the calling function's structure (e.g., something like if (null) { HandleError(); throw MyOwnException(); } vs. something like catch(bad_alloc) { HandleError(); throw MyOwnException(); }). Neither way of new error reporting provides additional information or additional inherent safety, and so neither way inherently makes programs somehow "safer" or "able to be more correct," assuming of course that the handling is coded accurately.

But what's the difference to a calling program that doesn't check for errors? In that case, the only difference is that the eventual failure will be different in mode but not severity. Either an uncaught bad_alloc will unceremoniously terminate the program (with or without unwinding the program stack all the way back to main()), or an unchecked null pointer will cause a memory violation and immediate halt when it's later dereferenced. Both failure modes are fairly catastrophic, but there's some advantage to the uncaught exception: It will make an attempt to destroy at least some objects, and therefore release some resources, and if some of those objects are things like a TextEditor object that automatically saves recovery information when prematurely destroyed, then not all the program state need be lost if the program is carefully written. (Caveat: When memory really is exhausted, it's harder than it appears to write code that will correctly unwind and back out, without trying to use a teensy bit more memory.) An abrupt halt due to use of a bad pointer, on the other hand, is far less likely to be healthy.

From this we derive Moral #1:

Moral #1: Avoid Nothrow New

Nothrow new does not inherently benefit program correctness or exception safety. For some failures, namely failures that are ignored, it's worse than an exception because at least an exception would get a chance to do some recovery via unwinding. As pointed out in the previous column, if classes provide their own new but forget to provide nothrow new too, nothrow new will be hidden and won't even work. In most cases nothrow new offers no benefit, and for all these reasons it should be avoided.

I can think of two corner cases where nothrow new can be beneficial. The first case is one that these days is getting pretty hoary with age: When you're migrating a lot of legacy really-old-style C++ application code written before the mid-1990s that still assumes that (and checks whether) new returns null to report failure, then it can be easier to globally replace "new" with "new (nothrow)" in those files - but it's been a long time now since unadorned new behaved the old way! The amount of such hoary old code that's still sitting around and hasn't yet been migrated (or recompiled on a modern compiler) yet is dwindling fast.

The second case for using nothrow new is if new is being used a lot in a time-critical function or inner loop, and the function is being compiled using a weaker compiler that generates inefficient exception handling code overhead, and this produces a measurable runtime difference in this time-critical function between using normal new and using nothrow new. Note that when I say "measurably" I mean that we've actually written a test harness that includes at least the entire piece of time-critical code (not just a toy example of new by itself), and timed two versions, one with new and one with new(nothrow). If after all that we've proved that it makes a difference to the performance of the time-critical code, we might consider new(nothrow) - and should at the same time consider other ways to improve the allocation performance, including the option of writing a custom new using a fixed-size allocator or other fast memory arena.^[3]

This brings us to Moral #2:

Moral #2: There's Often Little Point in Checking for New Failure Anyway

This statement might horrify some people. "How can you suggest that we not check for new failure, or that checking failures is not important?" some might say, righteously indignant. "Checking failures is a cornerstone of robust programming!" That's very true in general, but - alas - it often isn't as meaningful for new for reasons which are unique to memory allocation, as opposed to other kinds of operations whose failure should indeed be checked and handled.

Here are some reasons why checking for new failure isn't as important as one might think:

1. Checking new failure is useless on systems that don't commit memory until the memory is used.

On some operating systems, including specifically Linux,^[4] memory allocation always succeeds. Full stop. How can allocation always succeed, even when the requested memory really isn't available? The reason is that the allocation itself merely records a request for the memory; under the covers, the (physical or virtual) memory is not actually committed to the requesting process, with real backing store, until the memory is actually used. Even when the memory is used, often real (physical or virtual) memory is only actually committed as each page of the allocated buffer is touched, and so it can be that only the parts of the block that have actually been touched get committed.

Note that, if new uses the operating system's facilities directly, then new will always succeed but any later innocent code like buf[100] = 'c'; can throw or fail or halt. From a Standard C++ point of view, both effects are nonconforming, because the C++ standard requires that if new can't commit enough memory it must fail (this doesn't), and that code like buf[100] = 'c' shouldn't throw an exception or otherwise fail (this might).

Background: Why do some operating systems do this kind of "lazy allocation"? There's a noble and pragmatic idea behind this scheme, namely that a given process that requests memory might not actually immediately need all of said memory - the process might never use all of it, or it might not use it right away and in the meantime maybe the memory can be usefully "lent" to a second process which may need it only briefly. Why immediately commit all the memory a process demands, when it may not really need it right away? So this scheme does have some potential advantages.

The main problem with this approach, besides that it makes C++ standards conformance difficult, is that it makes program correctness in general difficult, because any access to successfully-allocated dynamic memory might cause the program to halt. That's just not good. If allocation fails up front, the program knows that there's not enough memory to complete an operation, and then the program has the choice of doing something about it, such as trying to allocate a smaller buffer size or giving up on only that particular operation, or at least attempting to clean up some things by unwinding the stack. But if there's no way to know whether the allocation really worked, then any attempt to read or write the memory may cause a halt - and that halt can't be predicted, because it might happen on the first attempt to use part of the buffer, or on the millionth attempt after lots of successful operations have used other parts of the buffer.

On the surface, it would appear that our only way to defend against this is to immediately write to (or read from) the entire block of memory to force it to really exist. For example:

// Example 1: Manual initialization
//
// Deliberately go and touch each byte.
//
char* p = new char[1000000000];
memset( p, 0, 1000000000 );

If the type being allocated happens to be a non-POD^[5] class type, the memory is in fact touched automatically for you:

// Example 2: Default initialization
//
// If T is a non-POD, this code initializes
// all the T objects immediately and will
// touch every (significant, non-padding) byte
//
T* p = new T[1000000000];

If T is a non-POD, each object is default-initialized, which means that all the significant bytes of each object are written to, and so the memory has to be accessed.^[6]

You might think that's helpful. It's not. Sure, if we successfully complete the memset() in Example 1, or the new in Example 2, we do in fact know that the memory has really been allocated and committed. But if accessing the memory fails, the twist is that the failure won't be what we might naively expect it to be: we won't get a null return or a nice bad_alloc exception, but rather we'll get an access violation and a program halt, none of which the code can do anything about (unless it can use some platform-specific way to trap the violation). That may be marginally better and safer than just allocating without writing and pressing on regardless, hoping that the memory really will be there when we need it and that all will be well, but not by much.

This brings us back to standards conformance, because it may be possible for the compiler-supplied ::operator new() and ::operator new[]() itself to do better with the above approach than we as programmers could do easily. In particular, the compiler implementer may be able to use knowledge of the operating system to intercept the access violation and therefore prevent a program halt. That is, it may be possible for a C++ implementer to do all the above work - allocate, and then confirm the allocation by making sure we write to each byte, or at least to each page - and catch any failure in a platform-specific way and convert it to a standard bad_alloc exception (or a null return, in the case of a nothrow new). I doubt that any implementer would go to this trouble, though, for two reasons: first, it means a performance hit, and probably a big one to incur for all cases; and second, new failure is pretty rare in real life anyway… which happens to lead us nicely to the next point:

2. In the real world, new failure is a rare beast, made nearly extinct by the thrashing beast.

As a practical matter, many modern server-based programs rarely encounter memory exhaustion.

On a virtual memory system, most real-world server-based software performs work in various parts of memory while other processes are actively doing the same in their own parts of memory; this causes increasing paging as the amount of memory in use grows, and often the processes never reach new failure. Rather, long before memory can be fully exhausted, the system performance will hit the thrash wall and just grind ever more unusably slowly as pages of virtual memory are swapped in and out from disk, and the sysadmin will start killing processes.

I caveat this with words like "most" because it is possible to create a program that allocates more and more memory but doesn't actively use much of it. That's possible, but unusual at least in my own experience. This also of course does not apply to systems without virtual memory, such as many embedded and real-time systems; some of these are so failure-intolerant that they won't even use any kind of dynamic memory at all, never mind virtual memory.

3. There's not always much you can do when you detect new failure.

As Andy Koenig pointed out in his 1996 article "When Memory Runs Low,"^[7] the default behavior of halting the program on new failure (usually with at least an attempt to unwind the stack) is actually the best option in most situations, especially during testing.

Sometimes when new fails there are a few things you can do, of course: If you want to record some diagnostic info, the new handler is a nice hook for doing logging. It is sometimes possible to apply the strategy of keeping a reserve "emergency memory" buffer; although anyone who does this should know what they are doing, and actually carefully test the failure case on their target platforms, because this doesn't necessarily work the way people think. Finally, if memory really is exhausted, you can't necessarily rely on being able to throw a nontrivial (e.g., non-builtin) exception; even throw string("failed"); will usually attempt a dynamic allocation using new, depending on how highly optimized your implementation of string happens to be.

So yes, sometimes there are useful things you can do to cope with specific kinds of new failure, but often it's not worth it beyond letting stack unwinding and the new handler (including perhaps some logging) do their thing.

What Should You Check?

There are special cases for which checking for memory exhaustion, and trying to recover from it, do make sense. Andy mentions some in his article referenced above. For example, you could choose to allocate (and if necessary write to) all the memory you're ever going to use up front, at the beginning of your program, and then manage it yourself; if your program crashes at all, it will crash right away before actually doing work. This approach requires extra work and is only an option if you know the memory requirements in advance.

The main category of recoverable new failure error I've seen in production systems has to do with creating buffers whose size is externally supplied from some input. For example, consider a communications application where each transmitted packet is prepended with the packet length, and the first thing the receiver does with each packet is to read the length and then allocate a buffer big enough to store the rest of the packet. In just such situations, I've seen attempts to allocate monstrously large buffers, especially when data stream corruption or programming errors cause the length bytes to get garbled. In this case, the application should be checking for this kind of corruption (and, better still, designing the protocol to prevent it from happening in the first place if possible) and aborting on invalid data and unreasonable buffer sizes, because the program can often continue doing something sensible, such as retrying the transmission with a smaller packet size or even just abandoning that particular operation and going on with other work. After all, the program is not really 'out of memory' when an attempt to allocate 2GB failed but there's still 1GB of virtual memory left!^[8]

Another special case where new failure recovery can make sense is when your program optimistically tries to allocate a huge working buffer, and on failure just keeps retrying a smaller one until it gets something that fits. This assumes that the program as a whole is designed to adjust to the actual buffer size and does chunking as necessary.

Summary

Avoid using nothrow new, because it offers no significant advantages these days and usually has worse failure characteristics than plain throwing new. Remember that there's often little point in checking for new failure anyway, for several reasons. If you are rightly concerned about memory exhaustion then be sure that you're checking what you think you're checking, because: checking new failure is typically useless on systems that don't commit memory until the memory is used; in virtual memory systems new failure is encountered rarely or never because long before virtual memory can be exhausted the system typically thrashes and a sysadmin begins to kill processes; and, except for special cases, even when you detect new failure there's not always much you can do if there really is no memory left.

Notes

1. H. Sutter, "To New, Perchance to Throw, Part 1" (C/C++ Users Journal, 19(3), March 2001).

2. Can be sung to the tune of "It's a Small World (After All)."

3. See also H. Sutter, "Containers in Memory: How Big is Big?" (C/C++ Users Journal, 19(1), January 2001) and H. Sutter, Exceptional C++ (Addison-Wesley, 2000).

4. This is what I've been told by Linux folks in a discussion about this on comp.std.c++. Lazy-commit is also a configurable feature on some other operating systems.

5. POD stands for "plain old data." Informally, a POD means any type that's just a bundle of plain data, though possibly with user-defined member functions just for convenience. More formally, a POD is a class or union that has no user-defined constructor or copy assignment operator or destructor, and no (non-static) data member that is a reference, pointer to member, or non-POD.

6. This ignores the pathological case of a T whose constructor doesn't actually initialize the object's data.

7. A. Koenig, "When Memory Runs Low" (C++ Report, 8(6), June 1996).

8. Interestingly, allocating buffers whose size is externally specified is a classic security vulnerability. Attacks by malicious users or programs specifically trying to cause buffer problems is a classic, and still favorite, security exploit to bring down a system. Note that trying to crash the program by causing allocation to fail is a denial-of-service attack, not an attempt to actually gain access to the system; the related, but distinct, overrun-a-fixed-length-buffer attack is also a perennial favorite in the hacker community, and it's amazing just how many people still use strcpy() and other unchecked calls and thereby leave themselves wide open to this sort of abuse.

Copyright © 2009 Herb Sutter