Robust Program Design - What to do when malloc fails?

If you have been writing C for any amount of time, you probably know the malloc function. It takes in a number of bytes, and returns a void pointer to some "allocated memory". But it can also fail in case no memory is available, in which case no allocation occurs and it returns a NULL pointer.

At first, you may write a simple if check next to such calls. If an out of memory error (OOM) occurs, you return from the main function with a non-zero status. But that obviously gets tedious and error-prone as soon as you have more than 1 call. So what should you do instead?

It may not come to you as a surprise that there are many different approaches, suited to different situations. We are programming, after all. To know how to act in an OOM, we need to first understand what it actually means when we get a NULL pointer.

How `malloc` works

Disclaimer: Any explanation of malloc's internals specifically refers to the glibc implementation.

malloc, being perhaps one the most important function in programming, obviously needs to be optimized. So there are different paths taken depending on how much memory you want, and the current state of the memory.

For any allocation, malloc will check multiple available sources of memory to determine what to use:

Thread-local cache ('tcache') - Using a lock for every allocation will be painfully slow, so instead, every thread gets a dedicated 'cache' which stores small chunks of memory that were recently freed. This way, a small allocation does not impact others in any way, and no extra memory is used.
'Fastbins' (for tiny allocations) - A list of small, available chunks of memory is checked per arena (a section of heap memory). This operates similarly to a stack, where the most recently added memory chunk is the first one to be processed (LIFO).
Sized bins - A best-fit search is made on a global, sorted list of available chunks of memory. Instead of using the first one it finds that satisfies the memory need, it finds the smallest one that does so, as to avoid wasting memory which could be otherwise used by other allocations. It can also combine different contiguous chunks in the bin to try and create a valid block to return.
Unused heap ('top chunk') - Since the free lists cannot accommodate the allocation, it then attempts to use the space at the very end of the heap memory.
Finally, if all of that failed, only then will it make a system call to retrieve more memory. This is in the form of brk(), where the heap memory region is expanded, or mmap(), which returns an independent block of memory.

After retrieving memory from any source, malloc splits it into 'chunks' with metadata, and organizes it into the 'bins' seen above, each with a different purpose. You may have noticed it uses both mmap() and brk(). Why is that? brk() is used for small allocations - typically less than 128KB. This is because the heap can be expanded easily and efficiently using it. With mmap(), it has a larger overhead and may be limited in count. However, memory from mmap() is independent and can be released easily using munmap(), while memory in the program heap is typically very fragment and the size can be held up by a tiny allocation at the very end. Both have their benefits and issues, so they are used when appropriate.

However, you are unlikely to actually be given actual memory from any system call. In general, programs don't end up using all the memory they allocate, so kernels usually 'lazily' give out memory - that is, only actually giving physical memory when you interact with that memory. But since memory is allocated in pages, it goes a step further - it only gives the pages that you touch, which are typically 4KB in size. That way, even if you allocate several MB, memory usage will stay low as long as you don't interact with each page. This is called demand paging.

Overall, it's more accurate to think of malloc as a 'memory reserver' rather than an 'allocator'. It's up to the kernel to actually give you the memory.

Because of this, the kernel can promise more memory that you actually have available, as it assumes programs don't make use of all the memory they demand for. This is called overcommit. So, under glibc, seeing a NULL return value would only occur if you requested an extreme amount of memory, if all contiguous available memory sections are too low (e.g. if you excessively allocate and de-allocate) or in a system with modified memory limits.

An alternative solution to running out of physical memory by the kernel is by using an OOM killer. It determines the worst offender for memory usage and kills that process, instead of a small program trying to get a single page. The only reason this is used is due to the aforementioned overcommit, where the kernel allowed the memory to be reserved, but it can't actually deliver on its promise, so it has no choice but to start killing processes.

But again, it doesn't immediately go for the destructive option. There are many different ways it tries to recover, before the last resort of killing programs:

Background memory reclamation - When a program exits without cleaning up resources, the kernel will do this itself. The reclamation is done in the background, until it reaches a safe threshold.
Direct reclamation - If the background work isn't keeping up and page count reaches a critical threshold, it blocks the process and starts evicting pages. This operates on pages like file pages, where the kernel caches file contents. This is just used to speed up file interactions by storing the file contents in RAM rather than using a disk access every time, but is optional.
Defragmentation - A different background kernel process works to try to defragment the memory and move free space together; also triggered by reaching that critical level.
OOM killer - If all else fails, the kernel has no choice but to kill processes.

As has been made clear, getting a memory reservation failure is rare on modern systems. And due to overcommit, even programs with robust failure checks everywhere can still be killed because they actually used the memory they were given. So should we even check for a NULL pointer at all? The answer is still absolutely yes, but what's more important to discuss is how you should check or handle such a case. Let's go over different approaches.

Handling an OOM error

The first and most well-known method of handling an OOM is just checking the return value, and returning an error in this case. While simple and technically working, it does introduce a development headache, as you now have to make sure to check every single call.


#include <stdlib.h>
#define LOGLEN 2

void some_work_function(void *memory)
{
	static int n = 0;
	static int worklog[LOGLEN];

	const int last = n++ ? 1024 : 131072;
	for (int i = 0; i < last; ++i) ((long *)memory)[i] = i;

	worklog[n] = last;
}

int main(void)
{
	void *reserved = malloc(1024 * 1024);
	if (!reserved) return EXIT_FAILURE;

	some_work_function(reserved);

	void *reserved2 = malloc(2 * 4096);
	if (!reserved2) return EXIT_FAILURE;

	some_work_function(reserved2);

	free(reserved);
	free(reserved2);
}

This is bad for maintenance - do you really want to write that everywhere? So the obvious answer would be to use a macro that does the check for you and returns whatever you pass in as an argument, right? This would work, but there still is another issue with this: if you are using it in a function that isn't main, you may have to change the return type to signal an error capability, and then change wherever you use this to also check for that error. This propagates all the way to your main function, or wherever the calls originated from. Furthermore, since we are trying to add a check outside of malloc, implementing it as a macro is not technically feasible for all scenarios (e.g. passing in as an argument directly).


#include <stdlib.h>
#define LOGLEN 2

// If you want maximum control, you could use this:
#define SAFE_MALLOC(size, name, return_value) malloc(size); if (name == NULL) return return_value

// Because we added a dynamic allocation to this function, we now
// have to change the return type and check for an error at any calls.
int some_work_function(void *memory)
{
	static int n = 0;
	static int *worklog = NULL;
	worklog = SAFE_MALLOC(sizeof *worklog * LOGLEN);

	const int last = n++ ? 1024 : 131072;
	for (int i = 0; i < last; ++i) ((long *)memory)[i] = i;

	worklog[n] = last;
	return 1;
}

int main(void)
{
	void *reserved = SAFE_MALLOC(1024 * 1024, reserved, EXIT_FAILURE);
	if (some_work_function(reserved) == 0) return EXIT_FAILURE;

	void *reserved2 = SAFE_MALLOC(2 * 4096, reserved2, EXIT_FAILURE);
	if (some_work_function(reserved2) == 0) return EXIT_FAILURE;

	free(reserved);
	free(reserved2);
}

The proper way to deal with this situation is to understand what it means. When malloc fails (for a reasonable request), this likely means the system is in a dangerous state and doing whatever it can to not die. So you should act accordingly instead of trying to run as normal. A person aware their system is dangerously low on memory is less concerned about how quickly their browser is running or how stable it is, but more on trying to save their data before anything bad happens.

If you are writing some initialization code where malloc is required, it makes sense to use the standard approach and just return an error value. That way, someone using it can check for this and decide whether to continue their program's execution. They may not require as much memory, so would be fine without it. However, if you are in the middle of your program, where many resources have already been allocated and are in use, your only real choice is to stop execution safely and as soon as possible - you likely won't be able to continue for long, in any case.

So the best solution is to wrap malloc with your own OOM handler. This means that everywhere else in your code, you can just reserve memory without worrying about failure - it is already handled for you. This results in clean and small code - a win-win!

But what should the handler look like? I've seen an example that looks like this before:


void *safe_malloc(size_t n)
{
	void *ptr = malloc(n);
	if (!ptr) abort();
	return ptr;
}

Let's go over why this isn't a great solution in general (although there's nothing stopping you from using it). The issue here is that it calls abort. What this does is generate a core dump, essentially writing the contents of program memory into disk. In a time where memory is likely low, that's just worsening the situation. Furthermore, this doesn't allow for any sort of emergency saving - what will be shown later on.

You likely don't want work to be lost, so is there some way of ensuring that even when memory is low, we can still save to some sort of 'backup' file and then cleanly terminate instead of going for the nuclear option and aborting termination on the spot? There is, but it has to be designed carefully. We need to ensure all the resources we need is already laid out for us, so we can just write directly to a file and exit as soon as possible.

Let's look at a way of doing this with POSIX functions, since it's the easiest to understand. First of all, instead of using the standard, formatted functions (e.g. fwrite) - which internally use a buffer (which we want to avoid) - we need to directly use system calls to handle the filesystem interactions. We also need to pre-allocate a sort of 'emergency buffer' which gives us enough memory to work with - any other threads in your program are still running, and may end up causing more page faults or allocations, worsening the situation. Finally, we also need to ensure the write is synchronized, so the data is actually saved to disk, not just stored in RAM.

Here's what that would look like. We first initialize all of the emergency resources and then allocate however we like. Whenever we get an OOM error, we deallocate the emergency buffer and then write directly to disk.


#define _GNU_SOURCE
#include <stdlib.h>
#include <unistd.h>
#include <stdio.h>
#include <fcntl.h>

struct {
	int important_data;
} save_data;

void *emergency_buffer;
int emergency_fd = -1;

#define EMERGENCY_BUFFER_SIZE (1024 * 1024)
#define EMERGENCY_FILE_NAME "oom_backup.bak"

// Call this at the end of your program, probably right before you exit.
void emergency_handler_deinit(void)
{
	free(emergency_buffer);
	close(emergency_fd);
}

// Call this early in your program, before any allocation happens.
int emergency_handler_init(void)
{
	// You can change the buffer size depending on your program.
	// If allocation fails here, just give up; there's no point.
	emergency_buffer = malloc(EMERGENCY_BUFFER_SIZE);
	if (!emergency_buffer) return 0;

	// O_DSYNC provides "synchronized I/O *data* integrity completion", so the file data
	// and metadata required for reading the file (e.g. file length) will be flushed to disk on write.
	// We don't care about the last modification time being saved, we just need a recovery file ASAP.
	emergency_fd = open(EMERGENCY_FILE_NAME, O_WRONLY | O_TRUNC | O_CREAT | O_DSYNC);

	// If this fails, give up too.
	if (emergency_fd == -1) {
		free(emergency_buffer);
		return 0;
	}

	// Allocate disk space for file. Any writes to the file in the given
	// range are guaranteed to not fail.
	int errno_value = posix_fallocate(emergency_fd, 0, EMERGENCY_BUFFER_SIZE);

	// If this fails, you could still continue, but there
	// would be no guarantee that the write would succeed.
	// That would kind of defeat the whole purpose.
	if (errno_value != 0) {
		emergency_handler_deinit();
		remove(EMERGENCY_FILE_NAME);
		return 0;
	}

	return 1;
}

int handle_oom(void)
{
	// Our own program now has a little bit more breathing space, so other
	// threads, libraries or hidden allocations don't ruin our attempt at saving the data.
	free(emergency_buffer);

	// Write directly to the file with a syscall.
	// Since we preallocated the space, this will always succeed,
	// as long as the number of bytes is at most EMERGENCY_BUFFER_SIZE.
	ssize_t wrote = write(emergency_fd, &save_data, sizeof save_data);
	close(emergency_fd);

	// Using O_DSYNC means it's as if we called fdatasync() after the write().

	// If we still failed for some reason (-1 or less than expected), we have no way of recovering.
	if (wrote < (ssize_t)(sizeof save_data)) return EXIT_FAILURE;

	// You could also do other backup things here, but usually you
	// would just hurry to create a recovery file.

	return EXIT_SUCCESS;
}

// Custom allocator - you should use one in general, so you
// can monitor allocations and test out behaviour in a low memory situation.
void *safe_malloc(size_t n)
{
	void *ptr = malloc(n);

	// Using _exit as we do not care about normal termination or cleanup,
	// just terminate right now. abort(), as previously mentioned, would be even worse.
	if (!ptr) _exit(handle_oom());

	return ptr;
}

// You can easily make a similar one for realloc() and calloc().

As you can see above, we make as many guarantees as possible for the OOM handler to work. We still need to code for every situation, but considering you only need to do that once and not worry about checking for a NULL pointer for most of your codebase, it's a much better choice in general. Obviously, if the program fails in the initial setup, when allocating a buffer whose entire purpose is to help with an OOM situation, the program is definitely in no shape to continue. Same with posix_fallocate() - this could be due to a filesystem error (unlikely), or if the disk is full (still unlikely), so you also have no choice but to either continue without that success guarantee (strongly do not recommend), or fail the program initialization.

As you can see here, by recognizing what it means to encounter an OOM error in C, you can make development much easier and avoid having to code for every possible failure scenario when allocating. But there is still one more way we can improve the stability of our program: Not using malloc at all.

Utilizing the stack

Think about how often you use the malloc function. It's likely that you use it too often. You can instead make assumptions about how much memory you will use. Even though malloc is heavily optimized for small, frequent allocations, not using it at all is faster. Plus, you won't have to worry about freeing after use or increased memory usage. In short, if you can avoid a dynamic memory allocation, you should do it.

First, how big is the stack in a reasonable system? Yes, it is possible to make it take up the entire address space, but you can't expect everyone to have a similar setup. For Windows, the stack size is 1MB and Linux has 8MB. Some systems have different stack limits for secondary threads, such as MacOS with 8MB for the main thread and 0.5MB for others.

Unless you are using recursion or have an insane call stack, you will go nowhere near that limit with normal usage. So let's not waste the space we were given! A couple of examples of utilizing the stack instead of dynamically allocating memory can be found below:

You often have a custom logging or print function that uses snprintf twice - once to determine the final size, and again after allocating that much (+1 for terminator). But if we make a reasonable assumption that any formatted text will not exceed a size of 4KB (adjustable), we can improve performance a little bit and avoid an allocation. Plus, you don't have to do any error checking or free() a pointer anymore!


#include <stdarg.h>
#include <stdio.h>

#define SEVERITY_ERROR 3
#define SEVERITY_WARNING 2
#define SEVERITY_MESSAGE 1

void (*user_log_function)(int severity, const char *description);

int malloc_logger(int severity, const char *format, ...)
{
	va_list list, copy;
	va_start(list, format);
	va_copy(copy, list);

	char *buffer = NULL;
	int ret = 0;

	int nchars = vsnprintf(NULL, 0, format, copy);
	va_end(copy);

	if (nchars <= 0) goto fail_1;
	buffer = calloc((size_t)nchars++, 1);
	if (!buffer) goto fail_1;

	vsnprintf(buffer, (size_t)nchars, format, list);

	user_log_function(severity, buffer);
	ret = 1;

fail_2:
	free(buffer);
fail_1:
	va_end(list);

	return ret;
}

void stack_logger(int severity, const char *format, ...)
{
	va_list list;
	va_start(list, format);

	char buffer[4096];
	vsnprintf(buffer, sizeof buffer, format, list);

	user_log_function(severity, buffer);
	va_end(list);
}

The stack-based logger is definitely much cleaner, and it produces the same result faster (4K chars is usually more than enough). There is one issue introduced by this though: race conditions. However, printing on different threads is already not a good idea, but you could add a lock to the function easily (mtx_t, pthread_mutex_t, etc).

Next up, for any Vulkan fans, code like this may look familiar:


// Somewhere we have...
uint32_t num_present_modes;
VkPhysicalDevice physical_device;

// Later on, in the actual code:

// Get number of presentation modes available.
VkResult result1 = vkGetPhysicalDeviceSurfacePresentModesKHR(physical_device, surface, &num_present_modes, NULL));
if (num_present_modes == 0); // Device does not have any.

// Allocate exact-size list to store presentation modes in.
VkPresentModeKHR *present_modes = malloc(sizeof *present_modes * num_present_modes);
VkResult result2 = vkGetPhysicalDeviceSurfacePresentModesKHR(physical_device, surface, &num_present_modes, present_modes));

It does exactly what we want - determine how many presentation modes are supported and allocates a list to store all of them. However, looking at the official specification, we see that there are only 7 presentation modes (at least at the time of writing), and VkPresentModeKHR is just an enum, so it's not really worth using the heap for. There is a caveat though - lifetime. If this is just a standalone function, returning the pointer would lead to a bug known as a dangling pointer, where it points to an invalid location, since a local array would only last as long as the scope it's in. So if you do switch to a stack-based version, make sure its stored in a more permanent data structure.


// Somewhere we have...
uint32_t num_present_modes = 10; // Note that we set the size explicitly.
VkPhysicalDevice physical_device;
VkPresentModeKHR present_modes[10]; // Additional present mode structure. Size of 10 for future-proofing, I guess.

// Later on, in the actual code:

// Get presentation modes available.
VkResult result1 = vkGetPhysicalDeviceSurfacePresentModesKHR(physical_device, surface, &num_present_modes, present_modes));
if (num_present_modes == 0); // Device does not have any.

// If the array is too small, the return value will be VK_INCOMPLETE. Since we have enough for
// every presentation mode, this is unlikely to ever happen.

Again, cleaner as we don't have to query the size and then allocate an array, similar to the last example. This can be applied to different Vulkan functionality, like getting the surface formats, pointers to VkPhysicalDevice, queue families, and more! There really are a lot of redundant allocations. This is assuming you can get around the lifetime issue, though.

Conclusion

There's so much more code you can simplify via a wrapper, but malloc is a great starting point, considering how often it is called. Overall, while it may be great developing software that produces results quickly (especially with the use of AI nowadays), making robust and maintainable code should remain a top priority, especially for software that needs to work consistently over a long time rather than as a one-off thing.

Sources:

Linux kernel internals (nice name)
Wikipedia: C dynamic memory allocation

How malloc works

Handling an OOM error

Utilizing the stack

Conclusion

How `malloc` works