Buffer Overflow and the Stack Canary Explained for CISSP

A buffer overflow happens when a program writes more data into a fixed block of memory than that block can hold, and the excess spills into the memory beside it. On the stack, the memory beside a buffer can include the return address, the value that tells the program where to go next. Overwrite it with a chosen value and the program may run an attacker’s code. A stack canary is a marker placed between the two, so the program can notice the overwrite before it uses that address.

It is one of the oldest ways software gets attacked. The Morris worm of 1988 spread partly by overflowing a buffer in the fingerd service, and the same class of flaw still appears in current advisories because the languages that allow it still run most operating systems, network devices and embedded firmware. For a CISSP candidate the useful view is a manager’s: which security property fails, where the control that prevents it sits in the software development lifecycle, and which of the familiar protections actually removes the flaw. Only one of them does.

What is a buffer overflow?

A buffer is a fixed amount of memory a program sets aside to hold data, such as a name typed into a form or a field read from a network packet. The size is decided when the code is written. The data arrives later, from whoever is supplying it.

A buffer overflow happens when the program copies that data into the buffer without checking that it fits. Nothing tells the program to stop at the edge of the buffer, so it keeps writing, and every byte past the end lands on whatever memory comes next.

Why some languages allow it and others do not

In languages such as C and C++, the programmer is responsible for checking lengths. The language will let a program write past the end of an array, and some older standard library routines copy input until the input ends rather than until the destination is full. Code that trusts a length value supplied by the sender has the same problem one step removed: the check exists, but it checks the attacker’s claim rather than the real size.

Memory-safe languages such as Rust, Java, C# and Go check bounds automatically. An attempt to write past the end of an array raises an error instead of corrupting the neighbouring memory. This is why moving new development to a memory-safe language is a genuine prevention rather than a mitigation: it removes the class of flaw rather than making it harder to use.

Stack and heap overflows

A buffer can live in two main places. Stack buffers are created for the duration of one function call and sit beside the bookkeeping that function needs to return. Heap buffers are allocated on demand and sit beside other allocations and the allocator’s own records. Both can be overflowed. The stack case is the one worth understanding first, because it shows most plainly how extra bytes become control of the program, and it is the case the stack canary was designed for.

How a stack buffer overflow overwrites the return address

When a function runs, it borrows a section of the stack called a stack frame. The frame holds the function’s local buffers and, among other things, the return address: the location of the code that called the function, so the program knows where to carry on when the function finishes.

In the layout used by the most common compilers and processors, a function’s local buffers sit below its saved return address, and a write that runs past the end of a buffer moves towards that address. There may be other local variables and a saved frame pointer in between; the simplified picture leaves them out, because they do not change what happens.

The honest case

Take a buffer of eight bytes, where one typed character fills one byte. Five characters arrive. They fill five of the eight bytes and stop. The return address is untouched. When the function finishes, the program reads the return address, goes back to the code that called it, and carries on.

The accidental overflow

Now twelve characters arrive. Eight fill the buffer. The other four land on the return address and replace it with the last four characters typed. The program never checked how much input it was given, so anything past the eighth character overwrites whatever sits next to the buffer.

When the function finishes, the program reads the return address as it always does. The value is now four typed characters that point nowhere useful, and the program most likely crashes. A crash is already a security failure, because the service has stopped for everyone who relied on it.

The deliberate overflow

Now bring in an attacker who has found the same weakness. They do not overrun the buffer by accident. They craft the input so its final bytes form an address of their choosing. When the function returns, the program reads the address and may carry on running whatever code sits there, which can be code the attacker supplied inside the input itself or code already present in the program.

The attacker did not need a password. The program did exactly what it was written to do: it copied the input, then returned to the address it found. That is what makes the flaw so serious. There is no failed login to alert on, and the code that runs does so with the full privileges of the program that was attacked.

How a stack canary detects the overwrite

Miners once carried a canary underground because the bird reacted to dangerous gas before they did. A stack canary plays the same part: it is the first thing to be harmed, and its condition is checked before anything worse happens.

A stack canary is a value the program places between a function’s buffers and its saved return address when the function starts. Before the function returns, the program checks that the value is unchanged. If it is, the return goes ahead. If it is not, something has written past the buffer, and the program stops rather than jump to an address it can no longer trust.

Replay the same twelve bytes. Now the canary sits between the buffer and the return address, so eight bytes fill the buffer, the next bytes change the canary, and the rest still reach the return address. The overwrite has happened exactly as before. The difference is the check at the end: it finds the canary changed, the program terminates, and the attacker’s address is never used.

Where the canary comes from

Programmers do not write canaries by hand. The compiler inserts them: it adds the placement when a protected function starts and the check before it returns. GCC and Clang do this with the -fstack-protector family of options and Microsoft’s compiler with /GS. Most current operating system distributions build their packages with the protection switched on, which is why the protection is often present without anyone having asked for it.

The value itself is chosen so an attacker cannot simply write it back unchanged. A random canary is picked when the program starts, so it differs between runs. A terminator canary contains bytes, such as a null byte, that stop the common string-copy routines, so an overflow that relies on those routines cannot reproduce it.

A detective control that fails secure

In control types terms, the canary is a detective control. It does not stop the write. It detects it in time and responds by terminating the program, which is a fail-secure outcome: when the check fails, the program denies itself the next step rather than run code it cannot trust.

That response has a cost worth naming. Terminating the program turns an attempted code execution into a crash, so the integrity and confidentiality impact is contained, but the availability impact remains. For a security manager, that is the right trade. A service that stops can be restarted; a service running an attacker’s code cannot be trusted again until it is rebuilt.

Worked example: the overflow found in an internal service

A company runs an internal service, written in C years ago, that accepts customer account references from its web portal. The reference field is a fixed 32-byte buffer. During a penetration test, the tester sends a reference several hundred characters long. The service terminates and the system log records that stack smashing was detected.

Reading that result correctly is the whole exercise.

• What the log shows. The canary worked. The overlong input overwrote the canary and would have reached the return address, and the check terminated the service before the return. That is detection, as designed.
• What the log does not show. The flaw is still there. The service still copies input without checking its length, and the canary only covers the stack. A variant that overflows a heap buffer, or that leaks the canary value first, may not be caught.
• What changes first. The development team adds a bounds check so the copy stops at 32 bytes, and rejects references of unexpected length or format at the point of entry. A static analysis rule for unbounded copy routines is added to the build, so the same pattern fails review elsewhere in the codebase, and the overlong input becomes a permanent regression test.
• What stays on. The canary, non-executable memory and address randomisation all remain enabled, because they protect against the flaws nobody has found yet. The service account is reduced to the permissions the service actually needs, applying least privilege so that any code an attacker does manage to run can reach as little as possible.

Notice the order. The platform protections bought time and turned a possible compromise into a crash. The fix happened in the code, and the governance change stopped the pattern coming back.

Stack canary vs ASLR vs DEP: what each protection does

The stack canary is one of three run-time protections that are routinely confused with each other and with the fix itself. Each works on a different step of the attack.

The first three rows are layers of defence in depth: each one closes a route the others leave open, and an attacker who defeats one still faces the rest. The last two rows are the only ones that remove the flaw. A supplier who answers a question about buffer overflows with a list of platform protections has answered a different question from whether the code was fixed.

Buffer overflows are also worth separating from their nearest software-security neighbour. A buffer overflow is an invalid operation: the program writes to memory it was never meant to touch. In a time of check to time of use flaw, every operation is valid and every check is truthful; the problem is timing. The two call for different fixes and are found by different tools.

Where the controls sit in the development lifecycle

A CISSP scenario question rarely names the control. It describes a situation and asks what should have happened, or what should happen next. The controls for buffer overflows sit at three distinct points, and matching the scenario to the point is most of the work.

In development: prevent the flaw

For software an organisation writes, prevention belongs in the development process:

• Secure coding standards that require bounds checking on every write, ban unbounded copy routines and require input validation at every trust boundary.
• Code review and static analysis to find unbounded copies and unchecked lengths before release.
• Security testing, including dynamic testing and fuzzing, which sends large volumes of malformed and overlong input to find the crashes an overflow causes.
• Memory-safe languages for new work where the platform allows it.

At run time: detect and limit

The stack canary, DEP and ASLR belong here. They are compiled or configured into the platform and they protect against flaws that were not found in development. Least privilege on the process belongs here too, because it limits what attacker-controlled code can reach if every other layer fails.

For software you buy: patch management

An organisation cannot add bounds checks to a vendor’s product. For bought software, commercial or open source, the control is patch management: tracking vendor advisories, applying fixes promptly and keeping the run-time protections enabled in the meantime. Asking suppliers how they prevent memory-safety flaws, and not only which protections their product enables, is part of the same control.

Where the stack canary falls short

The canary is effective at the job it was built for and silent outside it. Four limits are worth knowing:

1. It sees only what crosses it. An overwrite that changes a local variable or function pointer sitting between the buffer and the canary, or a heap overflow, never touches the canary. Compilers reduce the first risk by placing buffers above other local variables, but they cannot remove the second.
2. A leaked canary can be written back. If another flaw lets an attacker read memory, they can learn the canary’s value and include it, unchanged, in the overflowing input. The check then passes.
3. It responds with a crash. Detection ends in termination. Repeated overflows against a canary-protected service are still a denial of service.
4. It is only present where it was compiled in. Code built without the protection, including some older libraries and embedded firmware, has no canary at all, and nothing at run time warns that it is missing.

Each limit points back to the same conclusion. The canary and its companions reduce the chance that a flaw is exploited. Only fixing the code removes the flaw.

Conclusion

A buffer overflow is a write past the space reserved for the data. On the stack, the memory beyond that space includes the return address, which is why an overflow can decide what the program runs next rather than simply making it crash. The stack canary sits between the two and turns that outcome back into a crash, by checking the marker before the return address is used.

For the CISSP, keep the three points apart. Prevention belongs in development: secure coding, bounds checking, testing and memory-safe languages for software you write, and patch management for software you buy. The canary is a detective control that fails secure, and DEP and ASLR make exploitation harder; none of the three repairs the flaw. A scenario describing a program that stops itself because a value beside its return address changed is describing the canary, while a question about the most effective way to prevent the flaw in software you write points to the development process. The weakness is catalogued as CWE-121, stack-based buffer overflow, which sits under CWE-787, out-of-bounds write, in MITRE’s Common Weakness Enumeration.

Ready to practise telling those controls apart under exam conditions? Our LSM CISSP practice tests include scenario questions on secure coding failures, run-time protections and the difference between a control that detects a flaw and one that removes it, with full explanations for every answer.

Quick reference for the CISSP exam

A buffer overflow is a write past the end of a fixed buffer into adjacent memory. On the stack it can overwrite the saved return address and redirect execution. It is catalogued as CWE-121 (stack-based buffer overflow) under CWE-787 (out-of-bounds write).

The mechanism in three steps

• The unchecked write. The program copies input into a fixed buffer without checking that it fits.
• The overwrite. Bytes past the end of the buffer replace the saved return address.
• The redirected return. When the function finishes, the program jumps to the overwritten address: a crash if accidental, attacker-chosen code if crafted.

The distinctions most likely to matter

• Detecting against preventing. A stack canary detects the overwrite before the return and terminates the program. It does not stop the write.
• Mitigating against fixing. Stack canaries, DEP and ASLR make exploitation harder. Bounds checking, input validation and memory-safe languages remove the flaw.
• Software you write against software you buy. Prevention for your own code sits in the development process; for bought software it is patch management.
• CIA impact. Integrity first, availability on a crash, and all three if attacker code runs with the program’s privileges.

Spotting the topic in a scenario

Look for input longer than a field was designed for, a program written in C or C++, a crash on unusual input during testing, or a log entry recording that stack smashing was detected. If a program stops itself because a value next to its return address changed, that is the canary.

The control order

1. Fix the code: bounds checks, input validation, length-limited copy routines, memory-safe languages for new work.
2. Find it before release: coding standards, code review, static analysis, fuzzing and dynamic testing.
3. Patch what you buy and ask suppliers how they prevent memory-safety flaws.
4. Keep the run-time layers on: stack canary, DEP and ASLR.
5. Limit the blast radius with least privilege on every process that handles untrusted input.