I've been looking into building a basic game engine from the ground up and after making a list of features that are common to other engines, one of the bigger things is the fact that they have an embedded scripting language like lua or python.
My question is how is an embedded scripting language superior to just making a header file (or something of the like) that the user can include in a c++ file which gives them access to many of the functions and states. I'm sure there's a very good answer out there, I just haven't stumbled on it yet.
Also beyond why it's needed, what are languages like lua used for in things like game engines?
The reason for including a scripting language is to allow users to customize the behavior without having to recompile the code.
I'm not sure about what you are asking in the second part of the question. Are you asking what other languages are used, or are you asking what ways are languages like Lua used?
If you asked about what other languages are good for this, one such language is Tcl. Tcl was designed from the ground up to be an embedded scripting language, and is very mature and robust, and easily learned by non-technical people.
As for what scripting languages are good for ... configuration files is one way. By using a programming language rather than a text file with name/value pairs, it allows users to add logic to their start-up files. For example, maybe you allow users to assign different functions to keys on the keyboard; with a programming language they can add different functions for different computers. Or, if you're creating a game like a RPG, perhaps you can assign different keys for different character classes. If playing as a mage, F12 might be cast a spell, but if playing as a warrior f12 might be to do a finishing blow.
There are many ways to use scripting languages, and many different langages to choose from. It all boils down to allowing your users to customize the behavior of the game without having to recompile the code.
Lua is a far simpler language than C++, and all you need to edit it is a text editor. This puts the ability to script events and/or high level game logic in the hand of your designers and end users. Dynamic typing and garbage collecting allows them write very succinct code that focusses on game logic rather than all the systems-level housekeeping chores you get in a language like C++. It's also far easier to sandbox.
Lua is a popular choice because it's small, portable, hackable ANSI C code base; easy to embed, extend, and -- most importantly for game developers -- it has a minimal runtime footprint (one of the fastest interpreted languages). It's also a great combination of easy to learn/read/write syntax, but with powerful features like coroutines which can be very useful in games.
You might find this article by a Game developer useful in understanding why embedded languages are used.
http://www.grimrock.net/2012/07/25/making-of-grimrock-rapid-programming/
Another good reason, is unless you are sharing your source code with your game users and they are all C programmers, languages such as lua make it possible for users to extend the game, for example look at World of Warcraft.