Confusion of using “.” notation with __index and n

I am confused of the following two syntaxes using "."

From what I understand, __index is called when a key doesn't exist in a table but exists in its metatable. So why does the list table call __index and then assign itself to list.__index?

list = {}
list.__index = list

setmetatable(list, { __call = function(_, ...)
local t = setmetatable({length = 0}, list)
  for _, v in ipairs{...} do t:push(v) end
  return t
end })

function list:push(t)
  if self.last then
    self.last._next = t
    t._prev = self.last
    self.last = t
  else
   self.first = t
   self.last = t
  end
  self.length = self.length + 1
end 
  .
  .
  .
local l = list({ 2 }, {3}, {4}, { 5 })

Does Window.mt simply create a table? Why do we need Window = {} as a namespace here?

Window = {}  -- create a namespace    
Window.mt = {}  -- create a metatable
Window.prototype = {x=0, y=0, width=100, height=100, } 

function Window.new (o)  
    setmetatable(o, Window.mt)
    return o
end

Window.mt.__index = function (table, key)
    return Window.prototype[key]
end

w = Window.new{x=10, y=20}
print(w.width)    --> 100

So why does the list table call index and then assign itself to list.index?

Nowhere in your code does the list table call __index. The assignment part however is a common Lua idiom (aka. hack) to save some memory. Conceptually there are 4 different kinds of tables involved:

list objects (the tables created via {length=0} in your code)
a metatable (containing an __index field) that modifies the behavior of list objects when you try to access non-existing fields in the object
the list class, which holds all the methods for list objects (like the push method), and also serves as a constructor for list objects
a metatable (containing a __call field) for the list class, so that you can call the list table as if it were a function

As metatable fields always start with two underscores (__), and normal methods usually don't, you can put metatable fields and normal methods side by side into a single table without conflict. And this is what happened here. The list class table also serves as metatable for list objects. So using this trick you can save the memory you would normally need for the separate metatable (the size in bytes for Lua 5.2 on an x86-64 Linux is shown in square brackets in the table title bars, btw.):

Does Window.mt simply create a table?

No, {} creates a table. However, this new table is saved under key "mt" in the Window table, probably to give users of this Window "class" direct access to the metatable that is used for window objects. Given only the code you showed this is not strictly necessary, and you could have used a local variable instead.

Why do we need Window = {} as a namespace here?

In principle, you could store Window.mt, Window.new, and Window.prototype separately, but that would get cumbersome if you have multiple "classes" like Window. This way you can avoid name clashes, and using the Window "class" looks nicer.

Another reason might be that require can only return a single value from a module definition, and if you want to export multiple values (like new, mt, and prototype) from a module, you need a table to wrap them together (or use global variables, but that is considered bad style).