emacs lexical scoping and quoted variable name

Question

I was experimenting with interplay between Emacs lexical scoping (new feature of Emacs 24) and add-to-list and found the interplay confusing and I don't know how to make sense of it. Here is a minimal example, except I use set instead of add-to-list. (set is similar to add-to-list in that it usually takes a quoted variable name)

(eval
 '(progn
    (setq a "global")
    (let ((a "apple"))
      (defun my-print-a ()
        (print a)
        (set 'a "by set")
        (print a))
      (setq a "mature apple"))
    (let ((a "banana"))
      (my-print-a))
    (print a))
 t) ;; t for lexical scoping

Above code prints "mature apple", "mature apple", "by set" in order. The first print result, "mature apple", is as expected for lexical scoping (with support for lexical closures), nothing surprising to see here. But the results of the second and the third print are surprising to me. It's as if (set 'a "by set") is only recognizing and affecting the global binding of the name a.

Is this an intended behavior? Or is this a bug? If intended, how does one make sense of this behavior?

Am I right in assuming that it's always the global binding that set affects as long as lexical scoping is on?

With (eval '(progn ...) nil), things work as expected of dynamic scoping and the behavior of (set 'a ...) is the same as that of (setq a ...) in that case. Only when one uses lexical scoping and a quoted variable together, this gotcha shows up.

---

Update:

it seems to be an intended behavior according to the manual. On Void Variables, manual says

Under lexical binding rules, the value cell only holds the variable's global value, i.e. the value outside of any lexical binding construct. When a variable is lexically bound, the local value is determined by the lexical environment; the variable may have a local value if its symbol's value cell is unassigned.

and on Lexical Binding

functions like symbol-value, boundp, and set only retrieve or modify a variable's dynamic binding (i.e. the contents of its symbol's value cell).

symbol-value, boundp, set are functions usually called with quoted variable names ((symbol-value 'var) (boundp 'var) (set 'var 123)). These functions only get or set a symbol's value cell, and under lexical binding rules, the value cell only holds the global value. So under lexical binding, use of quoted variables only get or set global values (unless the variable is a special variable). Although it still seems odd in that the result is neither lexical (apple) nor dynamic (banana).

Answer 1

The code is not written in the way that Emacs expects a lexical-scoping-enabled program to be written.

http://www.gnu.org/software/emacs/manual/html_node/elisp/Definitions.html

defvar and defconst are special forms that define a symbol as a global variable—a variable that can be accessed at any point in a Lisp program.

(...)

In principle, you can assign a variable value to any symbol with setq, whether not it has first been defined as a variable. However, you ought to write a variable definition for each global variable that you want to use; otherwise, your Lisp program may not act correctly if it is evaluated with lexical scoping enabled.

http://www.gnu.org/software/emacs/manual/html_node/elisp/Lexical-Binding.html

Note that functions like symbol-value, boundp, and set only retrieve or modify a variable's dynamic binding (i.e. the contents of its symbol's value cell). Also, the code in the body of a defun or defmacro cannot refer to surrounding lexical variables.

http://www.gnu.org/software/emacs/manual/html_node/elisp/Setting-Variables.html

Special Form: setq [symbol form]...

This special form is the most common method of changing a variable's value. (...) The current binding of the symbol is changed.

(...)

Function: set symbol value

This function puts value in the value cell of symbol.

(...)

When dynamic variable binding is in effect (the default), set has the same effect as setq, apart from the fact that set evaluates its symbol argument whereas setq does not. But when a variable is lexically bound, set affects its dynamic value, whereas setq affects its current (lexical) value.

If we add defvar that defines a as a global variable, we can see that all the references to a in the my-print-a function turn out to be dynamically bound as the manual explains:

(eval
 '(progn
    (defvar a nil)
    (setq a "global")
    (let ((a "apple"))
      (defun my-print-a ()
        (print a)                       ; "banana"
        (set 'a "by set")              
        (print a))                      ; "by set"
      (setq a "mature apple"))
    (let ((a "banana"))
      (my-print-a))
    (print a))                          ; "global"
 t)