How does the Go1 compiler work?

Question

I've been dabbling with Go for about a month for a school project and I noticed the go/ast, go/token, go/parser, etc. packages in the src/pkg/go folder. However, the gc compiler was based on C files located in src/cmd/gc.

My question regards the new go command in Go1 that builds and runs programs: does this tool depend on the packages I referenced above? i.e. if I added a new token to /go/token/token.go, would it be recognized by the new go compiler?

Answer 1

Note (December 18th, 2013), there are plans to move the compiler from C to Go itself:

"Go 1.3+ Compiler Overhaul" (Russ Cox)

In that context packages like go/parser will be involved, and the "Phase 5" mentions:

Replace the front end with the latest (perhaps new) versions of go/parser and go/types.
Robert Griesemer has discussed the possibility of designing new go/parser and go/types APIs at some point, based on experience with the current ones (and under new names, to preserve Go 1 compatibility).
The work of connecting them to a compiler back end may help guide design of new APIs.

---

This is probably a testimony on how stable the language has become, since the old "A Tour of Go" (June 2012) mentioned before clearly stated:

The fact that Go wasn’t written in itself also made it much easier to make significant language changes.
Before the initial release we went through a handful of wholesale syntax upheavals, and I’m glad we didn’t have to worry about how we were going to rebootstrap the compiler or ensure some kind of backwards compatibility during those changes.

The question "Is there any plan to bootstrap Go in Go, to write the Go compiler in Go?" mentioned at the time (again, June 2012):

There’s no immediate plan. Go does ship with a Go program parser written in Go, so the first piece is already done, and there’s an experimental type checker in the works, but those are mainly for writing program analysis tools.

I’ve worked on bootstrapped languages in the past, and I found that bootstrapping is not necessarily a good fit for languages that are changing frequently. It reminded me of climbing a cliff and screwing hooks into the cliff once in a while to catch you if you fall.

Answer 2

does this tool depend on the packages I referenced above?

The 'go' tool does depend on those packages

if I added a new token to /go/token/token.go, would it be recognized by the new go compiler?

No.

Answer 3

The Go compiler is written in pure C and does not use the packages under go/. In the Go source tree, its lexer lives in src/cmd/gc/lex.c and its Bison grammar is src/cmd/gc/go.y.

The go/ packages are used in tools like godoc, gofmt, and various go tool subcommands. Maybe someday they can be used to write a Go compiler in Go as well, but no one's gotten very far on that path yet.