How to use Golang custom scanner string literals a

问题:

I have been trying to figure out how to implement what I originally thought would be a simple program. I have a text file of quotations that are all separated by ‘$$’

I want the program to parse the quotation file and randomly select 3 quotes to display and standard output.

There are 1022 quotes in the file.

When I attempt to split the file I get this error: missing '

I can’t seem to figure out how to assign $$ with a string literal, I keep getting:
missing '

This is the custom scanner:

onDollarSign := func(data []byte, atEOF bool) (advance int, token []byte, err error) {  
    for i := 0; i < len(data); i++ { 
        //if data[i] == "$$" {              # this is what I did originally
        //if data[i:i+2] == "$$" {    # (mismatched types []byte and string)
        //if data[i:i+2] == `$$` {    # throws (mismatched types []byte and string)
        // below throws syntax error: unexpected $ AND missing '
        if data[1:i+2] == '$$' {   
            return i + 1, data[:i], nil  
        }  
    }  

The string literal works fine if I only use one $.

For some reason only 71 quotations are loaded into the quotes slice. I'm not sure how to expand. To allow all 1022 quotes to be stored in memory.

I've been having a really difficult time trying to figure out how to do this. this is what I have right now:

package main
import (  
    "bufio"  
    "fmt"  
    "log"  
    "math/rand"  
    "os"  
    "time"  
)  

func main() {  
    rand.Seed(time.Now().UnixNano()) // Try changing this number!  
    quote_file, err := os.Open("/Users/bryan/Dropbox/quotes_file.txt")  
    if err != nil {  
        log.Fatal(err)  
    }  
    scanner := bufio.NewScanner(quote_file)  
    // define split function  
    onDollarSign := func(data []byte, atEOF bool) (advance int, token []byte, err error) {  
        for i := 0; i < len(data); i++ {  
            if data[i] == '$$' {  
                return i + 1, data[:i], nil  
            }  
        }  
        fmt.Print(data)  
        return 0, data, bufio.ErrFinalToken  
    }  
    scanner.Split(onDollarSign)  
    var quotes []string  

    // I think this will scan the file and append all the parsed quotes into quotes  
    for scanner.Scan() {  
        quotes = append(quotes, scanner.Text())  

    }  
    if err := scanner.Err(); err != nil {  
        fmt.Fprintln(os.Stderr, "reading input:", err)  
    }  
    fmt.Print(len(quotes))  
    fmt.Println("quote 1:", quotes[rand.Intn(len(quotes))])  
    fmt.Println("quote 2:", quotes[rand.Intn(len(quotes))])  
    fmt.Println("quote 3:", quotes[rand.Intn(len(quotes))])  
}  

回答1:

Using a scanner if you end up reading the whole file anyway is kind of convoluted. I'd read the whole file and then simply split it into the list of quotes:

package main

import (
    "bytes"
    "io/ioutil"
    "log"
    "math/rand"
    "os"
)

func main() {
    // Slurp file.
    contents, err := ioutil.ReadFile("/Users/bryan/Dropbox/quotes_file.txt")
    if err != nil {
            log.Fatal(err)
    }

    // Split the quotes
    separator := []byte("$$") // Convert string to []byte
    quotes := bytes.Split(contents, separator)

    // Select three random quotes and write them to stdout
    for i := 0; i < 3; i++ {
            n := rand.Intn(len(quotes))
            quote := quotes[n]

            os.Stdout.Write(quote)
            os.Stdout.Write([]byte{'\n'}) // new line, if necessary
    }
}

Using a scanner would make sense if you selected three quotes before reading the file; then you can stop reading after you have reached the last quote.

回答2:

In golang single quote ' is used for single chars(so called "runes" - internally it is an int32 with unicode code point), and double quote for strings which can be longer than 1 char: "$$".

So parser awaits a closing rune chanacter ' just after the first dollar sign.

Here's a good article: https://blog.golang.org/strings

UPDATE: If you want to avoid casting all data to string you may check this way:

...
   onDollarSign := func(data []byte, atEOF bool) (advance int, token []byte, err error) {  
        for i := 0; i < len(data); i++ {  
            if data[i] == '$' && data[i+1] == '$' {  /////   <----
                return i + 1, data[:i], nil  
            }  
        }  
        fmt.Print(data)  
        return 0, data, bufio.ErrFinalToken  
    } 
...

回答3:

I rewrote your split function based off the stdlib func bufio.Scanlines.

I haven't tested it thoroughly so you should exercise it. Also you should decide how you want to handle whitespace such as the newline at the end of the file.

func onDollarSign(data []byte, atEOF bool) (advance int, token []byte, err error) {

    // If we are at the end of the file and there's no more data then we're done
    if atEOF && len(data) == 0 {
        return 0, nil, nil
    }

    // If we are at the end of the file and there IS more data return it
    if atEOF {
        return len(data), data, nil
    }

    // If we find a $ then check if the next rune after is also a $. If so we
    // want to advance past the second $ and return a token up to but not
    // including the first $.
    if i := bytes.IndexByte(data, '$'); i >= 0 {
        if len(data) > i && data[i+1] == '$' {
            return i + 2, data[0:i], nil
        }
    }

    // Request more data.
    return 0, nil, nil
}

回答4:

Scanning for quotes (scanQuotes) is similar to scanning for lines (bufio.ScanLines). For example,

package main

import (
    "bufio"
    "bytes"
    "fmt"
    "os"
    "strings"
)

func dropCRLF(data []byte) []byte {
    if len(data) > 0 && data[len(data)-1] == '\n' {
        data = data[0 : len(data)-1]
        if len(data) > 0 && data[len(data)-1] == '\r' {
            data = data[0 : len(data)-1]
        }
    }
    return data
}

func scanQuotes(data []byte, atEOF bool) (advance int, token []byte, err error) {
    if atEOF && len(dropCRLF(data)) == 0 {
        return len(data), nil, nil
    }
    sep := []byte("$$")
    if i := bytes.Index(data, sep); i >= 0 {
        return i + len(sep), dropCRLF(data[0:i]), nil
    }
    if atEOF {
        return len(data), dropCRLF(data), nil
    }
    return 0, nil, nil
}

func main() {
    /*
       quote_file, err := os.Open("/Users/bryan/Dropbox/quotes_file.txt")
       if err != nil {
        log.Fatal(err)
       }
    */
    quote_file := strings.NewReader(shakespeare) // test data

    var quotes []string
    scanner := bufio.NewScanner(quote_file)
    scanner.Split(scanQuotes)
    for scanner.Scan() {
        quotes = append(quotes, scanner.Text())
    }
    if err := scanner.Err(); err != nil {
        fmt.Fprintln(os.Stderr, "reading quotes:", err)
    }

    fmt.Println(len(quotes))
    for i, quote := range quotes {
        fmt.Println(i, quote)
    }
}

var shakespeare = `To be, or not to be: that is the question$$All the world‘s a stage, and all the men and women merely players. They have their exits and their entrances; And one man in his time plays many parts.$$Romeo, Romeo! wherefore art thou Romeo?$$Now is the winter of our discontent$$Is this a dagger which I see before me, the handle toward my hand?$$Some are born great, some achieve greatness, and some have greatness thrust upon them.$$Cowards die many times before their deaths; the valiant never taste of death but once.$$Full fathom five thy father lies, of his bones are coral made. Those are pearls that were his eyes. Nothing of him that doth fade, but doth suffer a sea-change into something rich and strange.$$A man can die but once.$$How sharper than a serpent’s tooth it is to have a thankless child!` + "\n"

Playground: https://play.golang.org/p/zMuWMxXJyQ

Output:

10
0 To be, or not to be: that is the question
1 All the world‘s a stage, and all the men and women merely players. They have their exits and their entrances; And one man in his time plays many parts.
2 Romeo, Romeo! wherefore art thou Romeo?
3 Now is the winter of our discontent
4 Is this a dagger which I see before me, the handle toward my hand?
5 Some are born great, some achieve greatness, and some have greatness thrust upon them.
6 Cowards die many times before their deaths; the valiant never taste of death but once.
7 Full fathom five thy father lies, of his bones are coral made. Those are pearls that were his eyes. Nothing of him that doth fade, but doth suffer a sea-change into something rich and strange.
8 A man can die but once.
9 How sharper than a serpent’s tooth it is to have a thankless child!