Why is this function not breaking up this input st

Question

I'm trying to break up a string into "symbols" with C++ for further work. I haven't written anything in C++ for a long while, so forgive me if there is something inherently wrong with this code.

The purpose of the symbolize() function below is to break up a string, such as "5+5", into a vector of strings, eg {"5","+","5"}. It's not working. If you think the code is too messy, please suggest a way to simplify it.

Here's my code so far:

#include <iostream>
#include <string>
#include <vector>
#include <ctype.h>
#include <sstream>

using namespace std;

vector<string> symbolize(string);

int main(int argc, const char * argv[])
{

    string input;
    cin >> input;

    vector<string> symbols;

    symbols = symbolize(input);

    for(int i=0;i<symbols.size();i++){
        cout<<symbols.at(i) << endl;
    }

    return 0;
}


vector<string> symbolize(string input){
    int position = 0;
    char c;
    stringstream s;
    vector<string> symbols;
    enum symbolType {TEXT,OPERATOR}symbolType,charType;

    while(position < input.size()){
        c = input.at(position);
        if(isalnum(c))symbolType = TEXT;
        else symbolType = OPERATOR;
        charType = symbolType;

        while(symbolType == charType){
            s << c;
            position++;
            if(position>=input.length())break;
            c = input.at(position);
            if(isalnum(c)) charType = TEXT;
            else charType = OPERATOR;
        }

        symbols.push_back(s.str());
        s.clear();
    }

    return symbols;
}

Thanks for taking a look.

Edit: BTW, I should mention that the function returns the fist "token", eg "5+5" -> "5"

Edit2: I was mistaken. I just tried "5+5", and it returned {"5","5+","5+5"}. However, it only returns the first before a space. Sorry for the confusion!

Edit3: Thank you all! For those who may come across this page in the future, here's the code when everything's said and done:

#include <iostream>
#include <string>
#include <vector>
#include <ctype.h>
#include <sstream>

using namespace std;

vector<string> symbolize(string);

int main(int argc, const char * argv[])
{

    string input;
    getline(cin,input);

    vector<string> symbols;

    symbols = symbolize(input);

    for(int i=0;i<symbols.size();i++){
        cout<<symbols.at(i) << endl;
    }

    return 0;
}


vector<string> symbolize(string input){
    int position = 0;
    char c;
    //stringstream s;
    vector<string> symbols;
    enum symbolType {TEXT,OPERATOR}symbolType,charType;

    while(position < input.size()){
        stringstream s;
        c = input.at(position);
        if(isalnum(c))symbolType = TEXT;
        else symbolType = OPERATOR;
        charType = symbolType;

        while(symbolType == charType){
            s << c;
            position++;
            if(position>=input.length())break;
            c = input.at(position);
            if (isspace(c)||c=='\n'){position++; break;}
            if(isalnum(c)) charType = TEXT;
            else charType = OPERATOR;
        }

        symbols.push_back(s.str());
    }

    return symbols;
}

Answer 1

If you want to read an entire line instead of just one word, use getline instead of operator>>. See http://www.cplusplus.com/reference/string/getline/ for details, or just change line 14 to "getline(cin, input);".

Also, if you want to output "5", "+", "5" instead of "5", "5+", "5+5", you need to reset the stringstream each time through the loop, and clear doesn't do that. The simplest way around this is to just declare the stringstream in the outer loop and get rid of the clear call.

Answer 2

stringstream::clear doesn't clear the string buffer (only the error state).

you can use stringstream::str(x) to set the string buffer, so s.str(string()) or s.str("") instead of s.clear() will clear the string buffer.

Also, the operator<<(istream, ...) only reads until whitespace.

For reading you can try use:

• istream::get to read one character at a time; or;
• std::getline(istream,...) to read one line at a time; or;
• istream::read to read an arbitrary number of characters into a buffer.

http://en.cppreference.com/w/cpp/io/basic_istream

Answer 3

If you move stringstream s; inside the first while loop, you should achieve your aim.

s.clear() only resets the error state flags for the stringstream, it's not like std::string::clear()