How can I turn record array from arff file into nd

Question

ARFF documentation tells me that my file is being read as a record array but I can't seem to convert it to ndarray like a normal record array. There should be 11055 examples with 31 features.

>>> dataset.shape
(11055,)
>>> dataset[0]
(b'1', b'1', b'1', b'1', b'1', b'-1', b'1', b'1', b'-1', b'1', b'1', b'1', b'1', b'0', b'0', b'-1', b'1', b'1', b'0', b'1', b'1', b'1', b'1', b'1', b'1', b'1', b'1', b'1', b'0', b'1', b'1')
>>> dataset.dtype
dtype([('having_IP_Address', 'S2'), ('URL_Length', 'S2'), ('Shortining_Service', 'S2'), ('having_At_Symbol', 'S2'), ('double_slash_redirecting', 'S2'), ('Prefix_Suffix', 'S2'), ('having_Sub_Domain', 'S2'), ('SSLfinal_State', 'S2'), ('Domain_registeration_length', 'S2'), ('Favicon', 'S2'), ('port', 'S2'), ('HTTPS_token', 'S2'), ('Request_URL', 'S2'), ('URL_of_Anchor', 'S2'), ('Links_in_tags', 'S2'), ('SFH', 'S2'), ('Submitting_to_email', 'S2'), ('Abnormal_URL', 'S2'), ('Redirect', 'S1'), ('on_mouseover', 'S2'), ('RightClick', 'S2'), ('popUpWidnow', 'S2'), ('Iframe', 'S2'), ('age_of_domain', 'S2'), ('DNSRecord', 'S2'), ('web_traffic', 'S2'), ('Page_Rank', 'S2'), ('Google_Index', 'S2'), ('Links_pointing_to_page', 'S2'), ('Statistical_report', 'S2'), ('Result', 'S2')])

Basically, I am trying to turn this record array stored in dataset into a ndarray and reshape it to match the vector dimensions. The problem seems to be that the ndarray that I am left with is a list of objects with that long record dtype rather than a list of lists. I am just not sure how to convert that dtype into a list.

from scipy.io import arff
import urllib.request
import io
import numpy as np

# this just reads the arff from its URL 
url = "https://archive.ics.uci.edu/ml/machine-learning-databases/00327/Training%20Dataset.arff"
ftpstream = urllib.request.urlopen(url)
dataset, meta = arff.loadarff(io.StringIO(ftpstream.read().decode('utf-8')))

num_features = len(meta.names())
num_examples = dataset.shape[0]
dataset.view(np.ndarray).reshape(num_examples, num_features)

This last line causes error ValueError: cannot reshape array of size 11055 into shape (11055,31).

What I am ultimately looking to end up with is a ndarray with shape(11055,31) and a numeric dtype.

You can find the data here. But here is what the file looks like:

@relation phishing

@attribute having_IP_Address  { -1,1 }
@attribute URL_Length   { 1,0,-1 }
@attribute Shortining_Service { 1,-1 }
@attribute having_At_Symbol   { 1,-1 }
@attribute double_slash_redirecting { -1,1 }
@attribute Prefix_Suffix  { -1,1 }
@attribute having_Sub_Domain  { -1,0,1 }
@attribute SSLfinal_State  { -1,1,0 }
@attribute Domain_registeration_length { -1,1 }
@attribute Favicon { 1,-1 }
@attribute port { 1,-1 }
@attribute HTTPS_token { -1,1 }
@attribute Request_URL  { 1,-1 }
@attribute URL_of_Anchor { -1,0,1 }
@attribute Links_in_tags { 1,-1,0 }
@attribute SFH  { -1,1,0 }
@attribute Submitting_to_email { -1,1 }
@attribute Abnormal_URL { -1,1 }
@attribute Redirect  { 0,1 }
@attribute on_mouseover  { 1,-1 }
@attribute RightClick  { 1,-1 }
@attribute popUpWidnow  { 1,-1 }
@attribute Iframe { 1,-1 }
@attribute age_of_domain  { -1,1 }
@attribute DNSRecord   { -1,1 }
@attribute web_traffic  { -1,0,1 }
@attribute Page_Rank { -1,1 }
@attribute Google_Index { 1,-1 }
@attribute Links_pointing_to_page { 1,0,-1 }
@attribute Statistical_report { -1,1 }
@attribute Result  { -1,1 }


@data
-1,1,1,1,-1,-1,-1,-1,-1,1,1,-1,1,-1,1,-1,-1,-1,0,1,1,1,1,-1,-1,-1,-1,1,1,-1,-1
1,1,1,1,1,-1,0,1,-1,1,1,-1,1,0,-1,-1,1,1,0,1,1,1,1,-1,-1,0,-1,1,1,1,-1
1,0,1,1,1,-1,-1,-1,-1,1,1,-1,1,0,-1,-1,-1,-1,0,1,1,1,1,1,-1,1,-1,1,0,-1,-1
1,0,1,1,1,-1,-1,-1,1,1,1,-1,-1,0,0,-1,1,1,0,1,1,1,1,-1,-1,1,-1,1,-1,1,-1
1,0,-1,1,1,-1,1,1,-1,1,1,1,1,0,0,-1,1,1,0,-1,1,-1,1,-1,-1,0,-1,1,1,1,1
-1,0,-1,1,-1,-1,1,1,-1,1,1,-1,1,0,0,-1,-1,-1,0,1,1,1,1,1,1,1,-1,1,-1,-1,1
1,0,-1,1,1,-1,-1,-1,1,1,1,1,-1,-1,0,-1,-1,-1,0,1,1,1,1,1,-1,-1,-1,1,0,-1,-1
1,0,1,1,1,-1,-1,-1,1,1,1,-1,-1,0,-1,-1,1,1,0,1,1,1,1,-1,-1,0,-1,1,0,1,-1
1,0,-1,1,1,-1,1,1,-1,1,1,-1,1,0,1,-1,1,1,0,1,1,1,1,1,-1,1,1,1,0,1,1
1,1,-1,1,1,-1,-1,1,-1,1,1,1,1,0,1,-1,1,1,0,1,1,1,1,1,-1,0,-1,1,0,1,-1
1,1,1,1,1,-1,0,1,1,1,1,1,-1,0,0,-1,-1,-1,0,1,1,1,1,-1,1,1,1,1,-1,-1,1
1,1,-1,1,1,-1,1,-1,-1,1,1,1,1,-1,-1,-1,-1,-1,0,1,1,1,1,-1,-1,-1,-1,1,0,-1,-1
-1,1,-1,1,-1,-1,0,0,1,1,1,-1,-1,-1,1,-1,1,1,0,-1,1,-1,1,1,-1,-1,-1,1,0,1,-1
1,1,-1,1,1,-1,0,-1,1,1,1,1,-1,-1,-1,-1,1,1,0,1,1,1,1,-1,-1,0,-1,1,1,1,-1
1,1,-1,1,1,1,-1,1,-1,1,1,-1,1,0,1,1,1,1,0,1,1,1,1,1,-1,1,-1,1,-1,1,1
1,-1,-1,-1,1,-1,0,0,1,1,1,1,-1,-1,0,-1,1,1,0,1,1,1,1,1,-1,-1,-1,1,0,1,-1
1,-1,-1,1,1,-1,1,1,-1,1,1,-1,1,0,-1,-1,-1,-1,0,1,1,1,1,1,-1,0,-1,1,1,-1,-1

Answer 1

Looking at the file, we can see that all the fields are of categorical type, rather than numeric. Aside from that, your array is a regular ndarray with a complicated dtype. Since that's not something you can change, you will have to convert the structure and dtype of your array. The neatest approach (although not the most efficient) would be

dataset = np.array(dataset.tolist(), dtype=np.int8)

tolist will convert the array into a list of tuples, which the simple dtype int8 will then cause to be reassembled into a regular array.

This question was the basis for converting numpy array of string fields to numerical format.