EN231220191405
Theoretical explanation
The standardization of data for machine learning models involves the transformation of primary data so that their distribution has an average value of 0 and a standard deviation of 1. The average column value will be subtracted from each value in the data column, and then what will come out will be divided by the standard deviation of the data column . The described process applies to each column separately.
In [1]:
import tensorflow as tf
import pandas as pd
from sklearn import model_selection
import numpy as np
df = pd.read_csv('c:/TF/AirQ_filled2.csv', usecols=['CO(GT)','PT08.S1(CO)','C6H6(GT)','PT08.S2(NMHC)','NOx(GT)','PT08.S3(NOx)','NO2(GT)','PT08.S4(NO2)','PT08.S5(O3)','T','RH', 'AH'
,'Month','Weekday','Hours'])
df.head(3)
Out[1]:
In the previous post I built a linear regression model that had a square of 90
http://sigmaquality.pl/python/linear-regression-3/
I change the names of the columns to use the Tensorflow model
In [2]:
df.columns = ['CO_GT', 'PT08.S1_CO', 'C6H6_GT', 'PT08.S2_NMHC',
'NOx_GT', 'PT08.S3_NOx', 'NO2_GT', 'PT08.S4_NO2', 'PT08.S5_O3',
'T', 'RH', 'AH', 'Month', 'Weekday', 'Hours']
Standardization of data in Sklearn¶
In [3]:
df.head(2)
Out[3]:
Matrix created from the table above.
In [4]:
a = np.array(df)
a
Out[4]:
The average of the columns is:
In [5]:
np.mean(a, axis=0)
Out[5]:
The standard deviation of the columns is:
In [6]:
np.std(a, axis=0)
Out[6]:
We transform the data into a standard normal distribution with an average of 0 and standard deviation of 1.
In [7]:
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
scaled_data = scaler.fit_transform(a)
k = scaler.transform(df)
k
Out[7]:
The mean and standard deviation of the columns after standardization.
In [8]:
print("standard deviation: ",np.std(k, axis=0))
#k=k.astype(int)
print()
print("mean: ",np.mean(k, axis=0))
We substitute standardized data for Tensorflow
In [9]:
conc = np.vstack(k)
conc
Out[9]:
In [10]:
SKS = pd.DataFrame(conc)
SKS.columns = ['CO_GT', 'PT08.S1_CO', 'C6H6_GT', 'PT08.S2_NMHC',
'NOx_GT', 'PT08.S3_NOx', 'NO2_GT', 'PT08.S4_NO2', 'PT08.S5_O3',
'T', 'RH', 'AH', 'Month', 'Weekday', 'Hours']
In [11]:
SKS.head(3)
Out[11]:
Teraz dodajemy wartość wynikowa która nie została wystandaryzowana.
Now that we have standardized data, we can create a new Tensorflow linear regression model
Tensorflow linear regression model without standardization
We take the original data without standardization
Step 1. Divides the data into a test and training set</span>
In [12]:
df_train=df.sample(frac=0.8,random_state=200)
df_test=df.drop(df_train.index)
print(df_train.shape, df_test.shape)
In [13]:
df_train.head(3)
Out[13]:
Step 2. Converts data to Tensorflow format¶
In [14]:
COL = ['CO_GT', 'PT08.S1_CO', 'C6H6_GT', 'PT08.S2_NMHC',
'NOx_GT', 'PT08.S3_NOx', 'NO2_GT', 'PT08.S4_NO2', 'PT08.S5_O3',
'T', 'RH', 'AH', 'Month', 'Weekday', 'Hours']
features = [tf.feature_column.numeric_column(k) for k in COL]
features
Out[14]:
Step 3. Tensorflow Linear Regression Estimator
katalog: train_Wojtek
In [15]:
model = tf.estimator.LinearRegressor(model_dir="train_Wojtek", feature_columns=features)
Step 4. Defining how to feed the model and what is the result variable
In [16]:
FEATURES = ['CO_GT','PT08.S1_CO', 'C6H6_GT', 'PT08.S2_NMHC',
'NOx_GT', 'PT08.S3_NOx', 'NO2_GT', 'PT08.S4_NO2', 'PT08.S5_O3',
'T', 'RH', 'AH', 'Month', 'Weekday', 'Hours']
LABEL= 'CO_GT'
def get_input_fn(data_set, num_epochs=None, n_batch = 128, shuffle=True):
return tf.estimator.inputs.pandas_input_fn(
x=pd.DataFrame({k: data_set[k].values for k in FEATURES}),
y = pd.Series(data_set[LABEL].values),
batch_size=n_batch,
num_epochs=num_epochs,
shuffle=shuffle)
Step 5. Training the model
In [17]:
model.train(input_fn=get_input_fn(df_train,
num_epochs=None,
n_batch = 128,
shuffle=False),
steps=1000)
Out[17]:
Step 6. Model assessment
In [18]:
ev = model.evaluate(
input_fn=get_input_fn(df_test,
num_epochs=1,
n_batch = 356,
shuffle=False))
Calculation of R2
I make a prediction on a test set.
In [19]:
y = model.predict(
input_fn=get_input_fn(df_test,
num_epochs=1,
n_batch = 256,
shuffle=False))
In [20]:
import itertools
predictions = list(p["predictions"] for p in itertools.islice(y, 3000))
#print("Predictions: {}".format(str(predictions)))
Przekształcam wynik na dataframe
In [21]:
import numpy as np
conc = np.vstack(predictions)
ZHP = pd.DataFrame(conc)
ZHP.rename(columns={0:'y_pred'}, inplace=True)
kot = ZHP['y_pred'].values
kot = kot.astype('float32')
kot.dtype
Out[21]:
I agree on the data formats for the theoretical and empirical variable
In [22]:
y = df_test['CO_GT'].values
y = y.astype('float32')
y.dtype
Out[22]:
In [23]:
PZU = pd.DataFrame({'y': y, 'y_pred': kot })
PZU.dtypes
Out[23]:
In [24]:
def R_squared(y, y_pred):
residual = tf.reduce_sum(tf.square(tf.subtract(y,y_pred)))
total = tf.reduce_sum(tf.square(tf.subtract(y, tf.reduce_mean(y))))
r2 = tf.subtract(1.0, tf.div(residual, total))
return r2
In [25]:
residual = tf.reduce_sum(tf.square(tf.subtract(y,kot)))
total = tf.reduce_sum(tf.square(tf.subtract(y, tf.reduce_mean(y))))
r2 = tf.subtract(1.0, tf.div(residual, total))
r2
Out[25]:
In [26]:
sess = tf.Session()
a = sess.run(r2)
print('R Square parameter: ',a)
Tensorflow linear regression model with standardization
We are now substituting standardized data
WE STANDARDIZE INDEPENDENT VARIABLES ONLY BECAUSE WE WANT TO HAVE EASY TO READ THE RESULT IN THE LINEAR REGRESSION MODEL
Step 1. Divides the data into a test and training set
In [27]:
del SKS['CO_GT']
WKD = pd.concat([df['CO_GT'], SKS], axis=1, sort=False)
WKD.head(3)
Out[27]:
In [28]:
df_trainS=WKD.sample(frac=0.8,random_state=200)
df_testS=WKD.drop(df_trainS.index)
print(df_trainS.shape, df_testS.shape)
df_trainS.head(3)
Out[28]:
Step 2. Converts data to Tensorflow format
In [29]:
FOL = ['CO_GT', 'PT08.S1_CO', 'C6H6_GT', 'PT08.S2_NMHC',
'NOx_GT', 'PT08.S3_NOx', 'NO2_GT', 'PT08.S4_NO2', 'PT08.S5_O3',
'T', 'RH', 'AH', 'Month', 'Weekday', 'Hours']
PCK = [tf.feature_column.numeric_column(k) for k in FOL]
PCK
Out[29]:
Step 3. Tensorflow Linear Regression Estimator
katalog: train_Wojtek
In [30]:
model = tf.estimator.LinearRegressor(model_dir="train_Wojtek7", feature_columns=PCK)
Step 4. Defining how to feed the model and what is the result variable
Step 5. Training the model
In [31]:
model.train(input_fn=get_input_fn(df_trainS,
num_epochs=None,
n_batch = 128,
shuffle=False),
steps=1000)
Out[31]:
Step 6. Model assessment
In [32]:
ev = model.evaluate(
input_fn=get_input_fn(df_testS,
num_epochs=1,
n_batch = 356,
shuffle=False))
Calculation of R2
I make a prediction on a test set.
In [33]:
y = model.predict(
input_fn=get_input_fn(df_testS,
num_epochs=1,
n_batch = 256,
shuffle=False))
In [34]:
import itertools
predictions = list(p["predictions"] for p in itertools.islice(y, 3000))
In [35]:
import numpy as np
conc = np.vstack(predictions)
ZHP = pd.DataFrame(conc)
ZHP.rename(columns={0:'y_pred'}, inplace=True)
kot = ZHP['y_pred'].values
kot = kot.astype('float32')
kot.dtype
Out[35]:
In [36]:
y = df_testS['CO_GT'].values
y = y.astype('float32')
y.dtype
Out[36]:
In [37]:
PZU = pd.DataFrame({'y': y, 'y_pred': kot })
PZU.dtypes
Out[37]:
In [38]:
def R_squared(y, y_pred):
residual = tf.reduce_sum(tf.square(tf.subtract(y,y_pred)))
total = tf.reduce_sum(tf.square(tf.subtract(y, tf.reduce_mean(y))))
r2 = tf.subtract(1.0, tf.div(residual, total))
return r2
In [39]:
residual = tf.reduce_sum(tf.square(tf.subtract(y,kot)))
total = tf.reduce_sum(tf.square(tf.subtract(y, tf.reduce_mean(y))))
r2 = tf.subtract(1.0, tf.div(residual, total))
r2
Out[39]:
In [40]:
sess = tf.Session()
a = sess.run(r2)
print('R Square parameter: ',a)