Regression

Linear Regression

Prerequisites

linearity
homoscedasticity
multivariate normality
independence of errors
lack of multicollinearity

Principal component regression

Source

apply PCA (Dimensionality Reduction#Feature Selection) to generate principal components from the predictor variables, with the number of principal components matching the number of original features p
keep the first k principal components that explain most of the variance (where k < p), where k is determined by cross-validation
fit a linear regression model on these k principal components

Partial least squares (PLS) regression

when dependent variables are multiple and can be correlated
similar to Regression#Principal component regression but run PCA and regression in one go (apply PCA on both independent and dependent variables)

Logistic Regression

A logistic regression model:

l o g i t (p) = l n (\frac{p}{1 - p}) = β_{0} + β_{1} X_{1} + β_{2} X_{2} + β_{k} X_{k}

The odd ratio is the ratio of two groups' odds of some outcome:

o d d r a t i o = \frac{p_{a}}{1 - p_{a}} / \frac{p_{b}}{1 - p_{b}}

Odd ratios can be derived from logistic coefficients:

β_{x} = l o g i t (p_{a}) - l o g i t (p_{b}) = l n (o d d r a t i o)

Multivariate regression

Info

Multivariate regression != Multiple regression

R-Squared: coefficient of determination

R^{2} = 1 - \frac{s u m s q u a r e d r e g r e s s i o n (S S R)}{t o t a l s u m o f s q u a r e s (S S T)} = 1 - \frac{\sum (y_{i} - \hat{y_{i}})^{2}}{\sum (y_{i} - \overset{―}{y})^{2}}

It assesses the goodness of fit of regression models.
It represents the proportion of the variance for a dependent variable that's explained by an independent variable or variables in a regression model.
It can be negative in extreme cases, but usually it is in $[0, 1]$ .

You can regard the R-Squared as how much the total variance of y is captured by the model (rather than errors), i.e.

v a r (y) = v a r (X \hat{β}) + v a r (e)

R^{2} = \frac{v a r (X \hat{β})}{v a r (y)}

so it measures the goodness of fit, but it does not validate the model.

Adjusted R-Squared

Adjusted R-squared is a modified version of R-squared that has been adjusted for the number of predictors in the model.

A d j R^{2} = 1 - (1 - R^{2}) \frac{n - 1}{n - p - 1}

where $p$ is the number of regressors/predictors, $n$ is the sample size.

Why you need adjusted R-squared?
It is important, because adding independent variables will make the R-squared never decrease, then you cannot tell whether the increase of R-squared is due to the goodness of fit or more variables.
It includes the penalising factor that penalises you for adding independent variables that don't help your model.