Machine Learning Interview Questions

Supervised learning trains on labeled data, learning a mapping from inputs to known outputs — for example predicting house prices or classifying emails. Unsupervised learning works with unlabeled data to find structure on its own, such as clustering customers or reducing dimensionality. Reinforcement learning has an agent take actions in an environment and learn from rewards and penalties over time, as in game-playing or robotics.

Both are supervised learning tasks, but they differ in the type of output. Classification predicts a discrete category or label, such as spam versus not spam, or which of several classes an image belongs to. Regression predicts a continuous numeric value, such as a price, temperature, or age. The choice of model, loss function, and evaluation metric depends on which type of problem you have.

Parameters are values the model learns from the data during training, such as the weights in a linear model or the splits in a decision tree. Hyperparameters are settings you configure before training that control how learning happens, such as the learning rate, the number of trees, the value of k, or the regularization strength. Hyperparameters are typically chosen using the validation set, often through grid search, random search, or more advanced tuning methods.

In traditional programming you write explicit rules that transform input into output. In machine learning you supply examples of inputs and desired outputs, and the algorithm derives the rules — the model parameters — that best reproduce them. That makes ML the right tool when the rules are too numerous, too subtle, or too changeable to write by hand, such as recognizing objects in photos or ranking search results, and the wrong tool when a deterministic rule already exists.

Frame the problem and define a success metric tied to a business outcome, collect and explore the data, clean it and engineer features, split it into train, validation, and test sets, train and tune candidate models, evaluate on the held-out test set, then deploy, monitor, and retrain. In practice the data stages dominate the timeline, not the modeling. Interviewers ask this to check whether you have shipped a model or only fitted one in a notebook.

Semi-supervised learning uses a small labeled set alongside a large unlabeled one, propagating structure from the unlabeled data to improve on what the labels alone could support — useful when labeling is expensive, as in medical imaging. Self-supervised learning invents labels from the data itself, for example predicting a masked word or the next token, so that a model can pretrain on vast unlabeled corpora and then be fine-tuned on a small labeled task. Self-supervised pretraining is what makes modern language and vision models practical.

A parametric model commits to a fixed number of parameters regardless of dataset size — linear regression, logistic regression, and neural networks with a fixed architecture. A non-parametric model lets its effective complexity grow with the data, as k-nearest neighbors does by storing every training point and a decision tree does by growing more splits. Parametric models are faster and need less data but impose stronger assumptions; non-parametric models are flexible but need more data and more memory.

It states that averaged over all possible problems, no learning algorithm outperforms any other — every algorithm's advantage comes from assumptions that happen to match the problem at hand. In practice it means there is no universally best model, and claims that one algorithm always wins should be treated with suspicion. It is the formal justification for trying several model families against a common validation protocol rather than defaulting to a favorite.

Inductive bias is the set of assumptions a model uses to generalize beyond the training examples it has seen, since infinitely many functions fit any finite dataset. A convolutional network assumes translation invariance and local structure; a linear model assumes additive effects; a decision tree assumes axis-aligned splits. Choosing a model is largely choosing an inductive bias, and a mismatch between that bias and the true structure of the data is why the right architecture matters more than more tuning.

Batch learning trains on the full dataset at once and produces a fixed model that must be retrained from scratch to incorporate new data. Online (incremental) learning updates the model example by example or in small chunks as data arrives, which suits streaming systems, non-stationary data, and datasets too large for memory. The trade-off is stability: online models can be pulled off course by a burst of unusual traffic, so they need monitoring and often a bounded learning rate.

Overfitting happens when a model learns the training data too well, including its noise, so it performs strongly on training data but poorly on new, unseen data — it has high variance. Underfitting is the opposite: the model is too simple to capture the underlying pattern, so it performs poorly on both training and test data — it has high bias. The goal is a model that generalizes well, sitting between these two extremes.

Bias is error from overly simplistic assumptions that cause a model to miss real patterns (underfitting), while variance is error from being too sensitive to the training data and its noise (overfitting). Increasing model complexity typically lowers bias but raises variance, and simplifying it does the reverse. The tradeoff is about finding the balance that minimizes total error on unseen data.

Compare training and validation performance: a large and widening gap — near-perfect training accuracy with much weaker validation accuracy — is the signature of overfitting. Plotting both curves against training epochs or model complexity makes the divergence point obvious, and it is where early stopping should trigger. Cross-validation gives a more reliable read than a single split, especially on small datasets where one unlucky split can mislead you either way.

Get more or more varied training data, including through augmentation; simplify the model or reduce its capacity; add regularization such as L1, L2, or dropout; use early stopping; and use cross-validation so your model selection is not itself overfitted to one split. Ensembling — bagging in particular — also reduces variance. Start with the data: more representative data fixes overfitting more reliably than any hyperparameter.

Regularization adds a penalty on model complexity to the loss function to discourage overfitting by shrinking the weights. L1 regularization (Lasso) penalizes the sum of absolute values of the weights and tends to drive some weights exactly to zero, effectively performing feature selection. L2 regularization (Ridge) penalizes the sum of squared weights and shrinks them smoothly toward zero without eliminating them. Elastic Net combines both penalties.

Geometrically, the L1 constraint region is a diamond with corners on the axes, so the point where the loss contours first touch it is very often a corner — which means one or more coefficients are exactly zero. The L2 region is a sphere with no corners, so the contact point almost never lies on an axis and weights shrink toward zero without reaching it. In gradient terms, the L1 penalty has a constant-magnitude gradient that keeps pushing a small weight all the way to zero, while the L2 gradient shrinks in proportion to the weight and fades out.

Dropout randomly deactivates a fraction of neurons on each training step, so no unit can rely on any specific other unit and the network is forced to learn redundant, distributed representations. It behaves like training an implicit ensemble of thinned networks and averaging them at inference. Dropout is applied only during training; at inference all units are active and the activations are scaled so the expected output matches, which is why forgetting to switch the model to evaluation mode silently degrades predictions.

Early stopping monitors validation loss during training and halts once it stops improving for a set number of epochs (the patience), restoring the weights from the best epoch. It is a cheap and effective regularizer because it caps how far the model can drift into memorizing the training set. The main pitfall is stopping on training loss or on the test set — the first never rises, and the second leaks test information into model selection.

A learning curve plots training and validation error against the amount of training data or the number of epochs. If both curves plateau at a high error and sit close together, the model is underfitting and you need more capacity or better features. If training error is low but validation error is much higher, you are overfitting and more data or stronger regularization will help. It is the fastest diagnostic for deciding whether to spend effort on data or on the model.

It reliably reduces variance, so it helps most when the model is overfitting. It does very little for bias: if the model family cannot represent the underlying relationship, ten times the data will only estimate the wrong function more precisely. More data also cannot fix label noise, distribution mismatch between training and production, or leakage. The learning curve tells you which regime you are in before you spend money on collection.

Feature scaling brings features onto a comparable range, commonly through normalization (rescaling to a fixed range like 0 to 1) or standardization (rescaling to zero mean and unit variance). It matters because algorithms that rely on distances or gradients — such as k-nearest neighbors, SVMs, k-means, and gradient-descent-based models — can be dominated by features with larger numeric ranges. Tree-based models like decision trees and random forests generally do not require scaling.

When one class greatly outnumbers another, accuracy becomes misleading and models tend to ignore the minority class. Common remedies include resampling — oversampling the minority class (for example with SMOTE, which synthesizes new examples) or undersampling the majority class — and using class weights so errors on the minority class are penalized more heavily. It is also important to evaluate with metrics like precision, recall, F1, or area under the precision-recall curve rather than raw accuracy.

Feature engineering is turning raw data into inputs that expose the underlying signal to the model — extracting the day of week from a timestamp, taking a ratio of two columns, aggregating a user's last thirty days of activity, or binning a skewed variable. It matters because most models can only combine the features you give them in restricted ways, so a well-constructed feature can do more than a more powerful algorithm. On tabular problems it is usually where the largest gains come from.

Standardization (zero mean, unit variance) is the safer default: it tolerates outliers better than min-max scaling and suits algorithms that assume roughly Gaussian inputs, including linear models, SVMs, and PCA. Normalization to a fixed range suits inputs with known bounds, such as pixel values, and layers that expect bounded inputs. If the data has heavy outliers, a robust scaler based on the median and interquartile range beats both.

First find out why it is missing, because that decides the fix: missing completely at random can be imputed safely, while missingness that depends on the value itself carries information and may deserve its own indicator column. Common approaches are dropping rows or columns when the loss is small, imputing with the mean, median, or mode, model-based imputation such as k-nearest neighbors, and using algorithms that handle missing values natively like gradient-boosted trees. Always fit the imputer on the training set only, or you leak information from validation into training.

One-hot encoding creates a binary column per category and is the default for nominal variables with low cardinality. Ordinal or label encoding assigns integers and is correct only when the categories genuinely have an order, since otherwise it invents a false numeric relationship. For high-cardinality features, target encoding (replacing a category with a smoothed statistic of the target) or learned embeddings are more compact, but both risk leakage and must be computed inside the cross-validation fold.

Data leakage is any situation where information that would not be available at prediction time reaches the model during training, producing validation scores that collapse in production. It happens when you scale or impute using statistics computed over the whole dataset, when a feature is derived from the target, when duplicate or near-duplicate rows straddle the train and test split, and when time-ordered data is split randomly so the model sees the future. The tell is a model that performs suspiciously well — investigate rather than celebrate.

Statistics such as the mean, standard deviation, category frequencies, or imputation values summarize the data they are computed from. If you compute them over the full dataset, the validation and test rows have influenced the transformation, so your held-out estimate is optimistic. The correct pattern is to fit the transformer on the training fold, apply it to the others, and wrap the whole chain in a pipeline so cross-validation refits it per fold automatically.

Decide first whether an outlier is an error or a real extreme value — a negative age is a data bug, a very large transaction may be exactly what you are trying to predict. Errors should be corrected or removed; genuine extremes can be kept with a robust model or loss, capped through winsorizing, or compressed with a log transform. Note which models care: linear regression and k-means are pulled hard by outliers, while tree-based models and MAE-based losses are largely indifferent.

Feature selection removes uninformative or redundant inputs to reduce overfitting, training cost, and maintenance burden. Filter methods score features independently of the model using correlation, mutual information, or a chi-square test; wrapper methods such as recursive feature elimination search subsets by repeatedly retraining; embedded methods get selection for free from the model, as L1 regularization and tree-based importances do. Filters are cheap and scale, wrappers are accurate and expensive, embedded methods are the usual practical compromise.

One-hot encoding a feature with a hundred thousand levels produces a sparse matrix that most models handle badly and trees handle very badly. The options are grouping rare levels into an "other" bucket, target encoding with smoothing and out-of-fold computation, hashing the category into a fixed number of buckets, or learning a dense embedding as a layer in a neural network. Whichever you pick, the encoding must be fitted per fold and must have a defined behavior for unseen categories at inference.

SMOTE interpolates between minority-class neighbors, so if the minority class is noisy or overlaps the majority class it manufactures examples inside the wrong region and blurs the boundary. Applying it before the train-test split, or before cross-validation folds are drawn, leaks synthetic neighbors of test points into training and inflates the score badly. It also distorts predicted probabilities, so a model that needs calibrated outputs is often better served by class weights or by simply adjusting the decision threshold.

The training set is used to fit the model, the validation set is used to tune hyperparameters and compare models during development, and the test set is held out until the very end to give an unbiased estimate of real-world performance. Keeping the test set untouched prevents you from accidentally tuning to it, which would make your performance estimate overly optimistic. A common split is something like 60/20/20 or 70/15/15, depending on data size.

Cross-validation is a technique for estimating how well a model generalizes by repeatedly training and testing on different subsets of the data. In k-fold cross-validation, the data is split into k folds; the model trains on k-1 folds and is evaluated on the remaining one, rotating until each fold has served as the validation set, then the scores are averaged. It gives a more reliable performance estimate than a single split and uses the data more efficiently, which is especially valuable on small datasets.

Precision is the fraction of predicted positives that are actually positive — true positives divided by all predicted positives — and answers how trustworthy a positive prediction is. Recall is the fraction of actual positives the model correctly identified — true positives divided by all actual positives — and answers how many real positives were caught. The F1 score is the harmonic mean of precision and recall, giving a single balanced metric that is useful when you care about both, especially on imbalanced data.

A confusion matrix is a table that summarizes a classifier's predictions against the actual labels, with cells for true positives, true negatives, false positives, and false negatives. It lets you see exactly what kinds of mistakes the model makes rather than just an overall accuracy number. From it you can derive metrics like precision, recall, accuracy, and specificity.

Whenever the classes are imbalanced or the errors have different costs. On a dataset where 1 percent of transactions are fraudulent, a model that predicts "not fraud" every time is 99 percent accurate and completely worthless. Accuracy also hides the direction of the errors, which matters when a false negative on a disease screen is far more expensive than a false positive. Report precision, recall, and the confusion matrix alongside it, or use a metric matched to the cost structure.

It depends on which error is more expensive. Prioritize recall when missing a positive is costly and a false alarm is cheap — disease screening, fraud detection, safety alerts. Prioritize precision when acting on a false positive is costly or annoying — spam filtering that hides real mail, or an automated account suspension. The honest answer names the business cost first and derives the metric from it, rather than reaching for F1 by default.

The ROC curve plots the true positive rate against the false positive rate as the classification threshold sweeps from 0 to 1, and the area under it summarizes the curve in one number. AUC equals the probability that a randomly chosen positive is scored higher than a randomly chosen negative, so 0.5 is random guessing and 1.0 is perfect ranking. Because it is threshold-independent, it measures how well the model ranks rather than how well it classifies at any particular cutoff.

On strongly imbalanced data. The false positive rate in ROC has the large negative count in its denominator, so even thousands of false positives barely move it and the curve looks flattering. Precision uses predicted positives as its denominator, so PR-AUC reacts sharply to false positives and reflects what a user of the model would actually experience. A rule of thumb: if the positive class is rare and is the class you care about, report PR-AUC.

Not by defaulting to 0.5 — that is only optimal when classes are balanced and the two error types cost the same. Sweep the threshold over the validation set and pick the point that maximizes the quantity you actually care about: expected cost, F1, or recall subject to a precision floor. Then check that the threshold still holds on the test set, and re-derive it whenever the class balance in production shifts.

Stratified k-fold preserves the class proportions of the full dataset within every fold. You need it whenever classes are imbalanced or the dataset is small, because plain random folds can end up with very few — or zero — minority examples, which makes the per-fold scores erratic and can crash metrics that divide by the positive count. For grouped data, such as several rows per patient, you additionally need group-aware splitting so the same entity never appears in both train and validation.

Never with random k-fold, because shuffling lets the model train on the future and predict the past, which produces an estimate that cannot be reproduced in production. Use forward-chaining or rolling-origin validation: train on everything up to time t, validate on the window after it, then roll forward. Leave a gap between train and validation equal to the prediction horizon so lagged features cannot peek, and make sure every engineered feature uses only information available at the time of the prediction.

Mean absolute error is the average absolute difference and is in the same units as the target, so it is easy to explain. Mean squared error squares the errors, which penalizes large mistakes more heavily, and root mean squared error takes the square root to restore the original units. R-squared reports the fraction of variance the model explains relative to always predicting the mean, and MAPE expresses error as a percentage, which breaks down when actual values approach zero.

Prefer MAE when the data contains legitimate outliers that you do not want dominating the score, since squaring gives a single large error disproportionate weight. Prefer RMSE when large errors really are disproportionately bad — an underestimate of demand by a factor of ten in a supply chain, say. They also imply different optimal predictions: minimizing MAE targets the conditional median, minimizing MSE targets the conditional mean, which is why a model trained on one and judged on the other looks worse than it is.

R-squared is the proportion of variance in the target explained by the model, relative to the baseline of predicting the mean; it can go negative if the model is worse than that baseline. Its weakness is that adding any feature, even pure noise, never decreases it, so it rewards bloated models. Adjusted R-squared penalizes the number of predictors, so it rises only when a new feature contributes more than chance would, which makes it the fairer number for comparing models of different sizes.

A calibrated classifier's predicted probabilities match observed frequencies — among the cases it scores at 0.7, about 70 percent should be positive. It matters whenever the probability itself feeds a decision, such as expected-value pricing, risk scoring, or a downstream threshold on cost. Many models are poorly calibrated by default: SVMs and boosted trees tend to push scores toward the extremes, and Platt scaling or isotonic regression fitted on a held-out set corrects this without changing the ranking or the AUC.

Linearity between predictors and the target, independence of the errors, homoscedasticity (constant error variance across the range of predictions), normally distributed residuals (needed for the confidence intervals and p-values, not for the fit itself), and no severe multicollinearity among predictors. Violations do not always break prediction, but they invalidate the inferential statistics people quote from the model. Residual plots are the fastest way to check most of them.

Ordinary least squares has a closed-form solution — the normal equation — that minimizes the sum of squared residuals directly, and it is exact and fast for a few thousand features. It requires inverting a matrix that is O(p^3) in the number of features and is unstable when features are collinear, so for large or wide datasets libraries use gradient descent or QR and SVD-based solvers instead. Knowing that both routes exist, and why one is chosen, is the point of the question.

Multicollinearity is a strong linear relationship between two or more predictors, which makes the coefficient estimates unstable and their signs unreliable even though the model's predictions may be fine. Detect it with a correlation matrix or, better, the variance inflation factor, where values above roughly 5 to 10 signal a problem. Fix it by dropping or combining the redundant features, using PCA, or applying Ridge regularization, which is specifically well behaved under collinearity.

Logistic regression is a classification model that fits a linear combination of the features and then passes it through the sigmoid function to produce a probability between 0 and 1. It keeps the name because it regresses the log-odds of the positive class linearly on the inputs — the linear part is real, only the output is squashed. It remains a strong baseline because it is fast, its coefficients are interpretable, and it produces reasonably calibrated probabilities out of the box.

The sigmoid maps any real number to the range 0 to 1 as 1 / (1 + e^-z), turning the linear score z into a probability, and is symmetric around z = 0 where it outputs 0.5. The decision boundary is the surface where z = 0, which is a hyperplane in the feature space — so plain logistic regression can only draw a linear boundary. Curved boundaries require adding polynomial or interaction features, or switching to a model with a richer hypothesis space.

Composing MSE with the sigmoid produces a non-convex loss surface with local minima, so gradient descent is no longer guaranteed to find the global optimum. It also has weak gradients exactly where the model is confidently wrong, because the sigmoid saturates and its derivative approaches zero, which stalls learning. Log loss (binary cross-entropy) is convex in the parameters and its gradient reduces to the simple prediction-minus-label term, so confident mistakes produce large corrections.

A coefficient is the change in the log-odds of the positive class per one-unit increase in that feature, holding the others fixed; exponentiating it gives the odds ratio, which is what people usually want to quote. An odds ratio of 1.5 means the odds multiply by 1.5 per unit. Two cautions: the effect on probability is not constant, since the sigmoid is non-linear, and coefficients are only comparable across features if the features were standardized first.

Ridge adds an L2 penalty, shrinking all coefficients smoothly and handling correlated predictors gracefully by spreading weight across them, but it never removes a feature. Lasso adds an L1 penalty, which drives some coefficients to exactly zero and therefore performs feature selection, though with a group of correlated features it tends to pick one arbitrarily. Elastic Net mixes both, keeping Lasso's sparsity while retaining Ridge's stable treatment of correlated groups — which is why it is the usual default on wide, correlated data.

Either one-vs-rest, training one binary classifier per class and taking the highest score, or multinomial logistic regression with a softmax output that models all classes jointly and produces probabilities that sum to one. Softmax is usually preferable because the probabilities are coherent and it optimizes a single objective, while one-vs-rest can produce scores that do not compare cleanly across classifiers. One-vs-rest is still useful when the classes are not mutually exclusive, which is really a multi-label problem.

A decision tree is a single model that splits the data on feature thresholds to form a tree of decisions; it is easy to interpret but prone to overfitting. A random forest is an ensemble of many decision trees, each trained on a random subset of the data and features, whose predictions are averaged or voted. This reduces variance and usually generalizes much better than a single tree, at the cost of interpretability.

Both are ensemble methods that combine multiple weak learners, but they work differently. Bagging (bootstrap aggregating) trains models independently and in parallel on different bootstrapped samples, then averages or votes their predictions to reduce variance — random forests are a bagging method. Boosting trains models sequentially, with each new model focusing on the errors of the previous ones, reducing bias — examples include AdaBoost and gradient boosting methods like XGBoost.

It evaluates candidate splits on every feature and keeps the one that most reduces impurity in the resulting children. For classification the impurity measure is Gini or entropy, and the reduction is the information gain; for regression it is the reduction in variance or squared error. The search is greedy and local — it never reconsiders an earlier split — which is why a single tree is rarely globally optimal and why ensembles help so much.

Both measure how mixed the classes are in a node and both are zero for a pure node. Entropy uses a logarithm and Gini a sum of squared probabilities, so Gini is slightly cheaper to compute and is the default in most libraries. In practice they select the same split the overwhelming majority of the time, and the choice between them almost never matters compared with tree depth or the number of trees — worth saying, because it shows you know which knobs are real.

An unpruned tree keeps splitting until every leaf is pure, which means it has memorized the training set, including its noise. Pre-pruning stops growth early with constraints such as maximum depth, minimum samples per leaf, or a minimum impurity decrease; post-pruning grows the full tree and then removes branches that do not improve validation performance, as cost-complexity pruning does. Pruning is the main defense against a single tree's very high variance.

Bootstrapping rows alone still leaves every tree free to split on the same one or two dominant features, so the trees end up highly correlated and averaging them removes less variance than it should. Sampling a random subset of features at each split forces the trees to explore different structure, decorrelating them and making the ensemble average genuinely more stable. That extra randomness is exactly what separates a random forest from plain bagged trees.

Each tree in a random forest is trained on a bootstrap sample that omits roughly a third of the rows, and those held-out rows are that tree's out-of-bag set. Averaging each row's prediction over only the trees that did not see it gives an unbiased generalization estimate essentially for free, without a separate validation split. It is a genuine advantage of bagging, though it is not a substitute for a final held-out test set when you have also been tuning hyperparameters.

AdaBoost reweights the training examples after each round so that misclassified points matter more, and combines the weak learners with weights based on their accuracy. Gradient boosting instead fits each new learner to the residuals — formally, the negative gradient of a differentiable loss — so it generalizes to any loss function, including squared error, log loss, and quantile loss. Gradient boosting is the more flexible framework, and AdaBoost is essentially the special case using exponential loss.

It adds explicit L1 and L2 regularization on the leaf weights and a penalty on the number of leaves, so the objective itself controls complexity rather than relying on early stopping alone. It uses a second-order approximation of the loss for better split decisions, handles missing values by learning a default direction at each split, and builds a sparsity-aware histogram of candidate thresholds instead of scanning every value. Combined with cache-aware, parallel, out-of-core implementation, that is why it dominated tabular competitions; LightGBM and CatBoost extend the same ideas.

Choose a random forest when you want a strong result with almost no tuning, when you can parallelize training across trees, and when robustness to noisy labels matters — averaging independent trees is forgiving, while boosting will chase the noise. Choose gradient boosting when you need the last few points of accuracy on tabular data and are willing to tune learning rate, depth, and the number of rounds carefully with early stopping. Boosting usually wins on accuracy; forests usually win on effort and stability.

The default impurity-based importance is biased toward high-cardinality and continuous features, simply because they offer more candidate split points and can reduce impurity by chance. It is also computed on the training data, so it rewards features the model overfitted to, and it splits credit arbitrarily among correlated features. Permutation importance measured on held-out data, or SHAP values, gives a more trustworthy picture — and neither one establishes causation, only what the model relied on.

An SVM finds the hyperplane that separates the classes with the largest possible margin — the widest gap between the boundary and the nearest points of each class. Maximizing the margin is a form of structural risk minimization, which is why SVMs generalize well even on small, high-dimensional datasets. The soft-margin formulation allows some points to violate the margin, trading training errors against a wider, more robust boundary.

Support vectors are the training points that lie on the margin or violate it; they are the only points that determine the position of the decision boundary. Removing any other training point leaves the fitted model completely unchanged. That is why an SVM's model size depends on the number of support vectors rather than on the size of the training set, and why it can be memory-efficient on small data but slow at inference when nearly every point becomes a support vector.

Many datasets are not linearly separable in their original space but become separable after mapping into a higher-dimensional one. The kernel trick computes the inner product in that higher-dimensional space directly from the original features, via a kernel function, without ever constructing the mapped coordinates — so an RBF kernel gives you an effectively infinite-dimensional space at the cost of a single exponential per pair. The catch is that training scales between quadratically and cubically in the number of samples, which is why kernel SVMs are rare on large datasets.

C is the inverse of regularization strength: a large C penalizes margin violations heavily, producing a narrow margin that fits the training data closely and risks overfitting, while a small C accepts more violations for a wider, smoother boundary. Gamma, in an RBF kernel, sets how far a single training example's influence reaches — high gamma makes the boundary wrap tightly around individual points, low gamma makes it nearly linear. They interact, so they are tuned together on a grid, and both require scaled features.

KNN stores the training set and, to predict, finds the k closest training points under a distance metric and takes a majority vote for classification or an average for regression. Training is instantaneous, but every prediction costs O(n*d) in a naive implementation, so inference is the expensive part — the opposite of most models. It requires feature scaling, because an unscaled large-range feature dominates the distance, and it degrades badly in high dimensions as distances converge.

By cross-validation, not by rule of thumb. A small k gives a flexible, low-bias, high-variance model that follows noise; a large k smooths the boundary, raising bias and lowering variance, and at k equal to n it just predicts the majority class. Use an odd k for binary classification to avoid ties, and consider distance weighting so nearer neighbors count for more, which reduces the sensitivity to the exact value of k.

Because it does no work at training time beyond storing the data — there is no fitted function, no parameters, and no generalization until a query arrives. Eager learners such as logistic regression or a decision tree build an explicit model up front and then discard the training data. The consequence is that KNN's cost, memory footprint, and latency all scale with the training set, which is what usually rules it out in production without an approximate-nearest-neighbor index.

Naive Bayes applies Bayes' theorem to compute the probability of each class given the features, and picks the highest. The naive part is assuming that every feature is conditionally independent of the others given the class, which lets it multiply individual likelihoods instead of modeling a joint distribution — turning an intractable problem into a count. It trains in a single pass, needs very little data, and handles high-dimensional sparse input well.

Words in a document are obviously not independent, so the probability estimates it produces are badly miscalibrated — often pushed close to 0 or 1. But classification only needs the correct class to score highest, not the probabilities to be right, and the errors from the independence assumption frequently affect all classes in the same direction and cancel in the comparison. Combined with Laplace smoothing to stop an unseen word from zeroing out the whole product, that is why it remains a fast, respectable baseline for spam filtering and topic classification.

K-means is an unsupervised algorithm that partitions data into k clusters. It starts by placing k centroids, assigns each point to the nearest centroid, then recomputes each centroid as the mean of its assigned points, repeating until assignments stop changing. You must choose k in advance, often using methods like the elbow plot or silhouette score, and because results depend on the initial centroid positions it is typically run several times.

The curse of dimensionality refers to problems that arise when data has very many features. As dimensions grow, the volume of the space increases so fast that data becomes sparse, distances between points become less meaningful, and models need exponentially more data to generalize well. This is why dimensionality reduction techniques like PCA and careful feature selection are often used to keep models effective.

The elbow method plots within-cluster sum of squares against k and looks for the bend where additional clusters stop paying for themselves, though the bend is often ambiguous. The silhouette score is usually more decisive: it measures how much closer each point is to its own cluster than to the next nearest, and you pick the k that maximizes the average. The gap statistic compares your clustering against a random-uniform reference, and above all the clusters need to be interpretable to whoever will use them.

It assumes clusters are roughly spherical, similarly sized, and separated by distance, so it fails on elongated, nested, or density-varying shapes. It needs k in advance, is sensitive to initialization (which k-means++ mitigates by spreading the initial centroids), and is pulled around by outliers because centroids are means. It also requires scaled numeric features, since it depends entirely on Euclidean distance, and it forces every point into a cluster with no notion of noise.

Hierarchical clustering builds a tree of nested clusters — agglomerative methods start with each point as its own cluster and repeatedly merge the closest pair, divisive methods split from the top down. You do not have to choose k in advance; you cut the dendrogram wherever the merge distances jump. The trade-off is cost: it is typically O(n^2) or worse in memory and time, so it suits small datasets and exploratory analysis, while k-means scales to millions of points.

DBSCAN groups points that are densely packed — a point with at least minPts neighbors within distance eps is a core point, and connected core points form a cluster — and labels everything else as noise. It beats k-means when clusters are non-spherical or of very different sizes, when the number of clusters is unknown, and when the data contains outliers you want excluded rather than forced into a cluster. Its weaknesses are sensitivity to eps and poor behavior when cluster densities differ widely, which HDBSCAN addresses.

Principal component analysis finds an orthogonal set of directions — the principal components — ordered so that the first captures the most variance in the data, the second the most of what remains, and so on. It is computed from the eigenvectors of the covariance matrix, or equivalently by SVD, and projecting onto the top few components compresses the data with minimal loss of variance. It requires standardized features, because otherwise a large-scale variable dominates the covariance purely through its units.

Plot cumulative explained variance and keep enough components to reach a target such as 95 percent, or look for the elbow in the scree plot of eigenvalues. If PCA is a preprocessing step for a supervised model, the honest approach is to treat the component count as a hyperparameter and tune it against downstream validation performance rather than against a variance threshold. Remember that variance is not the same as usefulness — a low-variance direction can still carry the signal you need.

Feature selection keeps a subset of the original columns, so the result stays interpretable and you genuinely stop collecting the dropped inputs. PCA constructs new features as linear combinations of all the originals, which usually compresses better but produces components no domain expert can read, and you still need every original feature at inference time. Choose selection when interpretability or data-collection cost matters, PCA when you need decorrelated, compact inputs for a distance- or gradient-based model.

Both are non-linear embeddings designed to preserve local neighborhoods, which makes them excellent for seeing whether structure exists — and unreliable for anything quantitative. Cluster sizes and the distances between clusters in the plot are largely meaningless, the layout changes with the random seed, and t-SNE's perplexity parameter can manufacture apparent clusters in pure noise. Use them to generate hypotheses, never as evidence, and never fit a downstream model on t-SNE coordinates.

Gradient descent is an optimization algorithm that minimizes a loss function by iteratively moving the model's parameters in the direction opposite the gradient, the direction of steepest descent. The learning rate controls the step size: too large and it may overshoot or diverge, too small and it converges slowly. Common variants include batch gradient descent, stochastic gradient descent which updates on one example at a time, and mini-batch gradient descent which uses small batches.

Batch gradient descent uses the whole dataset per update, giving a smooth, accurate gradient but one very slow step and a memory requirement proportional to the data. Stochastic gradient descent updates on a single example, so it is fast and its noise can help escape poor regions, but the path oscillates and it cannot use vectorized hardware well. Mini-batch, typically 32 to 512 examples, is the practical default: enough averaging to be stable, small enough to fit in GPU memory, and large enough to keep the hardware busy.

It scales the size of each parameter update. Too large and the loss oscillates or diverges to NaN because each step overshoots the minimum; too small and training crawls, or stalls on a plateau before reaching a good solution. The usual diagnosis is the loss curve: a jagged or exploding curve means reduce it, a nearly flat curve means increase it. It is the single most important hyperparameter in deep learning, which is why learning-rate range tests and schedules exist.

Momentum accumulates an exponentially weighted average of past gradients, so the update keeps rolling through small oscillations and flat regions instead of zig-zagging across a ravine. RMSprop divides the step by a running average of recent squared gradients, giving each parameter its own effective learning rate so rarely updated parameters still move. Adam combines both, with bias correction for the early steps, which is why it is the default optimizer — though well-tuned SGD with momentum still generalizes slightly better on some vision tasks.

A schedule decays the learning rate over training — step decay, cosine annealing, or reduce-on-plateau — so early steps move fast and later steps settle finely into a minimum. Warmup does the opposite at the very start, ramping the rate up from near zero over the first few hundred or thousand steps. Warmup matters because at initialization the gradient estimates and the optimizer's running statistics are unreliable, and a full-size step can destabilize training, especially for large batches and transformer architectures.

The loss is what the optimizer minimizes, so it must be differentiable and well behaved with respect to the parameters. The metric is what you and the business judge the model by, and it can be non-differentiable — accuracy, F1, AUC, or revenue per session. They often differ: you train a classifier on cross-entropy but report F1. When they diverge badly, the fix is either a surrogate loss closer to the metric or a post-hoc threshold tuned against the metric.

Regression normally uses mean squared error, or MAE and Huber loss when outliers should not dominate. Binary classification uses binary cross-entropy (log loss); multi-class classification uses categorical cross-entropy over a softmax output. Multi-label classification uses per-label binary cross-entropy with sigmoid outputs, not softmax, since the labels are not mutually exclusive. Specialized tasks bring their own — hinge loss for SVMs, quantile loss for prediction intervals, focal loss for extreme class imbalance.

Cross-entropy measures the distance between the predicted probability distribution and the true one, reducing for a single label to the negative log of the probability the model assigned to the correct class. It punishes confident mistakes very harshly — a probability of 0.01 on the true class contributes a loss of about 4.6 — which is exactly the gradient signal you want. Paired with softmax, its gradient simplifies to prediction minus label, which is why the two are almost always implemented as one fused operation for numerical stability.

A batch is the group of examples processed in one forward and backward pass. An iteration, or step, is one such pass and therefore one parameter update. An epoch is one full sweep through the training set, so it contains dataset size divided by batch size iterations. Mixing these up is a common source of confusion when reading training logs or configuring a learning rate schedule, which is usually specified in steps rather than epochs.

A convex function has a single global minimum and no local minima, so gradient descent from any starting point converges to the optimum given a reasonable learning rate. Linear regression with squared error and logistic regression with log loss are convex in their parameters, which is why they are reproducible and need no initialization strategy. Neural networks are non-convex, so training depends on initialization, the optimizer, and the random seed — and the practical finding is that most minima reached in high dimensions are of similar quality.

Largely no — that is the classic misconception. In very high-dimensional loss surfaces, critical points are overwhelmingly saddle points rather than poor local minima, because a point needs every one of thousands of eigenvalues to be positive to be a minimum. The real difficulties are long flat plateaus around saddles, ill-conditioned ravines, and vanishing or exploding gradients, which is what momentum, adaptive optimizers, normalization layers, and gradient clipping are all designed to handle.

It takes a weighted sum of its inputs, adds a bias term, and passes the result through a non-linear activation function. The weights and bias are learned; the activation is a fixed choice of architecture. A layer is many such units computing in parallel, which is why a forward pass is a matrix multiplication followed by an element-wise function — and why GPUs suit the workload so well.

Because a composition of linear functions is itself linear: stacking a hundred layers with no non-linearity produces a model exactly as expressive as a single linear layer, just with more parameters to fit. The non-linearity is what lets depth build progressively richer representations and gives the universal approximation property. This is the single most common conceptual question about neural networks, and the answer is one sentence about composition.

Sigmoid squashes to 0-1 but saturates at both ends, killing gradients, and is not zero-centered, so it survives mainly as an output layer for binary probability. Tanh is zero-centered and therefore better behaved in hidden layers, but it still saturates. ReLU is the default for hidden layers because it does not saturate for positive inputs, is trivially cheap, and induces sparsity — its weakness is dying units that output zero forever, which leaky ReLU, ELU, and GELU address by allowing a small or smooth negative response.

Backpropagation computes the gradient of the loss with respect to every parameter by applying the chain rule backwards through the network, reusing the intermediate results cached during the forward pass. That reuse is the key: it makes the cost of the whole gradient about the same as one forward pass, rather than one pass per parameter. Backpropagation only computes gradients — it is the optimizer, such as SGD or Adam, that decides what to do with them.

In a deep network the gradient reaching an early layer is a product of many derivatives; if those terms are consistently below one — as with saturating sigmoid or tanh units — the product decays exponentially and the early layers stop learning. The fixes are non-saturating activations like ReLU, careful initialization such as He or Xavier scaling, normalization layers that keep activations in a healthy range, and residual (skip) connections that give the gradient a direct path backwards. Residual connections are what made networks hundreds of layers deep trainable at all.

The mirror image of vanishing gradients: when the repeated derivative terms are consistently above one, the gradient grows exponentially through the layers, producing enormous updates and a loss that jumps to NaN. It is most common in recurrent networks trained over long sequences. The standard remedy is gradient clipping — rescaling the gradient vector whenever its norm exceeds a threshold — along with better initialization and normalization layers.

Batch normalization standardizes each layer's pre-activations across the mini-batch, then rescales them with two learned parameters so the layer keeps its expressive power. It allows higher learning rates, reduces sensitivity to initialization, and has a mild regularizing effect from the batch noise. It behaves differently at training and inference — inference uses running averages, not batch statistics — which is why small batch sizes hurt it and why layer normalization is used instead in transformers and recurrent models.

Initializing all weights to zero makes every unit in a layer compute the same thing and receive the same gradient, so they never differentiate — symmetry is never broken. Initializing too large saturates activations and explodes gradients; too small and the signal vanishes as it propagates. Xavier (Glorot) initialization scales the variance by the number of input and output units for tanh-style activations, and He initialization doubles that variance for ReLU, which zeroes half its inputs.

A CNN slides small learned filters across the input, so the same weights detect a feature wherever it appears. That gives translation invariance and shares parameters, making it dramatically more efficient than a fully connected layer over pixels — and it encodes the prior that nearby pixels are related. Stacked layers build a hierarchy from edges to textures to object parts, which is exactly the structure natural images have.

Stride is how far the filter moves per step; a stride greater than one downsamples the output. Padding adds a border of zeros so the output keeps the input's spatial dimensions and edge pixels are not underweighted. Pooling — usually max pooling over a small window — downsamples by summarizing a region, which shrinks the feature maps, widens the effective receptive field, and adds a little local translation tolerance. Modern architectures often replace pooling with strided convolutions.

A recurrent network processes a sequence one step at a time, carrying a hidden state that summarizes everything seen so far, which is what lets it handle variable-length input. Plain RNNs cannot learn long-range dependencies because gradients vanish across many time steps. LSTMs add a cell state with input, forget, and output gates that let information flow across long spans largely unchanged; GRUs achieve much the same with two gates and fewer parameters. Both have since been displaced for most sequence tasks by attention-based models, which parallelize across the sequence.

Transfer learning reuses a model pretrained on a large general dataset and adapts it to your smaller, specific task, because the early layers have already learned features — edges, textures, syntax — that transfer broadly. The usual recipe is to replace the output head, freeze the backbone and train the head first, then unfreeze the upper layers and continue with a much lower learning rate so the pretrained weights are not destroyed. It is the standard approach whenever labeled data is limited, which is most of the time.

An embedding is a dense, low-dimensional vector that represents a discrete item — a word, a product, a user — such that items with similar meaning or behavior land close together in the space. It replaces sparse one-hot encoding, which carries no notion of similarity and explodes in width with vocabulary size. Embeddings are learned, either as a layer trained jointly with the task or taken from a pretrained model, and they are what makes semantic search and recommendation by nearest neighbor possible.

Attention lets a model decide, for each position it is producing, how much to draw from every other position, rather than compressing the whole input into one fixed vector as an RNN encoder must. Each position emits a query, and it is matched against keys from all positions to produce weights that mix the corresponding values. The practical consequence is a direct connection between any two positions regardless of distance, which removes the long-range dependency problem that limited recurrent models.

In self-attention the queries, keys, and values all come from the same sequence, so every token is re-represented as a weighted mixture of all tokens including itself, capturing context in a single parallel operation. A single attention distribution can only emphasize one kind of relationship at a time; multi-head attention runs several attention functions in parallel over different learned projections, so one head can track syntactic agreement while another tracks coreference, and their outputs are concatenated and projected. It is the same reason a CNN uses many filters per layer.

An RNN must process tokens in order, so training cannot be parallelized along the sequence and long-range gradients degrade. A transformer computes attention over all positions at once, so the whole sequence is processed in parallel on a GPU and any two tokens are one operation apart. That parallelism is what made it economical to train on internet-scale corpora, and scale is what produced the capability jump — the architecture removed the bottleneck, and the data did the rest.

Self-attention is permutation-invariant: it treats its input as a set, so without extra information the model cannot tell "the dog bit the man" from "the man bit the dog". Positional encodings inject order, either as fixed sinusoidal patterns added to the token embeddings or as learned position vectors, and modern variants such as rotary embeddings encode relative position directly inside the attention computation. Relative schemes generalize better to sequences longer than those seen in training.

Standard self-attention compares every token with every other token, so both the compute and the attention memory grow quadratically with sequence length — doubling the context quadruples the cost. That is why long-context models rely on engineering work such as memory-efficient attention kernels, sparse or sliding-window attention patterns, and key-value caching during generation. It is also why retrieval is often cheaper than a longer prompt: fetch the few relevant passages instead of paying quadratic cost on everything.

Pretraining runs a self-supervised objective — predicting masked or next tokens — over a very large unlabeled corpus, producing general representations at very high cost. Fine-tuning continues training that model on a much smaller labeled dataset for a specific task, at a low learning rate, so it adapts without discarding what it learned. Parameter-efficient methods such as LoRA train small adapter matrices instead of all the weights, which cuts memory and storage sharply while retaining most of the benefit.

Retrieval-augmented generation embeds a corpus into a vector index, retrieves the passages closest to the query at request time, and puts them in the model's context so the answer is grounded in real source text. It is preferable to fine-tuning when the knowledge changes often, must be attributable to a source, or is too large to memorize — you update an index rather than retrain. Fine-tuning is the better tool for changing behavior, format, or style; retrieval is the better tool for supplying facts.

Drift is the decay of a deployed model's performance as the world moves away from its training data. Data drift, or covariate shift, is a change in the distribution of the inputs — a new user segment, a new device mix — while the input-to-output relationship holds. Concept drift is a change in that relationship itself, so the same inputs now imply a different label, as fraud patterns change in response to your own defenses. The distinction matters: data drift may be fixable by retraining on recent data, concept drift often requires new features.

Monitor three layers. Operational health — latency, error rate, throughput — the same as any service. Data health — input schema, null rates, cardinality, and distribution distance against the training baseline using a measure such as population stability index or a KS test. And model health — prediction distribution, and true performance once labels arrive, which may be days later. Alert on the input distribution too, because it moves before accuracy does and it is the only early warning you get when labels are delayed.

Either on a schedule, which is simple and predictable, or triggered when monitoring shows drift or a performance drop past a threshold — the trigger-based approach avoids pointless retraining on stable problems and reacts faster on volatile ones. Whichever you choose, the retrained model must be validated against the incumbent on a fresh held-out window and rolled out behind a shadow deployment or a canary, not swapped in directly. Automatic retraining without a gate is how a poisoned or broken data pipeline silently reaches production.

Training-serving skew is any difference between how a feature is computed during training and how it is computed at inference — a different aggregation window, a different null-handling rule, a Python transform in the notebook that was reimplemented in Java for the service. The model then sees inputs it was never trained on, and the failure is silent because nothing errors. Prevent it by sharing one transformation pipeline across both paths, by serving features from a feature store that computes them once, and by logging production inputs and comparing their distributions with the training set.

Version the code, the data, and the environment together — a commit hash, a dataset hash or snapshot, and a pinned dependency lockfile or container image. Fix every random seed, including the framework, the data shuffler, and any sampling in feature engineering, and record the full hyperparameter set and metrics with an experiment tracker. Note that exact bitwise reproducibility on a GPU also needs deterministic kernels enabled, which costs speed, so most teams settle for reproducibility within noise and record the seed anyway.

Offline metrics are computed on historical data under the old model's decisions, so they cannot capture how users respond to the new one, feedback loops where the model changes the data it later trains on, or downstream effects on the metric the business cares about. A model with better AUC can lose money if its errors are concentrated where the stakes are highest. The answer is a staged rollout: shadow mode to compare predictions on live traffic without acting, then an A/B test measuring the business metric.

A feature store is a shared system that computes, stores, and serves features with the same definition to both training and inference — an offline store for historical point-in-time-correct training data and a low-latency online store for serving. It solves three recurring problems: training-serving skew from duplicated transformation logic, teams re-implementing the same features independently, and the difficulty of assembling a training set without accidentally including information from after the prediction time. It is worth the operational cost once several models share features; before that it is overhead.

Batch scoring runs the model on a schedule over a large set of records and writes the predictions to a store the application reads, which is simple, cheap, easy to monitor, and fine whenever predictions can be minutes or hours stale — churn scores, weekly recommendations. Real-time serving exposes the model behind an API and scores on request, which is required when the input only exists at request time, as with fraud checks or search ranking. Real-time brings latency budgets, autoscaling, and feature-freshness problems that batch does not have, so do not choose it by default.

Use global methods to describe overall behavior — permutation importance, partial dependence plots — and local methods to explain a single prediction, most commonly SHAP, which attributes the prediction to each feature with a game-theoretic guarantee of consistency, or LIME, which fits a simple surrogate model in the neighborhood of the point. For regulated decisions, an inherently interpretable model such as a regularized linear model or a shallow tree is often the safer choice. In every case, an explanation describes what the model used, not what causes the outcome.

Start by defining the fairness criterion, because they conflict mathematically — demographic parity, equal opportunity, and equalized odds cannot generally all hold at once, so the choice is a policy decision, not a technical one. Then measure the chosen metric per group on held-out data, and check the training data for historical bias and under-representation, since a model trained on biased decisions reproduces them faithfully. Mitigations act before training (rebalancing and reweighting), during it (fairness constraints in the objective), or after it (per-group thresholds) — and all of it needs ongoing monitoring, because a fair model can drift into unfairness.