Linear Algebra and Matrices

COMP1321 Mathematics II: Linear Algebra and Matrices

Introduction to Linear Algebra

Vectors

• Definition: A vector is a quantity with magnitude and direction, or more formally, an element of a vector space.

• Representation: Vectors can be represented as ordered tuples (e.g., in ) or column/row matrices.

• Column vector in

• Row vector in

• Transposition:

• Geometric Interpretation: Vectors can be visualized as displacements from the origin along coordinate axes.

Vector Addition and Scalar Multiplication

• Addition: Vectors are added component-wise.

• If and , then .

• Geometrically, vector addition forms a parallelogram or head-to-tail representation.

• Scalar Multiplication: A scalar multiplies each component of a vector.

• If and , then .

• Geometrically, scalar multiplication scales the magnitude of the vector.

• Subtraction: .

Linear Combinations of Vectors

• Definition: A linear combination of vectors is formed by multiplying vectors by scalars and adding the results.

• For vectors and scalars , a linear combination is .

• Geometric Interpretation:

• Linear combinations of one vector (e.g., ) form a line through the origin.

• Linear combinations of two vectors (e.g., ) form a plane through the origin (if and are linearly independent).

• Linear combinations of three vectors (e.g., ) can span the entire volume (if linearly independent).

Norm, Dot Product, Vector Spaces

Vector Spaces

• Definition: A vector space is a set with two operations, vector addition ( + ) and scalar multiplication (*), that satisfy eight axioms:

1. Associativity of addition:

2. Associative compatibility of scalar multiplication:

3. Commutativity of addition:

4. Identity element for addition: There exists such that for all .

5. Inverse element for addition: For every , there exists such that .

6. Identity element for scalar multiplication: for all .

7. Distributivity of scalar multiplication over vector addition:

8. Distributivity of scalar multiplication over field addition:

• Examples: , and the set of polynomials of a certain degree form vector spaces.

Norm of a Vector (Magnitude)

• Definition: The norm of a vector, denoted , represents its magnitude or length.

• Euclidean Norm (Geometric Vectors): Calculated using the Pythagorean theorem.

• For

• For

• Generalization: For non-geometric vectors (like polynomials), norms are defined based on axioms and context. For example, the norm for polynomials is .

Dot/Scalar Product

• Definition: The dot product (or scalar product) of two vectors results in a scalar value.

• Geometric Interpretation: , where is the angle between the vectors.

• Coordinate Form:

• For and .

• Applications:

• Checking for perpendicularity: If , the vectors are orthogonal (perpendicular).

• Calculating the angle between vectors: .

• Calculating the norm: .

• For Polynomials: The dot product of and can be defined as .

Unit Vectors

• Definition: A unit vector is a vector with a magnitude of one.

• Purpose: Unit vectors are used to store directional information and define coordinate systems (e.g., in ).

• Obtaining a Unit Vector: To get the unit vector in the direction of vector , divide by its magnitude: .

Cauchy-Schwarz Inequality and Triangle Inequality

• Cauchy-Schwarz Inequality: For any vectors . This is related to the fact that .

• Triangle Inequality: For any vectors . This states that the length of the sum of two vectors is less than or equal to the sum of their individual lengths.

Linear Maps and Matrices

Linear Maps

• Definition: A function between two vector spaces is a linear map if for any scalars and vectors in , the following holds:

• Examples:

• Stretching a vector: is linear.

• Non-linear transformations: is not linear.

Linear Maps as Matrices

• Representation: A linear map can be represented by an matrix .

• Transformation: Applying the linear map to a vector is equivalent to multiplying the matrix by the vector .

• Matrix Construction: The columns of the matrix are the images of the standard basis vectors under the linear map. For , if and , then .

Matrices and Matrix Arithmetic

Matrices

• Definition: A matrix is a rectangular array of numbers arranged in rows and columns, denoted as , where is the element in the th row and th column.

• Dimensions: A matrix with rows and columns is an matrix.

Matrix Transposition

• Definition: The transpose of a matrix , denoted , is obtained by swapping its rows and columns. If , then where .

• Example: If , then .

Matrix Arithmetic

• Scalar Multiplication: Multiplying a matrix by a scalar .

• Matrix Addition/Subtraction: Matrices must have the same dimensions. .

• Matrix Multiplication: For and , the product is an matrix where . This is equivalent to the dot product of the th row of and the th column of . Matrix multiplication is generally not commutative ( ).

• Example: Let and .

Special Matrices

• Identity Matrix (I): A square matrix with 1s on the main diagonal and 0s elsewhere. .

• Diagonal Matrix ( ): A square matrix where all off-diagonal elements are zero ( for ).

• Upper/Lower Triangular Matrix: A square matrix where all elements below (upper) or above (lower) the main diagonal are zero.

• Strict Upper/Lower Triangular Matrix: A triangular matrix where the diagonal elements are also zero.

• Permutation Matrix: A matrix obtained by permuting the rows of an identity matrix. It swaps rows/columns of vectors or matrices.

Inverse Matrices

• Definition: The inverse of a square matrix , denoted , is a matrix such that (the identity matrix).

• Existence: An inverse exists only for square matrices. A matrix has an inverse if and only if its determinant is non-zero.

• Obtaining the Inverse using EROS: Augment the matrix with the identity matrix . Apply Elementary Row Operations (EROS) to transform into the identity matrix . The matrix on the right side will become the inverse .

Systems of Linear Equations (SLES)

Representation

• General Form: A system of linear equations is a set of linear equations.

• Matrix Equation Form :

• Augmented Matrix Form ( ): A compact representation of the SLES.

Elementary Row Operations (EROs)

• Purpose: EROs transform the augmented matrix of an SLES into an equivalent system with the same solutions.

• Types:

1. Swapping two rows ( ).

2. Multiplying a row by a non-zero scalar ( ).

3. Adding a multiple of one row to another row .

Gauss Elimination (GE) Method

• Goal: To transform the augmented matrix into row echelon form (or reduced row echelon form) using EROs.

• Row Echelon Form:

• All zero rows are at the bottom.

• The first non-zero entry (leading entry or pivot) in each non-zero row is to the right of the leading entry of the row above it.

• For a square matrix, this results in an upper triangular form.

• Reduced Row Echelon Form:

• In addition to row echelon form, each leading entry is 1, and all other entries in the column containing a leading entry are zero.

• Solving: After reaching row echelon form, use backward substitution to find the values of the unknowns.

Singular SLES and GE

• Singular Matrix: A square matrix whose determinant is zero. Such a matrix does not have an inverse.

• Singular SLES: SLES where the coefficient matrix is singular. These systems have either no solutions or infinitely many solutions.

• No Solution: If EROs lead to a row of the form where , this represents a contradiction (e.g., ), indicating no solution.

• Infinite Solutions: If EROs lead to a row of all zeros , it indicates a dependent equation. If there are fewer non-zero rows (pivots) than variables ( ), there are infinitely many solutions. The number of free variables is . These solutions can be expressed parametrically.

SLES and (Hyper)Planes

• Geometric Interpretation: Each linear equation in an SLES can be represented as a hyperplane in Euclidean space ( ).

• In , a linear equation is a line.

• , a linear equation is a plane.

• Solutions:

• Unique Solution: The hyperplanes intersect at a single point.

• No Solution: The hyperplanes do not intersect.

• Infinite Solutions: The hyperplanes intersect along a line, plane, or higher-dimensional space.

Determinants

Definition and Interpretation

• Definition: The determinant is a scalar value associated with a square matrix.

• Geometric Interpretation: The absolute value of the determinant of a linear transformation matrix represents the scaling factor of volumes (or areas in 2D). A negative determinant indicates a reflection (orientation change).

• SLES and Invertibility:

• A square matrix has a unique inverse if and only if .

• If , the matrix is singular, and the SLES has either no solution or infinitely many solutions.

Calculation Methods

• 2x2 Matrix: For .

• 3x3 Matrix: For

• Cofactor Expansion: For an matrix , the determinant can be calculated by expanding along any row or column :

• Along row , where is the minor (matrix with row and column deleted).

• Along column .

• Triangular Matrices: The determinant of an upper or lower triangular matrix is the product of its diagonal entries.

• Using EROs:

1. Transform the matrix into row echelon form (upper triangular) using EROs.

2. Keep track of how EROs affect the determinant:

• Row swap: Multiplies determinant by -1.

• Scalar multiplication : Multiplies determinant by .

• Row addition : Does not change the determinant.

1. The determinant of the resulting triangular matrix is the product of its diagonals. Adjust this product based on the EROs performed.

Basis Vectors

Basis of a Vector Space

• Definition: A basis for a vector space is a set of linearly independent vectors that span .

• Properties:

• Linearly Independent: No vector in the basis set can be expressed as a linear combination of the others.

• Spanning Set: Every vector in the vector space can be expressed as a linear combination of the basis vectors.

• Dimension: The number of vectors in any basis of a vector space is the same and is called the dimension of the vector space.

Linear Independence

• Definition: A set of vectors is linearly independent if the only solution to the equation is .

• Checking for Linear Independence: Form a matrix where the vectors are columns. Solve the homogeneous system using GE. If the only solution is the trivial solution ( ), the vectors are linearly independent.

Spanning Set

• Definition: A set of vectors is a spanning set for a vector space if every vector can be written as a linear combination of these vectors.

• Checking for Spanning: For a vector , form the augmented matrix where columns of are the spanning vectors. Solve the system using GE. If a solution exists for every , the set spans .

Orthonormality

• Orthogonal: Two vectors and are orthogonal if their dot product is zero: .

• Normalized: A vector is normalized if its norm (magnitude) is one: .

• Orthonormal Set: A set of vectors is orthonormal if all vectors in the set are mutually orthogonal and each vector is normalized.

• For a set for , and for all .

• Canonical Basis: The standard basis vectors in (e.g., in ) form an orthonormal basis.

Eigenvalues and Eigenvectors

Definition

• Eigenvectors: Non-zero vectors that, when transformed by a matrix , result in a vector that is parallel to the original vector. Their direction is preserved (or reversed), only their magnitude changes.

• Eigenvalues: The scalar factor by which an eigenvector is scaled.

• Equation:

Obtaining Eigenvalues

• Characteristic Equation: Rearrange the eigenvalue equation: .

• For non-trivial solutions ( ), the matrix ( ) must be singular. Therefore, its determinant must be zero:

• This equation is called the characteristic equation, and solving it for gives the eigenvalues. The result is a polynomial in (the characteristic polynomial).

Obtaining Eigenvectors

• For each distinct eigenvalue , substitute it back into the equation .

• Solve this system of linear equations (which will be singular) for . The solutions represent the eigenvectors corresponding to . Eigenvectors are often expressed parametrically.

Eigenvalue-Eigenvector Workflow

1. Given a square matrix , form the matrix ( ).

2. Calculate the determinant: . This yields the characteristic polynomial.

3. Solve the characteristic equation for to find the eigenvalues.

4. For each distinct eigenvalue , solve the homogeneous system using GE to find the corresponding eigenvectors .

Diagonalization

Coordinate Transform Between Bases

• Concept: A vector can be represented using different sets of basis vectors. A transformation matrix allows conversion between these coordinate systems.

• Transformation Matrix: If and are two bases for a vector space, a vector can be written as or . A change-of-basis matrix relates these representations: , where the columns of are the basis vectors of expressed in terms of . The inverse matrix transforms coordinates from to .

Diagonalization of a Matrix

• Definition: A square matrix is diagonalizable if it can be expressed as , where is a diagonal matrix and is an invertible matrix.

• Using Eigenvectors and Eigenvalues:

1. Find the eigenvalues and corresponding eigenvectors of .

2. Construct the matrix whose columns are the eigenvectors: .

3. Construct the diagonal matrix with the eigenvalues on the diagonal, in the same order as their eigenvectors in .

4. Calculate the inverse of , denoted .

5. Then, .

• Conditions for Diagonalization: A matrix is diagonalizable if and only if it has linearly independent eigenvectors (where is ).

• Application: Diagonalization simplifies calculations involving powers of , as , and is easily computed by raising the diagonal elements to the power .