Math 4401 Numerical Methods

Notebook Author/Course Instructor: Barry McQuarrie
Date: Aug 2016
Lecture topic: Norms

Vector Norms

(R^2)

Here are two vectors in (R^2):

v1={-Pi,2};
v2={-8,-5};

Here is how we can define the (Subscript[l, p]) norm:

norm[v_,p_]:=((((∑_(i=1)^2)((Abs[v[[i]]])^p)))^(1/p))

Here are the norms for a variety of p vlaues:

TableForm[Table[{p,norm[v1,p],N[norm[v1,p]],norm[v2,p],N[norm[v2,p]]},{p,1,20}],TableHeadings->{None,{"p","||{-π,2}(Subscript[||, p])","||{-π,2}(Subscript[||, p])","||{-8,-5}(Subscript[||, p])","||{-8,-5}(Subscript[||, p])"}}]

Notice that as p ->∞, the (Subscript[l, p]) norm approaches the (Subscript[l, ∞]) norm, which explains its name.

Plot[{norm[v1,p],norm[v2,p]},{p,1,15},PlotStyle->Thick,FrameLabel->{"p","norm"},Frame->True,PlotRange->{All,All},Axes->False,ImageSize->{600,Automatic},BaseStyle->20,GridLines->{None,{8,Pi}},FrameTicks->{{{8,Pi},None},{Automatic,None}}]

Here is how we can define the infinity norm:

norminf[v_]:=Max[Abs[v]]

norminf[v1]
norminf[v2]

Notice that in the above we could even define the norm for noninteger values of p.

norm[v1,Pi]
N[%]

(R^15)

Here is how we can define a random vector in (R^15). The SeedRandom command just makes it so I get the same random vector whenever I run the code.

n=15;
SeedRandom[2]
v1=Table[RandomReal[{-5,2}],{i,1,n}]
v2=Table[RandomReal[{-3,4}],{i,1,n}]

norm[v_,p_]:=((((∑_(i=1)^n)((Abs[v[[i]]])^p)))^(1/p))

Here is a table of the norms for our two vectors.

TableForm[Table[{p,norm[v1,p],norm[v2,p]},{p,1,20}],TableHeadings->{None,{"p","||v1(Subscript[||, p])","||v2(Subscript[||, p])"}}]

And here is the Mathematica built-in command that relates to what we have just done.

Norm[v1,1]
Norm[v1,2]
Norm[v1]
Norm[v1,3]
Norm[v1,Infinity]

(R^2): Graphical representation of Norms

Clear[vecplot,normplot,vec,a,n]

This defines a vector that will have ||v(Subscript[||, p])=1.

a=-0.65;
b=1;
vec[p_]:={a,(((b-((Abs[a])^p)))^(1/p))};

Here is our norm:

norm[v_,p_]:=((((∑_(i=1)^2)((Abs[v[[i]]])^p)))^(1/p))

Let's check if the (Subscript[l, 565]) norm is one:

norm[vec[565],565]

Here is plot of a vector and its components.

vecplot[p_]:=Show[Graphics[{Red,Thick,Arrow[{{0,0},vec[p]}]}],Graphics[{Blue,Thick,Arrow[{{0,0},{vec[p][[1]],0}}]}],
Graphics[{Blue,Thick,Arrow[{{0,0},{0,vec[p][[2]]}}]}],Axes->True,PlotRange->{{-1.5,1.5},{-1.5,1.5}},AspectRatio->1,BaseStyle->20]

Here is a plot of where the vectors with norm equal to one will have their head.

normplot[p_]:=ContourPlot[norm[{x,y},p]==1,{x,-2,2},{y,-1.5,1.5},Axes->True,AspectRatio->1,PlotPoints->200,ContourStyle->{Blue,Thick}]

In the plots below, we plot regions of space for which ||v(Subscript[||, p])≤ 1, for p=1,2,3,4,5,6.

The vector v shown in red is a representative vector for which ||v(Subscript[||, p])= 1.

The blue vectors are the components of the vector in red.

Manipulate[Show[{normplot[p],vecplot[p]},PlotRange->{{-1.5,1.5},{-1.5,1.5}},BaseStyle->20],{p,1,20,1}]

As we approach the (Subscript[l, ∞]) norm, the region  for which ||v(Subscript[||, p])=1 approaches a square with width twice the length of the maximum component of the vector.

Show[{normplot[200],vecplot[200]},PlotRange->{{-1.5,1.5},{-1.5,1.5}},AspectRatio->1,BaseStyle->20]

Matrix Norms

(R^2): Natural Matrix Norm (Subscript[l, 1])

Here is a matrix in (R^2):

n=2;
A = {{1.45,0.96},{0.89,1.2}};
MatrixForm[%]

In (R^2), we know what all vectors with ||v(Subscript[||, 1])=1 look like. They will have tails on the origin and heads which lie on the diamond.

p=1;
Show[normplot[p]]

Vectors that satisfy this condition can all be written in the parametric form (a, 1-Abs[a]), -1≤a≤1. Below is a graphical representation of these vectors in quadrants II and I.

vec[a_]:={a,1-Abs[a]};

vecplot[a_]:=Show[Graphics[{Red,Thick,Arrow[{{0,0},vec[a]}]}],Axes->True,PlotRange->{{-1.5,1.5},{-1.5,1.5}},AspectRatio->1]

vectornormplot=normplot[1];

Manipulate[Show[{vectornormplot,vecplot[a]},BaseStyle->20],{a,-1,1,0.1}]

The vectors in quadrants III and IV can all be written in the parametric form (a, -(1-Abs[a])), -1≤a≤1.

The natural matrix norm describes how the matrix stretches unit vectors relative to that norm. The maximum amount of stretch is the norm of the matrix. Since we know the form of all the vectors that satisfy ||v(Subscript[||, 1])=1, we can compute the natural matrix norm fairly easily, since all we need is to calculate the vector norm for the vectors

A  (a, 1-Abs[a]), -1≤a≤1

and identify the length of the largest vector.

matrixnormplot=ParametricPlot[A.vec[a],{a,-1,1},Axes->True,PlotRange->{{-2,2},{-2,2}},AspectRatio->1,PlotPoints->200,AxesLabel->{"x","y"},PlotStyle->{Purple,Thick},BaseStyle->20]

Here is what happens to the vectors in the other two quadrants:

vec2[a_]:={a,-(1-Abs[a])};
matrixnormplot2=ParametricPlot[A.vec2[a],{a,-1,1},Axes->True,PlotRange->{{-2,2},{-2,2}},AspectRatio->1,PlotPoints->200,AxesLabel->{"x","y"},PlotStyle->{Purple,Thick},BaseStyle->20]

vecplot[a_,b_]:=Graphics[{Orange,Thick,Arrow[{{0,0},{a,b}}]}]

Show[matrixnormplot,matrixnormplot2,normplot[1],vecplot[1.45,0.9]]

The purple parallelogram is the shape that the matrix A distorts the diamond shape into.

The maximum stretch of a vector seems to be out to about (1.45,0.9).

Therefore, the natural norm (Subscript[l, 1]) for the matrix A is about |1.45| + |0.9| = 2.35.

To work this out exactly, we can sketch the norm, which is given by fcn2 below:

Clear[a]
fcn1=A.vec[a]
fcn2=(∑_(i=1)^2)Abs[fcn1[[i]]]
Plot[fcn2,{a,-1,1}]

We see the natural norm (Subscript[l, 2]) of the matrix A occurs at a=±1 (this is where the vetor is stretched the most, which is what the norm is).

fcn2/.a->-1
fcn2/.a->1

So the natural matrix norm (Subscript[l, 1]) for the matrix A is 2.34.

Of course, we want to connect this to a Mathematica built-in command.

Norm[A,1]

Excellent.

(R^2): Natural Matrix Norm (Subscript[l, 2])

Here is a matrix in (R^2):

n=2;
A = {{1.45,0.96},{0.89,1.2}};
MatrixForm[%]

In (R^2), we know what all vectors with ||v(Subscript[||, 2])=1 look like. They will have tails on the origin and heads which lie on the unit circle.

p=2;
Show[normplot[p],PlotRange->{{-1.5,1.5},{-1.5,1.5}},AspectRatio->1]

Vectors that satisfy this condition can all be written in the parametric form (cos θ, sin θ), 0≤θ≤2π. Below is a graphical representation of these vectors.

vec[θ_]:={Cos[θ],Sin[θ]};

vecplot[θ_]:=Show[Graphics[{Red,Thick,Arrow[{{0,0},vec[θ]}]}],Axes->True,PlotRange->{{-1.5,1.5},{-1.5,1.5}},AspectRatio->1]

vectornormplot:=ParametricPlot[{Cos[θ],Sin[θ]},{θ,0,2Pi},Axes->True,PlotRange->{{-1.5,1.5},{-1.5,1.5}},AspectRatio->1,PlotPoints->100,PlotStyle->{Blue,Thick}]

Manipulate[Show[{vectornormplot,vecplot[θ]}],{θ,0,2π,0.5}]

The natural matrix norm describes how the matrix stretches unit vectors relative to that norm. The maximum amount of stretch is the norm of the matrix. Since we know the form of all the vectors that satisfy ||v(Subscript[||, 2])=1, we can compute the natural matrix norm fairly easily, since all we need is to calculate the vector norm for the vectors

A (cos θ, sin θ), 0≤θ≤2π

and identify the length of the largest vector.

matrixnormplot=ParametricPlot[A.vec[θ],{θ,0,2Pi},Axes->True,PlotRange->{{-2,2},{-2,2}},AspectRatio->1,PlotPoints->200,AxesLabel->{"x","y"},PlotStyle->{Purple,Thick},BaseStyle->20];

Show[matrixnormplot,normplot[2],BaseStyle->20]

We can see that it looks like the length of the major axis on the above elliptical looking curve is (Sqrt[(1.8^2)+(1.5^2)])= 2.34. Therefore, the natural norm (Subscript[l, 2]) for the matrix A is about 2.34. How can we work this out exactly? Calculus! Let's find the maximum over θ.

fcn1=A.vec[θ]
fcn2=((((∑_(i=1)^2)((fcn1[[i]])^2)))^(1/2))
fcn3=(fcn2^2)
fcn4=D[fcn3,θ]
Solve[fcn4==0,θ]

Two of these produce minimums, and two will produce maximums.

fcn1/.θ->0.73144191898911`
fcn2/.θ->0.73144191898911`

We see the natural norm (Subscript[l, 2]) of the matrix A is actually 2.25887.

Of course, we want to connect this to a Mathematica built-in command.

Norm[A,2]

(R^2): Natural Matrix Norm (Subscript[l, ∞])

For the natural matrix norm (Subscript[l, ∞]), the process we previously used would be a bit cumbersome. Let's try it anyway!

Here is a matrix in (R^2):

n=2;
A = {{1.45,0.96},{0.89,1.2}};
MatrixForm[%]

In (R^2), we know what all vectors with ||v(Subscript[||, ∞])=1 look like. They will have tails on the origin and heads which lie on the square.

side1=ParametricPlot[{1,t},{t,-1,1},PlotStyle->{Thick,Blue}];
side2=ParametricPlot[{-1,t},{t,-1,1},PlotStyle->{Thick,Blue}];
side3=ParametricPlot[{t,1},{t,-1,1},PlotStyle->{Thick,Blue}];
side4=ParametricPlot[{t,-1},{t,-1,1},PlotStyle->{Thick,Blue}];
square=Show[side1,side2,side3,side4,PlotRange->All,AspectRatio->1]

The natural matrix norm describes how the matrix stretches unit vectors relative to that norm. The maximum amount of stretch is the norm of the matrix. Since we know the form of all the vectors that satisfy ||v(Subscript[||, 1])=1, we can compute the natural matrix norm fairly easily, since all we need is to calculate the vector norm for the vectors and identify the length of the largest vector. We just have to do it for each side of the square.

vec[t_]={1,t};
matrixnormplot1=ParametricPlot[A.vec[t],{t,-1,1},Axes->True,PlotRange->{{-3,3},{-3,3}},AspectRatio->1,PlotPoints->200,AxesLabel->{"x","y"},PlotStyle->{Purple,Thick}];

vec[t_]={-1,t};
matrixnormplot2=ParametricPlot[A.vec[t],{t,-1,1},Axes->True,PlotRange->{{-3,3},{-3,3}},AspectRatio->1,PlotPoints->200,AxesLabel->{"x","y"},PlotStyle->{Purple,Thick}];

vec[t_]={t,1};
matrixnormplot3=ParametricPlot[A.vec[t],{t,-1,1},Axes->True,PlotRange->{{-3,3},{-3,3}},AspectRatio->1,PlotPoints->200,AxesLabel->{"x","y"},PlotStyle->{Purple,Thick}];

vec[t_]={t,-1};
matrixnormplot4=ParametricPlot[A.vec[t],{t,-1,1},Axes->True,PlotRange->{{-3,3},{-3,3}},AspectRatio->1,PlotPoints->200,AxesLabel->{"x","y"},PlotStyle->{Purple,Thick}];

Show[matrixnormplot1,matrixnormplot2,matrixnormplot3,matrixnormplot4,square]

The purple parallelogram is the shape that the matrix A distorts the square shape into.

The maximum stretch of a vector seems to be out to about (2.5,2).

Therefore, the natural norm (Subscript[l, ∞]) for the matrix A is about |2.5| = 2.5.

To work this out more precisely, we can sketch the norm, which is given by fcn2 below, for a large value of p

Clear[a]
vec[t_]={1,t};
fcn1=A.vec[t]
fcn2=((((∑_(i=1)^2)((fcn1[[i]])^300)))^(1/300))
Plot[fcn2,{t,-1,1}]

We see the natural norm (Subscript[l, 2]) of the matrix A occurs at t=+1 (this is where the vetor is stretched the most, which is what the norm is).

fcn2/.t->1

So the natural matrix norm (Subscript[l, ∞]) for the matrix A is 2.41.

Of course, we want to connect this to a Mathematica built-in command.

Norm[A,∞]

Excellent.

A quicker way to calculate the (Subscript[l, ∞]) natural matrix norm is to make use of the theorem that states:

||A(Subscript[||, ∞])=(max_(||x(Subscript[||, ∞])=1))||A x(Subscript[||, ∞])=(max_(1≤i≤n))(∑_(j=1)^n)|(Subscript[A, i,j])|

(∑_(j=1)^2)Abs[A[[1,j]]]
(∑_(j=1)^2)Abs[A[[2,j]]]

(R^2): Spectral Norm

The spectral radius enjoys a relationship with the natural norm (Subscript[l, 2]) for a matrix, and provides an easier way of calculating this particular norm.

Here is how the spectral radius is calculated:

ρ[A0_]:=Module[{a1,λ},λ=Eigenvalues[A0];a1=Max[Abs[λ]] ;Return[a1]]

ρ[A]

Here is how the (Subscript[l, 2]) natural norm is found from the spectral radius:

spectralnorm[A0_]:=(Sqrt[ρ[Transpose[A0].A0]])

spectralnorm[A]
Norm[A,2]

The matrix we have been working with is divergent:

MatrixForm[A]
MatrixForm[(A^2)]
MatrixForm[(A^3)]
MatrixForm[(A^4)]

since the powers of A are not approaching the zero matrix. Therefore, the norm is not approaching zero as the power of A increases:

TableForm[Table[{i,spectralnorm[(A^i)]},{i,1,10}]]