Lec 4 MIT 18.085 Computational Science and Engineering I

for lecture four uh but during the break of course um two things I to catch up on uh one the Overflow room reminded me that I should repeat anything you say because otherwise it doesn't get transmitted or recorded so all or the speaking now to the Overflow room you just have to ask the question louder uh from for Mei remember just remind me real loud that I should repeat everything and the other thing that came up was that I hadn't drawn should have drawn the basic uh picture of uh that shows that the battery B is in this is this is our edge here that the battery B is in series with the uh resistor and the current source is in parallel so it's the this is It's the battery so the three parts of our our equation e equal B minus Au was Au comes from the voltage differences all that way B gives us the voltage difference across the battery so e is the voltage drop across the resistor and that's where ohms law operates so that's a figure I'm glad to add in okay so I'm coming back now to the third major application which is we have a system of equations so here here's the setup we have a system of equations Au U equal B but no solution so this is typically M equations n parameters and unknowns and unknown parameters to describe the model and M is bigger than n and very often much bigger than n so the a problem to think about is fitting a straight line I mean that's the most basic uh example this is going to lead us to these squares and uh this the most basic example of these squares is that we have uh some points that are close to a straight line but not really on it and we're looking for the best line so we're looking for this line something like that which has two parameters so our line is like C plus DT if if if if suppose we're tracking a satellite and poers off so it should fly in a straight line with constant velocity but our measurements B so our these are our measurements like B1 B2 B3 B4 and they're slightly off so there's noise here there there's error in the measurement say error E maybe I'll put a minus call it minus E so that so that e will then be B minus Au and I'll get back to the sign convention that that I had here so this this is measurement error this is error from the fact that our equations are not perfect if if we had perfect measurements then even though we had however many measurements we would still be we could still find a solution but in general are there's error here and there won't be a solution so the question is what to do and uh in the first chapter of linear algebra if you get a system of equations Au equal B you do elimination and you discover there is no solution while you give up but um I think we all understand that this is too important a problem to give up on so we need to get the best solution so we're looking for looking for the best solution so our framework will still start with a solution U so here let me draw our framework again just to see it solution U uh an error B minus Au U and then now we have I don't know quite what takes the place right now of this Matrix C so here and and then what would we expect here we'd expect a trans we we expect a transpose W is zero well well I expect I mean I'm getting in the habit of seeing a transpose there but of course we have to see why so this so we're these are our equations these are going to be our equations for so our problem so what's our problem our problem is find the best the best and I'll put it in quotes the best you the nearest we can come to solving this system and I'll call it you hat you hat will be the best one okay okay so really in a way now our unknown is now have a hat has a hat on it this is the error it produces there's a step to a w that we can for the moment take the identity Matrix can I put the identity Matrix in there for the moment and then there should be a step here but all that has to come naturally out of the problem so focus on the problem what's best I so I'd like to understand the problem from the point of view of minimizing the error so the error is so so the idea is minimize so one way to say best is minimize the error eal B minus Au in other words minimize what we miss by in the equations we we we can't solve the equations exactly we have errors our best line has an error E4 and a little error E3 I I drew them too close to being a line for you to see the errors there if I if I draw another example I'll make it far from a straight line so you can see there are four errors E1 E2 E3 E4 yeah let me draw again can I just draw again now so you see this is how the problem typically looks they almost look in a straight line but now let me draw it's enough to have maybe three um points because that's enough that'll give us so if I put in three points say there there and there now I really have S I want to find the best straight line and can you tell me which way it goes um it looks like maybe through here somehow you think somehow that that maybe is our best straight line and then the errors are E1 that's error one in equation one this is error two in equation two error three in equation three and what I want to do is minimize the error and what do I mean by that I mean I'll minimize the total error so let me put down what I'll actually minimize E12 plus e22 plus E3 2 okay so that's that's a decision there what the objective function would be we could have made a different choice and so this is this will lead us to ordinary least squares ordinary meaning that unweighted or ordinary unweighted these squares and then we'll come back to cases where we would want to wait the errors um uh unequally but here we have equal weights on and I made all the weights one so this is really the length of the vector squared it's I could also write it as B minus a u squared can it may be helpful if I just write it in all the different notations that we might write this quantity uh the or even even let me just keep going B minus a u transpose B minus Au everybody recognizes that when I take take uh the row Vector times the column Vector that'll produce just exactly E12 E2 s and E3 squ right the row times the column will give E12 E2 s and E3 squ and now if I write this out a little more I'm seeing U transpose a transpose Au U and uh B transpose Au U and uh U transpose a transpose B and then a constant term B transpose B that doesn't worry us I uh gosh I never wrote the same quantity so many ways um but this final way we see a transpose a appearing so we're we're uh okay okay so how do we minimize how do I find the U hat that makes this error as small as possible and again uh just to repeat we had we had a we made a decision to to make to to to decide on that quantity to minimize the advantages of that decision are quite important the quantity we've picked here is a pure quadratic second degree U transpose U appears second degree and first degree and constant but no cubic terms no other stuff and when we minimize a quadratic what kind of equations do we get linear that's the that's why that's one good reason why this is such a important choice that when we minimize this thing we'll end up with linear equations because derivatives of quadratics are linear we could introduce some waiting eventually with let me delete the UN weighted will will minimize some e transpose there'll be a waiting Matrix in there so that it won't just be sum of s but it will still be quadratic and minimizing will still be linear so weight at least squares is still going to give us linear equations and they're equations we've we've seen before so this is now the new example we're making the decision here uh to minimize a quadratic and now we just have to do it and we certainly expect that we can uh figure out how to minimize a quadratic okay in fact we've got various ways to do it we could do it by calculus so by calculus or by linear algebra well you know which is my favorite probably um but I'll give calculus a chance here um uh um so what does it mean by calculus I mean setting derivatives to zero the derivative this is my this is my if I Circle it that's the completely written out form for the for the objective function say the total squared error maybe I call it capital E so this is capital e is a function of U and U is n different components so we have a function of n variables a quadratic function of n variables and how do you minimize in calculus you set partial derivatives to zero right so calculus would take the derivatives of this quadratic set them to zero and um have linear equations okay the only thing stopping us or slowing us down is that it's a function of n variables and we've written it in Vector notation so so how do I so I want to take the derivative respect to U1 to zero the derivative respect to U2 to zero so I've got n equations the partial derivatives with respect to the gradient you could say of e the gradient is just the set of all the first derivatives and those have to be zero um I'm G to okay I'm going to go to linear algebra uh and then uh um because if I write the equations down then you won't listen to the linear algebra appro description of the problem and I think it's important um of course it'll Arise at the same equations but I I just want to see my problem geometrically I shouldn't even say linear algebra I should say geometry what is this asking me to do geometrically what what's the picture behind it well you could say there's one picture it's trying I'm trying to fit a a straight line through these points in this particular case with n equal to two unknowns and m equal three points that's one way to say that's the picture but that's not the picture I'm want you to think of I want to look at this I want a picture that really takes the big the big view so I'm trying to solve Au U equal B and the these this is a vector in M dimensional space so let's draw M dimensional space and well start with the origin and draw a b okay there's our Vector B so that's the right hand side that's the vector of measurements some ER are built in there or or unfortunately included uh but now what about now how do I draw a times U it's a Subspace it's a Subspace because I I'm allowing any U so so there So eventually of course I'm going to pin down a particular U hat that I that's the winner but what is a * U it's it's every time I'm multiply a matrix by a vector it's I'm I'm combining The Columns of the Matrix this is a combination of The Columns of the matrix it's a vector in the column space of the Matrix this is what's in the column space this is what the combinations of the columns give all the Au so so I have a sort of space let me kind of draw it this way this is all the Au all vectors Au and the name for this space because this is this is a combination of columns keep hammering away on that description it's called the column space and and we'll do an example in a minute so but but uh stay with me okay now when would these equations have a solution because that could happen Au equal B even if we had 50 equations and only two unknowns C and D there might be a solution I mean there's some chance of it what which which bees allow us to get an exact solution to this linear system if they're in the column space the B's in the column space are the vectors that are combinations of the columns that are perfect Aus and uh so for example in the column space of course lies column one whatever that might have been column one of my Matrix and also column two of my Matrix or up to n columns however many there are so The Columns of the Matrix are in the column space the combination what is the space then it's all combinations of these guys yeah and the vector B could happened to be in the space but the chances are against it because this is only an n-dimensional Subspace and the big space where B is that I'm drawing here is RM so this is this is M dimensional space that that this picture is lying in and this is an N dimensional column space because we're only n columns and the combinations of those will only give us an N dimensional Subspace do see I I why do I like this picture and then I'll do it do an example so I'll follow through on this example yeah let me follow through on this example right right now so so we have actual values B1 B2 and B3 those are our actual measurements well let's even give them numbers let's let's let's let's suppose that's the 0.1 one that looks like about one one and there's a point that looks like about the next measurement the next point is let's say 21 and the next well let's should I make that 31 maybe say does that your eyes say well maybe it's three steps over and this guy is four what shall that point be four three 4 three okay so I've got three points 1 1 31 and 43 do they lie on the line probably not but I I I don't rule out that possibility but it doesn't look likely okay now so what are my three equations that I won't be able to solve the three equations will be put a line through those three points so my three equations will be that this line should go through all three points and probably I won't be be able to do it okay so can I write those three equations down to go through that first point this is the txis so T is one and the height is one so the first equation would be C + 1 * d = 1 that would be my first equation let me write all three equations then you'll see the pattern second equation would be when T is three I would also like to get the height one for sure there's no straight line through these things right because I'm sorry I shouldn't even have been doubtful the these guys are at the that's horizontal and that certainly doesn't Okay so the second equation was I at T equals 3 I also want to get the height one so C + 3D T is three now I also want the height one but then at the same time when T is four I want to get the height to be three okay that's my a u equal B this is my a let's let's pull it out here a is the Matrix is 111 134 and U is the two unknown CD and B is the right side 113 okay that's the problem that's my problem those are my three equations I could have started I could have written those down at the very beginning but they represent trying to fit a straight line failing but doing the best possible so what's the best possible the best possible is going to make these so the there's going to be an error there's an E1 here and an E2 here and an E3 here that I'm going to have to live with so I'm looking for the best possible solution when there isn't one Okay now what's happening in this figure B is the vector what's B it's this right hand side 113 and what what's this column space column one is a is is 11 one do you see it and column two is what's column two 134 so I'm I'm I'm not like taking this fresh look at my problem I I'm given measurements 113 and I'm only allowing myself two parameters C and D so I'm only allowing myself to combine 111 the First Column with 1 134 the second column you can see how I'm producing combinations of 111 and 134 and I'm not going to reach B 113 I'm not going to hit it exactly so so what's the best choice what what's the in that picture what should I do I all the all my candidates are down in this column space this plane and yet the one thing I would really like to hit is not in the plane project project that's the key word project the best I can do is to take the closest point so let me move column one along over here because what's the winner the winner is drop a projection onto the plane say maybe it hits there let's say we call that P for projection and what's the so this is this is an a this is my best Au Au hat that is in the plane that is a combination of the Columns of a it's the best combination and what I'm missing by is e that's that's what I couldn't get right I the part that's out of the plane the out of plane part is the error I have to live with the inplane part is the part I can model so B is just e plus P so p is au or Au hat because it's the particular U that gives me that particular point in the plane okay I I like to keep that picture together with this one can you tell me where the P so p is a vector P1 P2 P3 the numbers I don't know yet but where would it Go in this picture where would so here's my picture I'm seeing all the good stuff I'm seeing the points that include the the A and the B here and then I'm I'm making errors that I'm going to live with where would the vector P be well so what is p p is a right hand is a is a right hand side of my equation that I can solve in other words I'm this is my equation that I would like to solve but can't the equation I can solve has I changed the right hand side to the nearest right hand side I can find that I cancel and then I call you hat the solution to that equation okay so let me ask again the question where would the P's come in to this so I see the E and I see the B's so the B's are the height the B's are the heights of the points the E's are the errors so where are the P's on the line on the line exactly the P's are on the line that's height is P1 P2 right on the line those Heights P3 and and I say sure if my measurements had been P1 P2 P3 I would have been golden that's the case when I could have solved the equation they would have been on a straight line that would have been perfect but I'm having to project down to get to that perfect situation I'm having to start with an imperfect noisy error situation and project down and the distance of that projection is the error E that I have to live with okay I still need to find an equation though I I'm happy you allowed me to draw this picture and see the problem this way but in the end uh so in this in this particular problem uh my a has two columns and N is two because a straight line has two degrees of freedom but of course I could uh this picture is going to work in in in higher Dimensions I might have more parameters than just two I might be fitting a plane or or some something in several dimensional space here so I might the m and n n doesn't have to be too I don't have to only fit straight lines but that's the the example to start with okay okay now I've got to get to the equation well I could save time and just write it down uh well or minimize this e yeah okay if I stay with this if I stay with this idea that's the best way yeah that why don't we do it a think way here uh so what's the rule e has to be perpendicular to all the columns brilliant that is exactly the key the vector e has to be perpendicular to every column because these columns are all in the plane right so can you write down let me write down then the equations of the First Column column one oh how am I going to well how how shall I there's column one so column one is perpendicular to e is that okay I don't never wrote it that way before but and also column two the 134 column is perpendicular to e do everybody knows why why do I do this sort of waving motion because inner product being zero is perpendicular I mean that's a wonderful fact we we really ought to back back up and see that but instead we'll only write it down uh that that that it's just amazing that to test whether that Vector is perpendicular to that vector or that vector or that vector or the whole plane is by inner product the inner product of that vector and these guys is zero okay now what have we got so it's perpendicular column one gave that zero perpendicular column two gave that zero that's two equations that's exactly what we wanted because we have two unknowns and what have we got we've got a transpose we've got the error is in the null space of a transpose beautiful that's that's and I'm I'm really mixing e and w here because the identity is I'm taking unweighted I'm doing ordinary these squares so right now there's no C appearing so this is the same as a transpose E equals z for the present okay everybody saw that that that's a transpose so that that's our equation can I can I so linear algebra led me directly to a transpose E equals zero okay I'm happy with that oh but let me plug in E and see what it really says then yeah it's the most important equation statistics right a transpose e is zero that's a transpose B minus Au is zero that's that's our equation and now if I just oh you hat for the winning guy the winning the winning uh projection P so this is yeah okay so um now shall I open that out a little and make that a transpose a you hat equals a transpose b key that's the that equation gets five stars if there was room uh that's the that's the normal equation in statistics we're doing regression linear regression everything is linear here it's the fundamental way that you it's the first thing you would do if you had a bunch of measurements and you wanted to fit a straight line you wanted you decided on a on a set of paramet so you have a bunch of measurements bunch of parameters and we want to choose the best parameters you had and that's this is the first equation you would arrive at and it would be exactly our our model here with the identity because right now we haven't uh thought of anything different okay okay so that that's key these normal equations all right maybe so just a moment or two on those normal equations do they have a solution well what's the what's the so we're back in our we understand this linear ality problem is that Matrix a C A transpose a our friend is it invertible is there a solution is there a unique solution does this give us an answer a The Columns of a have to be independent right right and if we set up our problem suitably they are and are these for this problem did I get it okay those columns are certainly independent when would I screw up well if I had a if I gave a third column in that same plane so that I now have allow myself a third a c and a d and an e but foolish because the projection didn't change nothing changed it's just just uh I've now uh haded a third column that was dependent on the others and the U hat would not be well determined because many combinations of those three columns could give the projection you with me on this we wouldn't do it of course unless we weren't watching or something it happen but if we set up the the problem correctly we would expect independent columns and uh this Matrix would be symmetric positive definite invertible everything perfect and and we would get the answer and we ought also to figure out what would be the what would the what would this capital E come out to be we ought also to figure out when we do you remember we really began with the minimization of this quantity and now the best the minimum value was when you when when I took this particular U this winning U and we should figure out okay then the winning e the bottom of the the bottom of the bowl so to speak for this function e is it zero no we can't we we can't get the total error down to zero we can't get e transpose e down to zero we could figure out actually so if we had this uh can I can could I leave that as a as a u I'll put it in the notes but but those will take a few days so maybe I just say it now in words if you've written down this problem carry it through to figure out what all these things are figure out what you had is figure out what the P which is a hat is and figure out what e is okay you you could you could do all those uh numerically as well as symbolically okay okay so that that is really leas squares and it's so uh uh far-reaching and uh omnipresent it's we meet all all all always okay uh but so now I'm ready in these minutes to start with a butt when would we not be happy with this this is like you you couldn't ask for a better uh numerical problem you're getting linear equations with a symmetric positive definite Matrix easy to write down and in fact with only two unknowns we could write everything but what um I guess so now I have to say okay when when would we wanted to to VAR from this um uh equation okay let's see uh let me so one common criticism of these squares is that it gives too much importance to uh an outlier so I'm I'm an outlier would be a point so suppose we had all these points pretty close to a line and then a fourth point was up there then the best line would change of course here we found or in the homework find the best line through these closest to these three points the best line okay now I have to think okay Inc comes a fourth measurement and composits up there okay if I want to use least so I know I have so many comments to make suppose I want to stay with Le ordinary Le squares then then what would I like to have a grasp of I would like to if with ordinary Le squares with that fourth point I would now have a new uh say one and that was it let's see maybe that was along about uh five and but the height and the thing was 100 that was a real outlier well no problem I could I could I would have a new Matrix a now with four rows and a new B with four components my system would be the same I'd find the best U the best straight line so two two comments about about that best straight line one is if I want to find it I would like to find it recursively I if I've if I've made a bunch of measurements and then in comes one more measurement I certainly don't want to redo a whole lot of calculations I would like to know what's the minimum update how could I update U hat how could I update cndd to take account of this one new measurement so so that's going to be a key Ty topic uh recursive lease squares recursive meaning sort of Step update lease squares if you like that's too much to tackle today but uh it's highly important and then we could even mention the calman filter which is going to come in the calman filter is updating and yet and at the same time the problem is changing the old problem isn't quite stable and I I I but it we get to Common filter so there are those important very important algorithmic questions of how do you how do you incorporate new measurements without recomputing a whole lot of numbers that you already did okay so that's then then I was going to make that would be staying with Le squares and and updating it now I was going to raise this criticism of these squares which may very well have met that that having this fourth point so far off changes the line radically that that the the by squaring the error we're giving tremendous uh importance to this outlier and a lot of uh applications that that would not give us the the CD the best the best u in these squares would not be the best u in our real lives in our real problem so for that reason these squares might not always be the winner can I just write down a a function that's more robust to outliers than ordinary leas squares so uh a function that's more robust to outliers might be and then you can say what's good and bad about it what might be just instead of squaring just take the first Power it's the square that now of course this isn't true anymore so there would be a different e to minimize and it will because we're just taking the first Power it would not overemphasize this uh outlier so what's why don't we just do this all the time gives us a unique right it's not quadratic anymore it's not quadratic anymore we could minimize it so I want to minimize this this sum and I could get some equations but those equations would not be linear if I'm minimizing that something that's not a quadratic in fact something that it's got Corners in it absolute values wow uh the equations will not be linear equations or they'll be piece-wise linear but the corners are going to change a sign somewhere so if I had to minimize this quantity then you know the projection I was measuring everything this perpendicular project everybody's eye immediately agreed that yeah that's the closest point that's the closest point provided we're talking ukian distance sum of squares for this that's not the projection anymore it's the the nearest the P that makes this small is not anymore that same one that our eye thinks of and our actually we get into a problem of linear programming which is so linear programming is prepared to deal with piece-wise linear equations and uh you can deal with that we that that's perfectly possible but that's uh actually we will touch linear programming and we could come back to this possibility this application um toward the end of these this series of lectures but for now uh we better stay with quadratic pure quadratics and therefore linear equations yeah so I guess I'm sort of saying that our equations up to now our linear equations have behind them has been a quadratic minimization problem and and here it is up front for these squares okay now the other comment to make we got this outlier it may have been way off based because that was measured by a different meter different uh different uh hardware and not reliable and maybe we knew that beforehand so that would be another way of putting in information that we have that that maybe the first three equations are fine but then this fourth equation which was D plus C + 5D equal 100 maybe that equation was much less reliable because the meter the measuring instrument was much less reliable than these so what would I do then so that that's this is now statistics and and uh the yeah the statistics of the error is now coming in up to now we've assumed that all the errors were identically distributed but now let's make recognize the possibility that the error from this fourth measurement might be might have much larger variance well I I'll always assume that the error has zero mean that our instrument is at least centered correctly but but it might be an old instrument it might whatever for whatever good reason this equation might be less reliable so how can we account for that that that's my question how how can I change the equations I'll still minimize oh no so now there's an E4 squar right because that's the error in equation four but what will I do what should I do well you know what's coming waited wait it I'll weight it somehow do I weit it more heavily or less heavily L less right less heavily I should wait it less heavily so then when I go to minimize this thing if I put a W for the moment for weight or something I I don't want to put that weight it's going to go in there if I weight it less heavily then when I minimize it doesn't count so much and we'll see exactly how that enters I guess the weight will be probably C4 squ up to now the weights were one C1 and C2 and C3 were all ones and that's why C was the identity Matrix now C is going to change C is going to change as soon as I introduce that weight c is now going to be one one one and this small weight because I'm not weighting it so heavily C4 okay so that's and now I can do the linear algebra repeat the linear algebra with this waiting Matrix coming in and what happens if I just repeat the linear algebra it let let me just say because you'll a c The Matrix C enters here there's a this this this C there's a c in our problem there's a matrix that tells me the weights and it enters here and it enters here and there's now I'm really ready to put five stars by that equation because that's now these squares with weights with correct with with with a weighting Matrix allowed and you see that it's the same equation that we met in an hour ago in uh networks it's the same equation with without the F so it's this is now the the key equation with with with a c but we have to decide on the C okay so that's the remaining thing what's what should we choose for C what should this weight be we know it should be smaller because the the the error is tend to be bigger it's a less this equation is less reliable in other words this equation is getting a weight of one and this equation is getting a weight of C4 small because E4 tends to be big because the measurement was poor okay now notice I I want to emphasize that we have to decide on C4 without knowing that particular sample the particular measurement of the particular satellite that's wor tracking it's the the C4 is small because the instrument is poor we have to know this the statistics of these errors in advance and that tells us what we should wait C4 by not the particular samples 1 1 3 and 100 I mean those are the numbers in the experiment but we had to choose C4 before that and and and what are we allowed to know about C4 we're allowed to know that it's that it's that it's coming from a poor instrument that the errors from that instrument have a higher variance so I'm now using that word variance the errors from in this fourth equation which we expect errors and the variance if we took many many samples the variance in those errors would be larger and then what's the right C4 the right C4 is one over the variance you see that that goes the right direction anyway the good C4 is one over Sigma 4^ squared that's the good that's the good um let's see am I getting no is is just C4 see C yes should be just Sigma 4 right do I want it You' already squ the error squ I've got the error squared right right that's right thanks yes yes yes I've got the errors squared right can you see the difference there this was an e let me complete that e so I'm gon to Sigma a Vari Sigma is a variance Sig s Sigma squ is a variance God Sigma squ is a variance yes yes Sigma squar is a variance right um so actually I wrote on the board here and I'll just because because this board is a little cleaner point to it and then come back to here so we have a matrix that gives us all the variances and it's the inverse of that Matrix that we want okay so now can I talk for a minute and it won't be long enough to do justice to variances but and co-variances but it's a start okay okay so what's our so we're we're speaking about now about variances and co-variance so that it's going to be really the co covariance Matrix and that Matrix is usually called Sigma in the in the literature that I read okay so that and then our choice for then the best C so we we I've got bunch of things to do here I have to say what is what are covariances what is this covariance Matrix and then I would really need to say why the the the smart choice is take C to be the inverse you can see of course that it's in the right direction that when the variance is large we want to wait to be small and if the variance was small that would mean the measurement was very reliable we would want to weight it heavily so this is the best this is called the best by smart I really mean the best linear unbiased estimate estimate B Lue it's sometimes written in the somebody noticed that made a word word and and in the textbook can maybe I could refer to the to I should have said where this material is in the applied math book so in the applied math book uh way back in section 14 is the ordinary Le squares this is the ordinary Le squares is section 1.4 and then weighted these squares and the reason for this weight we'll find later on in section 2.6 so so section 1.4 for what we've done till now section 2.6 for what we're now doing why should we choose this as the but but isn't it beautiful I mean that that that the appropriate waiting Matrix in our in our framework is exactly the inverse of this covariance Matrix so now I just have to say what's the covariance Matrix Sigma so now I'm talking about Sigma so Sigma is Sigma is a matrix and what's in it up here is Sigma 1^ squar which was one our model on the diagonal are the variances of the of the uh Sigma 4 squared okay so what's what's variance maybe can can we now we have to remember the facts of the main facts of Statistics right hooks law OHS law all those guys are gone uh now it's it's a statistical uh the the the constitutive laws from statistics now and this will be a sigma one two and a sigma 21 which will equal Sigma one2 The Matrix will be symmetric it'll it'll be it'll agree with Sigma transpose fortunately so and and very often well it's just since you know some of this let me say a little bit in words when would Sigma one two be zero because that's certainly we've we've been very happy to see diagonal matrices when when would our Matrix be diagonal when they're uncorrelated uncorrelated and we're just using these words without the writing down the math here but but Sigma these off diagonal Sigma I JS tell us correlations of measurement I with measurement J so that if somehow the two meters were hooked together in some way or we use the same meter in in arriving at uh several of these things then then then the errors if the errors were correlated we would have to reflect that fact okay now what is Sigma 1 squared can you just remind me of that Sigma 1 squared is an expected the average if we do many many samples so again if we do many many samples it's the average square of the error in equation one so so Sigma 1^ squar is okay room has to be made but this is okay Sigma 1^ squar is what it's the expected value of the first error squared and we've assumed here I've subtracted off the mean value but you didn't see it because I've assumed throughout that the expected value the mean error the expected value of each error is zero right so zero mean you some of you see this every day more more than I okay zero mean we're assuming so that I've subtracted off the mean but that was Zero then this is the average value the expected value of of the first error squared and of course Sigma 2 will be the expected value of the second error squared and what is Sigma one2 what is Sigma one two then it's the expected value of E1 * E2 E1 * E2 and if they're totally uncorrelated then if I knew E1 E2 could just as well have a plus sign as a minus sign so when I average overall samples I get zero so this is zero if they're uncorrelated because there's no if I know E1 I don't know anything about E2 but if they're positively correlated then when I know the first error I know I have an idea I know the second error tends to be in the same direction and I get a positive number if they're negatively correlated I get a negative number now I have to ask you about this Matrix oh I have to I recognize I have to put this in Matrix notation what is the whole covariance Matrix we got it entry by entry here we got the diagonal or the variances the off diagonal of the covariances now put the whole thing together in a matrix equals a matrix form it's the expected value of what what goes in here e transpose right e transpose got to deal with that shorthand I think CU because in the and it's if it's it's straightforward so what is e e is that column Vector of Errors e transposes a row Vector of Errors what happens when I multiply a column times a row what do I get then if I multiply a column by a row I get a matrix right so I take the column E1 E2 E3 E4 times the row E1 E2 E3 E4 and you see that the net result is a 4x4 Matrix and what's the one one entry E1 squ right and what's the one2 entry in this product of that I mean it's a perfectly legitimate operation it's 4 by 1 multiplying 1 by 4 and producing 4x4 and what does it produce in that one two position it produces E1 * E2 doesn't it and now we take the expected value of this and and I can come back to what that means the average over all samples so it's the expected value of this that produces Sigma and then I've circled Sigma inverse here because Sigma inverse I see that there's the inverse because that's the right choice for C okay so I'm I guess I'm making these definitions and uh looking to see what they mean I haven't identified the precise reason I mean we can see that roughly large Sigma should mean small C so we we like to have that inverse there but the fact that it's just exactly the inverse Matrix is fantastic okay and I was going to say one more thing about that Matrix or even about this Matrix here this e transpose Matrix tell tell me the properties of that Matrix I I never let the lecture end without saying okay what are the properties of these matrices if I if I take a matrix like uh e transpose yeah just tell me some interesting stuff so we had let me take an e to be say 421 transpose it 421 do the multiplication what do I get for that it'll be 3x3 16 8 4 842 421 right okay what kind of a matrix I got e transposed there symmetric for sure positive okay is it positive definite that's the key question is a combination each row is a combination of other rows so what's the rank of that Matrix only one what's its column space if I took the combinations of The Columns of of those three columns what when in three dimensions what would I have if I take any combination of those three columns can could I be anywhere or on a line the column space is only onedimensional the because one is the rank because one is the number of independent columns because one is the number of independent columns there you see okay so right so that's this is a rank one Matrix is it invertible no right as you said each row is a multiple of the other rows those rows are very dependent there're only one there's only rank one here so it's not invertible so it can it be positive definite not quite right exactly good all right let me just finish this thought but and and I have to repeat to the other uh the people uh the question was if could this be could this be our Matrix Sigma I hope not as you say if that's Sigma we're in trouble because that's not invertible so what all right first of all let's take this Matrix and then and then uh okay so that Matrix is symmetric and and it's not positive definite right because it's not invertible but what is it it's semi it's trying you could say but by itself it's not enough one column times one row has to be only rank one and only semi-definite okay now where does Sigma come in where does the posit because it better be positive definite or we're in serious trouble the thing is this this was e this was a particular sample but then when we average over all all samples the other samples aren't in the same 421 Direction other samples other e will be other vectors so we have a so our our our Sigma is is a is a sum of oh gosh I'll use the letter suum then uh it's a sum of all the e all the possible e e transpose multiplied by their probabilities right that's what I meant by expected value I take all the e transpose and multiply by the probability P of that then sum over all the all the errors that could happen times the probabilities of those errors and so since I've got lots of ease I'm not all hooked on to this single line and the thing does come out positive definite okay so that that's so we really did some fast stuff there we defined Sigma we got it in a matrix form we saw what a single single piece of it looks like e transpose but then when we take this expected value step this averaging step we're averaging a whole lot of different matrices that look like this the weights add to one the P's these these these probabilities add to one but we're mixing up different e transposes and the result is we come out positive definite so this is a symmetric positive definite Matrix and it's goes right into the framework it could be diagonal if these are uncorrelated errors or it could be none it could have off diagonal entries in which case the calculations won't be quite as nice as in the diagonal case but but it's the right Matrix to use there okay was there a final question before we call it a day no good uh maybe a question in the other room but too quiet so uh we're we're through for today and I'll see you next Thursday thanks