Lec 29 MIT 18.085 Computational Science and Engineering I

I thought I would start by maybe the most uh outstanding example uh in this direction of linear programming so linear programming it's an whole area it's linear algebra with inequalities that's what it really is about and um the whole area just sprang into existence after World War II or in the in the developments of World War II uh I guess the name the most famous name in linear programming is uh daning who created this uh Simplex method pretty unusual for a method to um survive fundamentally unchanged for such a long time that the that the first method suggested to solve linear programs should turn out to have such a a long life and still still very alive uh with competition now from uh a different uh a different class of methods but the I hope you'll feel that this subject is kind of fun uh I'm picking linear programming well I'm just creating an example here uh to show what Duality is about um and then the second lecture this afternoon please stay for part of it if possible uh it's about some integer programming problems uh like the marriage problem marriage usually have an integer number of marriages uh and in fact this problem is highly moral you have either zero or one uh marriages and and how many are possible and so on and the network flow problem maximal flow problem uh so that's the second lecture this will be uh linear algebra as I said let let me see what are the so what's the Primal problem the unknown is X so we minimize the cost which is C transpose X you see the linearity the cost depends linearly X1 up to xn might be the uh amounts of n different materials that we have to buy and um C's would be the costs per the unit costs of those materials so if we buy and materials in these amounts at these costs that's our total cost what are the constraints the constraints of are the constraints are one here's the key constraint is that the the vector X all the components have to be non- negative we can't use negative amounts of materials and the other constraint is that we have to satisfy some requirements and those will be expressed by axal B so there you go we here's our friend ax equal B appear again and now notice with M with few equations so hopefully many solutions we we're expecting lots of solutions if we have more unknowns than equations but we're of course looking for non- negative Solutions well then the question is are there any that's already a question are there any vectors X that's that that are solve the constraints this is called feasible if an X that satisfy the constraints is called feasible feasible X and then among the feasible X's we're looking to minimize the cost so that used the inputs a b and c and it's a pretty straightforward you can see the type of problem that that comes in the cost is linear the constraints are and there's this inequality constraint okay now I want to talk about the duel problem and the point is that in throughout this subject of optimization there's a twin problem that uses the same inputs ABC in in what looks a completely different way and yet yet their two are closely related it's it's this is one of the more subtle aspects of linear programming and all of optimization is this presence of dual variables well actually that theme of variables x's and y's has been fundamental in this whole from the start of these lectures we are are you remember in our Network flow problems we had potentials U and currents W and there was a certain Duality between those and in fact we had M MN potentials and M currents M nodes and M edges oh actually graphs are with nodes and edges are going to be Central in the uh second lecture today so we W aren't totally it's not toally news to us that uh that there's some reverse problem which turns out to be a maximization also linear problem of course and now it'll be the B transpose y so the B that was in the constraint is now popping into the cost if I call this cost or whatever income maybe and what are the constraints on this uh well not to anybody's surprise a transpose is going to show up and this Vector C is going to show up on the right hand side so it'll be a transpose y equal C okay have I got that right or do I want greater than or equal I bet I want greater than or equal here if here I yeah I better just check do I want less or equal or greater equal I think I want less or equal because not surprising the the uh we had a greater than or equal in in in the Primal problem well lesser equal shows up here but the main point is a transpose shows up again so we really uh I could stretch our little framework why do I say little our framework of uh three steps to include linear programming shall I do that that uh and then well so let me make some comments about this and and and do that yeah yeah how are the well at the moment these are two independent problems that I mean I I could hide the Dual problem never tell it never tell you about it and just discuss the Primal how to solve that I am revealing that there's a second problem which if I had five lectures on linear programming I probably wouldn't reveal so early that there's a second problem hiding around with the same inputs and also another linear program this is that's a linear programming problem too linear programming is linear costs linear equality and inequality constraints however I I can play I can massage one problem to fit another one I could uh I could uh I could have ax less or equal B there and play with it and and and introduce some slack unknowns to make up the equality sign and they would have to be non- negative and put them add them to the X's all I want to say is that the pr this fits the general form but others that don't look like they quite fit do as long as they have have linear costs and linear equations and inequalities is on like no no condition on why yeah it's it's it's interesting uh uh that's why you saw me goof and correct I have to think every time okay is there a condition let me let me tell you about Duality so duality so I'm really jumping here to the heart of the of the underlying math says that the maximum in the Dual problem equals the minimum in the Primal that the answer to this problem the the the maximum whatever is the Min is equal to the minimum cost so that uh um in a way this is a big help because it tells us us when to stop if we have a suppose we have an algorithm that gradually finds better and better X's this is what you really do in reality you you get a approximate X and you say is that the best I can do you look around it as calculus always does you say oh I can go off I can change that X to it to in to reduce the cost and so you go that way and then you don't know when to stop well there's several ways to know when to stop one way is when you get to a point x star let's say let's call let's call Y star the winner here and xstar the winner here so I'll sort of introduce the notation star for the optimizer the best then so one way to stop is to get to some xar you don't know that it's xstar yet you look around but every way you look is worse uh yes yeah this is the this is the minimum C so I mean that at the this is this says the the this quantity that you're trying to maximize when you do it equals this quantity that in this other problem you're Trying to minimize well you are naturally asking yourself what's the relation between these problems so let me prove right away weak Duality we what's what's always easy in these problems is to show that the maximum is is always below the minimum maximum in one problem that that look looks goofy right Max less or equal Min but this is the maximum in the jewel is always less or equal the minimum in the Primal we can show that right off the bat just from taking two two steps will do it but those are different problems so not too surprising and then um and then the the serious Duality thing is that actually when you push this quantity you're trying to maximize as high as it'll go and you pull this quantity you're trying to minimize this cost down as low as it'll go they meet and then you know you've got the answer because they can't pass each other according to this simple let me just prove that so the maximum in The Duel for for any so this take any X I want to show that this quantity cpose X is I I want to show that cpose X is always less or equal b b transpose y for for any for any allowable X for this is for any allowable X and Y so uh no yes maybe no maybe yes let's see what is it I'm I'm I just got them reversed here the thing I'm maximizing right thank you the thing I'm maximizing ing take that off the vide tape okay is B transpose y That's the quantity that I'm maximizing so I push that up but it doesn't matter any y that's acceptable will be below in the cost of Any X that's acceptable thanks and now that's for for any acceptable any shall I use the right word feasible X and Y any feasible X and Y this will work okay can we just take that step let's see if if if Y is feasible then let's see ax is B right if x is feasible ax is B right so I know that so now I'm going to connect those problems if if x is feasible then ax is B so this is ax transpose y right because B is ax okay now what am I going to do well anytime I see ax transposed like like one idea in my head write it as X transposed move the parenthesis write it as X transpose a transpose Y and move the parentheses so that I'm looking at a transpose was that a transpose what did I have there why yes it did thanks okay that's the key step so simple for matrices corresponds to integration by parts in Continuum problems that we met for a while okay and now why is this so that's okay right actually was that an even an equal no that was right that okay oh that was even an equal sign wasn't it that was even an equal sign because ax was exactly B okay now I'm ready for the step that this is less or equal this now why is that because a transpose y you don't mind what order this inner product comes in those are just two vectors I I I'm happy to write X transpose C if you if you would like let me write it there as X transpose C which is the same as C transpose X it's just just uh the dotproduct doesn't matter okay here here's the key step then obviously and why is that true well first of all a transpose y we have to it's beautiful how all the requirements come in a transpose Y is less or equal to C so does that make this true right away if a transpose Y is less or equal c not right away I need to know that I'm multiplying these this inequality a transpose y less or equal C by positive numbers if you take an inequality multiply by a negative number it reverses it but that's exactly where this comes in so oh and in a way yeah more than in a way here we can see is that all right so that so that any feasible one matched was less any any feasible y gave a b transpose y that was smaller than or equal to cpose X now our point is that so that's the easy step I mean right that just and there was a corresponding uh easy step when we were doing minimizing potential energy there was a something called a complimentary energy that involved the the flows the or or the uh or the stresses whatever that dual variable was and we could have found a weak Duality and then ultimately equality so so I Prov like you'll see that often to un to prove this that in One Direction it's it's easy that's the direction we took and but now to get equality is not so easy but actually when would equality hold so now this this actually not only gives us a easy argument but also gives us a condition for optimality when R when these equal when well we since we have all equal sign I only have to ask right there when does well it's when they're equal when they're equal I guess uh X transpose so but when will equality hold X transpose times now I'm just going to put them on the same side of the equation times C minus a transpose y has to be what well right it's always greater or equal zero from feasibility but to get equality so this is to get to hit to to get equality equal sign to be optimal so that that was a important word in there optimality that meant what's the condition like derivative equals zero in this in this world well it's this it's just that this has to be equal zero right I just I mean I'm just looking at the only place there was any question and saying if we want equality then we got to have equality at that step so this lesser equal followed immediately for any X and Y equality is only going to happen at the best ones so maybe I should put the star X star transpose C minus a transpose y star that's how we recognize the winner and and you see what's going on here this is a DOT product of two vectors and what's the deal with those vectors what do we know about those vectors well they're orthogonal but then each of those vectors is non- negative the x is non- negative and the C minus a transpose Y is non- negative so when could two non- negative vectors be orthogonal if I have a couple of vectors there's some sort of a complementary condition if if x is a is actually positive then x i sorry if if if a particular component of X is actually positive then the ice component of this guy has to be zero where and similarly if if if we if we if our a transpose Y is strictly below C on some component then the corresponding X has to be zero it's um it's really neat neat stuff in a way these X's are like L gr multipliers for these and somehow this this uh business of uh this is somehow uh um yeah it's well I don't know what words to use uh there's this but this is a typical feature showing up in linear programming that that connects the x's and the Y's at the winner so what I'm doing so far is pure algebra right I'm asking you I've asked you now for these minutes to just accept this these two problems the Primal and the Dual as as announced then to verify that sure enough by accepting those we got weak Duality and then identify the one place where equality has to hold to get du Duality I I haven't given you any help up to now with examples so it's time for me to turn to an example so you see what kind of questions these are and uh uh so I'm uh well this these four last four Le are really coming back from the uh applied math book introduction to applied math uh chapter 7 chapter 7 seven and eight really is where the last lectures are going to come from and uh including the um marriage problem and the and the uh other special network problems that that are following this afternoon uh but for an example I'm going to turn to my linear algebra book so that you have a sort of independent example uh uh this is one where well this is so now I'm going to this example so can I describe the example and and you'll see it as and then see it as a linear programming problem okay so we've got we so the problem is four homework problems if I write that up there it'll you won't realize this is example this isn't the homewor these are not the homework I'm not going to say the homework problems this is just there are four imaginary homework problems uh at minimum cost okay so what are the rules we've got uh three X's three so X1 is the work is the work done by a PhD on these problems X2 is the number of hours spent by um uh a grad student and X3 will be the number of hours spent on the problem by a computer okay so we can solve these problems by any combination of number of hours so now I have to tell you how much those cost and actually I used numbers that were unrealistically small but let's stay with them so this is the cost is $5 an hour for the PHD $3 an hour well that's unrealistically large most grad students would say and uh $8 an hour for the computer okay and then how fast can they solve the problems is another question the PHD can solve one problem per hour the this guy is equally fast one problem per hour and the computer is twice as fast it can solve two problems per hour okay okay now I've told you everything about the Primal problem and can we state it as a linear program so what's the cost total cost is the so we minimized we minimize the cost which is what well this is $5 an hour for X1 hours and the grad student spend it costs $3 an hour for X2 hours and the computer costs $8 an hour for X3 hours good so our cost is linear and that's linear cost is quite reasonable you know we're assuming then no efficiency of scale right if we had 100 problems the computer might be able to you know we could write a better program and solve them more efficiently or even the grad student might speed up or slow down or whatever but right now linear okay what's the constraint well the constraints are that they they can't work a negative number of hours and and interfere here so so X1 X2 and X3 are all all greater equals zero right okay but now what's the constraint about getting the four problems solved all right see so I just want to get four problems solved so if this person works for X1 hours and is able to solve one problem for I think here's the constraint that X1 + X2 + 2 x3s is four so this is the number of problems solved this this is an equation of the number of problems solves so this one works X1 hours solves one problem per hour total X1 problems X two problems this the computer works X3 hours but because it solves two problems an hour the number of problems it SES in X3 hours is to X3 is that okay so that our Matrix then has only one row it's the Matrix 112 m is one one one constraint and N is three and you're not allow allowed to solve the problem on the Fly here you have to listen to linear programming as as we go okay it fits this framework that that first of all there are many many solutions many many feasible points feasible point is a is a is a vector X that's non- negative required and that s satisfies the the one equation so we got three X's let me draw the feasible set so this is coming from introduction to applied math and here's a picture of the feasible set this is X1 Direction This is the X2 Direction This is the X3 Direction and I'm sort of Imagining the positive octant do you You2 uh so so what is that what's that equation give me in my picture if I graph the if I graph the points that solve that equation I get a plane I get a plane that's a that's a linear equation gives me a plane and but so that plane of course goes infinitely far I mean regular plane but I'm only interested allowed to look at the part of the plane that's in the in this positive octant so I'm seeing sort of a plane that well actually we could figure out what these Corners are you see this is the plane this this plane is there but it's we're only keeping the part that's that's a feasible right we're only keeping the points that are so what is this point here that's that's the point X1 equal four right yeah X1 is four and the other X's are zero so that's the point x if I wrot it wrot it as a vector 4 0 0 that's a corner maybe I call that corner p and what's this corner Q This is the point also four so it would be the point 040 where all the work all four problems are being solved by the a grad student and then this point up here say R has coordinates 0 0 because it's and then two because then the computer would work two hours now and any mixture in between the the triangle in between is any mixture where where they're all working well the point is that not that at the at the I'm looking for the xstar now the best the cheapest the cheapest who who should do this work here and the the point will be that one of these Corners is the winner that that if I and I'll think why but but let me emphasize that point the winner is at a corner the winner is at a corner so let me think so what does that mean in ordinary words that means that the winner only one of these X's at the optimal X star is not is working is not zero if and I only need like One X to to adjust because I only have one equation so so so a corner let me sort of say what a corner is corner is I have M nonzero x's and that and that gives me enough to solve the M equations axal B and the N minus m zero X's because I'm at a corner so that's it's we can visualize this thing in this 3D picture the visualization gets quite um harder in so in M and N dimensions in so my point was that the winner xstar is a corner some corner but the thing is so you might think okay just try all the corners evaluate the cost at every corner once you know the winner is a corner and but the thing is there're too many corners the number of Corners is n shoose m or something which is a very big number fast it's three here so there are three corners and we can check them all but generally you have to imagine so are you imagining n dimensionals space like this where n was three and then what do the constraints give us so so think of n dimensional space X1 to xn and then we have M equations the M equations what do they give in this space well every equation is linear so it's a plane so I've got M planes and so I've got this Octan and they got this octant positive direction in N dimensional space I I don't know why I'm saying this because I can't visualize it either but except it's it's sort of important to realize that the feasible set is a polyhedron it's it's hasn't got any curved boundaries everything is either they're the base planes which chop out the uh coming from X greater equal Z chop out the positive part or the constraint planes coming from the M constraints that cut through in some way so you got somehow a region that's created out of M plus n planes it's a it's a but it's can be quite complicated n plus n is pretty big number you just you're you're say we have oh I mean this wouldn't be big 20 equations 10 constraints so we're in 20 dimensional space we've got 30 planes all woofing through and you can see that the feasible set could be empty for example how could I how could I change this Pro how could I create a problem where the where there were no feasible X's out of this example well I could make that four a minus four four if I made that four a minus 4 then there wouldn't be any non- negative X's that would solve with a minus 4 but of course the problem with with minus four homeworks to do well often I have no homeworks but I never assign negative number of homework problems so anyway okay which is the winner but which is the best point well in this case I could just check the cost at every at every corner right oh and I have to say why is the winner at a corner the fact that the winner is at a corner is is a big help and it somehow depends on linearity you know if we have a linear function you know you if it's increasing it continues to increase until you bump up against the constraint or continues to decrease if we're trying to make something decrease like like suppose you said well maybe that's the winner in the middle there all three working away well but if I discover that to move from around from there in some direction is better then more is better more is better more is better until finally maybe I just make it hit an edge at that point it can't go it's not legal to go further because an X would get negative so at that point I uh look again okay but now my point is that we only have to consider corners and why is that restrict that to one of the edges and it's the same reason that's right drive drive to a corner that's the that's a fundamental reasoning yeah what one way to see it is is now to think of these costs if we got the so this is a good way to see not only that it's at a corner but but what what picks out that corner so the cost suppose I trying to do this job for a total cost of a million dollar it's not going to cost me a million dollars for these uh very uh parsimonious uh workers and only four homework problems but suppose I look at the equation 5x1 + 3x2 + 8x3 oh I don't even know if I could spend a million dollars yes I could could I I don't well I don't even know but anyway suppose I look at the X's that solve the that give me a cost of a million dollars yeah I I I don't think I could spend a million not feas they're not feasible right if you think of it that as another plane yes exactly that's what I want to do move it in that's right h a corner exactly exactly right so if I tried to spend a million dollars that would if I look at the equ the plane 5x1 plus 3x2 plus 8x3 equals a million that's off here somewhere well okay I'll reduce that let's say what am I trying to do minimum cost why don't I start at zero and move up that'd be smarter right okay so let me take the plane equals zero that's a plane right it's through the origin but I don't think it would touch the feasible set I can't get my problem solved for zero right I mean somebody's going to have to work and get paid here yeah so the This Plane would like are you imagining this is kind of the the it's a plane now it's not a tetrahedron that's just a plane and this plane would lie somewhere behind it not touching but then B says if I move if I increase this to one it'll move up it'll move parallel to itself of course because I'm just changing the right hand side but still I can't get them solved for $1 up to $2 and so on the the plane is moving you see this plane this cost plane sort of moving toward the feasible set but but not touching it at $2 is nobody I can't do it for $2 and then finally it touches the feasible set and of course it touches out a corner that's the minimum cost so the cost is I can't do it for cost zero I can't do it for cost one I can't do it for cost two I move that plane up and finally I can do it and a plane moving forward will hit a polyhedron at a corner that's that's the geometric way to visualize the re the reason why the winner is a corner so I just oh what would happen then it could hit the whole thing right it hit the whole right so that would mean that any of these plans would work so if I adjusted that if I adjusted these costs to be 1 one and two suppose I suppose I exactly you're picking out a degenerate case uh that if if if the PHD and grad student charge $1 an hour and the computer $2 an hour and they solve them exactly in proportion to their cost which would be imaginable then as you say uh This Plane would be parallel to this and it would boom it would hit the whole feasible set all at once and that would mean that in that degenerate case not only are the corners winners but even everything in between but still it's true that the winner is a corner well but not the only winner yeah yeah so it's it's aign yeah it's like it's like uh I mean we're dealing with linear equations here it's like that case would be like you know optimize when you're what's the maximum of that horizontal function well the maximum is whatever that height is but it's satisfied at it's all all the points are just as good so but of course you you know that that that's a a special special case so typically when a bunch of planes when planes move parallel to their s to themselves they'll hit a corner first but there would be degenerate cases where it could hit a whole Edge at once or even a whole flat face at once so that yeah yeah so there are degenerate cases always possible just like the feasible set might be empty there might be nobody that solves the equation if if I had a minus 4 here but I don't yeah so this will be a perfectly non degenerate completely normal case of linear programming and and let's see I was going to emphasize this so where who uses linear programming uh management I mean this is like a major application of of uh mathematics in uh management planning management science I guess is the right description uh so there's a uh but but also in our science uh the big the big uh requirement is the linearity here and of course lots of problems are not linear and then we have to escape from those requirements maybe I mentioned what is quadratic programming what is quadratic programming quadratic programming I'll include a quadratic Factor like X transpose KX in the cost so then I'm minimizing a quadratic subject to linear equations and constraints so that's a case of quadratic programming much closer to the rest of this course if I throw away those constraints then I'm right at the rest of this course minimizing a quadratic with linear constraints is exactly what we might have minimize that energy subject to Kirk off's current law or something like that uh or uh um the subject to the balance equation so so this is that this is the next step after linear programming would be quadratic programming and there would be a duel for that too and it would be another quadratic program but uh the real starting point here is linear programming and then this and that simplifies the the theory because it because we know the winner at a corner okay now which which corner does win the grad student right because the three corners are I could figure out the cost here the PHD is doing all the work solving all four problems working four hours for 20 bucks here the grad student is doing all the work working four hours $3 an hour for 12 here the computer is doing all the work only has to work two hours but its cost is $8 an hour so this is $16 also so 16 is that right yeah 16 20 or 12 so the winner is the optimum is there the grad student all right now but that was this was a small problem where we could check all the corners and my point was can't do that usually so now I speak about the Simplex method which um starts at some corner but I I need some preliminary method to find a corner suppose I start at corner P so start and the idea of the Simplex method will be to move uh to another Corner a neighboring corner what's a neighboring corner on a on a on a one of these high-dimensional deals a neighboring Corner means one of the X's becomes one of the X's that was Zero well look here you go here's the X at this corner what happens if I move to that corner this X One some X drops to zero and some other X enters you could say X1 leaves the basis is when I travel up here and X2 X3 rather enters the basis I'm using that word basis pretty freely X1 enter leaves the leaves the solution becomes zero and a different one X3 enters the solution becomes nonzero and the cost is reduced went from $20 down to 16 okay then I look around from this corner which way shall I go I go along an edge to the next corner so I could go along this Edge or along this Edge so I compute which way is the best way to go well when I go this way my computations are going to show I'm worse it costs more when I go this way it cost less I'm reducing the cost so of course I'll go that way until I can't go any further because I'm reducing reducing the more work the grad student does the more work that's taken away from the computer and given to the grad student much cheaper the more as I travel down here I'm giving more and more work to the grad student until I reached that point when I can't take away any more work from the computer the computer is is no longer doing anything it's dropped to zero so in this case the entering variable was the grad student number two and the leaving variable was the computer number three and it left to the point of dropping to zero the grad student moved up from zero to something whatever it whatever something it took to to satisfy the constraint and I at a new corner now I look around from that corner and the Simplex method of course doesn't know it's got to xstar so it would look around it would say there are two ways I can move from here but they both mean a cost increase so I must be at the winner so that's the that's the idea of the Simplex method start somewhere start at some Corner move travel along edges reducing the cost every time H until you get to an edge where you can't the cost cannot be reduced and then that's that's that's xstar the winner so that's the Simplex method now I have to tell you how the Simplex method chooses the edge to travel on right because in a big problem I'll be out a corner whole bunch of edges a whole bunch of possible entering variables whole bunch of x's that are willing to work and uh the simplex and and some of those will mean higher costs so they won't go those ways and some will mean lower costs so I will pick up which one which way to go which new X should enter and then I have to go along that edge until I come to that corner to that next corner and then I do it again so that's the that's the algorithm repeatedly you're at a corner you test directions you travel on the direction where the like steepest Ascent the cost goes down fastest per step along that edge and then you get to the next corner and you do it again is that a picture of yeah how do you know how do you know that you're not just at a local minimum very good question it's it's because of this linearity it's it's be from these linear equations and linear constraints you can't you you you can't get in uh trapped but uh and similarly if I had this quadratic one and was positive definite I couldn't get trapped but absolutely local Minima and so on are the are the you know they're the Big Challenge in in minimization problems in general you give me a function a cost function see most cost functions aren't linear you give me a cost function of F of 100 variables that you allow me to play with with and uh how do I how do you minimize a function of 100 variables boy that's the that's the zillion dollar question and if that function is convex then you so if the function is shaped like this maybe with Wiggles but still convex then you know there's only one minimum and the problem is much easier if the function is quite General it have Loop like a golf course its graph is some surface with ups and downs and ups and downs then when you find a local minimum you got to you can't just look around in a local minimum because everything looks upwards but at some other point there's a there's a thing much further down and and a whole lot of ideas have gone in to this subject and I guess there are a lot more to come because you somehow have to jog that that possible local minimum you know enough to get out of that little well and on with the job but uh how much do you jog by and you don't want if you spend all your time worrying about this you won't find that local minimum in the first place it's fascinating now okay one comment here and I not wasn't planning and and don't have time to develop interior Point methods but just so you you could guess what that means that means that we we travel in the interior we find a point so interior interior is interior of this feasible set find some place that solves the equation don't worry about the number of zeros and nonzeros this this had to do with Corners find a point find a direction is better travel in it decide when to stop and probably stop short of the boundary then you're still inside calculus still operate so to speak look around find another Point travel for a while find another point and eventually it in infinite many steps reach the winner I mean because you're always stopping short but getting closer and closer but of of course in reality we are happy to get within 10 to the minus something of the winner and and that's okay too so that's that's iterative and this is a finite algorithm right can't take well we know it's finite because the darn thing has only a finite number of corners finite number of edges and we're not going to go backwards on an edge because as we saw if forward on if we went forward on an edge that was because it improved the cost reduced the cost and we'll never travel backward on that edge so it's got to be finite but the number of edges is exponentially large for m&n and so there's a big I mean the big the theory of the Simplex method is does it actually take exponentially long as m&n increases or can it be done in polinomial time this is the one the the most important Pro problem where it was quite unclear even to experts whether um it whether it can be done in polinomial time meaning a power of m&n or whether the whether it grows like the number of Corners which is m n choose M and and grows exponentially well here's the answer to Quick report on that the Simplex method can in really hooked up cases take exponentially long so if I describe the Simplex the the clear straightforward Simplex method although its average cost and it's used all the time so of course it doesn't take exponentially long or or we wouldn't use it but its worst cost in in a in a extreme example could be exponential we we could travel along exponential number of edges before we finally settled on the winter and you see because traveling on an edge the co the cost the Computing cost is is the same however long The Edge so anyway uh but the but average costs are uh polinomial so a lot of people and the guy at MIT is looking at this problem again and found another way of describing average costs and and uh showing that that's polinomial and then there is a polom algorithm for linear programming you have to twist the Simplex method to make it polinomial but you can do it so there is a polinomial algorithm but most people happy with the Simplex method which is polinomial in all practical cases so this is this is polinomial actually this is polinomial too in getting arbitrary getting within a given Epsilon in polinomial time can I mention the name of the guy who uh Guy car marker who uh uh proposed I mean he he threw a bombshell into this wellestablished Theory um by discovering a method that that he for which he claimed high high efficiency so he's in his presentation of his new method it was beating the Simplex method which I mean there is a serious revolution in a important scientific subject because th this is these calculations are done a lot uh then well much more of course that led to many discussions much much further work to understand the theory behind it and now uh I think probably in reality most people most codes would just quickly use the Simplex method but uh interior methods are definitely competitive as always this will be better in some problems this will be better in other problems okay so I've done everything except to explain the Simplex method uh and I haven't explained this interior Point sort of there's a penalty anyway there's a penalty as you get near the border so you it stops you short and then you look again I could try to explain that Simplex method since we've got to this point in a few minutes let's can I take these few minutes to toh describe how the Simplex method works oh and I haven't done Duality I haven't looked at the and this is I want to do I haven't looked at the Dual problem so what's the Dual problem in this particular case b is just the number four I think there's only one y and what's the constraint in the Dual problem a transpose y less or equal C A was 112 so that's now transpose is 112 112 multiplying Y is less than or equal to what are the costs 5 3 and eight yeah I'd rather tell you about the Dual problem okay okay so that says y less or equal 5 y less or equal 3 2 y less or equal 8 and what's the biggest y oh you could tell me the answer we could solve that dual problem I want to maximize 4 Y where y has to be less or equal 5 also less or equal 3 and 2 y less or equal 8 so what's the smallest Yar y the largest y yes the largest y possible is no but but the Y has to solve all three of these conditions so it's three right the Y has to be less or equal five the Y has to be less essentially I have a cheater who's comp compare competing with the PHD for the money but he has to also compete with the grad student for the money and the grad student because he's cheaper is going to the cheater has to reduce his rate because he has to beat the grad student and he has to also beat what the computer is charging so this cheater if you allow me to use that word is is a coming in and saying I'll solve the problems for you pay me and he's going to solve four problems and he wants to set his rate as high as he can but he can't set the rate above what he's competing with and he's competing among other things with the grad student so the cheater can't set his rate per problem he he charges a rate per problem not per hour because he's just looking up the answers only takes him two seconds but per problem he has to has to uh beat or match yeah like like Auto Sales here he has to match what the other guys will do and the other guys will will solve so many this it's the computer solves a pro solves charges $4 a problem right because the computer charges $8 an hour but solves two problems so the computer is really solving charging $4 a problem so the cheater can't charge more than four but the grad student is only charging $3 a problem so what's the minimum the maximum here what's the what's the Y star the Y star is how much can the cheater charge three the maximum is y equal 3 this this is the tight constraint and the maximum is 12 and that's what of course the grad student charged he charged it took the grad student four hours where only took the cheater 4 seconds but the grad student the the cost was the same so that this was 12al 12 and um and we could even see that it was optimal do you see why it's optimal Whi which of these inequalities yeah do you see why you see the winning X xstar you remember xstar is 04 Z only the only the grad student works so two components of X are zero and in those two components the com the cheater uh has inequalities but in the component which matches the grad student in X the cheater has equalities you see it's it's just a beautiful subject this complementarity of the two problems and it gives there it it this because this is a theme through all of optimization the best algorithms are called Primal Jewel algorithms they pay attention to both both problems at once and uh Pro and and solve faster okay so we saw the the Dual problem there and you saw how it came out of the blue I mean somebody was prepared to do the same job and wanted to maximum income from that and uh but constrained by the the inequality uh the max equal the Min so that's the duality in in this example I I could uh yeah maybe I'll since since time's really up on this lecture I don't know whether to describe maybe I'll start by just maybe I'll just quickly describe how does the Simplex method operate yeah because I already have in words so let me just just repeat those words how does the Simplex method work okay we're at a corner okay now you might think it's natural to think okay I'm going to travel to another corner and I've got two edges to choose from and you might think okay the natural thing is figure out the other Corners their costs and choose the lowest that sounds totally reasonable but ites takes time to figure out where this corner is means solving a system of equations so the Simplex method doesn't do that it computes the sort of steepness of The Descent if if we let the if we let the grad student in to work for an hour how much will that reduce the cost if we let the computer in to work for an hour how much will that reduce theost cost we're we're figuring out the not knowing the length of the edge CU we don't know where the next corner is we're figuring out we're doing local problem around this corner we we're looking for the steepest Ascent that's that's the steepest Ascent is the first idea in in minimization go downhill go downhill as steeply as possible now it can happen and it actually does here that that the steepest descent is actually along this Edge but the edge so then you say okay I'll take that edge so that's the decision which the which the Simplex method makes which way is steepest once it's that tells us which corner it's which that tells us which new guy is coming into the picture then it solves the system ax equals B to find out where that corner is and it has and and somebody it's because it's a corner somebody has to stop working okay so it travels along there so there I've described the Simplex method without giving you the algebra of how it does it but we're talking linear equations here it it it's at a it's at a corner it's wondering okay who which X shall I let in which shall be the entering variable okay so it it says I've got a choice of edges and each along each Edge it can figure out locally what the reduction in cost would be if we let these in say for unit amount and then it chooses the one that looks the best the steepest but just as in steepest descent that doesn't mean it's going to hit the winner cuz this one could be steepest but shorter and and and it very likely is that the it doesn't mean the steepest one is probably not the right one but it's downhill anyway so you go downhill to this point and then again you apply the Simplex method you look around here well this way it's descending this way it's not even descending so of course the steepest is this way travel along here so the cost is at every every Simplex step you've got you've got linear equations to solve that's so that's why you the cost depends on the number of equations and then it depends on the number of steps you take and the point is that the of course this cost per the cost here which is significant is still polom right we have only m m equations here so but that's the in reality that's significant part of the cost is solving those equations and then then how many steps it takes can be exponentially large but in reality is not so there's the Simplex method for linear programming and there so that's the Practical side and the the basic side tells us the Winner's a corner and at the winning Corner the uh the max equals the min that's that's sort of my uh shot at linear programming here and what what's to be remembered well first of all it's it's uh described more fully in the applied math book um so you could look there for uh details of the method and some more mention of the excitement which came with car marker's ideas I can remember going to a meeting in Boston standing room only in a Boston hotel when he first announced his results heor uh no where was he at that time then he he actually did come to visit MIT for a year after that uh but his courses were not successful uh he he and so he I'm sure he wasn't that happy cuz nobody wants to give a class with one person or something uh and that's what it happened I think so then he went to AT&T so he's been at AT&T quite a while but my impression is perhaps he's he's uh that was quite a while ago and I think probably he's moved on but I'm not 100% sure uh but that that was at the time where the question was could you patent this I mean you know this meeting in the Boston Hotel was all the newspapers were all there and everything it was big news and uh that was at the time first time when I think he did patent it by of course patent you can't patent an algorithm which is what the Simplex method is but somehow the lawyers created a way you could patent the MCH the apparatus which would execute the algorithm which was of course just a computer but um it now it's a standard thing that stuff gets patented and this set that precedent uh I don't know about things here at the lab uh but in engineer math guys don't we're not business enough to do it but uh all my friends in the wavelet world um patent algorithms I mean the companies involv P patent those ideas uh and uh and uh it's now an established part of life well I'll pick up again at 3:30 if you like okay good thanks