In a previous post, I referred to the importance in environmental and natural resource economics of the technique of dynamic optimisation, also known as optimal control. However, the technique is difficult, and worked examples in textbooks or on the web often seem to pass over key points. Here I present my own example, which I describe as fully worked because it shows every step from the largely verbal statement of the problem to the optimal paths of the key variables and the maximum value of the objective functional, identifying some options and pitfalls along the way. It is intended for readers familiar with elementary algebra, calculus and static optimisation who have at least begun to study dynamic optimisation.
Capital is the only factor of production and is not subject to depreciation. The initial capital stock is . Output is at a rate , and may be used as consumption or investment , the latter being added to . The instantaneous utility function is . We are required to maximise social welfare from time to , where social welfare is defined as the integral of instantaneous utility subject to a continuous discount factor of per time period.
A Note on Notation
A widely used convention is that the subscript , as in , indicates discrete time, and that a variable in continuous time should be written as in . I find however that it saves a little keying time, and results in less cluttered formulae, to use the subscript approach for continuous time, and sometimes to omit the altogether when it is clear from the context. More conventionally, I use the notation to indicate a time-derivative, and for a second time-derivative.
I use Latex to display mathematical symbols and formulae. However, using Latex within a WordPress blog is not entirely straightforward, one problem being to obtain a satisfactory vertical alignment of symbols within text paragraphs. The commas which follow some symbols are a workaround which corrects vertical alignment in many (though not all) cases and seem to me preferable to the alternative of displaying symbols – like for example – with their base lower than that of the surrounding text.
Writing the Problem in Mathematical Formulae
Our problem statement above contains the symbols . The first question we should consider is whether we need all these for a precise mathematical formulation. It is clear that we can dispense with and relate directly to , writing the objective functional as:
We need which is clearly the state variable, but what is the control variable? Since , either of or determines the other. Nothing in the problem statement indicates that one is a choice variable and the other a residual. Either could be the control variable, but we do have to choose (because the method requires maximisation of the Hamiltonian or Lagrangian with respect to the control variable). Let us choose as the control variable (but Alternative 1 below will show that choosing leads to the same results). We therefore write the equation of motion as:
We also have the boundary conditions:
Does that complete the formulation of the problem? No!
If we rely on the formulation above, there is nothing to prevent negative consumption, with investment exceeding output and undefined (because the log of a negative quantity is undefined). There is also nothing to prevent negative investment. Thus the above formulation allows a time path in which capital is initially accumulated, but towards the end of the time period is run down to zero, enabling consumption to exceed output. That could be a desirable scenario if the capital is in the form of a good which can also be consumed. More typically, however, capital cannot be consumed and therefore consumption cannot exceed output, and the above formulation will therefore lead to erroneous results by permitting more consumption than is feasible. Indeed, there is nothing in the formulation to rule out the combination of infinite consumption and infinite negative investment.
We therefore add two constraints and, to prepare for writing the required Lagrangian function, rewrite each as a quantity to be less than or equal to a constant, in these cases zero:
Although we also require that capital should not be negative, we need not specify this as a further constraint since it is is implied by the combination of and , the latter following from the equation of motion together with constraint (5). Indeed, these imply the stronger condition . The combination of (1) to (5) completes the mathematical formulation of the problem.
The Value of W for Two Naïve Solutions
Before applying the method of optimal control, let us consider a couple of simple and feasible time paths for consumption and calculate the implied values of . The results will provide a benchmark against which we can compare our final result. Suppose first that there is no investment and all output is consumed. Then capital is always and consumption is always . Hence:
Now suppose that output is always divided equally between consumption and investment. Before we can calculate we need to find the time path of capital by solving the differential equation:
Making the standard substitution so that we have:
Hence for some constant :
Since we can infer that and so:
As we might expect, allocating half of output to investment, allowing capital to accumulate and increase output as time goes on, yields a higher than simply consuming all output. But there is no reason to expect that this value of is the maximum.
Necessary Conditions for a Solution
From (1) and (2) we obtain the Hamiltonian, introducing a costate variable :
This is a present value Hamiltonian because it retains the discount factor in the objective functional and so converts at any time to its present value, that is, its value at time . An alternative approach will be considered below. Because we have two inequality constraints, we must extend the Hamiltonian to form a Lagrangian, introducing two Lagrange multipliers and :
The expressions in brackets after the Lagrange multipliers are from the inequality constraints (4) and (5) with signs changed. The general rule here is that given a constraint and writing for the associated multiplier, the term to be included in the Lagrangian is .
Applying the maximum principle, we have to maximise the Lagrangian with respect to the control variable at all times. In this case, the Lagrangian is differentiable with respect to , so we can try to use calculus to find a maximum. But we also need to consider whether there might be a corner solution, that is, a solution at either of the limits of the constrained range of , which are and . We can rule out the possibility of a maximum at , since equals minus infinity. But there is no obvious reason why there should not be a maximum at for at least some values of , so we should keep this possibility in mind. Setting the derivative with respect to of the Lagrangian equal to zero we have:
The maximum principle also requires the conditions:
Although the effect of (14) is merely to repeat the equation of motion (2) it is standard practice to write it out at this point in the working. We also require the Kuhn-Tucker conditions in respect of the two inequality constraints, conditions (17) being known as the complementary slackness conditions.
Finally, there is the transversality condition. With a fixed terminal time, but terminal capital free subject to the implied condition , we have the situation known as a truncated vertical terminal line. Therefore we provisionally adopt the condition:
However, we will have to check that the resulting solution is consistent with the condition (and if not we must recalculate the solution with fixed at ). (12) to (18), with the provisos noted, constitute the necessary conditions for a maximum.
Sufficiency of the Necessary Conditions
We will test whether the Mangasarian conditions are satisfied. The basic conditions are:
(A) The integrand of the objective function, , must be differentiable and concave in the control and state variables, and , jointly.
(B) The equation of motion formula, , must be differentiable and concave in and jointly.
(C) If the equation of motion formula, , is non-linear in either or , then in the optimal solution we must have for all .
Considering these in turn:
Condition (A) is satisfied since, applying a calculus test for concavity:
We need not consider here since it does not occur in the integrand.
Condition (B) is satisfied since the formula is linear in both and and therefore concave, linearity being sufficient for concavity (there is no requirement for strict concavity).
Condition (C) is satisfied since, again, the formula is linear in both and .
For our problem, a further condition is needed for each of the inequality constraints, the general rule being that if a constraint is represented in the Lagrangian by the expression where is a constant, the required condition is that be jointly convex in the control and state variables:
(D) must be convex in and jointly.
(E) must be convex in and jointly.
These conditions are satisfied since the functions are linear (again there is no requirement for strict convexity).
Thus the Mangasarian conditions are satisfied, so we can conclude that the necessary conditions (12) to (18) are also sufficient for a maximum (and need not consider the more complex Arrow conditions).
Inferences from the Necessary Conditions
Using a common approach to simplification, we differentiate (13) with respect to time and then use (15) to substitute for :
Using (13) again we can eliminate and (but not ):
Collecting the terms in and using the complementary slackness condition (17) (which, since can never be zero as equals minus infinity, implies and therefore for all ):
Using the equation of motion (2) to substitute for :
Collecting terms in we have the differential equation:
Before proceeding we will explore two alternative approaches.
Alternative 1: Investment as the Control Variable
Suppose we take investment rather than consumption to be the control variable. The utility function is still which we will now have to write as , so the objective functional will be:
The equation of motion will be simply:
This is not tautologous since it implies that investment is the only cause of change in capital, eg there is no depreciation. The inequality constraints become:
Hence the Lagrangian is:
From the Lagrangian we derive the conditions:
We also have the complementary slackness conditions:
Differentiating (A5) with respect to time, using (A7) to substitute for , and substituting for :
Collecting terms in and using the first complementary slackness condition to eliminate and we have:
It can be seen that this is equation (26) above with signs reversed, so thereafter we can proceed as in the main line of reasoning.
Alternative 2: the Current Value Hamiltonian
When the objective functional contains a discount factor, an alternative method is to use the current value Hamiltonian. Where there are inequality constraints, this leads to a current value Lagrangian, which for our problem can be written:
where the multipliers are equal respectively to the original multipliers each multiplied by . In the necessary conditions, the equivalent of (13) is slightly simplified by the absence of the discount factor:
On the other hand the equivalent of (15) requires an extra term (the discount rate being ):
The difference between the coefficients in the terms in (15) and in (A14) may seem trivial, but it leads to additional complexity later in the reasoning. The equivalent of (24), which I re-write here for ease of reference:
is found to be:
The more complex coefficient of in turn makes it slightly more complicated to solve what below I call Case 2. This is not to argue against the current value approach, still less to suggest that it represents a pitfall. But whether on balance it simplifies matters, as is often suggested, seems to depend on the type of problem.
Solving the Differential Equation
Case 1: over some time interval.
Case 2: over some time interval.
Since Case 1 implies that is constant over the relevant interval, we can infer that over that period. Equation (26) therefore simplifies to:
The standard method for this type of differential equation is to make the substitution implying and . After dividing through by we are left with the equation:
By factorisation or by the quadratic equation formula, this is neatly solved by . Hence the solution to the differential equation (27) is:
where are constants to be found (generally a second order differential equation requires two constants of integration). Differentiating (29) with respect to time we can infer:
Having obtained equations (29) to (31) it is tempting to think that our work is almost complete. Putting in (29) we have:
Since investment right at the end of the time period can do nothing to increase consumption within the time period, we can infer that at . Hence, putting in (30):
Substituting into (31):
As expected, this yields a higher value of than either of the naïve solutions considered above. Nevertheless, this is not the time path that maximises . The fallacy here is the assumption that our Case 1 applies to the whole period . Just because at , it does not follow that at all .
We must also consider Case 2, . Using the second complementary slackness relation (17), this implies that:
Thus Case 2 is what we described above as a corner solution. Using the equation of motion (2) this implies that, within the relevant time range, and therefore . Hence the differential equation (26) reduces to:
Integrating with respect to , noting that can be treated as a constant since :
Which Case is Terminal?
We will now show that, as time approaches , the system must be in Case 2, with constant. This is what we would expect from economic reasoning, since there must be a time beyond which the effect of further investment in making possible higher output and consumption in the remainder of the time period is too small to compensate for the consumption that would be forgone in making that investment. To show this using the method of optimal control, we start from the transversality condition (18), . We can therefore reduce (13) at to:
Given the first complementary slackness relation (17), , this further simplifies to:
This implies that (otherwise would be infinite which is impossible given the problem data). So the system cannot be in Case 1 at , and must be in Case 2.
When Does the System Switch from Case 1 to Case 2?
Taking our Case 2 equation (36) at , and using (38) to substitute for :
From the equation of motion (2), and since in Case 2, we can substitute for :
Substituting for in (36):
While the system is in Case 2, is constant, so we can replace it by and therefore by :
Since Case 2, by definition, has , and since from (38) , the system will be in Case 2 while:
So we can infer that the system is in Case 1 during and in Case 2 during .
Solving Case 1
Having found the time period over which Case 1 applies, we can now determine the constants in equations (29) to (31). Taking (29) at we have:
Since the system switches to Case 2 at with , from (30) we have:
Substituting into (29 to (31), we have the time paths of the key variables over :
Although not essential to solve the problem, it may be of interest to note the time paths, over the same period, of the various multipliers. From the first complementary slackness relation (17) and because can never be zero, we can infer that , and from the definition of Case 1 we have . Substituting these values into (13):
The value of can be interpreted as the shadow price of the state variable, capital, that is, the amount by which could be increased if an extra unit of capital were available at time . It can be seen that this value at is , which may seem surprisingly small given the extra consumption over the whole period which an extra unit of initial capital would make possible, but can be shown to be correct given that depends on the log of consumption.
Solving Case 2
A feature of Case 2 is that remains constant. To find at what level it remains constant, we have simply to find its level at , when Case 1 switches to Case 2. Substituting into (55):
This is the value of over the period , and enables us to confirm that and therefore to accept the condition (18), , without qualification. Over the same period, and:
Turning to the multipliers, for the same reason as during Case 1. Substituting for in (43):
Thus increases gradually from at to at . The positive values of when indicate that if the constraint were relaxed then could be increased.
To obtain over the same period, we use (61) and (62) to substitute for and respectively in (13):
Thus falls from at to, as expected, at , at which point an extra unit of capital would have no effect within the time period on or .
Table 1 below shows the values of all the variables at integral time points over the whole period , convering Cases 1 and 2.
The Optimal Value of W
It remains to check that the optimal paths we have now identified do indeed result in a larger than our best so far – the obtained from our Pitfall 2. Summing the relevant integrals over the Case 1 and Case 2 periods we have:
The main source used in preparing this post was:
Chiang, A (1999) Elements of Dynamic Optimization Waveland Press, Illinois