Dynamic Programming Decision Making

All of us set certain goals and try to achieve them. What is the best way to keep track if you have taken the right decision?  Here is a quick method to keep track of your progress. Consider that you set a goal today, and you are trying to optimize all the actions in order to reduce the total cost. The cost here can be due to time, money, effort, social costs and others.

The idea is to incorporate a method to reach the optimum in a step by step manner that uses only the local information. We query that : "If I assume that I am optimal in the future, what action needs to be taken today to ensure a global optimum". This leads us to the answer that, the best way is to determine the best that you can do today, with the assumption that from tomorrow we are going to be optimal. This seems to be very suited to our current practice also. While we drive a car we put the gear to the right speed, so that we can endure the local roads and the nearby traffic. The goal might be distant but we always have to act optimally to reach there.  While spending we ensure that the basic needs for today is met. The savings also are done accordingly. Take the simple game of tic-tac-toe, here we are trying to see the local information in the game and trying to win or in most situations stop the opponent from winning !

Courtesy : http://experimental.designforfun.com

This approach of dynamic policy is widely employed in control systems and  communication. There are two things that we consider in this optimization problem, one is the value or the cost that is incurred due to our decision (actions)  and the second is the policy that we have come up with. So remember at the beginning of each day try to see that you are going to be optimum for 'today', because tomorrow will "take care for itself".

[Topic Courtesy :  Introduction to Stochastic Optimization, IITB]

On Decisions and Learning (Part II)

In the previous post, we have seen how the space and time constraints can affect the decision  making.  In this part, we extend the model to one of practical interest. Such a model should be robust across space and time variations. In effect we cover up the space and time and replace them by agents/players who affect our decision process. To illustrate with an example we take the case of a boy in say, Brazil and a similar aged boy in India. It is more probable that the Brazilian boy latches himself to football and the Indian boy would choose cricket with higher probability.

                     

The agents here is a representation of the culture and tradition - hence depends on the space. There can be agents representing time as well. For instance the decision on taking an umbrella before leaving the home depends on whether it is the rainy season or not. Rain is the agent here and hence represents time. Thus agents are the physical representation of the time-space (real) parameters. Another aspect of agents is that, there are certain agents that aid to the decision with more weight than certain other. Now there are decisions taken in accordance with logic, past experience, and predictions. Each of them has a correspondence with time - present, past and the future.

The process of making the decision can be summed up as \( f : (A_1,A_2, \cdots, A_k) \rightarrow  \mathcal{D} \). Here \(A|_1^k\) are the agents to be considered while taking the decision and the set \(\mathcal{D}  = d_1,d_2, \cdots, d_n\) are the decision choices. This model lacks the past learning that has been accumulated over time. To explain the learning in decision making, we can take the example of the man going out while it is raining (Agent = Rain). The man does not take the umbrella, and gets wet (Decision : No Umbrella). The next time he goes out in the rain he is reminded by the fact that he has got wet (which assumes a cost). Now it is possible to assign a cost of for the decision taken over a action set. Assume that the decision maker has learnt the decision taken over a set of agents till time \(T\). Now the system is opened and the learned costs are considered while taking the decision.