Great points to keep research going

I came across George H. Heilmeier’s catechism, which I think is very useful for keeping my research going. I post it here to remind me:

A set of questions credited to Heilmeier that anyone proposing a research project or product development effort should be able to answer.[1]

  • What are you trying to do? Articulate your objectives using absolutely no jargon.
  • How is it done today, and what are the limits of current practice?
  • What’s new in your approach and why do you think it will be successful?
  • Who cares?
  • If you’re successful, what difference will it make?
  • What are the risks and the payoffs?
  • How much will it cost?
  • How long will it take?
  • What are the midterm and final “exams” to check for success?

EM Algorithm: The way to solve the problem of “Chicken-Egg” in Machine Learning

One of the learning problem of Bayesian Network is given network structure but with some missing values, how to learn the better model to do prediction.

Why is it a “Chicken-Egg” problem? If we have complete data, then we can estimate the model parameters based on them, however our data has missing values. On the other hand, if we have the parameters for our model, then we can infer the values of missing data. But again, we do not know the parameters yet. So which one comes first? That’s the reason why it is like a “Chicken-Egg” problem.

EM stands for expectation maximization. EM can be understood as the solver of such “Chicken-Egg” problem. The high level idea is that:

1)First, let’s do not care too much about the quality of initial model. Just compute the model parameters based on the data with complete values.

2) OK, now we have our toy initial model. Then we can do inference! So we can fill in the missing values using our current model.

3) Now we have complete data with no missing values. so we can build a better model based on them. Then starting from here, you go back to the above two steps, again and again, until you see no change in your models.

There are theories to demonstrate that through above iterative procedure, our model will get better and better. In other words, we do not need to care about the initial model too much, and eventually we will get the better model.

Gibbs Classifier V.S. Bayes Optimal Classifier

Bayes Optimal Classifier: considering all the hypothesis, so is hopelessly inefficient

Gibbs Classifier: two steps

1. Choose just one hypothesis at random, according to the posterior P(h|D)   (D: data set; h: hypothesis)

2. Use this to classify new instance

Error rate comparison: assume target concepts are drawn at random from H according to priors on H. then

E[error of Gibbs] <= 2 * E[error of Bayes optimal]

What is scientist?

During a lunch with professor, I accidentally heard about a saying that “Scientist are those who know more and more about less and less…”, which sounds very philosophical.

Later on, I find the complete version, which is “Philosophers are people who know less and less about more and more, until they know nothing about everything. Scientists are people who know more and more about less and less, until they know everything about nothing”. It is from Konrad Lorenz.

From my shallow understanding, the difference between Philosophers and Scientist is due to the research target different: the former deals with high-level abstract object, which the latter works with concrete target.

Depth and Width about knowledge

Today I was lucky to have lunch with professor that I am working with. During the lunch, I asked a question that I had for a long time: “Now people talk about interdisciplinary study a lot, because different disciplinary seems to merge rapidly. On the one hand, people expect students to have broad knowledge about different  domains, but on the other hand, hope they have depth enough knowledge too. Students learning broad knowledge may face a dilemma while looking for jobs, since employers might think they do not have deep enough knowledge in either one of the domains they are familiar with. So what do you think as a student, should we purse depth or width?”

I rephrased Professor’s answer here, which I think is very enlightening: Depth is the concept from the western culture which wins great achievements by dividing problems into small components and establishing deep enough explorations on each of them. It started at industrial revolution. As hundreds of years past, knowledge structure has changed a lot and especially recently. There are increasing overlaps among various domains. Taking current popular “social computing” concept as an example, different sub-areas of computer science are trying to do research on it, e.g. data mining, machine learning, networking, human computer interaction. Meanwhile, social scientist, psychologist are also contributing a lot. So now it is really hard to find a person who has a depth knowledge on the topic like social computing. Width is coming to play significant role now. So no one is an expert who has so call depth knowledge on emerging themes.