Thursday, August 04, 2005

Pattern Recognition Redux

The holy grail of computing is pattern recognition. Nobody - but nobody - knows how to make software perform robust pattern recognition. There's some heuristic and algorithmic systems which can do some limited pattern recognition, but each one has to be carefully pounded out and optimized, and even then they are fragile creatures.

The holy grail is a robust pattern recognition system.

I might be biased. Maybe there are some other holy grails, such as quantum calculations. Hell, maybe you can get pattern recognition out of quantum computing. But there is no doubt: whoever creates a software system for pattern recognition will be a multi-billionaire within ten years.

You see, pattern recognition solves most of the problems we have today. If we had robust pattern recognition, we could do image analysis on a level unprecedented today. We could determine the relevance of information, then combine the information to a prediction - say, on the stock market. Or international politics. We could have a 100% accurate voice recognition system, and a 100% accurate language translation system. We wouldn't have to worry about software crashing, because a pattern-based development environment would be able to both optimize and bug-test itself on a level we can barely imagine. There are also untold military applications, and more besides that.

In short, pattern recognition is the building block of "hard AI" - a thinking machine.

Of course, it's also fantastically hard. No, neural nets don't solve it. They solve a tiny subset. No, relational DBs don't solve it. They solve a tiny subset. No, logic doesn't solve it. In fact, logic doesn't even solve a tiny subset.

Admittedly, we've only had the resources to really work on it for maybe fifteen years. Many problems have gone for centuries, only to suddenly be solved as circumstances permit. But, as I've mentioned, circumstances are coagulating and time compressing. Everything is moving faster. The need for this solution is far greater than our current ability to provide solutions.

My field is data analysis, and pattern recognition would solve it. Right now, our data-control resources are at the dying edge of their lifecycle. Our databases are flooded, our systems collapsing under the weight of a hundred million additions a day. Whether you're a government or a corporation, your data is killing you. At best, it is a leech, useless and slowly draining cash. At worst, it actively betrays you.

Software to manage data grows every year, but each growth is a painfully crafted, quite fragile heuristic. Costs are rising. Managing ten million a day can easily cost ten million a year, and that's unacceptable to me.

Unfortunately, although I've learned quite a lot, my endless forays into pattern recognition have never rewarded me with a swig of good ol' god juice, so I'm forced to think about methods which can actually be USED, instead of methods which are hopeless excursions into the land of mad mathematicians.

One solution is to analyze the heuristics and try to get a method which can generate heuristics. Although this would be more robust, it's also pretty damn near impossible.

The other option is to use humans.

Yes, the hu-mans do pattern recognition. Witness the hunam missiles and explosive space shuttles!

Since the beginning of time, humans have used their pattern recognition to earn a better life. Many of the things humans can do can now be emulated under controlled situations. For example, it wouldn't that hard to replace ALL McD's employees with robots and interfaces. It would, thanks to the government, be illegal. Which is kind of funny.

These days, the key advantage humans have over robots is their pattern recognition. And, in truth, most humans don't have significantly better pattern recognition than a lava lamp. Some humans have an advantage over robots. Others do not.

Since software cannot yet perform robust "learning", we need to get humans to perform it for us. Obviously, this is SOP. It's what you're getting paid to do, presuming you have a job that's more than $9/hour. In fact, most of the pattern recognition you do is to interface with other established pattern recognizers, which a software version really couldn't do very well anyhow.

My last post, on the Google game, demonstrated one method of substituting in human components for advanced software.

The basic idea is: you can do that anywhere. You just have to figure out how...

No comments: