Sunday, July 26, 2009

Data Infrastructure

Not game related

Recently I've been thinking about data infrastructure. Like your house has pipes and wires and vents, individuals are extending themselves using a data infrastructure that allows them to wrangle important data while offloading the boring, fiddly part.

Anyone reading this has a significant data infrastructure. We have vast stockpiles of sessions, cookies, login names, emails, and so forth. While these seem like passive storehouses of data, the truth is that programs are constantly accessing those storehouses and applying that data. While we may not personally use the infrastructure, it's there.

Working with systems that produce vast amounts of data is my day job, and in those situations the data infrastructure is much more significant. 99% of the data that is pumped into the database will never be seen by a human, but it is piped through an infrastructure and transformed into human-useful form through complex transforms.

This sort of data infrastructure is going to get more and more common as individuals start realizing that they want to monitor vast amounts of data without needing to manhandle all of it. For example, how many more years will it be before many people have a health monitor that scans their body's overall health on a day-to-day or even hour-to-hour or minute-to-minute basis? Simply looking down and seeing the present situation is useful, but to really be useful it has to have a history of data on you and the ability to determine what kind of changes mean what kind of things.

Not just health data: I can think of literally dozens of things that could pump data into our "data bloodstream". Power production and use. City contamination or health. Whereabouts of your friends and what instant-community events are around. Data trade on solving evolutionary programming problems. Or maybe that last one's just me.

The point is that we'll be developing more and more autonomous computing systems just to support the kind of monitoring we want to do. Figuratively, we're expanding our senses and memories to new areas.

I think its safe to say that many of the things we do manually today will be done automatically tomorrow. Like choosing backgrounds for your computer screens, or skipping boring tweets, or looking for interesting news.I even think we'll probably see a new kind of programming language which is far smarter about determining what you want to accomplish and the best way to accomplish it.

Not just data will be wrangled, though. I imagine that in a decade or two, when I walk into a coffee shop, my computer will figure out exactly how much caffeine I need and what other additives I'll want in my espresso... and order it for me.

What do you think?

3 comments:

Ellipsis said...

Of course there's an important caveat - just because something's more efficient doesn't mean people will actually use it.

Choosing desktop backgrounds is a solid example where I might simply want to manually look through a big pile of images, experiment with them, crop them, etc., until I have a background I'm happy with, because that makes it a kind of self-expression.

Similarly, I might simply not be in the mood for whatever my computer expects me to want at the coffee shop. Unless you're talking about a computer that can tell what I'm in the "mood" for, in which case I don't think a couple decades is a likely timeline.

Skipping boring tweets, though? That's definitely the future.

Craig Perko said...

Well, I'm not against not using these things, but I'm also not against not using email. Once something becomes extremely efficient, everyone will use it. The things we're used to doing - and even enjoying - will change in the same way that most people don't write actual letters any more.

I think that it's not at all difficult for computers perhaps five years from now to tell your mood. A simple webcam with some expression-deciphering software is within our grasp now, and I can't imagine it will stay under the surface for too much longer.

I just think we'll see a pretty radical change in the next ten years.

Ellipsis said...

Well there's a difference between telling if I'm happy or sad and whether I want a latte or a mocha (probably), and if it's not perfect at telling what I want, telling it not to order for me is just as much effort as ordering was for me in the first place.

In the case of email, the difference in convenience is huge, not just because you don't need to hand-write, but because your message arrives nearly instantaneously. With that kind of value, any technology will be adopted, but I'm not sure that organizing things for me which I can do now in a couple minutes on my own represents the same kind of value.

But we will probably see some radical changes soon, and maybe I'll be proven wrong.