There’s a fascinating spreadsheet going around showing AI algorithms “gaming the system”, that is, training for something that the creators of the system didn’t intend for. Here’s one amusing example that I’ll be referring to:

Neural nets evolved to classify edible and poisonous mushrooms took advantage of the data being presented in alternating order, and didn’t actually learn any features of the input images.

What, if anything, do examples like this show about our current systems and AI in general? Chris Hoffman claims:

A computer entity like Siri, Alexa, or Cortana doesn’t understand and think as we humans do. It doesn’t truly “understand” things at all.

Machine learning is all about assigning a task and letting a computer decide the most efficient way to do it. Because they don’t understand, it’s easy to end up with a computer “learning” how to solve a different problem from what you wanted.

I am sympathetic to the argument but think it goes too far. Consider the mushroom example. If you were learning to distinguish edible mushrooms from poisonous ones and always received images of them in alternate order then sure, you would likely discount that. The alternating order wouldn’t have any bearing as to what you learned. But what if you were out in the wild with a mushroom expert who pointed out every mushroom you two saw and its edibility, and it turned out they always alternated? You would certainly notice that. What would it tell you? Either that there is something spooky about how mushrooms grow where you are looking (perhaps they are planted) or that the “expert” is a liar. Either way it’s information that you would take into account.

Let’s look at another situation. Suppose you are learning about mushroom edibility again, and again you decide to look at a large data set in order to learn. As it turns out the images you look at only contain yellow and white mushrooms, which are always edible or poisonous respectively (in that data set). There’s nothing essential linking the colors and edibility, but you would certainly take it into account anyway unless corrected by an external source.

We could extend this farther. Suppose that you’ve learned how to identify all known mushroom species and know their edibility. You’re out one day picking them and come across a new species. It is poisonous but looks almost identical to an edible species, and you don’t take the difference into account because you think it’s irrelevant, it having never been a distinguishing factor for mushrooms before. You didn’t understand that the mushroom was poisonous. But how were you supposed to from the data set given to you?

A response would be that you merely had incomplete information about the mushrooms, while the AI had irrelevant information. And that’s true in some sense. But then, what is relevant information? If you learn to identify mushrooms by sight you aren’t directly learning about edibility. Rather you are learning a set of associations between appearance and edibility. The associations are almost always going to be correct, but the appearance doesn’t cause the mushroom to be edible or poisonous. If someone had a form of ESP that allowed them to directly perceive edibility, they might laugh at us for looking at “irrelevant appearances”.

What we can say about our current machine learning algorithms is that they’re efficient. They try to get the correct output with as little input or processing as possible. Just like us! We too try to separate signal and noise when making decisions, and we try to limit it to a few apparent-high-quality signals simply because our abilities are limited. This has often led us wrong with superstitious beliefs having no basis in reality, but it’s necessary when making the best of what we have.

It seems to me that what Hoffman was trying to get at is that the AI systems don’t have as much contextual information as us. And that is surely true. The scope of knowledge that they can have is very small. But that’s hardly evidence in favor of mindlessness. As I’ve said before I don’t think that current AI systems can be said to have a mind in a meaningful sense. But that’s not because they “learn the wrong thing”. In fact I think that if they did they would be more likely to, not less. Just as children so often do what their parents least expect of them, I think that “understanding” AI would be less understandable by us.