neural-netInteresting piece here reviewing the way some modern machine learning systems are unfathomable. This is because they learn how to do what they do, rather than being set up with a program, so there is no reassuring algorithm – no set of instructions that tells us how they work. In fact they way they make their decisions may be impossible to grasp properly even if we know all the details because it just exceeds in brute complexity what we can ever get our minds around.

This is not really new. Neural nets that learn for themselves have always been a bit inscrutable. One problem with this is brittleness: when the system fails it may not fail in ways that are safe and manageable, but disastrously. This old problem is becoming more important mainly because new approaches to deep machine learning are doing so well; all of a sudden we seem to be getting a rush of new systems that really work effectively at quite complex real world tasks. The problems are no longer academic.

Brittle behaviour may come about when the system learns its task from a limited data set. It does not understand the data and is simply good at picking out correlations, so sometimes it may pick out features of the original data set that work well within that set, and perhaps even work well on quite a lot of new real world data, but don’t really capture what’s important. The program is meant to check whether a station platform is dangerously full of people, for example; in the set of pictures provided it finds that all it needs to do is examine the white platform area and check how dark it is. The more people there are, the darker it looks. This turns out to work quite well in real life, too. Then summer comes and people start wearing light coloured clothes…

There are ways to cope with this. We could build in various safeguards. We could make sure we use big and realistic datasets for training or perhaps allow learning to continue in real world contexts. Or we could just decide never to use a system that doesn’t have an algorithm we can examine; but there would be a price to pay in terms of efficiency for that; it might even be that we would have to give up on certain things that can only be effectively automated with relatively sophisticated deep learning methods. We’re told that the EU contemplates a law embodying a right to explanations of how software works. To philosophers I think this must sound like a marvellous new gravy train, as there will obviously be a need to adjudicate what counts as an adequate explanation, a notoriously problematic issue. (I am available as a witness in any litigation for reasonable hourly fees.)

The article points out that the incomprehensibility of neural network-based systems is in some ways really quite like the incomprehensibility of the good old human brain. Why wouldn’t it be? After all, neural nets were based on the brain. Now it’s true that even in the beginning they were very rough approximations of real neurology and in practical modern systems the neural qualities of neural nets are little more than a polite fiction. Still, perhaps there are properties shared by all learning systems?

One reason deep learning may run into problems is the difficulty AI always has in dealing with relevance.  The ability to spot relevance no doubt helps the human brain check whether it is learning about the right kind of thing, but it has always been difficult to work out quite how our brains do it, and this might mean an essential element is missing from AI approaches.

It is tempting, though, to think that this is in part another manifestation of the fact that AI systems get trained on limited data sets. Maybe the radical answer is to stop feeding them tailored data sets and let  our robots live in the real world; in other words, if we want reliable deep learning perhaps our robots have to roam free and replicate the wider human experience of the world at large? To date the project of creating human-style cognition has been in some sense motivated by mere curiosity (and yes, by the feeling that it would be pretty cool to have a robot pal) ; are we seeing here the outline of an argument that human-style AGI might actually be the answer to genuine engineering problems?

What about those explanations? Instead of retaining philosophers and lawyers to argue the case, could we think about building in a new module to our systems, one that keeps overall track of the AI and can report the broad currents of activity within it? It wouldn’t be perfect but it might give us broad clues as to why the system was making the decisions it was, and even allow us to delicately feed in some guidance. Doesn’t such a module start to sound like, well, consciousness? Could it be that we are beginning to see the outline of the rationales behind some of God’s design choices?


  1. 1. Christophe Menant says:

    Perhaps we should be careful not to put in the same basket the incomprehensibility of neural networks and the incomprehensibility of human mind. The former is a determinist artificial agent (even if difficult to analyze), the latter is a biological entity that we do not understand because we lack understandings about the natures of life and self-consciousnessl. And I’m not sure that AGI tells more than AI on these concerns
    Also, talking about a system that does not understand data brings us back to the Turing Test that impicitly addresses the meaning of information.
    Bottom line, I feel we should be clear about three different levels of performances: life, human mind, artificial agents (with life => human mind => AAs). Each level exists as coming from the previous one. Can we pretend understanding human mind while not knowing the nature of life ? Not sure… (tried to address these items a few years ago in a short paper

  2. 2. SelfAwarePatterns says:

    On deep learning systems, I think exposing them to real world data, but only observing what output they produce for a while, might be a good approach. Essentially, this would be like having them serve an apprenticeship period, before allowing them to fly solo.

    Your last paragraph does sound like a type of introspection, which I think is a key piece of human consciousness, although it if only reports information to outside of the system rather than back into it, I’m not sure I would call it that. Such an AI monitoring system could have far more detailed and comprehensive knowledge of what is happening than the idiosyncratic and blind spot riven human version. In other words, machine consciousness might be a radically alien thing from the human variety.

  3. 3. Hunt says:

    Ultimately I think these systems will need psychologists. Asimov is again prescient, Susan Calvin, the protagonist of his stories, was a robopsychologist.

    The way to deal with these systems will be to examine them on the level of their own complexity, just as humans these days pass “psych evaluations” Normal human behavior can actually be aliased quite well by a number of abnormalities. The only way to tease them out, whether caused by corrupt belief systems, or deliberate deceit, is to interview the systems themselves (humans or advanced computers).

    Thinking you can have something like “quality assurance” with irreducible systems is having your cake and eating it too.

  4. 4. Jorge says:

    The key, I think is to combine neural networks with well-understood traditional algorithmic (and mechanical) safeguards. We have a neural net designed to drive vehicles, but at a higher level we have some extremely hard-coded instructions that can detect and override anything extremely stupid the neural network does. This is an excellent post though- it reminds me of those really weird “optical illusions for deep learning neural networks” from a couple of years ago, where random dots and lines were bafflingly misidentified completely.

    I (and many others) would be extremely hesitant to release AGIs “into the wild” simply to learn from the world. There’s nothing to suggest that there wouldn’t be catastrophic failures during the learning process either.

    Also, if Asimov seems prescient it is because his works were the TEMPLATE for our modern imagination about robots. A “real life imitating art” situation. In a way, that is higher praise than merely making a correct prediction.

    One thing to remember at the end of the day though is that even if we can’t resolve all these issues, the computers will still outperform humans in safety. Neural network AI can’t get drunk and drive. Or pilot a jet into a mountain because it’s sad about getting dumped by its girlfriend.

  5. 5. john davey says:

    ” This is because they learn how to do what they do, rather than being set up with a program, so there is no reassuring algorithm – no set of instructions that tells us how they work.”

    Well algorithms are used of course. But the algorithms are used to construct a data structure that is an inherently unreliable and untraceable source of information. The goal is to generate insights using certain types of mathematics where such insights are not evident from other forms of data analysis : similar approaches use linear programming, R, a whole host of data science tools. There is nothing significant about using a ‘neural net’ over any other form of data mining as far as I’m aware.

    The use of ‘neural’ in neural networks is linguistic coincidence and has nothing to do with brains. You might as well call it ‘teddy bear network’. That it cannot produce traceable decisions flows from it’s design, and in this it has exactly the same features as any other data mining tool.

    The only question of “intelligence” that arises is that of practitioners who would risk patients’ lives on the basis of data structures that they know to be unreliable.


  6. 6. Peter says:


    Yes, you’re right of course – good use of algorithms is very much part of it really.

  7. 7. john davey says:


    “Yes, you’re right of course – good use of algorithms is very much part of it really.”

    Its not just that – it’s that some data mining tools work on the basis that the typical computational rational flow control mechanisms – based upon ‘what if – then – else ‘ condition comparisons are too cludgy for sophisticated big data problems and consequently use too much CPU. It can also be terribly difficult to find correlations in data, and data mining tools are meant to help that (although they do appear to always need a big shove)

    Decision support systems like neural nets are not there to ‘be intelligent’ but to save CPU resource and ‘save’ on programming resource by hoping that the data structures obviate the need for endless condition comparison. They are also a cheapskate’s way of trying to avoid proper research and analysis of the data in my opinion. It’s a gamble at the best of times. Putting it into medical BI tools is lunacy unless profits are all you think about.


  8. 8. On the Interpretation of Artificial Souls | Three Pound Brain says:

    […] is fascinating in its own right, and Peter over at Consciousness Entities provides an excellent overview, but I would like to use it to flex a little theoretical muscle, and show the way the neural […]

Leave a Reply