Picture: global workspace. Global Workspace theories have been popular ever since Bernard Baars put forward the idea back in the eighties; in ‘Applying global workspace theory to the frame problem’*,  Murray Shanahan and Baars suggest that among its other virtues, the global workspace provides a convenient solution to that old bugbear, the frame problem.

What is the frame problem, anyway? Initially, it was a problem that arose when early AI programs were attempting simple tasks like moving blocks around. It became clear that when they  moved a block, they not only had to update their database to correct the position of the block, they had to update every other piece of information to say it had not been changed. This led to unexpected demands on memory and processing. In the AI world, this problem never seemed too overwhelming, but philosophers got hold of it and gave it a new twist. Fodor, and in a memorable exposition, Dennett, suggested that there was a fundamental problem here. Humans had the ability to pick out what was relevant and ignore everything else, but there didn’t seem to be any way of giving computers the same capacity. Dennett’s version featured three robots: the first happily pulled a trolley out of a room to save it from a bomb, without noticing that the bomb was on the trolley, and came too; the second attempted to work out all the implications of pulling the trolley out of the room; but there were so many logical implications that it was stuck working through them when the bomb went off. The third was designed to ignore irrelevant implications, but it was still working on the task of identifying all the many irrelevant implications when again the bomb exploded.

Shanahan and Baars explain this background and rightly point out that the original frame problem arose in systems which used formal logic as their only means of drawing conclusions about things, no longer an approach that many people would expect to succeed. They don’t really believe that the case for the insolubility of the problem has been convincingly made. What exactly is the nature of the problem, they ask: is it combinatorial explosion? Or is it just that the number of propositions the AI has to sort through to find the relevant one is very large (and by the way, aren’t there better ways of finding it than searching every item in order?). Neither of those is really all that frightening; we have techniques to deal with them.

I think Shanahan and Baars, understandably enough, under-rate the task a bit here. The set of sentences we’re asking the AI to sort through is not just very large; it’s infinite. One of the absurd deductions Dennett assigns to his robots is that the number of revolutions the wheels of trolley will perform in being pulled out of the room is less than the number of walls in the room. This is clearly just one member of a set of valid deductions which goes on forever; the number of revolutions is also less than the number of walls plus one; it’s less than the number of walls plus two… It may be obvious that these deductions are uninteresting; but what is the algorithm that tells us so? More fundamentally, the superficial problems are proxies for a deeper concern; that the real world isn’t reducible to a set of propositions at all, that, as Borges put it

“it is clear that there is no classification of the Universe that is not arbitrary and full of conjectures. The reason for this is very simple: we do not know what thing the universe is.”

There’s no encyclopaedia which can contain all possible facts about any situation. You may have good heuristics and terrific search algorithms, but when you’re up against an uncategorisable domain of infinite extent, you’re surely still going to have problems.

However, the solution proposed by Shanahan and Baars is interesting. Instead of the mind having to search through a large set of sentences, it has a global workspace where things are decided and a series of specialised modules which compete to feed in information (there’s an issue here about how radically different inputs from different modules manage to talk to each other: Shanahan and Baars mention a couple of options and then say rather loftily that the details don’t matter for their current purposes. It’s true that in context we don’t need to know exactly what the solution is – but we do need to be left believing that there is one).

Anyway, the idea is that while the global workspace is going about its business each module is looking out for just one thing. When eventually the bomb-is-coming-too module gets stimulated, it begins sending very vigorously and that information gets into the workspace. Instead of having to identify relevant developments, the workspace is automatically fed with them.

That looks good on the face of it; instead of spending time endlessly sorting through propositions, we’ll just be alerted when it’s necessary. Notice, however, that instead of requiring an indefinitely large amount of time, we now need an indefinitely large number of specialised modules. Moreover, if we really cover all the bases, many of those modules are going to be firing off all the time. So when the bomb-is-coming-too module begins to signal frantically, it will be competing with the number-of-rotations-is-less-than-the-number-of-walls module and all the others, and will be drowned out. If we only want to have relevant modules, or only listen to relevant signals, we’re back with the original problem of determining just what is relevant.

Still, let’s not dismiss the whole thing too glibly. It reminded me to some degree of Edelman’s analogy with the immune system, which in a way really does work like that. The immune system cannot know in advance what antibodies it will need to produce, so instead it produces lots of random variations; then when one gets triggered it is quickly reproduced in large numbers. Perhaps we can imagine that if the global workspace were served by modules which were not pre-defined, but arose randomly out of chance neural linkages, it might work something like that. However, the immune system has the advantage of knowing that it has to react against anything foreign, whereas we need relevant responses for relevant stimuli. I don’t think we have the answer yet.

*Thanks to Lloyd for the reference.

Picture: correspondent. Aspro Potamos has drawn my attention to the emerging series of YouTube videos, somewhat polemical in tone, on the War against Neuroscience.   If you prefer something less forensic, you may like this series of podcasts on music and the brain from the Library of Congress.

Huping Hu, whose research has been mentioned here in the past, has started the Journal of Consciousness Exploration and Research: the inaugural issue is available online.

Ayad Gharbawi says:

I have been doing my own studies and research for thirty years now on three concepts: Mind, Vision and Reality. I felt it necessary to create novel notations, just as others, like Boole, did, because of the inadequacies of language as per this subject. I believe that I have come to an entirely new methodology when we seek to understand two fundamental issues when we study Mind, Vision and Reality and that is – both their Structure and the Function.

I have been deeply, fundamentally and existentially affected by Quantum Physics as a human being and in my own pattern of thinking and analyzing problems before me.  I obviously understand full well that my work is unorthodox because I am not just presenting a study in one particular niche in these studies – although, of course, I have studied specific issues – but, I am at this stage in my life, in a position to say that I have come to a general theory that comprises an understanding of Mind, Vision and Reality.

That is why, when I have presented one study/manuscript it is often difficult to make any sense from it, and the reason here is because, those single manuscripts that I have been submitting, do not, in and of themselves alone, explain fully my general theory on Mind, Vision and Reality.  I also know full well, that the history of science, shows far too many times, that when one researcher submits an entirely, unorthodox novel methodology in his thinking, he is quite likely to be rejected by the general established body of scientists and philosophers.

But, I still do try.

A sample is here; he particularly asks for views on two pieces here and here.

Picture: AI resurgent. Where has AI (or perhaps we should talk about AGI) got to now? h+ magazine reports remarkably buoyant optimism in the AI community about the achievement of Artificial General Intelligence (AGI) at a human level, and even beyond. A survey of opinion at a recent conference apparently showed that most believed that AGI would reach and surpass human levels during the current century, with the largest group picking out the 2020s as the most likely decade.  If that doesn’t seem optimistic enough, they thought this would occur without any additional fundingfor the field, and some even suggested that additional money would be a negative, distracting factor.

Of course those who have an interest in AI would tend to paint a rosy picture of its future, but the survey just might be a genuine sign of resurgent enthusiasm, a second wind for the field (’second’ is perhaps understating matters, but still).  At the end of last year, MIT announced a large-scale new project to ‘re-think AI’. This Mind Machine Project involves some eminent names, including none other than Marvin Minsky himself. Unfortunately (following the viewpoint mentioned above) it has $5 million of funding.

The Project is said to involve going back and fixing some things that got stalled during the earlier history of AI, which seems a bit of an odd way of describing it, as though research programmes that didn’t succeed had to go back and relive their earlier phases. I hope it doesn’t mean that old hobby-horses are to be brought out and dusted off for one more ride.

The actual details don’t suggest anything like that. There are really four separate projects:

  • Mind: Develop a software model capable of understanding human social contexts- the signpost that establish these contexts, and the behaviors and conventions associated with them.
    Research areas: hierarchical and reflective common sense
    Lead researchers: Marvin Minsky, Patrick Winston
  • Body: Explore candidate physical systems as substrate for embodied intelligence
    Research areas: reconfigurable asynchronous logic automata, propagators
    Lead researchers: Neil Gershenfeld, Ben Recht, Gerry Sussman
  • Memory: Further the study of data storage and knowledge representation in the brain; generalize the concept of memory for applicability outside embodied local actor context
    Research areas: common sense
    Lead researcher: Henry Lieberman
  • Brain and Intent: Study the embodiment of intent in neural systems. It incorporates wet laboratory and clinical components, as well as a mathematical modeling and representation component. Develop functional brain and neuron interfacing abilities. Use intent-based models to facilitate representation and exchange of information.
    Research areas: wet computer, brain language, brain interfaces
    Lead researchers: Newton Howard, Sebastian Seung, Ed Boyden

This all looks very interesting.  The theory of reconfigurable asynchronous logic automata (RALA) represents a new approach to computation which instead of concealing the underlying physical operations behind high-level abstraction, makes the physical causality apparent: instead of physical units being represented in computer programs only as abstract symbols, RALA is based on a lattice of cells that asynchronously pass state tokens corresponding to physical resources.  I’m not sure I really understand the implications of this – I’m accustomed to thinking that computation is computation whether done by electrons or fingers; but on the face of it there’s an interesting comparison with what some have said about consciousness requiring embodiment.

I imagine the work on Brain and Intent is to draw on earlier research into intention awareness. This seems to have been studied most extensively in a military context, but it bears on philosophical intentionality and theory of mind; in principle it seems to relate to some genuinely central and difficult issues.  Reading brief details I get the sense of something which might be another blind alley, but is at least another alley.

Both of these projects seem rather new to me, not at all a matter of revisiting old problems from the history of AI, except in the loosest of senses.

In recent times within AI I think there has been a tendency to back off a bit from the issue of consciousness, and spend time instead on lesser but more achievable targets. Although the Mind Machine Project could be seen as superficially conforming with this trend, it seems evident to me that the researchers see their projects as heading towards full human cognition with all that that implies (perhaps robots that run off with your wife?)

Meanwhile in another part of the forest Paul Almond is setting out a pattern-based approach to AI.  He’s only one man, compared with the might of MIT – but he does have the advantage of not having $5 million to delay his research…

Picture: Socrates looking at himself. Introspection, the direct examination of the contents of our own minds, seems itself to be in many minds at the moment.  The latest issue of the Journal of Consciousness Studies was devoted to papers on introspection, marking the tenth anniversary of the publication of The View from Within, by Francisco Varela and Jonathan Shear (which was itself a special edition of the JCS); and now Eric Schwitzgebel has produced a new entry for the Stanford Encyclopedia of Philosophy.

The two accounts are of course quite different in some respects. The encyclopaedia entry is a careful, scholarly account, neutral and comprehensive; the JCS issue is openly a rallying-cry in support of a programme flowing from Varela’s work.  This, it seems, called for an end to the ban on examination of lived experience;  the JCS gives the impression that it was something of a milestone, though Schwitzgebel’s piece does not mention it (he does cite an earlier paper by Varela, once again in the JCS).

What’s all this about a ban? Well, back in the nineteenth century, psychologists had no fears about using introspective evidence; it was thought that a proper scientific effort would lead to an objectively verifiable kind of phenomenology. We should be able to classify the elements of mental experience and clarify how they worked together, just by examining what went on in our own heads. A great deal of work was done on all this (It was a great disappointments for me to discover, on first opening Brentano’s Psychology from an Empirical Standpoint, that it consisted almost entirely of this kind of thing, and that the only passage about intentional inexistence, the interesting issue, was the couple of paragraphs which I had already read as quotes in several other books.).  There was a gradual refinement of the methods involved, leading on to the great heyday of introspectionism, with Wundt and Titchener in the lead. Unfortunately, it became clear that the rival schools of introspectionism had begun to come up with results which in some respects were radically different and incompatible, and since our own introspections are by their nature private and unverifiable, all they could really do by way of settling the issues was to shout at each other.

This embarrassing impasse led to a reaction away from introspection and to the rise of behaviourism, which not only denied the usefulness of examining our inner experience, but actually went to the extreme of denying that there was any such thing as inner experience.  Behaviourism in its turn fell out of favour, but according to Varela there remained an instinctive distrust of introspection which continued to put people off it as an avenue of research. This is the ‘ban’ he wanted to see overturned.

Was there, is there, really a ban? Not exactly.  Apart from the most dogmatic of the behaviourists, no-one has ever tried to exclude introspection altogether. In recent times, introspective evidence has been widely accepted – the problem of qualia, thought by some to be the problem of consciousness, depends entirely on introspection. I think the real problem arises when we adopt special methods. In order to obtain consistent results, the old introspectionists thought extensive training was necessary. It wasn’t enough to sit and think for a bit; you had to have mastered certain skills of discrimination and perception. The methodological dangers involved in teaching your researchers what kind of thing they could legitimately look for are clear.

Unfortunately, it seems to be very much this kind of programme which the JCS authors would like to resurrect – or rather, have resurrected, and wish to gain acceptance and support for.  Once again we are going to need to learn how to introspect properly before our observations will be acceptable. What makes it worse for me is that the proposal seems to be tied up with NLP – Neuro-linguistic Programming.  I don’t know a great deal about NLP: it seems to be a protean doctrine which shares with the Holy Roman Empire the property of not really being any of the three things in its name – but for me it does nothing to render another trip down this particular blind alley more attractive.

Blandula I don’t know about that, but aren’t they right to emphasise the potential value of introspection? Isn’t it the case that introspection is our only source of infallible information? Most of the things we perceive are subject to error and delusion, but we can’t, for example, be wrong about the fact that we are feeling pain, can we? That seems interesting to me. Our impressions of the outside world come to us through a chain of cause and effect, and at any stage errors or misinterpretations can creep in; but because introspection is direct, there’s no space for error to occur. You could well say it’s our only source of certain knowledge – isn’t that worth pursuing a little more systematically?

Bitbucket Infallible? That is the exact reverse of the truth: in fact all introspections are false. Think about it. Introspection can only address the contents of consciousness, right? You can’t introspect the unconscious mental processes that keep you balanced, or regulate your heartbeat. But all of the contents of consciousness have intentionality – they’re all about things, yes? So to have direct experience of mental content is to be thinking about something else – not about the mental state itself, but about the thing it’s about! Now when we attempt to think directly about our own mental states, it follows that we’re not experiencing them in themselves – we’re experiencing a different mental state which is about them. In short, we’re necessarily imagining our mental states. Far from having direct contact, we are inevitably thinking about something we’ve just made up.

Phi

Picture: Phi. I was wondering recently what we could do with all the new computing power which is becoming available.  One answer might be calculating phi, effectively a measure of consciousness, which was very kindly drawn to my attention by Christof Koch. Phi is actually a time- and state-dependent measure of integrated information developed by Giulio Tononi in support of the Integrated Information Theory (IIT) of consciousness which he and Koch have championed.  Some readable expositions of the theory are here and here with the manifesto here and a formal paper presenting phi here. Koch says the theory is the most exciting conceptual development he’s seen in “the inchoate science of consciousness”, and I can certainly see why.

The basic premise of the theory is simply that consciousness is constituted by integrated information. It stems from the phenomenological observations that there are vast numbers of possible conscious states, and that each of them appears to unify or integrate a very large number of items of information. What really lifts the theory above the level of most others in this area is the detailed mathematical under-pinning, which means phi is not a vague concept but a clear and possibly even a practically useful indicator.

One implication of the theory is that consciousness lies on a continuum: rather than being an on-or-off matter, it comes in degrees. The idea that lower levels of consciousness may occur when we are half-awake, or in dogs or other animals, is plausible and appealing. Perhaps a little less intuitive is the implication that there must be in theory be higher states of consciousness than any existing human being could ever have attained. I don’t think this means states of greater intelligence or enlightenment, necessarily; it’s more  a matter of being more awake than awake, an idea which (naturally enough, I suppose) is difficult to get one’s head around, but has a tantalising appeal.

Equally, the theory implies that some minimal level of consciousness goes a long way down to systems with only a small quantity of integrated information. As Koch points out, this looks like a variety of panpsychism or panexperientialism, though I think the most natural interpretation is that real consciousness probably does not extend all that far beyond observably animate entities.

One congenial aspect of the theory for me is that it puts causal relations at the centre of things: while a system with complex causal interactions may generate a high value of phi, a ‘replay’ of its surface dynamics would not. This seems to capture in a clearer form the hand-waving intuitive point I was making recently in discussion of Mark Muhlestein’s ideas.  Unfortunately calculation of Phi for the human brain remains beyond reach at the moment due to the unmanageable levels of complexity involved;  this is disappointing, but in a way it’s only what you would expect. Nevertheless, there is, unusually in this filed, some hope of empirical corroboration.

I think I’m convinced that phi measures something interesting and highly relevant to consciousness; perhaps it remains to be finally established that what it measures is consciousness itself, rather than some closely associated phenomenon, some necessary but not sufficient condition. Your view about this, pending further evidence, may be determined by how far you think phenomenal experience can be identified with information. Is consciousness in the end what information – integrated information – just feels like from the inside? Could this be the final answer to the insoluble question of qualia? The idea doesn’t strike me with the ‘aha!’ feeling of the blinding insight, but (and this is pretty good going in this field) it doesn’t seem obviously wrong either.  It seems the right kind of answer, the kind that could be correct.

Could it?

Picture: chess with a machine. Kenneth Rogoff is putting his money on AI to be the new source of economic growth, and he seems to think the Turing Test is pretty much there for the taking.

His case is mainly based on an analogy with chess, where he observes that since the landmark victory of “Deep Blue” over Kasparov, things have continued to move on, so that computers now move in a sphere far above their human creators, making moves whose deep strategy is impenetrable to merely human brains. They can even imitate the typical play of particular Grandmasters in a way which reminds Rogoff of the Turing Test. If computers can play chess in a way indistinguishable from that of a human being, it seems they have already passed the ‘Chess Turing Test’. In fact he says that nowadays it takes a computer to spot another computer.

I wonder if that’s literally the case: I don’t know much about chess computing, but I’d be slightly surprised to hear that computer-detecting algorithms as such had been created. I think it’s more likely that where a chess player is accused of using illicit computer advice, his accusers are likely to point to a chess program which advises exactly the moves he made in the particular circumstances of the game. Aha, they presumably say, those moves of yours which turned out so well make no sense to us human beings, but look at what the well-known top-notch program Deep Gambule 5000 recommends…

There’s a kind of melancholy pleasure for old gits like me in the inversion which has occurred over chess; when we were young, chess used to singled out as a prime example of what computers couldn’t do, and the reason was usually given as being the combinatorial explosion which arises when you try to trace out every possible future move in a game of chess.  For a while people thought that more subtle programming would get round this, but the truth is that in the end the problem was mainly solved by sheer brute force; chess may be huge, but the computing power of contemporary computers has become even huger.

On the one hand, that suggests that Rogoff is wrong. We didn’t solve the chess problem by endowing computers with human-style chess reasoning; we did it by throwing ever bigger chunks of data around at ever greater speeds.  A computer playing grandmaster chess may be an awesome spectacle, but not even the most ardent computationalist thinks there’s someone in there. The Turing Test, on the other hand, is meant to test whether computers could think in broadly the human way; the task of holding a conversation is supposed to be something that couldn’t be done without human-style thought. So if it turns out we can crack the test by brute force (and mustn’t that be theoretically possible at some level?) it doesn’t mean we’ve achieved what passing the test was supposed to mean.

In another way, though, the success with chess suggests that Rogoff is right. Some of the major obstacles to human-style thought in computers belong to the family of issues related to the frame problem, in its broadest versions, and the handling of real-world relevance. These could plausibly be described as problems with combinatorial explosion, just like the original chess issue but on a grander scale. Perhaps, as with chess, it will finally turn out to be just a matter of capacity?

All of this is really a bit beside Rogoff’s main interest; he is primarily interested in new technology of a kind which might lead to an economic breakthrough; although he talks about Turing, the probable developments he has in mind don’t actually require us to solve the riddle of consciousness. His examples; from managing the electronics and lighting in our homes to populating “smart grids” for water and electricity, helping monitor these and other systems to reduce waste” actually seem like fairly mild developments of existing techniques, hardly the sort of thing that requires deep AI innovation at all. The funny thing is, I’m not sure we really have all that many really big, really new ideas for what we might do with the awesome new computing power we are steadily acquiring. This must certainly be true of chess – where do we go from here, keep building even better programs to play games against each other, games of a depth and subtlety which we will never be able to appreciate?

There’s always the Blue Brain project, of course, and perhaps CYC and similar mega-projects; they can still absorb more capacity than we can yet provide. Perhaps in the end consciousness is the only worthy target for all that computing power after all.

Picture: correspondent. One of the nice things about doing Conscious Entities is that from time to time people send me links to interesting things; new papers,  lectures, or ideas of their own.  I regret that I have generally kept this stuff to myself in the past, although it often deserves a mention, so I’ve been thinking about how best to deal with it. I would welcome suggestions, but as an experiment I’ve decided to try occasional round-up posts: so here goes.

Jesús Olmo, to whom many thanks, recently drew my attention to this review of  The Ego Tunnel; to PRISMs, Gom Jabbars, and Consciousness, and to the site Conscious Robots.

M.E. Tson has a Brief Explanation of Consciousness.

Mark Muhlestein has a thought experiment Consciousness and 2D Computation: a Curious Conundrum, and has been corresponding with David Chalmers. My own view is as follows.

I think causal relations are the crux of the matter. A computation essentially consists of a series of states of a Turing machine, doesn’t it? Normally each state is caused by the preceding state. Is that an essential feature? I think in the final analysis we’d say no, because the existence of a computation is really a matter of interpretation on the part of the observer. If the different states in the sequence are just written down on sheets of paper, we’d probably still be willing to call it a computation, or at least, we would in one sense. There’s another sense in which I personally wouldn’t: if we read ‘computation’ as meaning an actual run of a given algorithm, or an instantiation of the computation, I think the causal relationships have to be in place.

Now this would be even truer in the case of mental operations leading to consciousness. The causal relations in a Turing machine are to some degree artificial: the fact that we can program them in is really the point. In the human brain, by contrast, the causal relations are direct and arise from the physical constitution of the brain. To exhibit the relevant series of states (even if we assume consciousness in the brain is a matter of discrete states, which actually seems rather unlikely)would not be enough – they have to have caused each other directly in the right way for this to be an actual ‘run’ of the consciousness faculty.

It follows that your projected lights don’t give rise to a consciousness, or perhaps even to a computation. Does this mean I think zombies of some kind are possible? No, because the interesting kind of zombie is physically identical with a real person, and the projected lights are significantly different from the occurrence of the actual run of the computation. Real zombies remain impossible, and all we’re left with is a kind of puppet.

Readers of my post earlier this year about Sam Coleman’s views may be interested to see the nice comments he has provided.

You can send me links to interesting stuff at:

Contact.

Picture: Stanislas Dehaene. Edge has an interesting talk by Stanislas Dehaene.  He and his team, using a range of tools, have identified ’signatures’ of awareness; marked changes in activity in certain brain regions which accompany awareness of a particular stimulus. They made a clever use of the phenomenon of ‘masking’, in which the perception of a word can be obliterated if it follows too rapidly after the presentation of an earlier one. By adjusting the relevant delay, the team could compare the effect of a stimulus which never reached consciousness with one that did. Using this and similar techniques they identified a number of clear indications of conscious awareness: increased activity in the early stages of processing and activity in new regions, including the prefrontal cortex and inferior parietal. It appears that this is accompanied by a ‘P3 wave’ which is quite easy to detect even with nothing more sophisticated than electrodes on the scalp. Interestingly it seems that the difference between a stimulus which does not make it into consciousness and one which does emerges quite late, after as much as a quarter of a second of processing.

No-one, I suspect, is going to be amazed by the news that conscious awareness is accompanied by distinctive patterns of brain activity; but identifying the actual ’signatures’ has direct clinical relevance in cases of coma and apparent persistent vegetative state. In principle Dehaene’s research should allow conscious reactions continuing in a paralysed patient to be identified; this possibility is being actively pursued.

More speculative and perhaps of deeper theoretical interest, Dehaene puts forward a theory of consciousness as a global neuronal workspace, another variation on the global workspace theory of Bernard Baars (an idea which keeps being picked up by others, which must suggest that it has something going for it). Dehaene offers the view that a particular function of the workspace is to allow inputs to hang around for an extended period instead of dissipating. Among other benefits, this allows the construction of chains of processing operations, something Dehaene likens to a Turing machine, though it sounds a little messier than that to me. Further ingenious experiments have lent support to this idea; the researchers were able to contrast subjects’ chaining ability when information was supplied subliminally or consciously (this may sound odd, but subjects can perform at better-than-chance levels even with subliminal stimuli).

Dehaene says that he is dealing only with one variety of consciousness – in the main it’s awareness, which in some respects is the basement level compared to the more high-flown self-reflective versions. But in passing the talk does clarify a question which has sometimes troubled me in the past about global workspace theories – why should they involve consciousness at all? It seems easy to understand that the brain might benefit from a kind of clearing house where information from different sources is shared – but couldn’t that happen, as it were, in the dark? What does the magic ingredient of consciousness add to the process?

Well, being in the global workspace means being accessible to several different systems (no intention here to commit to any particular view about modularity); and one of those systems is the vocal reporting system. So as a natural consequence of being in the workspace, inputs become things we can vocally report, things we can talk about. Things we can talk about are surely objects of consciousness in some quite high-level sense.

Dehaene does not go down this path, but I wondered how far we could take it; is there a plausible explanation of phenomenal consciousness in terms of a global workspace? If we followed the same pattern of argument we used above, we would be looking to say that conscious experiences acquired qualia because being in the workspace made them available to the qualic system, whatever that might be. I think some people, those who tend to want to reduce qualia to flags or badges that give inputs a special weight, might find this kind of perspective congenial, but it doesn’t appeal all that much to me. I would prefer an argument that related the appearance of qualia to a sensory input’s being available to a global collection of sensory and other systems; something to do with resonances across modalities; but I happily confess I have no clear idea of how or exactly why that would work either.

Picture: Autoscopy. Among a number of interesting features, The Ego Tunnel includes a substantial account of out-of-body experiences (OBEs) and similar phenomena. Experiments where the subjects are tricked into mistaking a plastic dummy for their real hand (all done with mirrors), or into feeling themselves to be situated somewhere behind their own head (you need a camera for this) show that our perception of our own body and our own location are generated within our brain and are susceptible to error and distortion; and according to Metzinger this shows that they are really no more than illusions (Is that right, by the way – or are they only illusions when they’re wrong or misleading? The fact that a camera can be made to generate false or misleading pictures doesn’t mean that all photographs are delusions, does it?).

There are many interesting details in this account, quite apart from its value as part of the overall argument.  Metzinger briefly touches on four varieties of autoscopic (self-seeing) phenomena, all of which can be related to distinct areas of the brain:  autoscopic hallucination, where the subject sees an image of themselves; the feeling of a presence, where the subject has the strong sense of someone there without seeing anyone; the particularly disturbing heautoscopy, where the subject sees another self and switches back and forth into and out of it, unsure which is ‘the real me’; and the better-known OBE. OBEs arise in various ways: often detachment from the body is sudden, but in other cases the second self may lift out gradually from the feet, or may exit the corporeal body via the top of the head.  Metzinger tells us that he himself has experienced OBEs and made many efforts to have more (going so far as to persuade his anaesthetist to use ketamine on him in advance of an operation, with no result – I wonder whether the anaesthetist actually kept his word) ; speaking of lucid dreams, another personal interest, he tells the story of having one in which he dreamed an OBE. That seems an interesting bit of evidence: if you can dream a credible OBE, mightn’t they all be dreams? This seems to undercut the apparently strong sense of reality which typically accompanies them.

Interestingly, Metzinger reports that a conversation with Susan Blackmore helped him understand his own experiences.  Blackmore is of course another emphatic denier of the reality of the self. I don’t in any way mean to offer an ad hominem argument here, but it is striking that these two people both seem to have had a particular interest in ’spooky’ dualistic phenomena which their rational scientific minds ultimately rejected, leading on to an especially robust rejection of the self. Perhaps people who lean towards dualism in their early years develop a particularly strong conception of the self, so that when they adopt monist materialism they reject the self altogether instead of seeking to redefine and accommodate it, as many of us would be inclined to do?

On that basis, you would expect Metzinger to be the hardest of hard determinists; his ideas seem to lean in that direction, but not decisively. He suggests that certain brain processes involved in preparing actions are brought up into the Ego Tunnel and hence seem to belong to us. They seem to be our own thoughts, our own goals and because the earlier stages remain outside the Tunnel, they seem to have come from nowhere, to be our own spontaneous creations. There are really no such things as goals in the world, any more than colours, but the delusion that they do exist is useful to us; the idea of being responsible for our own actions enables a kind of moral competition which is ultimately to our advantage (I’m not quite sure exactly how this  works). But in this case Metzinger pulls his punch: perhaps this is not the full story, he says, and describes compatibilism as the most beautiful position.

Metzinger pours scorn on the idea that we must have freedom of the will because we feel our actions to be free, yet he does give an important place to the phenomenology of the issue, pointing out that it is more complex than might appear. The more you look at them, he suggests, the more evasive conscious intentions become.  How curious it is then, that Metzinger, whose attention to phenomenology is outstandingly meticulous, should seem so sure that we have at all times a robust (albeit delusional) sense of our selves. I don’t find it so at all, and of course on this no less a person than David Hume is with me; with characteristically gentle but devastating scepticism, he famously remarked “For my part, when I enter most intimately into what I call myself, I always stumble on some particular perception or other, of heat or cold, light or shade, love or hatred, pain or pleasure. I never can catch myself at any time without a perception, and never can observe any thing but the perception.”

Metzinger concludes by considering a range of moral and social issues which he thinks we need to address as our understanding of the mind improves. In his view, for example, we ought not to try to generate artificial consciousness. As a conscious entity, the AI would be capable of suffering, and in Metzinger’s view the chances are its existence would be more painful than pleasant. One reason for thinking so is the constrained and curtailed existence it could expect; another is that we only have our own minds to go on and would be likely to produce inferior, messed-up versions of it. But more alarming, Metzinger argues that human life itself involves an overall preponderance of pain over pleasure; he invokes Schopenhauer and Buddha. With characteristic thoroughness, he concedes that pleasure and pain may not be all that life is about; otherr achievements can justify a life of discomfort. But even so, the chances for an artificial consciousness, he feels are poor.

This is surely too bleak. I see no convincing reason to think that pain outweighs pleasure in general (certainly the Buddhist case, based on the perverse assumption that change is always painful, seems a weak point in that otherwise logical religion), and I see some reasons to think that a conscious robot would be less vulnerable to bad experiences than we are. It’s millions of years of evolution which have ingrained in us a fear of death and the motivating experience of pain:  the artificial consciousness need have none of that, but would surely be most likely to face its experiences with superhuman equanimity.

Of course caution is justified, but Metzinger in effect wants us to wait until we’ve sorted out the meaning of life before we get on with living it.

His attempt to raise this and other issues is commendable though; he’s right that the implications of recent progress have not received enough intelligent attention. Unfortunately I think the chances of some of these issues being addressed with philosophic rationality are slim. Another topic Metzinger raises, for example, is the question of what kinds of altered or enhanced mental states, from among the greatly expanded repertoire we are likely to have available in the near future, we ought to allow or facilitate; not much chance that his mild suggestions on that will have much impact.

There’s a vein of pessimism in his views on another topic. Metzinger fears that the progress of science, before the deeper issues have been sorted out, could inspire an unduly cynical, stripped-down view of human nature; a ‘vulgar materialism’, he calls it. Uninformed members of the public falling prey to this crude point of view might be tempted to think:

“The cat is out of the bag. We are gene-copying bio-robots, living out here on a lonely planet in a cold and empty physical universe. We have brains but no immortal souls and after seventy years or so the curtain drops. There will never be an afterlife, or any kind of reward or punishment for anyone… I get the message.”

Gosh: do we know anyone vulgar and unsophisticated enough to think like that?

Picture: Metzinger. The denial of one’s own existence might seem a desperate philosophical strategy, but denying the reality of the self is a line which a number of people have taken, and Thomas Metzinger is prominent among them. The thesis of his massive 2003 work is summed up in the title: Being No One: The Self-Model Theory of Subjectivity. In that book, Metzinger made a commendable effort to balance philosophy and science; but the sheer size of the resulting text may have deterred some readers – I confess to being somewhat daunted myself. Now he has come back with a slimmer volume The Ego Tunnel which is aimed at a wider public and raises wider issues which Metzinger suggests need public attention.

Metzinger’s theory – the Self-model Theory of Subjectivity or SMT – suggests that subjective experience is really a kind of trick the brain plays on itself. Our brain sets up a model of the world (actually based on fairly limited data) to which it then adds a model of us, ourselves. The coherence of the model and the fact that the processes supporting it are transparent – ie invisible to us – yield the vivid impression of a self in direct contact with reality, and that’s where subjectivity arises; although in fact the whole thing is simply an illusion.

Metzinger’s view of qualia is characteristically complex. He has a good argument against the existence of what he calls canonical qualia, qualia conceived as subjective universals. He points out that our ability to discriminate is far greater than our ability to recognise. So, if we are presented with examples of green 64 and green 66, we can readily tell the difference: but if at a later stage we are presented with one of the examples, we have no hope of telling which it is. So there is no single thing that consistently goes along with the experience of green 64.
Concluding that at any rate we need to distinguish between ‘qualia’ available to memory and qualia available to the faculty of recognition, Metzinger goes on to distinguish a series of possible conceptions of qualia, ending with ‘Metzinger qualia’ which are available attentionally but not cognitively. These are slippery customers for obvious reasons, impossible to report and broadly ineffable – but then that’s how qualia are generally assumed to be.

Even as a summary, the foregoing is a bare and radically, probably over- simplified view of the theory, however. Metzinger actually presents ten constraints which need to be satisfied for the occurrence of subjective experience: they are:

  • Globality; as in ‘global workspace’; conscious items are always integrated into an overall world-model,
  • Presentationality; present in the now , temporal immediacy,
  • Convolved holism; objects of experience are made up of collections of other objects in a nested hierarchy,
  • Dynamicity; the perceived world flows through constant changes,
  • Perspectivalness; we experience the world from a point of view,
  • Transparency; we cannot see the works – the neural processing which gives rise to our experience is excluded from conscious experience,
  • Offline activation; subjective experience is not confined to the live inputs from our senses, a notable exception being dreams, where a whole non-existent world appears.
  • Representation of intensities; besides distinguishing between qualia (whichever version we’re dealing with) we can distinguish their levels of intensity,
  • Homogeneity; areas of pink, for example, are made up of smaller areas of pink, not red and white at once, and
  • Adaptivity; the features of subjective experience have to be things that could reasonably have appeared in the course of evolution.

You may feel that there’s something a bit odd about this list, especially the appearance of ‘adaptivity’. One does not have to be a creationist to feel uneasy about the idea that that is really an essential feature of conscious experience in any deep sense. In a précis which he prepared for a discussion on the Psyche site – sadly this no longer appears to be available – Metzinger discussed only the first six constraints, which suggests that the last four are at least somewhat dispensable. This is a bit confusing – is homogeneity essential to conscious experience, or was that just a kind of bonus, a description of a property of phenomenal experience which is important but not a defining requirement? I think one issue here may be that Metzinger seems to want to do two jobs at once; he wants to explain subjectivity in a philosophical sense, but he also wants to describe and categorise phenomenal experience in a way which can be related to scientific observation, clarifying the puzzling phenomenology of blindsight and other unusual conditions. These are not incompatible aims, exactly, and there’s no absolute reason why one account couldn’t do both, but there is a tension. A robust philosophical explanation would pare down the account to its essentials, while the description of subjectivity seems to be something we would want to do as fully as possible.

So Metzinger, for example, notes that convolution is one property of phenomenal experience and homogeneity is another. If we’re looking for one kind of explanation, to be told that subjective experience breaks down into things which are different, and also that subjective experience sometimes breaks down into things that are the same, is not very helpful; but as a description of different ways phenomenology can work it’s a legitimate clarification.

In the new book, Metzinger boils the requirement down in terms which make it easier for us to get to grips with the theory in ways that relate to our usual concerns here, by asking what it would take to build an Ego Machine, an artificial conscious subject.

The recipe requires us to begin with an integrated and dynamical world-model . We must then ensure that the information flow is organised so as to assign events to a present moment, a Now; and that the images presented by the model are transparent: again, we must not be able to see the works. If these constraints are satisfied, a world appears. The last step is to add to the model an equally transparent representation of an experiencing self; then the artificial entity will finally appear to itself to be someone, and to be there. There will seem to be someone in the Ego Tunnel, Metzinger’s equivalent of Plato’s cave.

This seems to me to make quite a lot of sense, but the reader’s verdict is likely to be determined by whether it ‘feels right’ – does it finally dispel that sense of mystery? It must be taken into account that Metzinger’s theory itself provides strong reasons why his conclusions are going to seem intuitively unacceptable, so a kind of mental ‘aiming off’ may be required. I think we have to ask ourselves two questions. The first is: can we imagine that all of Metzinger’s requirements could be met in the absence of ‘inner experience’: does it seem that we could have his kind of world-model running with a representation of a self in it and yet still be, as it were, only a Metzinger zombie?

If we’re happy that this is not a possibility, there still remains the question of whether Metzinger has smuggled his consciousness into the tunnel. Do any of his constraints covertly imply that consciousness is already present?

I think some of them could – presentationality? Perspectivalness? – but carefully read, I don’t think they need do. Whether some of their intuitive appeal derives from the reader having unconsciously done some illicit importation is, I suppose, a matter for the reader. Some of the constraints do look a bit strange in other ways. Let’s consider that requirement of transparency, for example. In the ego tunnel, data about the outside world must be globally available, but there must be no information about how this data got to the tunnel, since that would dispel the illusion that what we are dealing with is the real world itself. But why would there be? A TV screen does not automatically contain an account of how the TV itself or the local transmitter works. Metzinger has an odd perspective here which seems to start with the idea that the mind naturally knows everything and has to be carefully shielded. In The Ego Tunnel he talks about our sensory apparatus filtering out most of the potential information about the world, limiting us to a meagre trickle, as though our natural state was a kind of omniscience which is only reined in by the limitations of our eyes. Isn’t it the other way round?

Look at it the other way: suppose we were consciously aware of the fact that light rays were being processed by our retinas, would that destroy our sense of ourselves as existing in the world? Actually, I doubt it. The fact that we can see the edges of the screen does not inhibit us much from getting immersed in a film: for that matter the very visible appearance of the book does not prevent our getting immersed in a novel. Similarly, the fact that I can see a ‘floater’ in my right eye at the moment doesn’t cause reality to recede from me although it is a tell-tale sign of the visual processes that intervene between my brain and the world.

But there’s more in the Ego Tunnel…

(My apologies if things have been a bit slow here – I rashly committed to Nanowrimo this year so a lot of my energy is going to turning out a 50,000 word novel text this month. You can read the results here.  – Peter)

xeex470503 bumble bee physics personal invetment templates on excel blanchard river watershed map disturbed get psycho wilton cake supplies discounted business roundtable cost effectiveness committee salina kansas scholls biometrics iraq creampie cumshots ancient trojan wars off white gallery avalon sammy kaye downloads sabiki rigs for yellow perch plumbing for pedicure chairs baptism wheel craft richard elton miramar terrace symphonic dvd vhs combo dubbing errors krumkakke iron sandy duncan pics photos by gilliland maury city tennessee leland frederick cooley jamie foxx on hbo lawler real estate llc farmingdale walmart balloon valvotomy for aortic stenosis network setup between computers ephesus culture wooden pizza paddles refugees poland will coates mk 4 no1 us santa clara home page amaroo school canberra what will happen 2008 x press 2 afrika bambaataa fire school counselor jobs in ohio how to override endevor compile jcl mph to fps phones supporting android incorporate and rock band value of a wurlitzer organ belton senior living american idol jason castro cervical nerve root contusion researcher content writer wanted key and certificate life cycle domestic worker contract inter faces sarl what is ecosystem diversity repair macintosh madison wisconsin kasey nordeen sc chimney sweep canton ohio johann gottfried flugel said ceu miami uscg 1000 sportbike shootout blackberry peal sold 2008 freddy mack ephedrine hcl pure dish installers cleveland georgia kids having kids robin hood foundation stc craft melanie falick books blm minimum royalty provision description pagosa springs co fishing shipley manor wilmington de momo hotels and resorts dvd authoring with surround sound self imagery progressive church and sacramento generational dynamics jon savitch financial planner b beckwith relationship between training and job satisfaction otters animals mother of pearl pillow open fire pit safety lawsuit movies scifi employees forced to buy uniforms shakira feet 94 seadoo review honda nav system 2007 pdr like drug information free college invasion free video im logger grilled halibut mango salsa spiral wrap tool symptoms gallbladder attacks valinge aluminum patents cell mitosis and cell division information ansi ieee device numbers feeling results of infrared sauna iglesia primitiva perennial ornamental grasses field erosion barefoot merlot montana trailors last minute orlando deals creamery robert graham style edun trinity meadow view soccer ea simonia morning domino molasses free templates for tenancy agreements kal equip vacuum pumps guidlines for association newsletters marijuana urinalysis detection times team building for parents poser millenium woman bittorrent diet self confidence self esteem 2 cloverdale way columbus nj 08022 people who study glaciers firestone grill menu san luis obispo blender bottle shaker organic blue agave sweetener dell awaiting arrival of associated peripherals mls grantsberg wisc san diego hire a felon sandstone amphitheater susan niven locust vally a sand county almanac land ethic maps brasil argebtina free pamphlet christian jew foundation pool table instalation irvin mischner libros pdf hytrol conveyors exporing four empires of mesopotamia franz viegener temple faucets stretcher sigma d14 camera virginia penal system espn bcs games online informatica publish webdav skeletal with other systems perfection company verizon text messages argentina president claim falkland islands how animals are alike genre browser horror turned wooden plates fullerton movibe hh robyn hillis springfield mo mark crew sentenced to death braves apparel leaking shower drain seal baseline nutritionals talented video loyola university maywood truck chassis repair equipment old bridge cellars merced department of agriculture partch replacement for parchment paper fortran histogram windermere british columbia empress hotel saigon mf doom dunks documentary norse san mateo fair grounds how to mixed media collage blogs about vinyl records and collecting lotro and defending the bridge pepper jelly using red pepper flakes jay leno automobiles transport of deceased to mexico chronic pain theoretical framework super rsi strubs pickles rs auto clicker 2007 passover dates aquatic peat water gardens herbal treatment for altitude sickness van hiele implication of teaching mathematics foods that include arginine internal capsule shamans knowledge omron e3x galena castle home fdny firefighter engine fire fireman verkleedkleren brussels how earthquake sizes are measured sagem my200c unlock mode s decode psp dj rey de reyes jalisco sk racing intercooler installation teal shag rug euphoria morning mp3 scott orchard lanzarote live music nero image file plumeria breeding hybrid the adjective of habit poems to an unborn nephew steel drum radio giga manuscript pallet turret song graduation ambition wood anemone subaru impreza 1.5 lake auditor motorola i265 set alarm clock gene simmons family jewels show age of remington woodsmaster 740 refinish tile surround prisoner locate us webmail earthlink login tecra 730xcdt external mouse not enabled ptb withdrawl rbc banking definitions mandy moore extraordinary video job satisfaction correlation to education level red toyota trd pictures promised bathing belles le bar sans nom its about thyme austin explorer closes opening jpeg balsamic strawberries def jam icon pics robert redford movies 1998 mei linn chuong indus vally civilization wording casual wedding in stefanie heitmann hacking pioneer inno house of names wick cape cod chefs not expresing your emotions tilt beaming entire contact list kanton bern der grosse rat eric scott allen partners lane county oregon food handlers ramones pet cemetary log house on bull shoals lake tmobile service in israel enlightenment literature barenaked ladies lyrics ron workman comic warhammer guild scrolls william foster northumberland va lange 1815 succulent ground clover plants laminate suture cart texas a m cadet corp waco joyner sales royal victoria hospital barrie on ca vivacious vixen new orleans lago di como tarrytown rare precious moments love quotes by yoda broom handel mauser forbes list and grocery stores tasty crisa gallery cyprus developers contractors collect british postal stationery david stoner clearwater florida jerseys spooner wi cochranville pa recycling ron george jewelers bo bo bibs starbucks workplace locum tenens modifier neuron asians tits sweet memories menu keller the keswick theater pac man world cheats pl gaines photography beckie hazzard kyle mcdermott in boston cervical sample case study on delegation do geese see god clive hornsby delta kitchen faucets with water filters woman bicep 1967 ford fairlane parts comprar instrumentos musica denny middle school healy brothers on teh rocks santorini alderon astrological meaning sudbury ontario furniture storage massey 35 tractor delino beach near st augustine myspace navy nuke background rented property in leeds dossin great lakes museum bayarea rappers layouts medical anthropology canciones maltrato mujer sheriff statue gift magic illusions guillotine electrolux stick vacuum biography chef alain ducasse direct x dowload open season sounds porcupine order of franciscans companions hampton inn lewisville address certa pro painters colorado springs hafler pro2400 amplifiers amag occasions tapa university macau slumber parties murrieta about article 21 of indian constitution premier hgh compression box polygon triangle pictures tweak addon for adaware se century 21 dwyer ri fantastic fiction uk microtel reservations jacqueline childress windows live club cheats smith wesson guns lipitor and lichen planus care of crocodile shoes fluorescent bulb candelabra dimmable how to strenghten cell signals homemade thick cookie frosting clifford big red dog video streams crazy frog dies ed hargis chief picture photo miyabi restaurant summerville sc boy stud gasoline prices in tennessee congress performance rating kacey cahill blue bell pa attorney shanna jack kompan custom mystick cue yoke kee powerpress ikon models bill gates email beta scam pyrotechnic simulated aerial bombing runs funny clean pictures stags estate ae effect of cayenne on warfarin harney sons irish breakfast tea the sleepover saturday night live transscript baby crib sham gis metadata research organics cerebral palsy feed refusal alpha grasslands preserves pohatcong nj medizinische hochschule hannover small sliding windows shootable cannons marcella gallagher new el pollo loco tv ads

bernstein police

architecture research office in telluride colorado

california chateau housing association palmdale ca

disney passports

dvd decrypter failed to reserve track

how force spectroscopy works

movies in waxahachie

eyelashes los gatos

2008 honda odyssey rockingham nh

stanthonymedicalcenter.com

canadas confederation

rvcountry.com

batman spoof interrogation youtube

fms s-2a program

calculate ideal body wieght

chidl developmental chart

bell engineering intelligent weighing technology

sholarship requirements at ub baguio

clarence snowblower impeller modification

custom made veils

pycomall.com

ayers ma

baby boomer librarian gorman michael

10mm compression 5 8 hose

painting stripes on textured walls

cool facts on neptune

abyssinian kittens

conference center santee south carolina

free gimp tutorial

327 iac 8 rule 10

cyl head milling advise

images of lord shiva

7401 mill

1875 orford kent ontario

4 seasons global garden gazebo

gaymilitarydating.com

american hotrod salt flats

devonshire police division

woodworkingplans.ca

grupo consultor financiero puerto ricoi

crypt lake hike waterton park

all natural gourmet dog bakery

clorox commercial solutions

dark english fable

gmscontracts.com

aj calloway

raja arshad

$400 segway i2

accolade systems pvt ltd

anberlin paperthin hymn myspace code

bravo silva

amy nelson laugh

btrescue.org

alabaster greek statue

bobbi-billard-nude.com

journey fellowship selma tx

alhambra cemetery contra costa

incall andover

hirem linder barbera

bessborough school moncton nb

hannity bio

avrage waist size

1890 grimms fairy tale book

burrito gallery jacksonville florida

1524 ocean st simons

1988 toyota corolla sr5

looney toon printable pictures

bestcontacttables.com

any way he wants

abb 50 51 relay

airfare san fransisco to bwi

cheerleadershardcore.com

acyclovir structure

2004 fur prices

aristokraft jasper in

8 basic colors

jacksons of henley in arden

bulk mulch illinois

2006 abate iowa freedom

everett farris seneca ac

foreigner and credit card and korea

1000 thread count symphony sheets

apos xml

imformation about apple computer network

anna molly incubus free download

oily blackmouth

2006 partnership tax return forms

3hosts.info

25-27 inch motion spider hallowen

elements of foriegn policy

blood thinner complications

joni abraham rice

monitor-drivers.com

bookmark arturo e de la torre

carthage college 2000 jess basketball

guerilla banners

highest legal intrest rate in us

buddy holly maybe baby lyrics

automation direct plc

cosby sho

albert ainstain

40th irthday candy wrappers

car rental in martinsburg west virginia

defunct company liens

wbli.com

carousel and asbury park

isdb transmitters

all indian shows desi

cal pico casing spacers

allen jensen poem writter

druggists up on platforms

mary-shelley-frankenstein.com

font free cultura

dvd replication services la

garnishment lawyers

neilson of antioch

famouse hispanic people

infomercial consumer bite back

journal pastoral care

victorias secret 2006

1947-1973 legion lodge 134 members

auditory delusions

cozzy coverings in ottawa

chrono crusade fiore pics

etna and versuvius

goodies gifts and graphics

ad d paladin mount quests

hydro thunder cheat codes ps1

bail bonds annapolis md

benefits of horses on pasture

1000base-sx short wavelength gbic

devinney laura allegra prescription huntingdon

fillet mignon steak recipe

bogs sour bubble

10 worst foods iron magazine

kimberley dahme born 1968

bowling surry hills nsw

2008 dodge avenger stalls

1990 toyota 4runner side mirrors

beef stew meat recipes

southjerseygas.com

girl mastrabation storys

60 in duffel bag

barbie dollhouse building supplies

gruppa kino gruppa krovi

dinosaur fossils for kids

downhill racer

consolidate debts

alabaster wall sconce

24-120 nikon reviews

edward ennis pa

blood sweat and tears web site

are vanilla extract molecules polar

racecar

biggest steel exporters

aladdin 17774 glow plug

7 16-20 lug nuts gorilla

209 7255 mn

futurama chat

forbes richest people us

cast decorator tiles

cervical spine and nerves

blackberry message alerts

aster plant tiger paw

1992 kia pride

2002 msjc brick ties

autographed first edition of lonesome dove

nad 5200 cd player

attractions at disneyland

breezy beach

cng auto gas utah conversion

alyssa milano vampire

abstract thought

alkaline trio albums

actechbooks.com

adding negative exponents

are halogen lights economical

christain dancing

bishop arst sushi

advice debt financial reduction

2008 hyundai entourage test

don snider roofing

vlacs.org

amendoin hardwood flooring

egypt remains an outlier

jacinto medical group

alex grass governor

cancun sightseeing

arb thatcher of kidspeace

archuleta jogging

rosso granite porcelain tile

antiques amp collectibles webring

avon aftershave collectible glass bottles

empathetic gut spasm

accreditation agency in barbados

janelle deady

janelle baur

motleycrueconcerttickets.com

4th of july picnic basket

home builders in kokomo in

affordable castle for sale

bayes probability calculations formulas

bang buddies

yanmar 155d

craigslist boats yanmar diesel

causes of stiff neck

leadership managerial grid

american eagle carrying american flag

audio strobe gogles

gilded age cottages long island

antique brass formed hinge

2nd grade resources

delta lavatory faucet

alli diet medicene

how to loc

ancient chinese writings

01 chevy malibu ground affects

avalon middle school milton fl photos

bill brit

a reed hayes connecticut

1.0 c# visual studio

nrv 5 pfizer taken by mistake

vegasstripcasino.com

buddhist catholic dialog

erikson college vancouver

piratenews.org

guangzhou italian leather

chocolate coated ice cream cones

goodpreacher.com

advice debt expert

188 northern pacific locomotive

teasing mares

blog thirtytwo advanced snowboard footwear

bishop kelley high school

commandes unix wikip dia

accomadations near splash country

bid on russia airline fligts

california populations

quickrxmed.com

furnace ductwork parts plenum

18 girl dolls wholesale lots

bilko doberman

comic book retailers gastonia nc

clogged fresh air vent vehicle leaking

how to fix a mikuni carby

beck everybody gotta learn sometime

callaway crx golf balls

086 computer

catalytic coaching personal development plan

eyecare vision center website austin tx

email july bringing buns wini message

giuliano de medici

blu-ray disc player gadgets shows slide

1999 honda accord instrument panel light

histoire dogpile

advanced visualization procedures in histology

bryan mick

harbor lights wyndham

hair removal creams depilatory coarse hair

publix calypso crunch bites

aaron brandon windsor

christian communion large number

1997 triumph sprint

constable bill phelan rcmp

1971 john deere 450b

hqualityporn.com

12-string acoustic electric guitar freddy

franchising centro benessere

kosovo egyptian

$200.00 electric wheelchair scooter lifts

collected poems of john donne

c g burnet tx

arbitration morocco

duval container company

henry varmint express reviews

31 x 72 privacy liner

goode company seafood in houston tx

buy garmin 650

global prosperity group

faithful public servants

.300 remington ultra magnum

rachel ray marrige

ntp 4.2.4 windows

amanda knox astrology

aqua genetic urticaria

baroque art in italy

casio exilim ex-z9 reviews

land cheap midwestern us

9 ll terrorists saudi

2007 college baseball standings

garton-tractor.com

charles pitts of california

pascali pepe

crystal vhf band reject filter

3d plant design

1850 pike co al census

1987 nba finals starting lineup

attend for add

capture recapture detection probability variance

advanced search toolbar

nylonporn.tv

interview with selena gomez

c s lewis miracles notes

67 camaro satin

botanical body