Picture: Google chatbot. Bitbucket I was interested to see this Wired piece recently; specifically the points about how Google picks up contextual clues. I’ve heard before about how Google’s translation facilities basically use the huge database of the web: instead of applying grammatical rules or anything like that, they just find equivalents in parallel texts, or alternatives that people use when searching, and this allows them to do a surprisingly good – not perfect – job of picking up those contextual issues that are the bane of most translation software. At least, that’s my understanding of how it works.  Somehow it hadn’t quite occurred to me before, but a similar approach lends itself to the construction of a pretty good kind of chatbot - one that could finally pass the Turing Test unambiguously.

Blandula Ah, the oft-promised passing of the Turing Test. Wake me up when it happens – we’ve been round this course so many times in the past.

Bitbucket Strangely enough, this does remind me of one of the things we used to argue about a lot in the past.  You’ve always wanted to argue that computers couldn’t match human performance in certain respects in principle. As a last resort, I tried to get you to admit that in principle we could get a computer to hold a conversation with human-level responses just by the brutest of brute force solutions.  You just can a perfect response for every possible sentence. When you get that sentence as input, you send the canned response as output. The longest sentence ever spoken is not infinitely long, and the number of sentences of any finite length is finite; so in principle we can do it.

Blandula I remember: what you could never grasp was that the meaning of a sentence depends on the context, so you can’t devise a perfect response for every sentence without knowing what conversation it was part of.  What would the canned response be to;  ‘What do you mean?’  – to take just one simple example.

Bitbucket What you could never grasp was that in principle we can build in the context, too. Instead of just taking one sentence, we can have a canned response to sets of the last ten sentences if we like - or the last hundred sentences, or whatever it takes. Of course the resources required get absurd, but we’re talking about the principle, so we can assume whatever resources we want.  The point I wanted to make is that by using the contents of the Internet and search enquiries, Google could implement a real-world brute-force solution of broadly this kind.

Blandula I don’t think the Internet actually contains every set of a hundred sentences ever spoken during the history of the Universe.

Bitbucket No, granted; but it’s pretty good, and it’s growing rapidly, and it’s skewed towards the kind of thing that people actually say. I grant you that in practice there will always be unusual contextual clues that the Google chatbot won’t pick up, or will mishandle. But don’t forget that human beings miss the point sometimes, too.  It seems to me a realistic aspiration that the level of errors could fairly quickly be pushed down to human levels based on Internet content.

Blandula It would of course tell us nothing whatever about consciousness or the human mind; it would just be a trick. And a damaging one.  If Google could fake human conversation, many people would ascribe consciousness to it, however unjustifiably. You know that quite poor, unsophisticated chatbots have been treated by naive users as serious conversational partners ever since Eliza, the grandmother of them all. The internet connection makes it worse, because a surprising number of people seem to think that the Internet itself might one day accidentally attain consciousness. A mad idea: so all those people working on AI get nowhere, but some piece of kit which is carefully designed to do something quite different just accidentally hits on the solution? It’s as though Jethro Tull had been working on his machine and concluded it would never be a practical seed-drill; but then realised he had inadvertently built a viable flying machine. Not going to happen. Thing is, believing some machine is a person when it isn’t is not a trivial matter, because you then naturally start to think of people as being no more than machines.  It starts to seem natural to close people down when they cease to be useful, and to work them like slaves while they’re operative. I’m well aware that a trend in this direction is already established, but a successful chatbot would make things much, much, worse.

Bitbucket Well, that’s a nice exposition of the paranoia which lies behind so many of your attitudes. Look, you can talk to automated answering services as it is: nobody gets het up about it, or starts to lose their concept of humanity.

Of course you’re right that a Google chatbot in itself is not conscious. But isn’t it a good step forward?  You know that in the brain there are several areas that deal with speech;  Broca’s area seems to put coherent sentences together while Wernicke’s area provides the right words and sense. People whose Wernicke’s area has been destroyed, but who still have a sound Broca’s area apparently talk fluently and sort of convincingly, but without ever really making sense in terms of the world around them. I would claim that a working Google chatbot is in essence a Broca’s area for a future conscious AI. That’s all I’ll claim, just for the moment.

Picture: global workspace. Global Workspace theories have been popular ever since Bernard Baars put forward the idea back in the eighties; in ‘Applying global workspace theory to the frame problem’*,  Murray Shanahan and Baars suggest that among its other virtues, the global workspace provides a convenient solution to that old bugbear, the frame problem.

What is the frame problem, anyway? Initially, it was a problem that arose when early AI programs were attempting simple tasks like moving blocks around. It became clear that when they  moved a block, they not only had to update their database to correct the position of the block, they had to update every other piece of information to say it had not been changed. This led to unexpected demands on memory and processing. In the AI world, this problem never seemed too overwhelming, but philosophers got hold of it and gave it a new twist. Fodor, and in a memorable exposition, Dennett, suggested that there was a fundamental problem here. Humans had the ability to pick out what was relevant and ignore everything else, but there didn’t seem to be any way of giving computers the same capacity. Dennett’s version featured three robots: the first happily pulled a trolley out of a room to save it from a bomb, without noticing that the bomb was on the trolley, and came too; the second attempted to work out all the implications of pulling the trolley out of the room; but there were so many logical implications that it was stuck working through them when the bomb went off. The third was designed to ignore irrelevant implications, but it was still working on the task of identifying all the many irrelevant implications when again the bomb exploded.

Shanahan and Baars explain this background and rightly point out that the original frame problem arose in systems which used formal logic as their only means of drawing conclusions about things, no longer an approach that many people would expect to succeed. They don’t really believe that the case for the insolubility of the problem has been convincingly made. What exactly is the nature of the problem, they ask: is it combinatorial explosion? Or is it just that the number of propositions the AI has to sort through to find the relevant one is very large (and by the way, aren’t there better ways of finding it than searching every item in order?). Neither of those is really all that frightening; we have techniques to deal with them.

I think Shanahan and Baars, understandably enough, under-rate the task a bit here. The set of sentences we’re asking the AI to sort through is not just very large; it’s infinite. One of the absurd deductions Dennett assigns to his robots is that the number of revolutions the wheels of trolley will perform in being pulled out of the room is less than the number of walls in the room. This is clearly just one member of a set of valid deductions which goes on forever; the number of revolutions is also less than the number of walls plus one; it’s less than the number of walls plus two… It may be obvious that these deductions are uninteresting; but what is the algorithm that tells us so? More fundamentally, the superficial problems are proxies for a deeper concern; that the real world isn’t reducible to a set of propositions at all, that, as Borges put it

“it is clear that there is no classification of the Universe that is not arbitrary and full of conjectures. The reason for this is very simple: we do not know what thing the universe is.”

There’s no encyclopaedia which can contain all possible facts about any situation. You may have good heuristics and terrific search algorithms, but when you’re up against an uncategorisable domain of infinite extent, you’re surely still going to have problems.

However, the solution proposed by Shanahan and Baars is interesting. Instead of the mind having to search through a large set of sentences, it has a global workspace where things are decided and a series of specialised modules which compete to feed in information (there’s an issue here about how radically different inputs from different modules manage to talk to each other: Shanahan and Baars mention a couple of options and then say rather loftily that the details don’t matter for their current purposes. It’s true that in context we don’t need to know exactly what the solution is – but we do need to be left believing that there is one).

Anyway, the idea is that while the global workspace is going about its business each module is looking out for just one thing. When eventually the bomb-is-coming-too module gets stimulated, it begins sending very vigorously and that information gets into the workspace. Instead of having to identify relevant developments, the workspace is automatically fed with them.

That looks good on the face of it; instead of spending time endlessly sorting through propositions, we’ll just be alerted when it’s necessary. Notice, however, that instead of requiring an indefinitely large amount of time, we now need an indefinitely large number of specialised modules. Moreover, if we really cover all the bases, many of those modules are going to be firing off all the time. So when the bomb-is-coming-too module begins to signal frantically, it will be competing with the number-of-rotations-is-less-than-the-number-of-walls module and all the others, and will be drowned out. If we only want to have relevant modules, or only listen to relevant signals, we’re back with the original problem of determining just what is relevant.

Still, let’s not dismiss the whole thing too glibly. It reminded me to some degree of Edelman’s analogy with the immune system, which in a way really does work like that. The immune system cannot know in advance what antibodies it will need to produce, so instead it produces lots of random variations; then when one gets triggered it is quickly reproduced in large numbers. Perhaps we can imagine that if the global workspace were served by modules which were not pre-defined, but arose randomly out of chance neural linkages, it might work something like that. However, the immune system has the advantage of knowing that it has to react against anything foreign, whereas we need relevant responses for relevant stimuli. I don’t think we have the answer yet.

*Thanks to Lloyd for the reference.

Picture: correspondent. Aspro Potamos has drawn my attention to the emerging series of YouTube videos, somewhat polemical in tone, on the War against Neuroscience.   If you prefer something less forensic, you may like this series of podcasts on music and the brain from the Library of Congress.

Huping Hu, whose research has been mentioned here in the past, has started the Journal of Consciousness Exploration and Research: the inaugural issue is available online.

Ayad Gharbawi says:

I have been doing my own studies and research for thirty years now on three concepts: Mind, Vision and Reality. I felt it necessary to create novel notations, just as others, like Boole, did, because of the inadequacies of language as per this subject. I believe that I have come to an entirely new methodology when we seek to understand two fundamental issues when we study Mind, Vision and Reality and that is – both their Structure and the Function.

I have been deeply, fundamentally and existentially affected by Quantum Physics as a human being and in my own pattern of thinking and analyzing problems before me.  I obviously understand full well that my work is unorthodox because I am not just presenting a study in one particular niche in these studies – although, of course, I have studied specific issues – but, I am at this stage in my life, in a position to say that I have come to a general theory that comprises an understanding of Mind, Vision and Reality.

That is why, when I have presented one study/manuscript it is often difficult to make any sense from it, and the reason here is because, those single manuscripts that I have been submitting, do not, in and of themselves alone, explain fully my general theory on Mind, Vision and Reality.  I also know full well, that the history of science, shows far too many times, that when one researcher submits an entirely, unorthodox novel methodology in his thinking, he is quite likely to be rejected by the general established body of scientists and philosophers.

But, I still do try.

A sample is here; he particularly asks for views on two pieces here and here.

Picture: AI resurgent. Where has AI (or perhaps we should talk about AGI) got to now? h+ magazine reports remarkably buoyant optimism in the AI community about the achievement of Artificial General Intelligence (AGI) at a human level, and even beyond. A survey of opinion at a recent conference apparently showed that most believed that AGI would reach and surpass human levels during the current century, with the largest group picking out the 2020s as the most likely decade.  If that doesn’t seem optimistic enough, they thought this would occur without any additional fundingfor the field, and some even suggested that additional money would be a negative, distracting factor.

Of course those who have an interest in AI would tend to paint a rosy picture of its future, but the survey just might be a genuine sign of resurgent enthusiasm, a second wind for the field (’second’ is perhaps understating matters, but still).  At the end of last year, MIT announced a large-scale new project to ‘re-think AI’. This Mind Machine Project involves some eminent names, including none other than Marvin Minsky himself. Unfortunately (following the viewpoint mentioned above) it has $5 million of funding.

The Project is said to involve going back and fixing some things that got stalled during the earlier history of AI, which seems a bit of an odd way of describing it, as though research programmes that didn’t succeed had to go back and relive their earlier phases. I hope it doesn’t mean that old hobby-horses are to be brought out and dusted off for one more ride.

The actual details don’t suggest anything like that. There are really four separate projects:

  • Mind: Develop a software model capable of understanding human social contexts- the signpost that establish these contexts, and the behaviors and conventions associated with them.
    Research areas: hierarchical and reflective common sense
    Lead researchers: Marvin Minsky, Patrick Winston
  • Body: Explore candidate physical systems as substrate for embodied intelligence
    Research areas: reconfigurable asynchronous logic automata, propagators
    Lead researchers: Neil Gershenfeld, Ben Recht, Gerry Sussman
  • Memory: Further the study of data storage and knowledge representation in the brain; generalize the concept of memory for applicability outside embodied local actor context
    Research areas: common sense
    Lead researcher: Henry Lieberman
  • Brain and Intent: Study the embodiment of intent in neural systems. It incorporates wet laboratory and clinical components, as well as a mathematical modeling and representation component. Develop functional brain and neuron interfacing abilities. Use intent-based models to facilitate representation and exchange of information.
    Research areas: wet computer, brain language, brain interfaces
    Lead researchers: Newton Howard, Sebastian Seung, Ed Boyden

This all looks very interesting.  The theory of reconfigurable asynchronous logic automata (RALA) represents a new approach to computation which instead of concealing the underlying physical operations behind high-level abstraction, makes the physical causality apparent: instead of physical units being represented in computer programs only as abstract symbols, RALA is based on a lattice of cells that asynchronously pass state tokens corresponding to physical resources.  I’m not sure I really understand the implications of this – I’m accustomed to thinking that computation is computation whether done by electrons or fingers; but on the face of it there’s an interesting comparison with what some have said about consciousness requiring embodiment.

I imagine the work on Brain and Intent is to draw on earlier research into intention awareness. This seems to have been studied most extensively in a military context, but it bears on philosophical intentionality and theory of mind; in principle it seems to relate to some genuinely central and difficult issues.  Reading brief details I get the sense of something which might be another blind alley, but is at least another alley.

Both of these projects seem rather new to me, not at all a matter of revisiting old problems from the history of AI, except in the loosest of senses.

In recent times within AI I think there has been a tendency to back off a bit from the issue of consciousness, and spend time instead on lesser but more achievable targets. Although the Mind Machine Project could be seen as superficially conforming with this trend, it seems evident to me that the researchers see their projects as heading towards full human cognition with all that that implies (perhaps robots that run off with your wife?)

Meanwhile in another part of the forest Paul Almond is setting out a pattern-based approach to AI.  He’s only one man, compared with the might of MIT – but he does have the advantage of not having $5 million to delay his research…

Picture: Socrates looking at himself. Introspection, the direct examination of the contents of our own minds, seems itself to be in many minds at the moment.  The latest issue of the Journal of Consciousness Studies was devoted to papers on introspection, marking the tenth anniversary of the publication of The View from Within, by Francisco Varela and Jonathan Shear (which was itself a special edition of the JCS); and now Eric Schwitzgebel has produced a new entry for the Stanford Encyclopedia of Philosophy.

The two accounts are of course quite different in some respects. The encyclopaedia entry is a careful, scholarly account, neutral and comprehensive; the JCS issue is openly a rallying-cry in support of a programme flowing from Varela’s work.  This, it seems, called for an end to the ban on examination of lived experience;  the JCS gives the impression that it was something of a milestone, though Schwitzgebel’s piece does not mention it (he does cite an earlier paper by Varela, once again in the JCS).

What’s all this about a ban? Well, back in the nineteenth century, psychologists had no fears about using introspective evidence; it was thought that a proper scientific effort would lead to an objectively verifiable kind of phenomenology. We should be able to classify the elements of mental experience and clarify how they worked together, just by examining what went on in our own heads. A great deal of work was done on all this (It was a great disappointments for me to discover, on first opening Brentano’s Psychology from an Empirical Standpoint, that it consisted almost entirely of this kind of thing, and that the only passage about intentional inexistence, the interesting issue, was the couple of paragraphs which I had already read as quotes in several other books.).  There was a gradual refinement of the methods involved, leading on to the great heyday of introspectionism, with Wundt and Titchener in the lead. Unfortunately, it became clear that the rival schools of introspectionism had begun to come up with results which in some respects were radically different and incompatible, and since our own introspections are by their nature private and unverifiable, all they could really do by way of settling the issues was to shout at each other.

This embarrassing impasse led to a reaction away from introspection and to the rise of behaviourism, which not only denied the usefulness of examining our inner experience, but actually went to the extreme of denying that there was any such thing as inner experience.  Behaviourism in its turn fell out of favour, but according to Varela there remained an instinctive distrust of introspection which continued to put people off it as an avenue of research. This is the ‘ban’ he wanted to see overturned.

Was there, is there, really a ban? Not exactly.  Apart from the most dogmatic of the behaviourists, no-one has ever tried to exclude introspection altogether. In recent times, introspective evidence has been widely accepted – the problem of qualia, thought by some to be the problem of consciousness, depends entirely on introspection. I think the real problem arises when we adopt special methods. In order to obtain consistent results, the old introspectionists thought extensive training was necessary. It wasn’t enough to sit and think for a bit; you had to have mastered certain skills of discrimination and perception. The methodological dangers involved in teaching your researchers what kind of thing they could legitimately look for are clear.

Unfortunately, it seems to be very much this kind of programme which the JCS authors would like to resurrect – or rather, have resurrected, and wish to gain acceptance and support for.  Once again we are going to need to learn how to introspect properly before our observations will be acceptable. What makes it worse for me is that the proposal seems to be tied up with NLP – Neuro-linguistic Programming.  I don’t know a great deal about NLP: it seems to be a protean doctrine which shares with the Holy Roman Empire the property of not really being any of the three things in its name – but for me it does nothing to render another trip down this particular blind alley more attractive.

Blandula I don’t know about that, but aren’t they right to emphasise the potential value of introspection? Isn’t it the case that introspection is our only source of infallible information? Most of the things we perceive are subject to error and delusion, but we can’t, for example, be wrong about the fact that we are feeling pain, can we? That seems interesting to me. Our impressions of the outside world come to us through a chain of cause and effect, and at any stage errors or misinterpretations can creep in; but because introspection is direct, there’s no space for error to occur. You could well say it’s our only source of certain knowledge – isn’t that worth pursuing a little more systematically?

Bitbucket Infallible? That is the exact reverse of the truth: in fact all introspections are false. Think about it. Introspection can only address the contents of consciousness, right? You can’t introspect the unconscious mental processes that keep you balanced, or regulate your heartbeat. But all of the contents of consciousness have intentionality – they’re all about things, yes? So to have direct experience of mental content is to be thinking about something else – not about the mental state itself, but about the thing it’s about! Now when we attempt to think directly about our own mental states, it follows that we’re not experiencing them in themselves – we’re experiencing a different mental state which is about them. In short, we’re necessarily imagining our mental states. Far from having direct contact, we are inevitably thinking about something we’ve just made up.

Phi

Picture: Phi. I was wondering recently what we could do with all the new computing power which is becoming available.  One answer might be calculating phi, effectively a measure of consciousness, which was very kindly drawn to my attention by Christof Koch. Phi is actually a time- and state-dependent measure of integrated information developed by Giulio Tononi in support of the Integrated Information Theory (IIT) of consciousness which he and Koch have championed.  Some readable expositions of the theory are here and here with the manifesto here and a formal paper presenting phi here. Koch says the theory is the most exciting conceptual development he’s seen in “the inchoate science of consciousness”, and I can certainly see why.

The basic premise of the theory is simply that consciousness is constituted by integrated information. It stems from the phenomenological observations that there are vast numbers of possible conscious states, and that each of them appears to unify or integrate a very large number of items of information. What really lifts the theory above the level of most others in this area is the detailed mathematical under-pinning, which means phi is not a vague concept but a clear and possibly even a practically useful indicator.

One implication of the theory is that consciousness lies on a continuum: rather than being an on-or-off matter, it comes in degrees. The idea that lower levels of consciousness may occur when we are half-awake, or in dogs or other animals, is plausible and appealing. Perhaps a little less intuitive is the implication that there must be in theory be higher states of consciousness than any existing human being could ever have attained. I don’t think this means states of greater intelligence or enlightenment, necessarily; it’s more  a matter of being more awake than awake, an idea which (naturally enough, I suppose) is difficult to get one’s head around, but has a tantalising appeal.

Equally, the theory implies that some minimal level of consciousness goes a long way down to systems with only a small quantity of integrated information. As Koch points out, this looks like a variety of panpsychism or panexperientialism, though I think the most natural interpretation is that real consciousness probably does not extend all that far beyond observably animate entities.

One congenial aspect of the theory for me is that it puts causal relations at the centre of things: while a system with complex causal interactions may generate a high value of phi, a ‘replay’ of its surface dynamics would not. This seems to capture in a clearer form the hand-waving intuitive point I was making recently in discussion of Mark Muhlestein’s ideas.  Unfortunately calculation of Phi for the human brain remains beyond reach at the moment due to the unmanageable levels of complexity involved;  this is disappointing, but in a way it’s only what you would expect. Nevertheless, there is, unusually in this filed, some hope of empirical corroboration.

I think I’m convinced that phi measures something interesting and highly relevant to consciousness; perhaps it remains to be finally established that what it measures is consciousness itself, rather than some closely associated phenomenon, some necessary but not sufficient condition. Your view about this, pending further evidence, may be determined by how far you think phenomenal experience can be identified with information. Is consciousness in the end what information – integrated information – just feels like from the inside? Could this be the final answer to the insoluble question of qualia? The idea doesn’t strike me with the ‘aha!’ feeling of the blinding insight, but (and this is pretty good going in this field) it doesn’t seem obviously wrong either.  It seems the right kind of answer, the kind that could be correct.

Could it?

Picture: chess with a machine. Kenneth Rogoff is putting his money on AI to be the new source of economic growth, and he seems to think the Turing Test is pretty much there for the taking.

His case is mainly based on an analogy with chess, where he observes that since the landmark victory of “Deep Blue” over Kasparov, things have continued to move on, so that computers now move in a sphere far above their human creators, making moves whose deep strategy is impenetrable to merely human brains. They can even imitate the typical play of particular Grandmasters in a way which reminds Rogoff of the Turing Test. If computers can play chess in a way indistinguishable from that of a human being, it seems they have already passed the ‘Chess Turing Test’. In fact he says that nowadays it takes a computer to spot another computer.

I wonder if that’s literally the case: I don’t know much about chess computing, but I’d be slightly surprised to hear that computer-detecting algorithms as such had been created. I think it’s more likely that where a chess player is accused of using illicit computer advice, his accusers are likely to point to a chess program which advises exactly the moves he made in the particular circumstances of the game. Aha, they presumably say, those moves of yours which turned out so well make no sense to us human beings, but look at what the well-known top-notch program Deep Gambule 5000 recommends…

There’s a kind of melancholy pleasure for old gits like me in the inversion which has occurred over chess; when we were young, chess used to singled out as a prime example of what computers couldn’t do, and the reason was usually given as being the combinatorial explosion which arises when you try to trace out every possible future move in a game of chess.  For a while people thought that more subtle programming would get round this, but the truth is that in the end the problem was mainly solved by sheer brute force; chess may be huge, but the computing power of contemporary computers has become even huger.

On the one hand, that suggests that Rogoff is wrong. We didn’t solve the chess problem by endowing computers with human-style chess reasoning; we did it by throwing ever bigger chunks of data around at ever greater speeds.  A computer playing grandmaster chess may be an awesome spectacle, but not even the most ardent computationalist thinks there’s someone in there. The Turing Test, on the other hand, is meant to test whether computers could think in broadly the human way; the task of holding a conversation is supposed to be something that couldn’t be done without human-style thought. So if it turns out we can crack the test by brute force (and mustn’t that be theoretically possible at some level?) it doesn’t mean we’ve achieved what passing the test was supposed to mean.

In another way, though, the success with chess suggests that Rogoff is right. Some of the major obstacles to human-style thought in computers belong to the family of issues related to the frame problem, in its broadest versions, and the handling of real-world relevance. These could plausibly be described as problems with combinatorial explosion, just like the original chess issue but on a grander scale. Perhaps, as with chess, it will finally turn out to be just a matter of capacity?

All of this is really a bit beside Rogoff’s main interest; he is primarily interested in new technology of a kind which might lead to an economic breakthrough; although he talks about Turing, the probable developments he has in mind don’t actually require us to solve the riddle of consciousness. His examples; from managing the electronics and lighting in our homes to populating “smart grids” for water and electricity, helping monitor these and other systems to reduce waste” actually seem like fairly mild developments of existing techniques, hardly the sort of thing that requires deep AI innovation at all. The funny thing is, I’m not sure we really have all that many really big, really new ideas for what we might do with the awesome new computing power we are steadily acquiring. This must certainly be true of chess – where do we go from here, keep building even better programs to play games against each other, games of a depth and subtlety which we will never be able to appreciate?

There’s always the Blue Brain project, of course, and perhaps CYC and similar mega-projects; they can still absorb more capacity than we can yet provide. Perhaps in the end consciousness is the only worthy target for all that computing power after all.

Picture: correspondent. One of the nice things about doing Conscious Entities is that from time to time people send me links to interesting things; new papers,  lectures, or ideas of their own.  I regret that I have generally kept this stuff to myself in the past, although it often deserves a mention, so I’ve been thinking about how best to deal with it. I would welcome suggestions, but as an experiment I’ve decided to try occasional round-up posts: so here goes.

Jesús Olmo, to whom many thanks, recently drew my attention to this review of  The Ego Tunnel; to PRISMs, Gom Jabbars, and Consciousness, and to the site Conscious Robots.

M.E. Tson has a Brief Explanation of Consciousness.

Mark Muhlestein has a thought experiment Consciousness and 2D Computation: a Curious Conundrum, and has been corresponding with David Chalmers. My own view is as follows.

I think causal relations are the crux of the matter. A computation essentially consists of a series of states of a Turing machine, doesn’t it? Normally each state is caused by the preceding state. Is that an essential feature? I think in the final analysis we’d say no, because the existence of a computation is really a matter of interpretation on the part of the observer. If the different states in the sequence are just written down on sheets of paper, we’d probably still be willing to call it a computation, or at least, we would in one sense. There’s another sense in which I personally wouldn’t: if we read ‘computation’ as meaning an actual run of a given algorithm, or an instantiation of the computation, I think the causal relationships have to be in place.

Now this would be even truer in the case of mental operations leading to consciousness. The causal relations in a Turing machine are to some degree artificial: the fact that we can program them in is really the point. In the human brain, by contrast, the causal relations are direct and arise from the physical constitution of the brain. To exhibit the relevant series of states (even if we assume consciousness in the brain is a matter of discrete states, which actually seems rather unlikely)would not be enough – they have to have caused each other directly in the right way for this to be an actual ‘run’ of the consciousness faculty.

It follows that your projected lights don’t give rise to a consciousness, or perhaps even to a computation. Does this mean I think zombies of some kind are possible? No, because the interesting kind of zombie is physically identical with a real person, and the projected lights are significantly different from the occurrence of the actual run of the computation. Real zombies remain impossible, and all we’re left with is a kind of puppet.

Readers of my post earlier this year about Sam Coleman’s views may be interested to see the nice comments he has provided.

You can send me links to interesting stuff at:

Contact.

Picture: Stanislas Dehaene. Edge has an interesting talk by Stanislas Dehaene.  He and his team, using a range of tools, have identified ’signatures’ of awareness; marked changes in activity in certain brain regions which accompany awareness of a particular stimulus. They made a clever use of the phenomenon of ‘masking’, in which the perception of a word can be obliterated if it follows too rapidly after the presentation of an earlier one. By adjusting the relevant delay, the team could compare the effect of a stimulus which never reached consciousness with one that did. Using this and similar techniques they identified a number of clear indications of conscious awareness: increased activity in the early stages of processing and activity in new regions, including the prefrontal cortex and inferior parietal. It appears that this is accompanied by a ‘P3 wave’ which is quite easy to detect even with nothing more sophisticated than electrodes on the scalp. Interestingly it seems that the difference between a stimulus which does not make it into consciousness and one which does emerges quite late, after as much as a quarter of a second of processing.

No-one, I suspect, is going to be amazed by the news that conscious awareness is accompanied by distinctive patterns of brain activity; but identifying the actual ’signatures’ has direct clinical relevance in cases of coma and apparent persistent vegetative state. In principle Dehaene’s research should allow conscious reactions continuing in a paralysed patient to be identified; this possibility is being actively pursued.

More speculative and perhaps of deeper theoretical interest, Dehaene puts forward a theory of consciousness as a global neuronal workspace, another variation on the global workspace theory of Bernard Baars (an idea which keeps being picked up by others, which must suggest that it has something going for it). Dehaene offers the view that a particular function of the workspace is to allow inputs to hang around for an extended period instead of dissipating. Among other benefits, this allows the construction of chains of processing operations, something Dehaene likens to a Turing machine, though it sounds a little messier than that to me. Further ingenious experiments have lent support to this idea; the researchers were able to contrast subjects’ chaining ability when information was supplied subliminally or consciously (this may sound odd, but subjects can perform at better-than-chance levels even with subliminal stimuli).

Dehaene says that he is dealing only with one variety of consciousness – in the main it’s awareness, which in some respects is the basement level compared to the more high-flown self-reflective versions. But in passing the talk does clarify a question which has sometimes troubled me in the past about global workspace theories – why should they involve consciousness at all? It seems easy to understand that the brain might benefit from a kind of clearing house where information from different sources is shared – but couldn’t that happen, as it were, in the dark? What does the magic ingredient of consciousness add to the process?

Well, being in the global workspace means being accessible to several different systems (no intention here to commit to any particular view about modularity); and one of those systems is the vocal reporting system. So as a natural consequence of being in the workspace, inputs become things we can vocally report, things we can talk about. Things we can talk about are surely objects of consciousness in some quite high-level sense.

Dehaene does not go down this path, but I wondered how far we could take it; is there a plausible explanation of phenomenal consciousness in terms of a global workspace? If we followed the same pattern of argument we used above, we would be looking to say that conscious experiences acquired qualia because being in the workspace made them available to the qualic system, whatever that might be. I think some people, those who tend to want to reduce qualia to flags or badges that give inputs a special weight, might find this kind of perspective congenial, but it doesn’t appeal all that much to me. I would prefer an argument that related the appearance of qualia to a sensory input’s being available to a global collection of sensory and other systems; something to do with resonances across modalities; but I happily confess I have no clear idea of how or exactly why that would work either.

Picture: Autoscopy. Among a number of interesting features, The Ego Tunnel includes a substantial account of out-of-body experiences (OBEs) and similar phenomena. Experiments where the subjects are tricked into mistaking a plastic dummy for their real hand (all done with mirrors), or into feeling themselves to be situated somewhere behind their own head (you need a camera for this) show that our perception of our own body and our own location are generated within our brain and are susceptible to error and distortion; and according to Metzinger this shows that they are really no more than illusions (Is that right, by the way – or are they only illusions when they’re wrong or misleading? The fact that a camera can be made to generate false or misleading pictures doesn’t mean that all photographs are delusions, does it?).

There are many interesting details in this account, quite apart from its value as part of the overall argument.  Metzinger briefly touches on four varieties of autoscopic (self-seeing) phenomena, all of which can be related to distinct areas of the brain:  autoscopic hallucination, where the subject sees an image of themselves; the feeling of a presence, where the subject has the strong sense of someone there without seeing anyone; the particularly disturbing heautoscopy, where the subject sees another self and switches back and forth into and out of it, unsure which is ‘the real me’; and the better-known OBE. OBEs arise in various ways: often detachment from the body is sudden, but in other cases the second self may lift out gradually from the feet, or may exit the corporeal body via the top of the head.  Metzinger tells us that he himself has experienced OBEs and made many efforts to have more (going so far as to persuade his anaesthetist to use ketamine on him in advance of an operation, with no result – I wonder whether the anaesthetist actually kept his word) ; speaking of lucid dreams, another personal interest, he tells the story of having one in which he dreamed an OBE. That seems an interesting bit of evidence: if you can dream a credible OBE, mightn’t they all be dreams? This seems to undercut the apparently strong sense of reality which typically accompanies them.

Interestingly, Metzinger reports that a conversation with Susan Blackmore helped him understand his own experiences.  Blackmore is of course another emphatic denier of the reality of the self. I don’t in any way mean to offer an ad hominem argument here, but it is striking that these two people both seem to have had a particular interest in ’spooky’ dualistic phenomena which their rational scientific minds ultimately rejected, leading on to an especially robust rejection of the self. Perhaps people who lean towards dualism in their early years develop a particularly strong conception of the self, so that when they adopt monist materialism they reject the self altogether instead of seeking to redefine and accommodate it, as many of us would be inclined to do?

On that basis, you would expect Metzinger to be the hardest of hard determinists; his ideas seem to lean in that direction, but not decisively. He suggests that certain brain processes involved in preparing actions are brought up into the Ego Tunnel and hence seem to belong to us. They seem to be our own thoughts, our own goals and because the earlier stages remain outside the Tunnel, they seem to have come from nowhere, to be our own spontaneous creations. There are really no such things as goals in the world, any more than colours, but the delusion that they do exist is useful to us; the idea of being responsible for our own actions enables a kind of moral competition which is ultimately to our advantage (I’m not quite sure exactly how this  works). But in this case Metzinger pulls his punch: perhaps this is not the full story, he says, and describes compatibilism as the most beautiful position.

Metzinger pours scorn on the idea that we must have freedom of the will because we feel our actions to be free, yet he does give an important place to the phenomenology of the issue, pointing out that it is more complex than might appear. The more you look at them, he suggests, the more evasive conscious intentions become.  How curious it is then, that Metzinger, whose attention to phenomenology is outstandingly meticulous, should seem so sure that we have at all times a robust (albeit delusional) sense of our selves. I don’t find it so at all, and of course on this no less a person than David Hume is with me; with characteristically gentle but devastating scepticism, he famously remarked “For my part, when I enter most intimately into what I call myself, I always stumble on some particular perception or other, of heat or cold, light or shade, love or hatred, pain or pleasure. I never can catch myself at any time without a perception, and never can observe any thing but the perception.”

Metzinger concludes by considering a range of moral and social issues which he thinks we need to address as our understanding of the mind improves. In his view, for example, we ought not to try to generate artificial consciousness. As a conscious entity, the AI would be capable of suffering, and in Metzinger’s view the chances are its existence would be more painful than pleasant. One reason for thinking so is the constrained and curtailed existence it could expect; another is that we only have our own minds to go on and would be likely to produce inferior, messed-up versions of it. But more alarming, Metzinger argues that human life itself involves an overall preponderance of pain over pleasure; he invokes Schopenhauer and Buddha. With characteristic thoroughness, he concedes that pleasure and pain may not be all that life is about; otherr achievements can justify a life of discomfort. But even so, the chances for an artificial consciousness, he feels are poor.

This is surely too bleak. I see no convincing reason to think that pain outweighs pleasure in general (certainly the Buddhist case, based on the perverse assumption that change is always painful, seems a weak point in that otherwise logical religion), and I see some reasons to think that a conscious robot would be less vulnerable to bad experiences than we are. It’s millions of years of evolution which have ingrained in us a fear of death and the motivating experience of pain:  the artificial consciousness need have none of that, but would surely be most likely to face its experiences with superhuman equanimity.

Of course caution is justified, but Metzinger in effect wants us to wait until we’ve sorted out the meaning of life before we get on with living it.

His attempt to raise this and other issues is commendable though; he’s right that the implications of recent progress have not received enough intelligent attention. Unfortunately I think the chances of some of these issues being addressed with philosophic rationality are slim. Another topic Metzinger raises, for example, is the question of what kinds of altered or enhanced mental states, from among the greatly expanded repertoire we are likely to have available in the near future, we ought to allow or facilitate; not much chance that his mild suggestions on that will have much impact.

There’s a vein of pessimism in his views on another topic. Metzinger fears that the progress of science, before the deeper issues have been sorted out, could inspire an unduly cynical, stripped-down view of human nature; a ‘vulgar materialism’, he calls it. Uninformed members of the public falling prey to this crude point of view might be tempted to think:

“The cat is out of the bag. We are gene-copying bio-robots, living out here on a lonely planet in a cold and empty physical universe. We have brains but no immortal souls and after seventy years or so the curtain drops. There will never be an afterlife, or any kind of reward or punishment for anyone… I get the message.”

Gosh: do we know anyone vulgar and unsophisticated enough to think like that?

xeex470503 bumble bee physics personal invetment templates on excel blanchard river watershed map disturbed get psycho wilton cake supplies discounted business roundtable cost effectiveness committee salina kansas scholls biometrics iraq creampie cumshots ancient trojan wars off white gallery avalon sammy kaye downloads sabiki rigs for yellow perch plumbing for pedicure chairs baptism wheel craft richard elton miramar terrace symphonic dvd vhs combo dubbing errors krumkakke iron sandy duncan pics photos by gilliland maury city tennessee leland frederick cooley jamie foxx on hbo lawler real estate llc farmingdale walmart balloon valvotomy for aortic stenosis network setup between computers ephesus culture wooden pizza paddles refugees poland will coates mk 4 no1 us santa clara home page amaroo school canberra what will happen 2008 x press 2 afrika bambaataa fire school counselor jobs in ohio how to override endevor compile jcl mph to fps phones supporting android incorporate and rock band value of a wurlitzer organ belton senior living american idol jason castro cervical nerve root contusion researcher content writer wanted key and certificate life cycle domestic worker contract inter faces sarl what is ecosystem diversity repair macintosh madison wisconsin kasey nordeen sc chimney sweep canton ohio johann gottfried flugel said ceu miami uscg 1000 sportbike shootout blackberry peal sold 2008 freddy mack ephedrine hcl pure dish installers cleveland georgia kids having kids robin hood foundation stc craft melanie falick books blm minimum royalty provision description pagosa springs co fishing shipley manor wilmington de momo hotels and resorts dvd authoring with surround sound self imagery progressive church and sacramento generational dynamics jon savitch financial planner b beckwith relationship between training and job satisfaction otters animals mother of pearl pillow open fire pit safety lawsuit movies scifi employees forced to buy uniforms shakira feet 94 seadoo review honda nav system 2007 pdr like drug information free college invasion free video im logger grilled halibut mango salsa spiral wrap tool symptoms gallbladder attacks valinge aluminum patents cell mitosis and cell division information ansi ieee device numbers feeling results of infrared sauna iglesia primitiva perennial ornamental grasses field erosion barefoot merlot montana trailors last minute orlando deals creamery robert graham style edun trinity meadow view soccer ea simonia morning domino molasses free templates for tenancy agreements kal equip vacuum pumps guidlines for association newsletters marijuana urinalysis detection times team building for parents poser millenium woman bittorrent diet self confidence self esteem 2 cloverdale way columbus nj 08022 people who study glaciers firestone grill menu san luis obispo blender bottle shaker organic blue agave sweetener dell awaiting arrival of associated peripherals mls grantsberg wisc san diego hire a felon sandstone amphitheater susan niven locust vally a sand county almanac land ethic maps brasil argebtina free pamphlet christian jew foundation pool table instalation irvin mischner libros pdf hytrol conveyors exporing four empires of mesopotamia franz viegener temple faucets stretcher sigma d14 camera virginia penal system espn bcs games online informatica publish webdav skeletal with other systems perfection company verizon text messages argentina president claim falkland islands how animals are alike genre browser horror turned wooden plates fullerton movibe hh robyn hillis springfield mo mark crew sentenced to death braves apparel leaking shower drain seal baseline nutritionals talented video loyola university maywood truck chassis repair equipment old bridge cellars merced department of agriculture partch replacement for parchment paper fortran histogram windermere british columbia empress hotel saigon mf doom dunks documentary norse san mateo fair grounds how to mixed media collage blogs about vinyl records and collecting lotro and defending the bridge pepper jelly using red pepper flakes jay leno automobiles transport of deceased to mexico chronic pain theoretical framework super rsi strubs pickles rs auto clicker 2007 passover dates aquatic peat water gardens herbal treatment for altitude sickness van hiele implication of teaching mathematics foods that include arginine internal capsule shamans knowledge omron e3x galena castle home fdny firefighter engine fire fireman verkleedkleren brussels how earthquake sizes are measured sagem my200c unlock mode s decode psp dj rey de reyes jalisco sk racing intercooler installation teal shag rug euphoria morning mp3 scott orchard lanzarote live music nero image file plumeria breeding hybrid the adjective of habit poems to an unborn nephew steel drum radio giga manuscript pallet turret song graduation ambition wood anemone subaru impreza 1.5 lake auditor motorola i265 set alarm clock gene simmons family jewels show age of remington woodsmaster 740 refinish tile surround prisoner locate us webmail earthlink login tecra 730xcdt external mouse not enabled ptb withdrawl rbc banking definitions mandy moore extraordinary video job satisfaction correlation to education level red toyota trd pictures promised bathing belles le bar sans nom its about thyme austin explorer closes opening jpeg balsamic strawberries def jam icon pics robert redford movies 1998 mei linn chuong indus vally civilization wording casual wedding in stefanie heitmann hacking pioneer inno house of names wick cape cod chefs not expresing your emotions tilt beaming entire contact list kanton bern der grosse rat eric scott allen partners lane county oregon food handlers ramones pet cemetary log house on bull shoals lake tmobile service in israel enlightenment literature barenaked ladies lyrics ron workman comic warhammer guild scrolls william foster northumberland va lange 1815 succulent ground clover plants laminate suture cart texas a m cadet corp waco joyner sales royal victoria hospital barrie on ca vivacious vixen new orleans lago di como tarrytown rare precious moments love quotes by yoda broom handel mauser forbes list and grocery stores tasty crisa gallery cyprus developers contractors collect british postal stationery david stoner clearwater florida jerseys spooner wi cochranville pa recycling ron george jewelers bo bo bibs starbucks workplace locum tenens modifier neuron asians tits sweet memories menu keller the keswick theater pac man world cheats pl gaines photography beckie hazzard kyle mcdermott in boston cervical sample case study on delegation do geese see god clive hornsby delta kitchen faucets with water filters woman bicep 1967 ford fairlane parts comprar instrumentos musica denny middle school healy brothers on teh rocks santorini alderon astrological meaning sudbury ontario furniture storage massey 35 tractor delino beach near st augustine myspace navy nuke background rented property in leeds dossin great lakes museum bayarea rappers layouts medical anthropology canciones maltrato mujer sheriff statue gift magic illusions guillotine electrolux stick vacuum biography chef alain ducasse direct x dowload open season sounds porcupine order of franciscans companions hampton inn lewisville address certa pro painters colorado springs hafler pro2400 amplifiers amag occasions tapa university macau slumber parties murrieta about article 21 of indian constitution premier hgh compression box polygon triangle pictures tweak addon for adaware se century 21 dwyer ri fantastic fiction uk microtel reservations jacqueline childress windows live club cheats smith wesson guns lipitor and lichen planus care of crocodile shoes fluorescent bulb candelabra dimmable how to strenghten cell signals homemade thick cookie frosting clifford big red dog video streams crazy frog dies ed hargis chief picture photo miyabi restaurant summerville sc boy stud gasoline prices in tennessee congress performance rating kacey cahill blue bell pa attorney shanna jack kompan custom mystick cue yoke kee powerpress ikon models bill gates email beta scam pyrotechnic simulated aerial bombing runs funny clean pictures stags estate ae effect of cayenne on warfarin harney sons irish breakfast tea the sleepover saturday night live transscript baby crib sham gis metadata research organics cerebral palsy feed refusal alpha grasslands preserves pohatcong nj medizinische hochschule hannover small sliding windows shootable cannons marcella gallagher new el pollo loco tv ads afro american mental illness history
contrabanddays.com
as pyridoxine hydrochloride
10 project objectives microsoft migration scope
arthur miller and marilyn monroe
bigvuetv.com
herbs that my interact with lipitor
c label wedges
old presque lighthouse
carmella bing porneskimo
licensed cattery oxfordshire
blue gray self storage midlothian va
1959 dodge power wagon interior
macquarie bank winter vac
freelance architectural business
atomic symbol jpeg
asp search type find locate
easysender.com
free printouts for wood working
2008 fox gear
ancient civilizations famous people
huffy volta
mercyjobs.com
birth control and diarrhea
calvary chapel emerging church
acceptable surface rust on laminations
klonopin medication
castor bean plant india
cause effect fcat questions
1984 essay questions
realdolls.com
100 meter dash times
brett a kaplan baseball
kihei homes for sale
brass precision turned components
sp teri
definition of a-a gradient
frequent heartburn gas
accidentally delete dinner family time
downtownnashvillenissan.com
2005 silverado quad
english for beginners nouns verbs pronouns
duro test corp
clarendon bt bold free
featured chris boat bay harbor
overstock and closeout sporting goods
her-first-lesbian-experience.com
2002 intrepid
cost of vela smooth lipolite
carotid artery plaque
halibut anchorage charter
hubble telescope nebula pictures
ais australia independent schols
gotta keep you separated
biltmor pool ladders
95sx halloween 2004
baby phat casual shoes
amazon giant catfish
accidental suicide
pueblocountyjail.com
adam west eating taffy wav
amleia attwater rhodes
keds shoes popular early 90s
difference between gatorade and powerade
artista beach villas
barbara peters tonawanda
a1 buttstock screw sale
elena costas
1920 s fashion history
pat tung
gym stowe vermont
green road harford county maryland
4wd how it works trooper
afco precast
art le sean
polycom quad nt1
chartwells spartanburg 6 lunch menu
club djs
fung shui for motivation
an arkansas razorback pattern
bonding bushing armored cable
ties2.net
aryepiglottic cancer trials
addie sayers shamrock middle school
msmguide.com
2000 exit polls gore bush
alabama gambling
application of title vii
annabelle lettering
commspeed.net
anderson d leather sleigh bed
280zx stats
berklee tower of power ensemble
first hacked satelite superbowl message
varicose eczema treatment
scifichannel.com
girls silver signet ring
free chapter 13 plan forma
dodger charger x-pipe
albums by anne lennox
arcserve mcafee slow 2007
hopw to use ridgid locking pin
2006 infiniti g35x grill
fabt.com
incorrect 1099
jazz saxaphone
aions mmorpg
ear taping bullmastif
2008 new song releases
orvis bed covers
denver landforms and wildlife
1973 mustang power steering cooler
1998 subaru forester power mirror
1885 lybrook ambassador to algeria
horticultural sand
3104 blaine grand island nebraska
carmel holding linden mi
sgivp.com
alpaca farm solvang
parenthood a divine stewardship
hawking ha09sdp antenna
upward bound christian athletics
decrypt qam
polycom computer input
betta fish and soits out food
9buckbigcocks.com
catholic liturgical music calendar
2000 silverado knock sensor location
archdiocesesantafe.org
dragon acupunture santa clarita
add book guest napster record
explain acidic alkaline to kids
books like meg cabot
musee des beaux arts questions
do freon leak sealers work
2004 nissan frontier replacement keyless entry
beowulf analysis
cheap ralph lauren perfume glamorous
micrometer optical
breyer horse dolls
michael parks weighing risks
big dog k-9 voodo rip
abe f march said
black and white six ring jordans
adjustable bling ring
crusaders fighting qualities
cat-man-doo xtra large bonito flakes
does nucleic acid consist of phosphorus
1950 slam dunk contest winner
hotels in mechanicsburg
care giver redding ca
117 laureston st brockton ma
american primary documents
acidic human saliva
excellence apartments usf tampa fl
beverly connor lindsay chamberlain
barbara keefe cranberry hospice
amman bhajans
alma wright
congress of neurosurgery
nahl.com
452 varmint in style stock
museo archeologico nazionale
cast of rosewood movie
actor gale harold
blasto walkthroughs
materials for constructions in concrete industry
benign colon polyp
7.1 channel home theater speaker systems
0 riellys auto parts
adhesive packaging
ancient india death dying practices
newtek.com
354 chrysler piston
abba bail bonds riverside
biblical greek scholars explanation of monogenes
all weather wicker and rattan
10 spinning rod
halo mod melee effect
history of thc
webcamtranx.com
neuroma vs rsd
gametrailers com gt faction list
approved fall restraint lanyard length
pghboe.net
alyssa milano poison ivy 2
lymph nod chart of pelvic area
juggling accessories
american wetlands
1970 nfl football schedule
acceptable center of gravity rc airplane
2001 volkswagon passat garage door opener
louis xiv its the girl rapidshare
autocenterlive.com
decorative sockets uk
consult ink limited
ladyboy-crush.com
david robert crews
laborers.org
certified psychiatric rehabilitation practitioner
hormone imbalance after giving birth
bonnie johnson nampa idaho
dave sato
15th security forces squadron
95 olds 3.1 vacuum pcv tee
jake jake for goodness sake
a very special accoustic christmas
cruiseshipjob.net
cookbook lydia
fuckingdickhead.com
morningstar farms coupon
abp anti-siphon outdoor faucet suppliers
behind the shad
christian guevara
historical astronomical events pleaides
jboss web application exploit
thunderstorm july 18 2007 patchogue ny
crf 250 graphics
18 gauge tools freight harbor oilless
charlie nubiles galleries
theanimegallery.com
eaten girls
swandc.com
anton raphael mengs
5 in 1 crib dresser
clomipramine vicodin
aids population in jamaica
ixtapa temperatures
bnai torah emily weiss
1982 hunter yachts
continent of the ninth
abi ofarim reichstadt father
asians pulled skirts
2007 information stae of minnesota
hexagon grids
kiawah island inn resort
addis great lakes auction
ansonia song lyric
cjhosting.net
aaa travel patrice
telscape.com
alum research wetlands
baltimore cork
crab creole
agence immobili re marseille
candian lynx
20 pin ic device driver
2006 taxes late
1 pe pipe
skim coat with epoxy mortar
agusta sports
mcdougal littell geometry
coat chicken egg whites
starwood canada microsoft
remo banjo heads
ds game reviews at eb games
limo4hire.com
metis canada powerpoint presentations
doormat fluke rigs
art nouveau religious locket
lyrics someday soon
2001 mazda tribute door handle schematic
american cosmetic surgery network
shot suitable for bears
childrens union generals uniforms
naturalnirvana.com
90s rap and hip hop
buffy fanfic anything to save her
incheon kr
a picture of a jammed finger
homeofhardcore.com
causes of swolen lymph nodes
8th blackbird chicago
calphalon bakeware
catherine fine ritual abuse
alaska oil leases
911 truth books
patch resting potential
american style karate log
broad leaf feild balsom
armani black code m
shawnadee.com
1980 honda cx500 custom luggage rack
find the willamette meridian
bread of life homeless mission houston
fifth element uptown minneapolis
articles about fundamentals of organizing
a65 bsa popping out of gear
backyard activity toys
composite entity
2 7 8 mayhew thread dimensions
alter email receive service water
143 north drive pittsburgh
fetish-pix.net
2008 senior bowl game recap
dixiekiss.com
directtv terre haute
beads jewellery supplies beadwork patterns
amber atkins muskegon
makeup-fx.com
$1 stock trades
dsc 832 security system battery
bail hearings federal district court
blogspot elle
ps2cheatcodes.com
cartridge for lexmark x1270
2003 honda foreman rubicon 500
brent allan barth onsted mi
monstergulf.com
15 fixed mortgage rate yr
china executes drug offical
convert amv to avi
con val high school nh