Time slices and streams

Smooth or chunky? Like peanut butter, experience could have different granularities; in practice it seems the answer might be ‘both’. Herzog, Kammer and Scharnowski here propose a novel two-level model in which initial processing is done on a regular stream of fine-grained percepts. Here things get ‘labelled’ with initial colours, durations, and so on, but relatively little of this processing ever becomes conscious. Instead the results lurch into conscious awareness in irregular chunks of up to 400 milliseconds in duration. The result is nevertheless an apparently smooth and seamless flow of experience – the processing edits everything into coherence.

Why adopt such a complex model? What’s wrong with just supposing that percepts roll straight from the senses into the mind, in a continuous sequence? That is after all how things look. The two-level system is designed to resolve a conflict between two clear findings. On the one hand we do have quite fine-grained perception; we can certainly be aware of things that are much shorter than 400ms in duration. On the other, certain interesting effects very strongly suggest that some experiences only enter consciousness after 400ms.

If for example, we display a red circle and then a green one a short distance away, with a delay of 400ms, we do not experience two separate circles, but one that moves and changes colour. In the middle of the move the colour suddenly switches between red and green (see the animation – does that work for you?). But our brain could not have known the colour of the second circle until after it appeared, and so it could not have known half-way through that the circle needed to change. The experience can only have been fed to consciousness after the 400ms was up.

A comparable result is obtained with the intermittent presentation of verniers. These are pairs of lines offset laterally to the right or left. If two different verniers are rapidly alternated, we don’t see both, but a combined version in which the offset is the average of those in the two separate verniers. This effect persists for alternations up to 400ms. Again, since the brain cannot know the second offset until it has appeared, it cannot know what average version to present half-way through; ergo, the experience only becomes conscious after a delay of 400ms.

It seems that even verbal experience works the same way, with a word at the end of a sentence able to smoothly condition our understanding of an ambiguous word (‘mouse’ – rodent or computer peripheral?) if the delay is within 400ms; and there are other examples.

Curiously, the authors make no reference to the famous finding of Libet that our awareness of a decision occurs up to 500ms after it is really made. Libet’s research was about internal perception rather than percepts of external reality, but the similarity of the delay seems striking and surely strengthens the case for the two-level model; it also helps to suggest that we are dealing with an effect which arises from the construction of consciousness, not from the sensory organs or very early processes in the retina or elsewhere.

In general I think the case for a two-level process of some kind is clear and strong, and well set out here. We may reasonably be a little more doubtful about the details of the suggested labelling process; at one point the authors refer to percepts being assigned ‘numbers’; hang on to those quote marks would be my advice.

The authors are quite open about their uncertainty around consciousness itself. They think that the products of initial processing may enter consciousness when they arrive at attractor states, but the details of why and how are not really clear; nor is it clear whether we should think of the products being passed to consciousness (or relabelled as conscious?) when they hit attractor states or becoming conscious simply by virtue of being in an attractor state. We might go so far as to suppose that the second level, consciousness, has no actual location or consistent physical equivalent, merely being the sum of all resolved perceptual states in the brain at any one time.

That points to the wider issue of the Binding Problem, which the paper implicitly raises but does not quite tackle head on. The brain gets fed a very variable set of sensory inputs and manages to craft a beautifully smooth experience out of them (mostly); it looks as if an important part of this must be taking place in the first level processing, but it is a non-trivial task which goes a long way beyond interpolating colours and positions.

The authors do mention the Abhidharma Buddhist view of experience as a series of discrete moments within a flow; we’ve touched on this before in discussions of findings by Varea and others that the flow of consciousness seems to have a regular pulse; it would be intriguing and satisfactory if that pulse could be related to the first level of processing hypothesised here; we’re apparently talking about something in the 100ms range which seems a little on the long side for the time slices proposed; but perhaps a kind of synthesis is possible..?

30 thoughts on “Time slices and streams

  1. Peter,

    I’m glad you posted this demonstration. It is empirical evidence that is important for our understanding of consciousness. Here is how I explain this phenomenon in terms of the retinoid model of consciousness:

    In the retinoid model, selective visual attention consists in the projection of added neuronal excitation in retinoid space by selective excursions of the heuristic self-locus (HSL). Here’s how patterns of self-locus activation explain the phi phenomenon:

    1. When the first dot flashes on (S1), HSL moves to the spatial locus of S1.

    2. When S1 turns off and the second dot (S2) flashes on after a blank interval of ~ 30 ms up to ~200ms, HSL moves over intervening autaptic neurons to the new spatial locus of S2.?

    3. Over the trajectory of S1 to S2, neuronal HSL excitation plus excitation from the decaying S1 combine to create a moving trace of heightened autaptic-cell activity in retinoid space.?

    4. We see phi motion between successively flashed dots because there really is a path of moving neuronal excitation in retinoid space that is induced by the heuristic self-locus in the spatial interval between S1 and S2.

  2. I don’t see it move, TBH. Is this like old woman/young woman illusion pictures? Am I an X man now? But seriously, perhaps changing the colours might help (it’s a lot like traffic lights now – so they don’t seem to move just as much as traffic lights don’t). Also could have the bottom left orb move toward the top right corner just a little bit and fade to nothing instead of blink to nothing. Then the top right could appear to fade in while heading toward the top right, then stop, then head toward the bottom left while fading out. Yeah, fiddley to do – I just didn’t want to say ‘I don’t see it’ without giving any suggestions as to how it could get me.

    Really it opens up a few questions of what ‘summerisations’ are in play (or heuristics, as others would put it). Like, why see two different things there at all? Why even see two different things, instead of the fine grain grid of LCD pixels? Reminds me of Ferris Beulers day off when his friend is looking at an artwork and looks ever closer at it, but entering only into the tiny details of it. But yeah, why see even two orbs, let alone one orb that travels from one point to the other? There are levels of unmasking to be had, here.

    It’s certain though that if instead of seeing a grid of pixels (or a grid of atoms), a creature might do better in terms of survival if it saw orbs. Well, if it saw fruit (rather than the abstract ‘orb’) rather than a grid of atoms.

    And these summerisations (sorry to make up words!) might be definitional of the creature. Brain augmentation would definitely alter or remove the summerisations, thus removing the creature from existance to various degrees, even if externally it seems to be there as normal. Transhumanism? Uncanny.

  3. Count me as one of those who’ve never experienced any motion in the color phi experiment; but apparently, some people just don’t. Here’s an interactive gadget to let you play around with the effect—none of the settings amount to anything other than ‘one disk flashing, then another disk flashing’ to me, I’m afraid. Whatever editing needs to go on in order to introduce the appearance of motion, it seems my brain just doesn’t bother—which is really typical of the bugger.

  4. Arnold, no, there’s still no motion if I use the same color for both disks. However, there’s also the issue that the first time I saw this, I already was aware that I would be seeing two flashing lights, but ought to believe seeing motion, so I didn’t go into this as a naive observer. Those of you capable of seeing the motion, does/did knowing the actual mechanics produce any change?

  5. Jochen,

    I saw motion before I had any knowledge of the phi phenomenon. I remember, as a child, being intrigued by moving advertising signs that showed illuminated messages or pictorial images at night moving along a large display surface. This is a common application of the phi effect. Have you never seen these as patterns in motion?

  6. Arnold:

    I saw motion before I had any knowledge of the phi phenomenon.

    Sorry if I wasn’t clear. I didn’t intend to imply that knowledge of the phenomenon serves to bias an observer towards experiencing it, but the other way around—I meant to ask whether, in those that experience the phenomenon, knowledge of the underlying mechanism changes the experience (since I first encountered it having this knowledge, and didn’t get the effect most people describe).

    As for the billboard signs, I presume you mean those neon lights type things where two or three different illumination patterns form the stages of a rudimentary animation? Those are not really common around here; in fact, I think the only examples I know come from films. And I’m not sure I would say I really experience motion—it’s more that I get that motion is supposed to be implied by the succession of images. I also don’t see any ‘interpolating’ images between the ones of the animation.

  7. If I squint I can convince myself that I see motion. Maybe the 400ms isn’t quite set in stone? What happens (for those who see no motion) if the interval is changed? 375ms? 425ms?

  8. Using Jochen’s link, if I bring the two orbs closer together then I start to see motion (especially if the orbs overlap). Like a young lady/old lady illusion image where you can adjust from seeing one to the other, here there’s some room to adjust from seeing movement to seeing two separate lights going on and off, but as they got closer together I found that harder to do, even to the point of simply dismissing the idea of thinking of it as two orbs.

    Thinking on it the interval is probably some ‘coincidence equation’ or something – I mean, how often in nature do two separate things go off in such co-ordination? They usually don’t. So when two images go off with a certain short time interval between them, they the coincidence equation decides to not treat it as a coincidence and as the one object. As they say, a heuristic. So you can start to rap your knuckle against the inside of your own capacities boundaries.

    Probably flocks of birds and schools of fish exploit this in taking off together when startled by potential predators, making it harder for predators to track an individual target.

  9. Jochen, but surely you don’t see cartoons on TV as just a series of unrelated static images? This is really no different. Perhaps turn up the framerate?

  10. Callan:

    Jochen, but surely you don’t see cartoons on TV as just a series of unrelated static images?

    No, I don’t; but part of the phi-effect seems to be (if I understand the description correctly) that one experiences some form of interpolation between the images, with the disk changing color in between. That’s a distinct effect that I don’t seem to get. E.g., the illusion of motion in cartoons could be explained by my visual system itself just sampling the visual field at some fixed rate, such that anything that changes faster than that is simply experienced as a continuous motion (I know that this isn’t the actual explanation, or at least, that it’s too simplistic, but it’s a model consistent with the data). But such a model wouldn’t work for either the usual phi or color-phi effects.

  11. I also see two disks appearing and disappearing. Maybe it’s my ignorance but perhaps this is another toy examples that doesn’t necessarily apply to anything relevant outside the lab?

    That would be the case for Libet’s supposed experiments about “decisions” as well so there’d be some consistency at the least.

  12. i see a transition in side vision but i’m not sure i see a change of colour. Red + Green = yellow, I can’t say I can see yellow but in side vision i definitely sense movement. If i look straight on I see two flashing circles.

    “ut our brain could not have known the colour of the second circle until after it appeared, and so it could not have known half-way through that the circle needed to change. ”

    Well – unless the brain got used to the idea after four or five cyles of flashes and knew what to expect. It’s the same with the verniers – good luck with proving it’s not the rhythm of the flashing that causes the effect but some magical 400ms delay.

  13. Jochen,

    So the phi effect is supposed to be different from how cartoons work? I’m not sure how?

    I’m wondering if it’s because the illusion of movement is easier to detect and that’s why it’s getting another name – part of the brain is fooled and part of the brain can see the fooling, whereas in a cartoon it’s undetectable? Otherwise I don’t know why there’s a different name for it? Or is it supposed to turn yellow or something in between or even have a third image simply projected between the two? Some actual illusionous orb created in the viewers vision? I’m a bit confused here!

  14. In the explanation given by the retinoid model, there should be no representation of a change of color in the motion of S1 to S2 because it is just the contribution of S1 excitation that is experienced on the path to S2. I have never seen a change of color in the path of motion. I believe that such reports are based on a judgement that the color must have changed along the way. But a judgement is not a perception.

  15. Callan:

    So the phi effect is supposed to be different from how cartoons work? I’m not sure how?

    Well, to explain the illusion of motion in a cartoon, one really only needs to assume that (say) one’s visual system refreshes with a certain rate; so when we see ‘motion’, we really always see just a series of static images (note that this isn’t actually how vision works, but it would be a plausible mechanism at least for this effect). So once the cartoon has a higher rate of repetition than that, we couldn’t distinguish it from any other motion, so we experience it as we would that motion.

    But this isn’t sufficient for the phi phenomenon: there, we effectively have just two slides of an animation, with the object being first at one point, and then at another; but apparently, some people see it moving between the two locations. But with the mechanism such as above, we would still see it ‘jump’, even if the time between the animation frames were smaller than the visual system’s refresh rate. A continuous movement (i.e. the experience thereof) would require the displacement of the object along its trajectory in successive steps.

  16. Thanks for the explanation, Jochen. Somehow some intermediary moving object is seen in the the white space between the two orb positions? I didn’t expect that.

  17. At least that’s what explanations like the one on wikipedia have me thinking (see the pictures). Although the page on the ‘regular’ phi phenomenon seems to throw that into doubt:

    Although both lines are perceived to be stationary and simultaneous, motion is perceived between them. This motion is described as having direction (from the earlier presented line to the later presented line) but to not be bound to an object. It was therefore also described as ‘pure’ motion, that is motion that is not bound to an object.

    Also, for the record, I do perceive motion in the example on the wikipedia page, with the empty spot ‘going around’ the circle. But still, there’s nothing in the color phi demonstration, certainly no hint that anything ‘changes color midway’.

  18. Don’t be so hard on yourselves, we all experience phi-like phenomena whenever we watch a movie. An object is displayed for about 40 ms, then another picture is displayed, with changed position on the screen – we perceived a moving person or a car.
    It’s the same thing, really.

  19. Arnold

    1. When the first dot flashes on (S1), HSL moves to the spatial locus of S1.

    ?2. When S1 turns off and the second dot (S2) flashes on after a blank interval of ~ 30 ms up to ~200ms, HSL moves over intervening autaptic neurons to the new spatial locus of S2.?

    3. Over the trajectory of S1 to S2, neuronal HSL excitation plus excitation from the decaying S1 combine to create a moving trace of heightened autaptic-cell activity in retinoid space.?

    4. We see phi motion between successively flashed dots because there really is a path of moving neuronal excitation in retinoid space that is induced by the heuristic self-locus in the spatial interval between S1 and S2.

    Is there any expirical evidence for the claim nr 2? Do we really know that the excitation moves through the neurons in the retinotopic topology of the visual cortex?

  20. Ihtio,

    Here is empirical evidence supporting my claim:

    J Cogn Neurosci. 2006 Jul;18(7):1174-80.
    Images of illusory motion in primary visual cortex.
    Larsen A1, Madsen KH, Lund TE, Bundesen C.

    Abstract

    Illusory motion can be generated by successively flashing a stationary visual stimulus in two spatial locations separated by several degrees of visual angle. In appropriate conditions, the apparent motion is indistinguishable from real motion: The observer experiences a luminous object traversing a continuous path from one stimulus location to the other through intervening positions where no physical stimuli exist. The phenomenon has been extensively investigated for nearly a century but little is known about its neurophysiological foundation. Here we present images of activations in the primary visual cortex in response to real and apparent motion. The images show that during apparent motion, a path connecting the cortical representations of the stimulus locations is filled in by activation. The activation along the path of apparent motion is similar to the activation found when a stimulus is presented in real motion between the two locations.

  21. It’s glib of me, but it really would be interesting to see what neurons are firing when these intermediary images are being ‘seen’. I suspect it would be fairly deep into the visual parts of the brain – in a late processing point.

  22. Thanks, Arnold.

    Looking through the references I’ve found another interesting paper:

    Akselrod, M., Herzog, M. H., & Ö?men, H. (2014). Tracing path-guided apparent motion in human primary visual cortex V1. Scientific reports, 4.

    Which can be read openly from Nature website.

    What’s interesting is that the retinotopic activation can follow curved paths as well.

  23. ihtio,

    Yes, the retinoid model of consciousness/perception explains how “filling in” by excursions of the heuristic self locus can follow any kind of path. I have argued that the solving of a visual maze requires the tracing of open pathways by controlled excursions of our heuristic self locus in retinoid space.

  24. Pingback: Communicating Animation – referendumpd.com

  25. Pingback: Speaking Animation – alkonoch-ms.ru

Leave a Reply

Your email address will not be published. Required fields are marked *