There is a popular thought experiment that goes like this. Suppose that neurologists characterized each neuron's inputs and outputs exactly, and were able to engineer a functional equivalent. That is, an artificial device whose inner workings may or may not be similar to those of a natural neuron, but whose behavior, seen in terms of its responses to inputs, was identical to that of a neuron. Now suppose that the neurons that comprise your brain were replaced with these artificial neurons, one by one. Once your entire brain was cut over to the artificial neurons, you should have a brain system whose functioning at the neuronal level is identical to that of the brain you were born with, but whose workings are entirely artificial, and as such, able to be characterized with an algorithm of some sort.
This thought experiment (called Neuron Replacement Therapy, or NRT for short) is intended to put anti-physicalists and anti-functionalists like me in the uncomfortable position of having to say either that the resulting artificial brain is not conscious (and if not, at what point in the gradual neuron replacement does consciousness disappear, and when it does, does it wink out all at once or fade out gradually), or that the artificial brain maintains its consciousness, and therefore full-blown consciousness is realizable by a machine.
I agree that there is nothing magic about organic or biological systems. There is no reason that consciousness must be manifested in a biological system. Indeed, as a panpsychist, I would argue that consciousness in some form is likely manifested all the time in all kinds of matter. The problem with the thought experiment is that it presumes exactly the functionalist reductionism that is, in my opinion, at the heart of the matter. It assumes that what makes a brain a mind does so purely by virtue of the complex interaction of lots of blind little autonomous parts, each not knowing or caring about the others, as long as each has the right interface presented to it.
No one knows the details of the relationship between neurons as neuroscientists characterize them and consciousness, but as I have argued, thoughts come whole. A given percept, thought, or moment of consciousness is what it is in its entirety, all at once, or not at all. When I perceive or think about a dog, I see the whole dog, nose to tail, all at once. That dog idea is indivisible, one of nature's primitives, like a quark or an electron. It is, to use Gregg Rosenberg's term, a natural individual, a partless whole. It must be, because that is how it appears, and as with the seeing red argument, this appearance can not be illusory. Because it has no parts, you can not swap some of its parts out in favor of "functionally equivalent" parts.
Even if my dog idea is an example of some kind of fundamental holism occurring in nature, couldn't it still be generated in some way by the orderly, lawful interactions of smaller parts? Possibly, in some sense, but it could not turn out to simply be the orderly, lawful interactions of smaller parts. There is no sense in which the blind interactions of lots of little parts necessarily come together into a single, unified percept, whereas my idea of the dog is manifestly a single unified percept. The interactions of the parts may functionally emulate the percept, and they may support it somehow, but they alone can not be it. Assuming that there will, ultimately, turn out to be necessary relations between the physical world as we understand it and consciousness, the physical correlates of consciousness would have to display or allow for the kind of holism that our thoughts manifest. Where in the physical world might we find this kind of inherent wholeness, as opposed the to just may-be-seen-as wholeness that functional analysis of systems of parts gives us?
Physical systems in states of quantum entanglement display the necessary holism. Further, I have speculated that as quantum mechanics contains the only currently known gap in the causal closure of the physical, the indeterminacies of quantum mechanics are, in fact, the fence around these natural individuals that modern science has built, with a sign that says, "Something funny is going on in here, and we can never know what". Maybe consciousness occurs in bursts, in the collapse of quantum superpositions, as Hameroff and Penrose claim. Maybe not.
For the moment, however, let us set aside my strong suspicions about quantum mechanics. Perhaps my speculations about quantum mechanics are completely wrong. Perhaps consciousness is some kind of hitherto undiscovered field or force that is modulated or generated by neurons. Maybe Gregg Rosenberg is right, and consciousness is built into the mesh of causation itself. Moreover, no matter how this question is answered, the quantum superposition or force or field that is consciousness could be something that spans lots of neurons, as Hameroff and Penrose believe, or it could be something that happens inside a single neuron, as suggested by Jonathan Edwards.
Whatever thoughts, percepts, or moments of consciousness turn out to be, neurons have evolved to generate or exploit them in some way. But it is they, (these fields, forces, superpositions, collapses thereof, or whatever) who do the heavy lifting, at least in the "what is it like to see red" sense. They quite probably do a lot of the heavy lifting in a lot of other ways as well, including some we might call straightforwardly cognitive.
If the the artificial neurons in the NRT thought experiment can also exploit or generate these things, then great - consciousness is preserved in the artificial brain. If not, not, and the NRT thought experiment fails. If the field or force or superposition that is consciousness spans multiple neurons, it will be something that can not be carved up and characterized in terms of quantifiable inputs and outputs between neurons, and algorithmic functions that map between the two, and the NRT hypothesis is untenable.
If, on the other hand, it is something that occurs entirely within a single neuron, with explicitly quantifiable interactions between the neurons, then the artificial neurons will likely not be able to emulate natural neurons with an explicitly specified algorithm, for the usual arguments against epiphenomenalism. It is most likely that the non-algorithmic stuff in the neuron guides the neuron's behavior in non-algorithmic ways.
If, on the other hand, the stuff of consciousness (force, field, whatever) happens inside individual neurons, and the neuron's extrinsic behavior with regard to the rest of the world (including other neurons) is emulatable with an algorithm (the epiphenomenal case) the end result of NRT will be a zombie. All of its neuronal behaviors and motor outputs will be identical to those of a conscious mind, but it will not, in fact, be conscious, at least in the "what is it like to see red" sense.
In the latter case, (which I consider unlikely) how will the transition from conscious mind to zombie occur? I don't know. We can speculate on may ways consciousness could fade out as the biological neurons were replaced by artificial ones. Consciousness probably would not wink out all at once. I do not believe that we each consist of one continuous consciousness, always operating at the same level. Just as we operate on autopilot to some degree or another at various times throughout any given day, perhaps as our real neurons were cut over to the artificial ones, the autopilot moments would become more and more frequent and complete in their internal blankness. As the (fewer and fewer) moments of true consciousness happened, they might come complete with the (erroneous) impression of complete, continuous consciousness. This might be simply a result of the moments of consciousness having some sort of cognitive access to zombie memories and extrapolating consciousness onto them. Whatever the scientific basis for consciousness is, there may be lots of consciousnesses active at any one time in our brains. Maybe as our neurons are cut over, there would be fewer and fewer consciousnesses happening within us. The question of how the transition would actually happen is wide open - we just can't know right now. But there is nothing we do know that makes the transition an impossibility.
The bottom line is that even if you succeed in creating a conscious machine, you have no less a mystery on your hands than you do with conscious people, and you are no closer to characterizing consciousness in algorithmic or functional terms. And if you do create something that behaves as if it were conscious and whose workings are entirely specified by an algorithm, there is nothing obviously incoherent in my claim that you will have created a zombie.