It could very quickly become racist, but it isn't necessarily. The Masters of Scale podcast recently did a nice episode on using a voice clone of Reid Hoffman to help with various dubs and edits: https://mastersofscale.com/meet-reid-ish-the-story-of-clonin...
There's a segment where the various staff/producers all try to imitate Reid's style, and then apply the voice transfer model to here their impression in his voice.
I've also used an open source TTS model conditioned on an actor whose voice I like to read the transcript from a podcast host whose voice I have a hard time tolerating. Here's where you start to edge closer to possible racism, echo chambering, etc. It's certainly a gray area, and I haven't navigated the moral shoals and reefs of replacing one voice with another* I like more – which is why I won't publish anything about how I did it.
(*A random sample: Better to not listen to what a person has to say, or to listen to your voice-cloned version of their words? What about if you literally can't understand them? Non-native speakers already rewrite their words using ChatGPT. Possibly (likely) only okay if they do it themselves...)
