Many conlangers like to exoticize their language by adding non-European features. But this betrays the same Eurocentric bias as Esperanto does. Esperanto was meant to be a Human language, but ended up as an European language. Klingon was meant to be a non-Human language, but deliberately takes features from non-European languages. The same underlying assumption is that Human = European. The very notion of “exotic” is generally understood to be racist and Eurocentric.
On top of this, is the very fact of taking features of e.g. Dahalo not an act of cultural appropriation? Under the usual rules, yes. This makes about any attempt at constructing languages by people having white privilege thoroughly racist, if not White Supremacist. Think of it, geeks!
oogenhand 
“Esperanto … ended up as an European language”. I’m sorry but I think you’re wrong. Esperanto’s originator was a speaker of a Semitic language. Esperanto has characteristics in common with Turkish and Chinese.See for example: http://claudepiron.free.fr/articlesenanglais/europeanorasiatic.htm
I will blog it.
In a story that I am currently writing, I based its language on Cantonese, which is my native language. It is set in an world that is somewhat like Earth, but in a more sci-fi setting.
I sometimes wonder if basing my company on Cantonese would be racist, even as I am a Cantonese. Is it so?
It wouldn’t be cultural appropriation, i think. However, you could be internalizing white supremacy if your conlang reinforced racist stereotypes. I think it would depend on the expert you ask.