From the man page under "color.ui" (version
"This variable determines the default value for variables such as
color.diff and color.grep that control the use of color per command
family. Its scope will expand as more commands learn configuration
to set a default for the --color option. Set it to always if you
want all output not intended for machine consumption to use color,
to true or auto if you want such output to use color when written
to the terminal, or to false or never if you prefer git commands
not to use color unless enabled explicitly with some other
configuration or the --color option."

So, setting it to false should do the trick, but maybe you have something that overrides it somewhere?