Hi Guys,

I have been working on a project to generate color palette using a deep learning model. Idea is to feed the model famous color palettes collected across the internet and use that to generate new color palettes. Idea is to generate new color palettes. User can also specify some colors in the palette and have to generate rest of the colors. Example of palettes from the dataset are:

​

Colors from dataset

I have tried following things:

Bidirectional GRU based model: I converted the problem of generating color into fill in the blank problem. I represented input as 15 numbers (5 color palette and each color has 3 RGB values) and each number ranges from 0 to 256. where 0-255 are color values and 256 is for mask. So input to model is randomly masked vector and output is unmasked values. Here is an example with mask [0,0,0,0,0,1,0,1,0,0,1,0,0,1,0]:

Input : [0, 185, 252, 124, 122, 256, 48, 256, 189, 48, 256, 113, 178, 256, 116]

Output: [0, 185, 252, 124, 122, 125, 48, 170, 189, 48, 127, 113, 178, 143, 116]

The task of the model would be to predict values which are 256. Idea is similar to Masked LM (e.g. BERT) but without attention or other advanced layers. Model is embedding layer, a few GRU layers and one dense layer with softmax loss. Before training model outputs random garbage:

First two colors are hints – outputs before training

For inference, I would do something like beam search. So I would give give my masked vector to model and get prediction. Then select one masked value and replace it with prediction. Selection of class is random based on probabilities from last layer. Then repeat this step until there is no masked value remaining.

Output after training:

First two colors are hint – GRU after training

The problem with this model was that, model would predict dull color (for comparison see colors from dataset or generated by GAN). It felt like model was trying to just average out all the colors. So I went for different approach.

GAN-based Approach: In this approach, I tried to normalize my color from 0-255 to -1 to +1. generator would also output a vector of size 15 from -1 to +1. Then discriminator would discriminate the colors and so on. This worked like a charm. Generator outputs sometimes surprisingly good colors and sometimes it fails as well.

Outputs before training:

GAN: Color generated before training

Outputs after Training:

Palettes look pretty nice

But main problem in this approach was that I could not specify the hint as mentioned above. Like if user provides a few values and wants rest of colors to be generated, that would not be possible. To work around this, I tried a trick from style gan paper.

First one is given a color, find its latent space. I tried training a simple network which would predict latent space of randomly generated colors. It didn’t work that great. Then I tried gradient search. e.g. input vector to be optimized and generator weights are locked but it would get stuck in local minimum. Finally, I went for genetic search. Start with random population, find, colors which are closer to hint. This gave pretty good colors but given the reasonable amount of time it really did converge very close but not exact to hint color. Fitness score would be MSE between generated color and hinted color only for hinted values.

This posed another challenge. In some cases, model would always converge to just a few palettes given a specific hint. To solve this I included diversity in the fitness function e.g. try to maximize distance from previously generated colors as well. Now colors do not get very close to hint.

Following is example with locking and diversity correction:

​

First two colors should have been same as provided in above (GRU) example

As you can see first two colors do not stay perfectly close to original colors.

Currently, last mentioned method, with tiny bit of error is currently live. But I wanna know if someone knows better technique to solve this problem.

Thanks