Pix2Code: Automating Front-End development
Certainly not the end of Front-end developers, but it is a good start.
Paper link: https://arxiv.org/abs/1705.07962
Several days ago, Tony Beltramelli published pix2code. Pix2Code aims at automating the mockups-to-code process using Artificial Intelligence.
The generated code is not really compilable, but it can be converted into Web, iOS, Android or anything that can be used in an application.
Training
Pix2Code uses a combination of 3 Neural Networks:
- A LSTM (Long-Short Term Memory) network that takes a sequence of tokens as input (tokens being terminals in the language chosen)
- a CNN (Convolutional Neural Network) used to engineer features out of the mockup.
- A LSTM’ that takes (q_t, p) as input and outputs a Softmax vector of tokens. The result (x_t+1) is then fed into the next iteration. The output vector is a mapping of probability to a terminal token.
The restricted language complexity of the DSL (Domain Specific Language) allows hot vector encoding at the input and a Softmax vector at the output. The LSTM network is fed with a window of tokens of size T (the paper specifies that T=48 is a good window size following empirical tests).
Sampling
After training the network, we can feed it any mockup and generate DSL code.
The empty state is an array of 48 vectors with the last one being <START>. The generating process ends when the generating network outputs the <END> token.
The generated DSL code can be compiled into a target code such as HTML (Web), Objective-C (iOS), Java (Android).
Results
Although the loss curves seem really promising, the author only shared cherry-picked examples (such as the one at the top of this article). A 77% accuracy is advertised by the author.
Restrictions of the algorithm so far:
- No dataset of GUI to code is available online
- The code is not provided (yet) by the author
- The code generated is restricted to a subset language (DSL)
- The colors and text are not captured by the AI
This paper is a good start and an interesting use of Convolutional Neural Nets for writing software. This echoes with some other recent papers (DeepAPI, DeepCoder) that hope to automate some parts of software engineering is the near future.
Interestingly enough, to my knowledge, nobody has been able to generate front-end code using Adversarial Neural Network so far. Surely we’ll see more of those in the short term.
