[back to blog]

Gradient origin networks

This work has been accepted at ICLR 2021. Gradient Origin Networks (GONs) are comparable to Variational Autoencoders in that both compress data into latent representations and permit sampling in a single step. However, GONs use gradients as encodings meaning that they have a simpler architecture with only a decoder network:

In Gradient Origin Networks, unknown parameters in the latent space are initialised at the origin, then the gradients of the data fitting loss with respect to these points are used as the latent space. At inference, the latent vector can be sampled in a single step without requiring iteration. Here are some random spherical interpolations showing this:

In practice, we find GONs to be parameter-efficient and fast to train. For more details, please watch the YouTube video and read the paper in the links in the project page. There is also a Google Colab link if you want to try running the code in your browser.

Citation

Please cite the article with:

@inproceedings{bond2020gradient,
  title     = {Gradient Origin Networks},
  author    = {Sam Bond-Taylor and Chris G. Willcocks},
  booktitle = {International Conference on Learning Representations},
  year      = {2021},
  url       = {https://openreview.net/pdf?id=0O_cQfw6uEh},
  keywords  = {Conference}
}