From ANU to Canva and PyTorch: The Award-Winning Work Behind Smarter AI Vision

By Benjamin Hosking

How does a machine see? To perceive the world, artificial intelligence needs a way to understand the various elements in an image and what they represent. Computer vision researchers employ a technique called representation learning, which uses neural networks to automatically learn from visual datasets and translate images into vectors: numerical representations of the meaning of an object, allowing an AI to distinguish a bird from an airplane.

Datasets can suffer from bias or be too small, affecting the performance of the AI learning from that data. When real-world data is limited, researchers like ANU School of Computing Associate Professor Liang Zheng step up to augment it while avoiding the introduction of bias.

“No matter how you want a computer to perceive the world or generate one, you will rely on a vector representation of the image,” Zheng said. “I’m trying to improve the vector representation by creating better training data.”

Zheng created a method that allows for generation of synthetic data that expands the training set for an AI without introducing bias. Zheng and his team thought of hiding patches of an image to synthesize what might be in that patch, similar to how one might fill in what might be hidden behind objects. They applied random rectangular masks to images to augment datasets and simulate occlusions.

For example, a robot might want to locate a person behind a sofa. Thanks to the augmented data that provided examples of occlusion, the robot stands a better chance of successfully searching in such settings.

Unexpected applications for ANU research

Zheng’s approach had an impact well beyond his initial expectations. Now included in Meta’s open-source deep learning framework PyTorch, AI researchers around the world have used it to detect fabric defects, glaucoma, and crop disease, estimate sites of amplified seismic activity, and even recognise individual sharks.

Zheng was recently awarded the Australian Academy of Science’s Brian Anderson Medal recognising his work in computer vision. Named after ANU Emeritus Professor Brian Anderson, the medal honours outstanding contributions to research in information and communications technology.

“I met [Anderson] the other day,” said Zheng, who sees Anderson as an inspiration. A widely used form of generative AI builds on Anderson’s contributions going as far back as the 1980s. “That reflects how I feel about ANU’s research – not just short-term curiosity, but also long-term impact.”

Zheng’s interest in computer vision stems partially from his undergraduate studies in biology at Tsinghua University. Before graduating, he decided to pursue a PhD. A faculty supervisor in electrical engineering asked Zheng if he was interested in a biologically inspired way of approaching computer vision based on how the human brain works.

“I was attracted by this idea to design algorithms to let computers perceive the world,” Zheng said. “I switched from biology to computer vision. I had to learn programming from scratch and a lot of computer science knowledge.”

After a two-year postdoc at the University of Technology Sydney, Zheng joined the ANU in 2018. He received several fellowships from the Australian Research Council and served as program co-chair for ACM Multimedia’s 2024 conference, leading the review process for over 4000 submissions. He mentions fellow faculty members Lexing Xie, Stephen Gould, Hongdong Li, and Richard Hartley as mentors for him at the ANU.

“We have eight faculty members in computer vision and work closely together,” Zheng said. “ANU also has the best machine learning and AI groups in Australia.”

Collaborating with industry to advance image generation

Beyond his research at the ANU, Zheng supports Australian software company Canva’s design generation platform. He helps develop fundamental algorithms that power their text-to-image models.

“Whether you want to generate or perceive something, the underlying principles of machine learning are the same,” Zheng said.

Zheng’s partnership with Canva on image generation started a couple years after one of his PhD students, Jaskirat Singh, began working on generative AI in 2023. When Canva decided to build a team of researchers, Zheng was the right person.

“I learn from my students a lot, and I realized [Singh’s] research was very cool,” Zheng said. “Shortly after that, ChatGPT and Midjourney came out. Everyone was so excited about generative AI from that point.”

With Canva colleagues in San Francisco, London, Vienna, Sydney, and Beijing, Zheng enjoys working closely with a global set of collaborators to improve generative AI. His push for global collaboration has also led to the recent establishment of a partnership between the School of Computing and Singapore’s Agency for Science, Technology, and Research Institute (A*STAR), sending up to ten PhD students to do one to two years of fully funded research with A*STAR.

Preparing students for the future of AI research in Australia

Zheng enjoys teaching the next generation of machine learning researchers, which requires him to stay on the cutting edge of current AI models. One project for Advanced Topics in Machine Learning involved students training their own nano-GPT model to tell stories in five sentences.

Projects like these prepare computing students for cutting-edge roles in industry. Zheng sees research integration with industry as a positive characteristic of the AI field, as algorithms he developed as a university researcher find applications in industrial scenarios, while industry provides researchers with real-world problems to solve.

“The problems aren’t siloed, and people work together,” Zheng said. “Australia has a much stronger industry ecosystem compared with five years ago. Our PhD students can find research jobs in Australia rather than go to the USA or China.”

Referring to companies with valuations above one billion dollars, he said, “I’m very confident we will have more unicorns in the next few years.”