Neural networks are some of the most promising artificial intelligence (AI) models. These systems process data similarly to the human brain, passing information through a complex network of nodes in the same way information goes through layers of neurons. That makes them capable of solving complicated problems in minimal time, which is particularly advantageous in the medical industry. Drug discovery is a vital but challenging process.
Dong et al. (2019) and Tay et al. (2022) train on a mixture of denoising tasks with different attention masks (full, causal and prefix attention) to bridge the performance gap with next token pretraining on generative tasks.
