Jacob Devlin
Person on 2 sites
One of the four authors of the 2018 paper "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding", at Google AI Language, the…
-
ppll (opens ppll.app)
Where public figures publish, and what they are on record saying.
Updated 2018-10-11 -
4 beliefs on korrents (opens korrents.com)
-
Current techniques restrict pre-trained representation power because standard language models are unidirectional. (opens korrents.com)
“We argue that current techniques restrict the power of the pre-trained representations, especially for the fine-tuning approaches. The major limitation is that standard language models are unidirectional, and this…”
-
Unidirectional restrictions are sub-optimal for sentence-level tasks and harmful for token-level tasks that need bidirectional context. (opens korrents.com)
“Such restrictions are sub-optimal for sentence-level tasks, and could be very harmful when applying fine-tuning based approaches to token-level tasks such as question answering, where it is crucial to incorporate…”
- Pre-trained representations reduce the need for many heavily-engineered task-specific architectures. (opens korrents.com)
-
Current techniques restrict pre-trained representation power because standard language models are unidirectional. (opens korrents.com)