By default, WordPress relies on the Gravatar service to display avatars for post and comment authors. Whether the avatars are displayed or not is determined by the theme you’re using. Gravatar is a centralized service where you can register with your email address and select an avatar. When you use your email on a website that has Gravatar support, the avatar you uploaded to Gravatar will be used by the site. Gravatar is owned by Automattic, and was added to WordPress in version 2.5, back in March 2008.
If you or your users find it counterintuitive that you need to use a third-party website to manage your avatar, there are numerous WordPress plugins that replace Gravatar with the ability to upload avatars directly on your site. One of the most popular ones is the WP User Avatars plugin by John James Jacoby.