Updated 14-Sep-2023
Image optimization is a big deal when it comes to Google PageSpeed. There are several things to consider with images: compression, image dimensions , file format, and number of files.
Image Compression
Some server-side compression using plugins is ok, but it really is better to just do this as a part of the image file creation workflow. There are many tools for this. Find one that works for you.
Sizing of Image Dimensions and avoiding CLS
Images should be sized properly, so that they are correct for the viewport. Media queries can help with this, or just optimize the viewport for mobile and do responsive layout of images. This combined with things such as tiled images provides a consistent look and feel for the viewer, with whatever device and viewing distance is being used. That way desktop views don't appear as bloated versions of mobile formats, and mobile devices don't suffer from very small or partial graphics.
There are ways of hiding content via toggles or images inside of right-swipe lightboxes that work for mobile, if there is too much content for a given screen and viewing scenario.
Google PageSpeed complains about Cumulative Layout Shift (CLS) as a Core Web Vital, and resizing images is one aspect of that.
File Formats for Images: JPG, PNG, SVG, WebP
File formats should be chosen based on browser support and compressed size, and no other metric. Some images are compressed better with JPG than with PNG, and some the reverse. SVG is generally smaller, but again browser support is paramount, and after that simply chose the smallest file size.
Ultimately WebP and SVG should rule the day, but that is not the case now. For WebP using WordPress there are many plugins that promise compression and WebP file support. Test for the best approach.
SVG as Emerging Standard
SVG is now fairly well supported by browsers, and even in those browsers and browser versions where less than full support is available, only a few SVG features won't work.
Moving to SVG images will make things much easier for vector and font-based graphics. Within tools such as the venerable Inkscape
Image Sprites
Image sprites are single images that have multiple images on them, which are combined with CSS to display one part of the image in a particular location, and other parts in other locations. The main advantage is that a single file is loaded, and can contain many images. Very useful for multiple small images such as things like social media buttons, navigation bars, and the like.
PNG sprites should not to be confused with an APNG animated PNG, but same concept.
Image Sprite File Formats: GIF, PNG, SVG
Sprites were originally done with the GIF file format, but there are PNG sprites as well as SVG sprites, aka inline SVG](https://css-tricks.com/svg-sprites-use-better-icon-fonts/).