After trial and error with all the settings in Outlook, there is a setting/checkbox in Define Send/Receive Groups --> Edit --> Receive mail items. There are two radio buttons. One is for Download complete items including attachments for subscribed folders and the other is for Use the custom behavior defined below. I chose the second setting and then selected ONLY the Inbox. This way ALL of the subscribed folders don't get synched when the sync happens. Back on the main Outlook screen, if a folder is clicked, it shows Synching folder xyz at the bottom of the screen and then updates that folder in real time to see any recent additions or deletions in the folder.
On this screen I also unchecked Get folder unread count for subscribed folders since that was recommended in a lot of search results I found. I don't know how much this impacts the speed.
The default setting when adding a new IMAP email account is for the Download complete items including attachments for subscribed folders to be ON. For smaller mailboxes or ones without too many folders, maybe this is OK, but as the mailbox grows, it adds considerable time to the sync.
With these settings, when clicking Send/Receive All Folders, the sync happens almost instantaneously.