It really is not. At most it can be tedious, because it has large surface area of protocols and formats. Most of them other than SMTP and message format are optional.
Minimal mail system involves just postfix with config in a bunch of files, delivering mail to a folder on the server, and pretty much everything on top, like SPF,DKIM,DMARC,... is optional.
You can ssh to your server from anywhere and run mutt there, or install a webmail if you're into web stuff. That's pretty much all that's needed for a basic thing. Security wise, there's not much to do wrong in this setup. Mail server is open for reception by default as it should be and submission is only allowed from localhost.
Webmail can be protected via HTTP auth and ssl. As long as your password is secure, noone will get there that way.
Rolling your own crypto is a much bigger can of worms.
It's not the hard part, it's simply impossible to ensure from your side. You can have everytrhing righy, and they'll still block you based on IP address.
You can configure your dmarc so the big mail systems all provide you with reports on delivery failures. I get lots of them when I reply to a mailing list that doesn't deal with spf or dkim sensibly.
Not too much, unless you are completely new to server environments and networking.
You'll likely experience security breaches rolling your own crypto unless you are an expert in cryptology.
You'll also experience security breaches running your own email server if you also don't know what you're doing, but it's far easier to learn what it takes to configure an email server correctly from a security perspective than rolling your own crypto.
The bigger problems with running your own email server is NAT/CGNAT, ISP restrictions, and the fact that the big players will still ignore you due to only caring about the other big players.
