Apart from what he points out in the article ("The PIC client is susceptible to a MitM (Man In The Middle) attack because it does not verify if the public key sent by a server is from a trusted party")
1. Weird encryption which seems to be XORing the plaintext against the same encrypted IV?
2. Using random padding instead of PKCS5 or similar?
3. Using memcmp instead of CRYPTO_memcmp or similar?
2. The padding is definitely concerning; see if you can devise an attack from it.
3. That's a real issue, but I'm not counting it (for my own count) since a memcmp timing channel against a general purpose CPU over IP is extraordinarily difficult to exploit.
Well, on closer inspection it seems like the MAC isn't actually an HMAC, but just SHA3. And since the padding isn't verified (because it can't be) that leaves it open to simple extension/truncation attacks (Crytopals challenges 29 and 30)?
Nope. It's safe to make a MAC out of a keyed SHA3 hash (one of the criteria for the SHA3 contest was that none of the hashes be vulnerable to length extension). Fun fact: that's also true of the "truncated" SHA2 variants (SHA2-384 and the like).
1. Weird encryption which seems to be XORing the plaintext against the same encrypted IV?
2. Using random padding instead of PKCS5 or similar?
3. Using memcmp instead of CRYPTO_memcmp or similar?