How do you, personally, know that the chain is correct? Have you ever checked the hashes? Have you ever checked the transactions? Have you ever proved that the math they use is correct? Have you ever read the code for any BTC client?
You're trusting an awful lot of assumptions much more complex than 'the longest chain is right'.
You're trusting an awful lot of assumptions much more complex than 'the longest chain is right'.