chillydawg, I take hinkley's assertion to be more of a warning to be careful about biases that may have inadvertently been introduced through the testing methodology. I think it's impossible for us to know without seeing the test you use, but it's always possible that the requirements aren't as clear as you think they are, or potentially what you deem as trivial may be biased from recent direct experience on a problem that you've then turned into a test.

I think these tests are a great tool, but like anything we all just need to be aware and careful of the biases introduced. Also, we need to be mindful that really good developers who are employed, have families etc, may not have an over abundance of time to spend on these, so if 10 companies each send a challenge that takes 4 hours, and it's unclear if the assessment will lead to further interviews, some better candidates may give up on the process or be selective about the ones they complete.

I think this creates an additional bias, that the people who complete your assessments may be the ones with the most time to invest in the process (ie the currently unemployed).

At my current job, we have an "onsite" challenge that we used to hire some Cuban devs. We went to Cuba (from Mexico) and gave the candidates a codebase for Tetris in an interpreted language they did not know. They have 2 hours to transform that Tetris into a Nibbles (snake) game.

The catch? There is no internet in Cuba, so they had to do it using a book, and asking questions to a programmer who was helping them there.