I'm sorry but the problem is a trivial one. Even if you've never read a file, never made a http call in code or written to a file, these are all things a 5 minute stack overflow away.

I absolutely would not hire someone who failed that test. Your defence of it seems bizarre to me.

chillydawg, I take hinkley's assertion to be more of a warning to be careful about biases that may have inadvertently been introduced through the testing methodology. I think it's impossible for us to know without seeing the test you use, but it's always possible that the requirements aren't as clear as you think they are, or potentially what you deem as trivial may be biased from recent direct experience on a problem that you've then turned into a test.

I think these tests are a great tool, but like anything we all just need to be aware and careful of the biases introduced. Also, we need to be mindful that really good developers who are employed, have families etc, may not have an over abundance of time to spend on these, so if 10 companies each send a challenge that takes 4 hours, and it's unclear if the assessment will lead to further interviews, some better candidates may give up on the process or be selective about the ones they complete.

I think this creates an additional bias, that the people who complete your assessments may be the ones with the most time to invest in the process (ie the currently unemployed).

At my current job, we have an "onsite" challenge that we used to hire some Cuban devs. We went to Cuba (from Mexico) and gave the candidates a codebase for Tetris in an interpreted language they did not know. They have 2 hours to transform that Tetris into a Nibbles (snake) game.

The catch? There is no internet in Cuba, so they had to do it using a book, and asking questions to a programmer who was helping them there.

I think this is a good point. There's a real risk that a homework you think is simple is a little more complicated because you're making assumptions. I think it's good to have a few current team members actually code it up (not just whiteboard it), and also to talk with a few candidates to see how long it took them, to calibrate this.

It's also better to aim small (a la FizzBuzz) and walk up the complexity slowly, as opposed to starting with "Solve this ticket from our prod JIRA queue, with tests please" (which I have actually been given as a test!)

I like to aim for a very simple task, like proposed above, with maybe one non-obvious part or edge case (which the spec mentions, so it's not a trick question). Just enough to demonstrate a) clean coding for the simple parts, and b) ability to solve something without using a completely naive solution.

"[...] perf issues that nobody else believes exist."

Uhuh.

You'd be surprised how many people stop looking at the perf numbers once all the tall tent poles are gone. They have no idea or tools for digging beyond that. I've worked on several projects where the customer still wanted the app to be 2-10 times faster (or worse, a competitor was 2-3 times faster) and people look at the charts and announce that they've done everything that can be done. Some hide in plain sight but others are just putting in work. For instance code duplication can hide a tall tent pole because it's chopped into five pieces.

I had a shared library running on a bunch of systems. One was VxWorks. Cross compiled. I knew nothing about VxWorks except that the code needed to run about 8x faster. I got about 2.5x out of my code by improving locality of reference problems that appeared to be intrinsic but just required experience to see. For the rest I filed a bug report to WindRiver and five to the cross compiler writer. At the end of this process I knew more about the VxWorks administration than anybody else in the group.

People give up, even when there's a clear business case for continuing.