Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

I want it to not hang forever, not spam the console with pages of exception tracebacks, and not require a ton of boilerplate process-management code to accomplish the above. Ideally it would also allow me to handle other exceptions (i.e. besides KeyboardInterrupt) that occur both in the main process and child processes. I've never figured out how to do this with multiprocessing, despite lots of attempts.


Don't know what you did wrong, but you should be able to catch exceptions at their appropriate level and shut down gracefully.


multiprocessing was designed to mirror the threading library... so the exceptions are specifically set up not to cross thread/process boundaries. But the same strategies to handle multithreaded exceptions and keyboardinterrupt should apply.

If you want the child processes to shutdown when the main process goes down, you should be able to just set .daemon = True on the process objects before you start them.

If you want exceptions in the children thread to propagate up to the main thread and then handle, looks like you'd just need to send the exception across a queue or something in multiproccessing. In the new futures library, your future (wrapping the process) has a return value property you can try to access. If the child process ran into an exception, that exception would be raised in the main process when you tried to access that return value.


I know it should be possible, but I've never figured out how to do it, nor have I ever seen any example code using multiprocessing that does it. If you know of some example code that properly catches and handles exceptions in any process, I'd love to see it.


The concurrent.futures module handles exceptions well, waiting to raise them until you call ``future.result``.

Or did I misunderstand the problem?


My complaints are about the multiprocessing module, and my hope is that concurrent.futures will solve some of them.


The latter library has a better design in some respects, but isn't quite as pleasant as ``Pool.imap_unordered`` when you're not worried about exceptions in the subprocesses/threads. However, I've been using ``concurrent.futures`` more in the last year.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: