Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Just curious - have you guys used the hadoop contrib stuff with Luigi? We use it almost exclusively to kick off Hadoop jobs and when I went to refactor some of my predecessors stuff that just kicked off a raw process that called 'hadoop jar' to the hadoop contrib stuff, I ran into a lot of weird issues (largely, the way arguments get passed to the hadoop job).

Just was curious if many other people were using the hadoop contrib stuff successfully, or if I am trying to use something that isn't very well supported.



No, we haven't used Luigi with Hadoop. For batch processing we use containerized jobs with a simple job queue, like described in the blog article.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: