Saturday, August 24, 2013

Scalding - WordCount example in local mode











Scala IDE based on eclipse
scalding on scala 2.9


How to run scalding on eclipse

1. install eclipse indigo ( preferable j2ee edition, but add maven plugin -m2e plugin fromupgrade repistory - Help ->Install new software))
2. In Help->Install New software -> add a site http://download.scala-ide.org/sdk/e37/scala29/stable/site
and install scala ide plugin
http://scala-ide.org/download/current.html
We will work with scalding template created byAmit Nithan
http://hokiesuns.blogspot.co.il/2012/07/running-your-scalding-jobs-in-eclipse.html
it already contains needed scalding dependencies
Onec you followed an article and scalding was tested in local mode your next step is to run it on real hadoop cluster.
In order to run Maven's package or other commands from eclipse do:
  • right-click project
  • run as
  • run configurations..
  • double click maven build (to create a new configuration)
  • give a name for configuration e.g. package
  • click variables
  • select "selected_resource_loc" and click ok
  • write your goal e.g. "package" or "clean package"
  • run
The next time when you want to package another project, you can use this configuration again:
  • right-click project
  • run as
  • run configurations..
  • select your maven configuration
  • run
ENJOY:)

4 comments:

  1. Followed instructions to the letter but can't get a fat jar built.. any other details in the pom or runtime configuration that you've changed to get this to work?

    ReplyDelete
    Replies
    1. not at all.
      if you want you can send me a screenshot with the error,,,,
      rami.mankevich@gmail.com

      Delete
    2. Please check you run with Indigo eclipse..and relevant scala plugin.
      Thanks

      Delete
  2. This comment has been removed by the author.

    ReplyDelete