Saturday, October 12, 2013

Apache kafka on windows





After downloading apache kafka i failed starting it up on windows with existing scripts.
I made some adjustments for some of scripts and now the broker successfully starts.

Anyone who needs it feel free to do'nload:

https://docs.google.com/file/d/0B83rvqbRt-ksUEVUY1ZNTTRfcVk/edit?usp=sharing

or download updated scripts only:

https://docs.google.com/file/d/0B83rvqbRt-ksR2VPb2JzeHA1U2M/edit?usp=sharing

Just unrar , start zookeeper and server.






Fastest REST service




Targer
Handling 100K tps per core on 64 bit linux (RH) json based service.

After deep investigations

Comparisons of following webservers
undertow with Servlet
tomcat with Servlet
jetty with Servlet

Frameworks
play http://www.playframework.com/
spray.io http://spray.io/

Low level frameworks
Netty
ZeroMQ


Conclusions

1. In order to reach the load we have to release service io thread as soon as possible .
2. Request with single entry vs request with bulk mode.
3. zero calcualtion in io thread.

Thats how i  reached the performance torget.
Final architecture:
1. Jetty with servlet based serviec (POST implementation)
2. Bulk mode with 100K per request.
3. Release of request ASAP ( return 200 as soon as possible - then processing )
4. Still synchronous servlet.
5. Jmeter load testing .

Measuring
On server side by definining int[120] and doing System.currentTimeInMillis() / 1000 and incrementing apropriate variable in array :
myArray[System.currentTimeInMillis() / 1000] ++;
then printing once in 2 minutes and zeroing.

Limitation on single core
taskset -c 0 mycommand --option  # start a command with the given affinity
taskset -c 0 -p 1234             # set the affinity of a running process
BTW :
When process was already running taskes -p <PID>  didnt work.

Future investigation

1. Asynch servlet.
2. Akka based async service.
3.Netty + RESTEasy framework

Friday, October 4, 2013

REST Webservice on NETTY RESTEasy



I had to implement some simple REST service
Requirments were:
1. low latancy
2. highly scalable
3. robust
4. high throughput
5. simple
6. JSON based parameters passing
Ive started with Tomcat with servlet on top of it and got bad throughput.
Ive tried Jetty but still - throughpit was terreble.
Then i decided to use Netty with some REST back on top of it.
Will do benchmarking and update you soon....

Resteasy
http://www.jboss.org/resteasy

Service :


@Path("/message")
public class RestService {

@Path("/test")
@POST
@Consumes("application/json")
@Produces("application/json")
public Response addOrder(Container a)
{
return Response.status(200).entity("DONE").build();
}

}

@XmlRootElement
public class Container
{

private ArrayList<Parameter> parameters = new ArrayList<>();
public void AddParameter(Parameter p)
{
this.parameters.add(p);
}
@XmlElement
public ArrayList<Parameter> getParameters() {
return parameters;
}
public void setParameters(ArrayList<Parameter> parameters) {
this.parameters = parameters;
}

}

@XmlRootElement
public class Parameter {

private String name;
private int age;
@XmlElement
public String getName() {
return name;
}
public void setName(String name) {
this.name = name;
}
@XmlElement
public int getAge() {
return age;
}
public void setAge(int age) {
this.age = age;
}
}

public class Server 
{
public static void main(String[] args) {
ResteasyDeployment deployment = new ResteasyDeployment();
  Map<String, String> mediaTypeMappings = new HashMap<String, String>();
  mediaTypeMappings.put("xml", "application/xml");
  mediaTypeMappings.put("json", "application/json");
  deployment.setMediaTypeMappings(mediaTypeMappings);
  NettyJaxrsServer netty = new NettyJaxrsServer();
   netty.setDeployment(deployment);
   netty.setPort(TestPortProvider.getPort());
   netty.setRootResourcePath("");
   netty.setSecurityDomain(null);
   netty.start();
  deployment.getRegistry().addPerRequestResource(RestService.class);

}

}

Client :

public class ClientMain {
public static void main(String[] args) {
Client client = ClientBuilder.newBuilder().build();
       WebTarget target = client.target("http://localhost:8081/message/test");
       Container c = new Container();
       Parameter param = new Parameter();
       param.setAge(11);
       param.setName("RAMI");
       c.AddParameter(param);
       Response response = target.request().post(Entity.entity(c, "application/json"));
       String value = response.readEntity(String.class);
       System.out.println(value);
       response.close();  
}
}