As Dr. Evil found out after spending a few decades in a cryogenic freezer, perspectives on quantity change very quickly. Ever since the explosion of mobile apps and the growing number of services that can deal with humongous amounts of data, we need to re-define our concepts of what ‘Big Data’ means. This goes for developers who want to write the Next Great App, as well as those who want to write services to support it.

One of the best ways of connecting with mobile application customers is via remote push notifications. This is available on both Android (GCM) and iOS (APNS). These services allow developers to send messages directly to their users and this is an extremely valuable tool to announce updates, send personalized messages and engage directly with the audience. Google and Apple provide services that developers can send push messages to and they in turn deliver those messages to their users.

The Problem

It’s not unusual for apps these days to have in the order of millions and even tens of millions of users. Testing a Push Notification backend can be extremely hard. Sure, you can set up a few test devices to receive messages but how do you know how long it would take your backend to send out a large number of push messages to the Google and Apple servers? Also, you don’t want to risk being throttled or completely blacklisted by either of those services by sending a ton of test data their way.

The Solution

The solution is to write a mock server that’s able to simulate the Google/Apple Push Notification Service, and a test client to hit it with requests.

The Google service is completely REST based, so a script that executes a lot of curls in a loop can do that job. Also, it’s fairly straightforward to write a simple HTTP server and accepts POSTs and sends back either a 200 or some sort of error code.

Apple’s APNS, however, presents a few challenges. It’s a binary format listed here. Since the protocol is binary, you need to write some sort of mock client that can generate messages in the specified format. At Flurry, we’ve been playing around with Node.js to build scalable services and it’s fairly straightforward to setup an Apple APNS test client and server.

The Client

The client.connect() method connects to the mock server and generates test data. The Buffer object in Node is used to pack the data into a binary format to send it over the wire. Although the protocol lets you specify a token size, the token size has been set to 64 bytes in the client since that’s typically the token length that gets generated. Also, in our experience, the APNS server actually rejects tokens that aren’t exactly 64 bytes long. The generateToken() method generates 64 byte hex tokens randomly. The payload is simple and static in this example. The createBuffer method can generate data in both the simple and enhanced format.
 
What good is a client without a server, you ask? So without further ado, here’s the mock server to go with the test client.

The Server

After accepting a request, the server buffers everything into an array and then reads the buffers one by one. APNS has an error protocol, but this server only sends a 0 on success and a 1 otherwise. Quick caveat: Since the server stores data in a variable until it gets a FIN from the client (on ‘end’) and only then does it process the data, the {allowHalfOpen:true} option is required on createServer so that the client does not automatically close the connection.

4th grade budding girl geek in the making 2nd row 2nd girl from the left

TechWomen brings emerging women leaders in Science, Technology, Engineering and Mathematics  from the Middle East and Africa together with their counterparts in the United States for a professional mentorship and exchange program. TechWomen connects and supports the next generation of women entrepreneurs in these fields by providing them access and opportunity to advance their careers and pursue their dreams.

http://www.techwomen.org/

An organized rack makes a world of difference in tracing and replacing cables, easily removing hardware, and most importantly increasing airflow. By adopting good cabling habits, your hardware will run cooler and more efficiently and ensure the health and longevity of your cables. You also prevent premature hardware failures caused by heat retention. Good cabling practices don't sound important but it does make a difference. It's also nice to look at or show off to your friends/enemies.

When cabling, here are some practices Flurry lives by:

Label everything

There has never been a situation where you've heard someone say, "I wish I hadn't labeled this." Labeling just makes sense. Spend the extra time to label both ends of the network and power cables. Your sanity will thank you. If you're really prepared, print out the labels on a sheet ahead of time so they'll be ready to use.

Cable length

When selecting cable length, there are two schools of thought. There are those who want exact lengths and those who prefer a little extra slack. The majority of messy cabling jobs are from selecting improper cable lengths so use shorter cables where possible. A good option is custom made cables. You get the length that you need without any excess. This option is usually expensive in either time or money. The other option is to purchase standard length cables. Assuming that you have a 42U rack, the furthest distance between two network ports is a little over six feet. In our rack build outs, we've had great results using standard five foot network cables for our server to switch connections. 

Cable management arms

When purchasing servers, some manufacturers provide a cable management arm with your purchase. They allow you to pull out your server without unplugging any cables. For this added benefit, they provide bulk, retain heat, and reduce proper airflow. If you have them, we suggest that you don't use them. Under most circumstances, you would unplug all cables before you pull out a server anyway.

No sharp bends

Cables do require a bit of care when being handled. A cable's integrity can suffer with any sharp bends so try to avoid this. In the past, we have seen port speed negotiation and intermittent network issues cause by damaged network cables.

Use mount points

As you group cables together, utilize anchor points inside of the rack to minimize stress on cable ends. Prolonged stress on the cable ends can cause the cable and socket it's connected in to break. Power ends are also known to unplug. The weight of the bundled power cables can gradually unplug it at any moment. Using anchor points will help alleviate directed stress to the outlet.

Less sharing

Isolate different types of cables (power, network, kvm, etc) into different runs. Separating cable types will allow for easy access and changes. Bundled power cables can cause electromagnetic interference on surrounding cables so it would be wise to separate power from network cables. If you must keep copper network and power cables close together, try to keep them at right angles. Standing at the back of the rack, network cables are positioned on the left hand side of the rack while power cables are generally on the right in our setup.

Lots and lots of velcro

We've seen the benefits of velcro cable ties very early on. It's got a lot of favorable qualities that plastic zip ties do not. They're easy to add/remove and also retie. They're also great when mounting bundled cables into anchor points inside of the racks. If your velcro ties come with a slotted end, do not give into the urge to thread the velcro into the ends. It's annoying to unwrap and rethread. Don't be shy to cut the velcro to length, either; using just the right length of velcro can make it easier to bundle and re-bundle cables. 

Now that you have these tips in mind, let's get started on cabling a Flurry rack.

1. Facing the back of a 42U rack, add a 48 port switch in about the middle of the rack (position 21U (21st from the bottom). Once you have all your servers racked, now the fun part being, cabling. Let's start with the network.

 2. From the top most server, connect the network cable to the top left port of your switch, which should be port 1.

3. As you go down the rack, connect the network cables on the top row of ports from left to right on the switch (usually odd numbered ports). Stop when you've reached the switch.

4. Using the velcro cable ties, gather together the cables in a group of ten and bundle the cabled groups with the cable ties. Keep the bundle on the left hand side of the rack. You will have one group of ten and one group of eleven that form into one bundled cable.

5. For the bottom set of servers, start with the lowest server (rack position 1U) and connect the network cable to the bottom left most port on the switch.

6. Starting from the bottom up, connect the network cables on the bottom row of ports from left to right on the switch (usually even numbered ports).

7. Doing the same as the top half of the rack, gather together the cables in a group of ten and bundle the cabled groups with the cable ties. Keep these bundles on the left hand side of the rack. You'll end up with two bundles of ten that form into one bundled cable. Look pretty decent?

8. Now, lets get to power cabling. In this scenario, we will have three power distribution units (pdus), one on the left and two on the right side of the rack. Starting from the top of the rack, velcro together five power cables and plug them into one of the pdu strips on the left side of the rack from the top down.

9. Take another two sets of four bundled power cables and plug them into the other pdu strips on the right hand side also following the top to bottom convention. You should end up with a balanced distribution of power plugs.

10. Take a bundle of six power cables and plug them into the pdu strip on the left hand side.

11. Take another two sets of four power cables and plug them into the two pdu strips on the right hand side.

12. Start from the bottom up, bundle the power cables in groups of five. You will end up with two sets of five power cables and a bundle of four.

13. Plug the bundle of four power cables into the pdu on the left hand side.

At this point, you can take a step back and admire your work. Hopefully, it looks sort of like this:

Good cabling can be an art form. As in any artistic endeavor, it takes a lot of time, patience, skill, and some imagination. There is no one size fits all solution, but hopefully this post will provide you with some great ideas on your next rack build out.

Any program that pulls data from a large HBase table containing terabytes of data spread over many nodes will need to put a bit of thought into the retrieval of this data. Failure to do this may mean waiting for and subsequently processing a lot of unnecessary data, to the point where it renders this program (whether a single-threaded client or a MapReduce job) useless. HBase’s Scan API helps in this aspect. It configures the parameters of the data retrieval, including the columns to include, start and stop rows and batch sizing.
 
The Scan can also include a filter which can be the most impactful way to improve performance of scans of an HBase table. This filter is applied to a table and screens out unwanted rows from being included in a result set. A well-designed filter is performant and minimizes the data scanned and returned to the client. There are many useful Filters that come standard with Hbase, but sometimes the best solution is to use a custom Filter tuned to your HTable's schema.

Before your custom filter can be used, it will have to compiled, packaged in a jar, and deployed to all the regionservers. Restarting the HBase cluster is necessary for the regionservers to pick up the code in their classpaths. Therein lies the problem – an HBase restart takes a non-trivial amount of time (although rolling restarts mitigate that somewhat) and the downtime is significant with a cluster as big as Flurry's.

This filter will have to be pushed to the regionservers to be run server-side. The sequence of how the custom filter gets replicated on the regionservers is as follows:

1. The client serializes the Filter class name and its state into a byte stream by calling the CustomFilter's write(DataOutput) method.
2. The client directs the byte array to the regionservers that will be part of the scan.
3. Server-side, the RegionScanner re-creates the Scan, including the filter, using the byte stream. Part of the stream is the filter’s class name, and the default classloader uses this fully qualified name to load the class using Class.forName().
4. The filter is then instantiated using its empty constructor and configured using the rest of the filter byte array (using the filter's readFields(DataInput) method (see org.apache.hadoop.hbase.client.Scan for details).
5. The filter is then applied as a predicate on each row.

(1) myfilter.jar containing our custom filter resides locally in the regionservers' classpath
(2) the custom filter is instantiated using the default Classloader

Once deployed, this definition of our custom filter is static. We can make an ad hoc query using the combination of filters, but if we need to add, extend or replace a custom filter, it has to be added to the regionserver’s classpath and we have to wait for its next restart before those filters can be used.

The ProxyFilter passes its own class name to be instantiated on the server side, and serializes the custom filter after.

On the regionserver the ProxyFilter is initialized in the same way as described in the previous section. The byte stream that follows should minimally contain the filter name and its configuration byte array. In the ProxyFilter's readFields method, the relevant code looks like this.

This is very much like how the default Scan re-creates the Filter on the regionserver with one critical difference – it uses a filterModule object to obtain the Class definition of the custom filter. This module retrieves the custom filter Class and returns it to ProxyFilter for instantiation and configuration.

There can be different strategies for retrieving the custom filter class. Our example code copies the jar file from the Hadoop filesystem to a local directory and delegates the loading of the Filter classes from this jar to a custom classloader [3].

To configure the location of the directory the module searches for the filters.jar in HDFS, add the following property in hbase-site.xml.

<property>
<name>flurry.hdfs.filter.jar</name>
<value>/flurry/filters.jar</value>
</property>
(1) The custom filter jar resides in a predefined directory in HDFS

With the ProxyFilter in place, it is now simply a matter of placing or replacing the jar in the Hadoop FS directory to pick up the latest filters.

Reloading Filters
When a new Scan is requested on the server side, this module first checks up on the filter.jar. If this jar is unchanged, the previously loaded Classes are returned. However, if the jar has been updated, the module repeats the process of downloading it from HDFS, creating a new instance of the classloader and reloading the classes from this modified jar. The previous classloader is dereferenced and left to be garbage collected. Restarting the HBase cluster is not required.

The HdfsJarModule keeps track of the latest custom filter definitions using a separate classloader for the different jar versions

Custom classloading and reloading can be a class-linking, ClassCastException minefield, but the risk here is mitigated by the highly specialized use case of Filtering. The filter is instantiated and configured per scan and its object lifecycle limited to the time it takes to do the scan in the regionserver. The example uses the child-first classloader mentioned in a previous post on ClassLoaders that searches for a configured set of URLs before delegating to its parent classloader [2].

• The example code has a performance overhead as it makes additional calls to HDFS to check for the modification time of the filter jar when a filter is first instantiated. This may be a significant factor for smaller scans. If so, the logic can be changed to check the jar less frequently.
• The code is also very naïve at this point. Updating the filter.jar in the Hadoop FS while a table scan is happening can have undesired results if the updated filters are not backward compatible. Different versions of the jar can be picked up by the RegionServers for the same scan as they check and instantiate the Filters at different times.
• Mutable static variables are discouraged in the custom Filter because they will be reinitialized when the class is reloaded dynamically.

Extensions
The example code is just a starting point for more interesting functionality tailored to different use cases. Scans using filters can also be used in MapReduce jobs and coprocessers. A short list of possible ways to extend the code:

• The most obvious weakness in the example implementation is the ProxyFilter only supports one jar. Extending that to include all jars in a filter directory will be a good start. [4]
• Different clients may expect certain versions of Filters. Some versioning and bookkeeping logic will be necessary to ensure that the ProxyFilter can serve up the correct filter to each client.
• Generalize the solution to include MapReduce scenarios that use HBase as the input source. The module can load the custom filters at the start of each map task from the MR job library instead, unloading the filters after the task ends.
• Support other JVM languages for filtering. We have tried serializing Groovy 1.6 scripts as Filters but performance was several times slower.

Using the Proxyfilter as a generic extension point for custom filters allows us to improve our performance without the hit of restarting our entire HBase cluster.

Writing correct multithreaded code is difficult, and writing Android apps is no exception. Like many mobile platforms, Android's UI framework is single threaded and requires the application developer to manage threads with no thread-safe guarantee. If your app is more complicated than "Hello, World!" you can't escape writing multithreaded code. For example, to build a smooth and responsive UI, you will have to move long running operations like network and disk IO to background threads and then return to the UI thread to update the UI.

Thankfully, Android provides some tools to make this easier such as the AsyncTask utility class and the StrictMode API. You can also use good software development practices such as adhering to strict code style and requiring careful code review of code that involve the UI thread. Unfortunately, these approaches require diligence, are prone to human error, or only catch errors at runtime.

CheckThread for Android

CheckThread is an open source project authored by Joe Conti that provides annotations and a simple static analysis tool for verifying certain contracts in multithreaded programs. It's not a brand new project and it's not very difficult to use, but it hasn't had a very high adoption for Android apps. It offers an automated alternative to exclusively using comments and code review to ensuring no bugs related to the UI thread are introduced in your code. The annotations provided by CheckThread are: @ThreadSafe, @NotThreadSafe, @ThreadConfined

ThreadSafe and NotThreadSafe are described in Java Concurrency in Practice, and CheckThread enforces the same semantics that book defines. For the purposes of this blog post, the only annotation that we'll be using is ThreadConfined.

Thread confinement is a general property of restricting data or code to access from only a single thread. A data structure confined to the stack is inherently thread confined. A method that is only ever called by a single thread is also thread confined. In Android, updates to the UI must be confined to the UI thread. In very concrete terms, this implies that any method that mutates the state of a View should only be called from the UI thread. If this policy is violated, the Android framework may throw a RuntimeException, but also may simply produce undefined behavior, depending on the specific nature of the update to the UI.

CheckThread supports defining thread policies in XML files, so while it would be possible, it's not necessary to download the source of the Android framework code and manually add annotations to it. Instead, we can simply define a general thread policy to apply to Android framework classes.

Time for an Example

The following example demonstrates how to declare a thread policy in XML, annotate a simple program and run the CheckThread analyzer to catch a couple of bugs.

CheckThread’s XML syntax supports patterns and wildcards which allows you to concisely define policies for Android framework code. In this example we define two Android specific policies. In general this file would contain more definitions for other Android framework classes and could also contain definitions for your own code.

The first policy declares that all methods in Activity and its subclasses that begin with the prefix “on” should be confined to the main thread. Since CheckThread has no built-in concept of the Android framework or of the Activity class we need to inform the static analyzer which thread will call these methods.

The second policy declares that all methods in classes ending with “View” should be confined to the main thread. The intention is to prevent calling any code that modifies that UI from any other thread than the UI thread. This is a little bit conservative since there are some methods in Android View classes that have no side-effects, but it will do for now.

Having defined the thread policy, we can turn to our application code. The example app is the rough beginnings of a Hacker News app. It fetches the RSS feed for the front page, parses the titles and displays them in a LinearLayout.

This first version is naive; it does network IO and XML parsing in Activity.onCreate. This error will definitely be caught by StrictMode, and will likely just crash the app on launch, so it would be caught early in development, but it would be even better if it were caught before the app was even run.

In this code, we make a call to the static method getHttp in the IO utility class. The details of this class and method are not important, but since it does network IO, it should be called from off the UI thread. We’ve annotated the entire class as follows:

This annotation tells CheckThread that all the methods in this class should be called from the “other” thread.

Finally, we’re ready to run the static analyzer. CheckThread provides several ways to run the analysis tool, including Eclipse and Intellij plugins, but we’ll just use the Ant tasks on the command line. This is what CheckThread outputs after we run the analyzer:

As you can see, CheckThread reports an error: we’re calling a method that should be confined to the “other” thread from a method that’s confined to “MAIN”. One solution to this problem is to start a new thread to do network IO on. We create an anonymous subclass of java.util.Thread and override run, inside of which we fetch the RSS feed, parse it and update the UI.

We’ve annotated the run method to be confined to the “other” thread. CheckThread will use this to validate the call to IO.getHttp. After running the analyzer again, we discover that there’s a new error reported:

This time, the error is caused by calling the method setText on a TextView from a different thread than the UI thread. This error is generated by the combination of our thread policy defined in XML and the annotation on the run method.

Instead, we could call the Activity.runOnUiThread with a new Runnable. The Runnable’s run method is annotated to be confined to the UI thread, since we’re passing it to a framework method that will call it from the UI thread.

Finally, CheckThread reports no errors to us. Of course that doesn’t mean that the code is bug free, static analysis of any kind has limits. We’ve just gotten some small assurance that the contracts defined in the XML policy and annotations will be held. While this example is simple, and the code we’re analyzing would be greatly simplified by using an AsyncTask, it does demonstrate the class of errors that CheckThread is designed to catch. The complete sample project is published on Github.

The Pros and Cons of Annotations

One drawback that is probably immediately obvious is the need to add annotations to a lot of your code. Specifically, CheckThread's static analysis is relatively simple, and doesn't construct a complete call graph of your code. This means that the analyzer will not generate a warning for the code below:

While this may appear to be a significant problem, it's my opinion that in practice it is not actually a deal breaker. Java already requires that the programmer write most types in code. This is seen by some as a drawback of Java (and is often cited incorrectly as a drawback of static typing in general). However there are real advantages to annotating code with type signatures, and even proponents of languages with powerful type inference will admit this, since it's usually recommended to write the type of "top-level" or publicly exported functions even if the compiler can infer the type without any hint.

The annotations that CheckThread uses are similar; they describe an important part of a method's contract, that is whether it is thread safe or should be confined to a specific thread. Requiring the programmer to write those annotations elevates the challenge of writing correct multithreaded code to be at the forefront of the programmer's mind, requiring that some thought be put into each method's contract. The use of automated static analysis makes it less likely that a comment will become stale and describe a method as thread safe when it is not.

The Future of Static Analysis

The good news is that the future of static analysis tools designed to catch multithreaded bugs is looking very bright. A recent paper published by Sai Zhang, Hao Lü, and Michael D. Ernst at the University of Washington describes a more powerful approach to analyzing multithreaded GUI programs. Their work targets Android applications as well as Java programs written using other GUI frameworks. The analyzer described in their paper specifically does construct a complete call graph of the program being analyzed. In addition, it doesn't require any annotations by the programmer and also addresses the use of reflection in building the call graph, which Android specifically uses to inflate layouts from XML. This work was published only this past summer, and the tool itself is underdocumented at the moment, but I recommend that anyone interested in this area read the paper which outlines their work quite clearly.

Many Java developers use a development environment different from the target deployment environment.  At Flurry, we have developers running OS X, Windows, and Linux, and everyone is able to contribute without thinking much about the differences of their particular operating system, or the fact that the code will be deployed on a different OS.

There's Always a Catch

Unfortunately no technology is magical and while this set up rarely causes issues, it can. One interesting case arises when the developer is using the Eclipse 1.6 compiler compliance and the target environment uses Sun's 1.6 JDK compiler.  For example at Flurry, during development we rely on Eclipse's JDT Compiler, but the code we ship gets compiled for deployment on a Jenkins server by Ant using Sun's JDK compiler. Note that both the developer and continuous integration environment are building for Java 6, but using different compilers. 

Let's See an Example

Compiling this code using javac in the Sun 1.6 JDK gives this compiler error:

"Write Once, Run Anywhere" never made any promises about using different compilers, but the fact that our toolchain was using a different compiler than our build server never bore much thought until now.

Possible Solutions

The obvious solution is to have all developers work on the same environment as where the code will be deployed, but this would defer developers from using their preferred environment and impact productivity by constraining our development options. Possible solutions we have kicked around :

1. Have ant compile using the Eclipse incremental compiler, (using flags  -Dbuild.compiler=org.eclipse.jdt.core.JDTCompilerAdapter and of course -Dant.build.javac.target=1.6). This side steps the problem by forcing the continuous integration system to use the same compiler as developer laptops, but is not ideal as this was never an intended use of the Eclipse compiler. 
2. Move to the 1.7 JDK for compilation, using a target 1.6 bytecode. This solves this particular issue, but what happens in the future?
3. Change the code to compile under Sun's JDK. This is not a bad option but will cost some speed of development found in the simplicity of Eclipse's built in system. 

My experience has been that Eclipse is a well worn path in the Java world, and its a little surprising that this hasn't come up before for us given the heavy use of generics (although there are lots of generics issues which have been logged over at bugs.sun.com, like http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6302954 which has come up for us as a related issue - the "no unique maximal instance exists" error message). 

Switching to use the Eclipse compiler for our deployable artifacts would be an unorthodox move, and I'm curious if anyone out there reading this has done that, and if so, with what results.

Here at Flurry we are big proponents of unit testing and TDD (Test-driven Development) - we believe it makes for tighter code that is less prone to behaving in unexpected ways.  We also believe in extensive Regression Testing - making sure the changes and code we add don’t break existing functionality.

Recently we have begun moving parts of the Flurry backend over to a new set of services running on NodeJs.  NodeJS is a fast, flexible application framework that is especially suited to developing RESTful API services.

Restify

To help us build these new backend components we’re using the NodeJS “Restify” framework.  Restify provides an extensive amount of low-level functionality to help you correctly implement RESTful services - much in the same way the “Express” NodeJS module helps you rapidly develop web-based applications - for those of you familiar with Express, you’d feel right at home in Restify.  We’ve found Restify to be a fantastic framework for development - easy to use, understand and implement.

Mocha

To facilitate unit testing, we’re using the Mocha Javascript unit testing framework for everything we do and have found it to be flexible and easy to use.  Mocha makes it really easy to write unit tests for your NodeJS code - so easy in fact, we decided we wanted to use Mocha for our regression testing.

After some trial and error, we have settled on a fairly simple setup, which we have found works well and is easy to implement.  The following steps outline the process, and for this small tutorial we’ll build the requisite “Hello World” API that simple returns “Hello World” when called.

Before we get started, we first want to make sure that both Restify and Mocha are installed for use in our new Node project:

Once those are installed, we’re ready to create our sample “Hello World” API service, as well as setup the Mocha regression test cases.

Here’s the contents of the app.js file that we will be using for testing:

You can see that unlike other app.js examples you may have seen, this one is very small and simply makes a function call to StartServer() which has been exported from the start_server.js file.  This function simply starts the server - we’ll cover that below.

Start up the NodeJS service in Mocha

Before we can do any regression testing against our “Hello World” service, we must first start up the Hello World service.  To do this, we are going to create a special “before” Mocha hook - this will run before any of the other regression test files are run by Mocha, and will allow us to start the service so it can be tested.

Within your directory, create a sub directory called “test”.  All of our regression test and unit test files are going to be located inside.  Create a new file called “01-start-server.js” with the following contents:

This file will be picked up as the first file in the directory (that's why we started the filename with 01), and the before() function will be executed, which in turn requires and runs our StartServer() function.   The StartServer() function is defined in the start_server.js file:

It's purpose is to actually initialize the Restify listeners and start the service.

Create a Mocha regression test

Now that we have a way to automatically start the service before we need to test it, we can go about the business of writing our regression test cases.  Since our Hello World service is so simple, we’re just going to have one test - we’re going to test to be sure our call to the /hello service returns an HTTP response code of “200”, indicating our request was “OK”:

Initializing an HTTP Client

At the very top of our file that contains the test cases for our Hello World service you can see we are using a feature of Restify - the Restify JSON Client.  The JSON Client is a small wrapper around NodeJS’s http module that makes it easy to call URL based web services that return their content in JSON format.  The JSON client will automatically JSON.parse the response body and make it available for your use (as the “data” parameter in our callback function).

Once we’ve created our client, we can then use the client to make a GET call to our /hello service URL.  If we encounter an error connecting to the service, our “err” parameter will contain the error.  The “data” parameter will contain the data returned from the call, so we will want to check that to be sure it contains the data we requested. 

Running the Regression Test

Now that we have our test in place, the next step is to actually run it, which is as easy as typing "mocha" in your project directory:

Mocha will first run the "01-start-server.js" file in the test directory - this starts our service.  Next, it will move on to the service_hello_tests.js file and run our solitary regression test.  If the service responds as we have outlined in our test, the test will be marked as passed by Mocha.

Using this simple setup we can add as many additional tests as needed - either extending our "hello" service, or writing additional test cases for new services.

Using Mocha for both unit testing and regression testing allows us to save time by only having to deal with one testing framework - Mocha is flexible enough to make running unit tests in more complex scenarios fairly straight forward and easy to do.  Now if only it could write your unit tests for you :)

Flurry is a rapidly growing company in every meaning of the word including customers, teams, and applications.  We have 650 million unique users per month from mobile applications that embed our SDKs which create over 1.6 billion transactions per day.  Our HBase cluster with over 1,000 nodes holds many petabytes of data and is rapidly growing.

While Flurry's growth is great news for our customers, employees, and investors, it creates challenges for Release Management to support the growing number of applications, the growing number of developers, and the growing number of  development and test environments.  We need to manage all of that well and do so quickly and reliably.

In short, we need continuous integration to rapidly build, deploy, and test our applications.

Deployment Tools @ Flurry
 
To support our continuous integration, we setup three core tools.

• Source Control: We use Github Enterprise to manage our source code and various configuration files. We use a variation of the Git Flow development process where all features are developed on individual branches and every merge is in the form of a pull request (which is also code reviewed).
• Continuous Build: We use Jenkins to build code and deploy to our QA and test environments and to run our JUnit tests.  Jenkins is set up to automatically run JUnit tests when code developers check-in code and when they create pull requests for their branches.  Jenkins also runs Selenium tests with SauceLabs every night against the QA environments and Production.
• Task Tracking: We use Jira (with the Greenhopper agile plugin) for ticket management for planning and tracking enhancements and bug fixes.  All three tools are well integrated with various plug-ins that allow them to share information and to trigger actions.

Challenges at Scale
 
Our setup for continuous integration has served us well but has some challenges.

• Too Many Jobs: We have more than 50 Jenkins jobs.  We have over 130 deployment scripts and more than 1,600 configuration files for the CI tools and applications.  Each new application and each new QA environment adds to the pile.  While we are whizzes at writing bash scripts and Java programs, this is clearly not scalable in the long term.
• Slow Deployments: For security reasons, our Jenkins server cannot deploy war files and config files directly to Production servers.  For Production deployments, we run a Jenkins job that copies the files built by other jobs to a server in the data center over a locked-down, one-way secure tunnel.  Ops staff then manually runs various scripts to push the files to the Production servers and restart them.  This is inefficient in terms of time and people resources.
• Test Overrun: Our JUnit test suite has over 1,000 test cases which take about an hour to run.  With the increase in the number of developers, the number of test runs triggered by their pull requests is clogging the Jenkins build server. We have biweekly releases to Production which we would like to be able to cut down to a daily basis or at least every few days.  The blocker to this is that the build, deploy, test, analyze, and fix cycle is too long to allow that.

Improving the Process: Design for Speed

The speed of an engineering team is directly related to the speed of release and deployment so we needed to get faster. We have taken a number of steps to address the challenges.

• We optimized our JUnit test cases by removing unnecessary sleep statements and stubbed out the deployments to our test CDN which reduces network wait time.
• We upgraded the build system to bigger, faster hardware and parallelized the JUnit test runs so that we can run multiple test jobs at the same time.
• We added dedicated Jenkins slave servers that can share the burden during times of heavy parallel building. 

Overall we have reduced the time to run the entire test suite to 15 minutes.

To make it easier to manage the Jenkins jobs, we removed a lot of old jobs and combined others using parameterized builds.  We renamed the remaining Jenkins jobs to follow a standard naming convention and organized them into tabbed views.  We now have a dozen jobs laid out where people can find them.

Jenkins

All of the improvement steps have helped, but we needed more fundamental changes.

Improving the Process: RPM Based Software Deployments
 
We changed our build and deployment process to use RPM repositories where every environment has its own local repository of RPMs.  In the new process, the Jenkins job builds the war files then bundles up each war file along with its install script.  The job also builds RPMs for each application's config files, email templates, data files and the config files for HBase and Hadoop.  Once all of the RPMs are built, the job rsyncs the RPMs to the target environment's repo.  It then runs ssh yum install against each server in the environment to get it to update itself.  Once all the servers are updated, the job restarts all of the servers at once.  The job is parameterized so that users can build and deploy a single app, a set of apps, all apps, or just some config files.

The developers have access to Jenkins so that they can update the QA environments at will without having to involve Release Engineering.

The RPM-based build and deployment process gives us several advantages.  The install scripts are embedded into the RPMs which reduces the cluttered hierarchy of scripts called by the Jenkins jobs.  The same tools and processes for deployments in the Dev and QA environments can now be safely used in Production.

By having a repo for each environment, we only have to deploy the RPMs once to that repo.  Each sever in the environment then pulls the RPMs from its repo.  This save a lot of time and network bandwidth for our remote environments whose servers used to get files directly from Jenkins.

RPMs support dependencies which instruct yum to deploy a group of other RPMs before deploying the given RPM.  For example, we can set an application's RPM to be dependent of the application's config file RPM, so that when we install the application, yum automatically installs the config file RPM.  The dependency feature also allows us to set up a parent RPM for each class of server where the parent RPM is dependent on all of the application RPMs that run on that class of server.  We simple execute yum install with the parent RPM, and yum downloads and installs all of the application RPMs and their dependent config file RPMs needed for that server.  In the future we will add dependencies for Java, Jetty, and various OS packages to the parent RPMs.  This will allow us to kick start a new server and fully provision it at the push of a button.

Conclusion

As with any change in process and tools, there were a few gotchas.  The Jenkins slave server was easy to set up, but there were a lot of tools and configurations needed to support our junit test runs that had to be copied from the Jenkins master server.  We also found a few places where the concurrent junit tests runs stepped on common files.

Overall, the changes have sped up and cleaned up our build and deployments.  They have allowed us to better manage what we have and to handle our future growth.

The Flurry Operations Team handles 4,428 spinning disks across 1107 servers among a team of 6 awesome operations engineers.  Since Flurry is a startup, we don’t have an on-site tech team to handle all of the hardware issues that happen at the datacenter.  As we’ve grown from 400 disks to over 4000, we've improved our process of handling servers experiencing disk hardware failures.

The most common hardware alerts we receive are from the “Self-Monitoring, Analysis and Reporting Technology”, better known as a SMART alert.   This tool tests and monitors disks and will detect and report on potential disk issue, hoping to warn the admin before a disastrous disk issue appears.  (Find out more about SMART errors).

Flurry lives and dies by the data stored in our hadoop and hbase cluster, so when a disk issue happens we need to respond quickly and decisively to prevent data loss and/or performance impacts.  We generally find that we receive critical and non-critical alerts on around 1% of active cluster disks each month, not all of which need immediate attention.

Monitoring 400 disks: SMART error detected on host

When we were a humble cluster of 100 servers it was easy to log into a box, gracefully stop the hadoop services, unmount the drive, and start the hadoop daemons back up.  Most of the actionable alerts we saw were High Read Errors or Uncorrectable Sectors, which tend to indicate a potentially fatal disk issues.

Hadoop tends to let the filesystem handle marking the sectors as bad and/or unreadable, forcing a read to occur on another replica.  Hadoop is pretty good about moving the block mapping but it can increase the read latency, and generally degrades the overall performance of the cluster.  Did I already mention that we don't like performance degradation?

Monitoring 1200 disks: Find those bad drives fast

Our first datacenter expansion in 2011 consisted of a buildout of an additional 200 servers.  Each server has 4 x 1TB drives which are utilized in the cluster, that’s 800 disks in this buildout.   During pre-production diagnostic tests, we had a 0.5% failure rate of the new disks.  

Once the initial issues were resolved, we deployed the servers into production.  The 200 new servers had an average of 2.67 disks going bad per month for the period before our next data center buildout.  Our original 400 disks started reporting 2 new issues a month.  That’s jumping from 0.3% to 0.6% disk issues a month, possibly degrading due to their age.

Monitoring 2400 disks: Throwing more servers in the mix

Four months later, we needed to double our infrastructure to handle all of the new data that we were processing for our analytics.  This time, we were adding in 1200 new disks to the cluster with the same amount of issues.  The pre-production diagnostics tests only shook out 0.02% of the bad disks.

At this time, we started seeing our drive SMART checks increasing from <1% to 1.3% failures a month.  This was also during the Holiday App Boom as seen here and here. We were spending too much time ferrying drives back and forth from the office to the datacenter and started questioning our diagnostics, urgency and response of SMART errors, and steps to replace a drive.

Our servers have temperature indicators we started to manually monitor and started noticing the new servers were running around 84°F on idle, which we tend to see higher hardware failure rates.  We started graphing the temperatures and noticed they increased to 89°F as we started to bring servers into production.  There was a lot we needed to do and not enough resources to do it, other than bug the NOCs to come up with strategies to bring us down to 77°F.

Monitoring 4800 disks: Finally some cooling relief

1. higher temperatures definitely seemed to contribute to higher failure rates
2. we had a burn in time for those first 2400 disks
3. the load across the disks had lightened after such a large expansion

Monitoring N disks: Scripts and processes to automate our busy lives

We've also improved our process for taking out servers with SMART alerts by creating a script which smartd will call when there's an issue.  In order to automate this, we've allowed the smartd check to take out servers at will. Modifying the smartd.conf script a bit, we use the check to call our shell script which does a few checks to gracefully stop the hadoop and hbase processes. This spreads out the existing data on the effected disks to healthy servers. We've also included a check to make sure the number of servers we take down does not exceed our hadoop HDFS replication factor, which further prevents the increase in the risk of removing multiple replicated blocks of data. Once all is complete, the script will notify the Operations team of the tasks performed or skipped. We have open sourced this script on our Github account here so you can fork and use it yourself.

What about the physical disks? Instead of having an engineer go down and take out disks from each server, we plan on utilizing our Remote Hands to perform that task for us, so we can focus on handling the broader-reaching issues. There were times where we batched up disk collection and engineers would carry 30 drives back to the office (walking barefoot, uphill both ways).  

As always, we're trying to do things more efficiently.  A few improvements we have in the plans include:

1. Having the script unmount the bad disk and bring the server back into production.
2. The script will email Remote Hands with the server, disk, location and issue, for them to swap the bad drive.
3. Once the disk is swapped, mount the new drive and return the server into production.
4. Adapting the script to handle other hardware alerts/issues (network cards, cables, memory, mainboard)
   

We've learned from those grueling earlier days, and continue to make hardware management a priority.  With a small team managing a large cluster, it's important to lean on automating simple, repetitive tasks as well as utilizing the services you are already paying for.  I, for one, welcome our new robotic overlords.

Prologue: The Blind Men and the Elephant
As members of the Hadoop community, we here at Flurry have noticed that at many Hadoop-related gatherings, it is not atypical for about half of the attendees to be completely new to Hadoop. Perhaps someone asked them “Should we use Hadoop for this?”, and they didn’t know how to answer. Maybe they had heard of Hadoop and planned to research it, but never found the time to follow up. Out in the Hadoop community at large, some of the answers I have heard to the question “What is Hadoop?” include the comparison of Hadoop to a tape drive, and that the most important thing about Hadoop is that it is an accelerator for Moore’s Law.

1. Split the input files and prepare input key/values from them.
2. Run one Map task per input split unit. There is no communication between Map tasks.
3. “Shuffle”: Group the output according to output keys and distribute to nodes that will run Reduce task(s). This is the only communication step.
4. Run Reduce tasks.
5. Write Output

Like HDFS, Hadoop Map/Reduce has a master/slave architecture. The master node is called the Jobtracker, and the slave nodes are called Tasktrackers. The Jobtracker distributes Map and Reduce tasks to Tasktrackers. Tasktrackers execute the Map and Reduce tasks and handle the communication tasks required to move data around between the Map and Reduce phases.

1. Files are loaded from HDFS
2. An implementation of the Hadoop interface InputFormat determines which files will be read for input. It breaks files into pieces which are implementations of the Hadoop interface InputSplit (typically FileSplit instances). A very large file may be split into several such pieces. A single Map task is created per split.
3. Each Map task generates input key-value pairs by reading its FileSplit using the  RecordReader instance provided by the InputFormat.
4. Each Map task performs computation on the input key-value pairs. There is no communication between Map tasks.
5. Each Map task calls OutputCollector.collect to forward its results to the Reduce task(s).
6. (Optional) Run a Combiner on the Map node - this is a lot like a Reduce step, but it only runs on the data on one node. It is an optimization.
7. Shuffle: outputs from Map tasks are moved to Reducers. There can be user-controlled via a custom Partitioner class.
8. Sort: the map results are sorted before passing to the Reduce step.
9. The Reduce step is invoked. The reducer recieves a key and an iterator over all map-output values associated with the key.
10. Reduce invokes OutputCollector.collect to send final values to the output file.
11. A RecordWriter instance outputs the results to files in HDFS.

Note here that the storage architecture of HDFS drives the parallelization of the Map/Reduce job. The alignment of the data distribution and computing distribution enables the work in the Map phase to run locally to the data, which in turn makes Hadoop perform very well for this type of work. Note further that Hadoop Map/Reduce does not require that its input files reside in HDFS, nor is HDFS only usable by Map/Reduce programs. The two combine to produce a powerful and simple mode for parallel computing.

Those of you familiar with the Hadoop word count example will recognize that this code is similar. This example expects input files that are lists of IP addresses, one per line. It processes all the files in the (specified) input directory in HDFS using FileInputFormat. The Map task parses each line (passed in as input values) into the four numbers of an IPv4 address, and checks to see if the address is in the major IP address block for Vatican City. If the address is in the Vatican City IP block, the Mapper outputs the IP address as the output key and 1 as the output value. The Reducer receives all of the Map output values for a given IP address in the Vatican City IP block and sums up the counts for each.

Conclusion
HDFS and Hadoop Map/Reduce combine in a supportive architecture to form a foundation for high-performance parallel computing on clusters of commodity machines. If you are trying to decide whether or not to use Hadoop for a project, asking yourself “Can we use Map/Reduce to attack this problem” is a fruitful line of inquiry that can provide a compelling and defensible answer.

Lyric is an Open Source implementation of Linear Regression in pure javascript, built on the great javascript matrix library Sylvester by James Coglan and the popular JQuery library. Feel free to use it to implement your own trending and prediction charts or read more below on how it's built and how to use it. You can download or fork it on Github (Lyric) in the Flurry Github account.

What is Linear Regression?

Given a dataset including a dependent and an explanatory variable, linear regression is a statistical technique used to create a model (equation) that best represents the relationship between the variables. This model can then be used to predict additional values that will follow the same pattern. It is a technique commonly used to add trend lines to charts of time series data and in Machine Learning to predict future values from a given training set of data.

For example, given the following set of noisy timeseries data:

Typically, linear regression is implemented using an optimization approach such as Gradient Descent which starts with a rough approximation and improves the accuracy over a large number of iterations. While such an approach will optimize the model, it can be slow based on the number of iterations required. In some cases the problem can be greatly simplified and solved in closed form using a derivation called the Normal Equation. 

Lyric uses the Normal Equation to make it fast and efficient, as it should work for most applications. 

Using Lyric

First, make sure your data is represented in the form of a 2xN Array comprised of elements with an 'x' and 'y' value. The x value should be the explanatory and the y the dependent variables.

Then you need to have Lyric determine the best equation to represent this data. The equation is known as the model and you can build it using the following:

Now that you have your model, you will want to apply it to a set of inputs. The newInput should be a 1xN array containing only the explanatory variable values you would like to calculate the dependent values. This will result in a new 2xN array which will include the resulting series. 

The following is a complete example which, given some values for the explanatory values 1 through 5, estimates the values of 6 through 8:

If you wanted to create a trend line, such as in the example above, you would simply apply the model to the same x values you provided in the input to build the model. 

For more information on using Lyric (and more advanced options) please refer to the Lyric README.

How is Lyric implemented?

Lyric implements the normal equation using a series of matrix operations implemented by Sylvester. However, before the matrix operations can be applied, the input series x (explanatory values) must be modified to represent the degree of polynomial we want to use to fit the data. To do this, we simply take the vector of x values and create a matrix where each row is the values of x raised to a given power. For example, given the power = 3 (for a 3rd degree polynomial) the output O will be of the form:

O[0] = x^0  i.e. all ones

O[1] = x^1  i.e. the same as x

O[2] = x^2  i.e. all elements of x squared

O[3] = x^3  i.e. all elements of x cubed

If you are familiar with linear algebra, you'll recognize that this represents an equation of the form:

Ax^3+Bx^2+Cx+D

Once the input is in this form, the actual matrix operations are fairly simple, following the normal equation steps. 

The resulting theta matrix is the values of the constants A, B, C and D from the above equation. Then, by multiplying future values of x by the same theta matrix we can predict y values. 

Learn More

If you're like to learn more about Linear Regression, the Machine Learning class offered by Coursera reviews it in high detail (as well as many other machine learning topics). 

Last week we released a major update to our ad serving infrastructure, which provides enhanced targeting and dynamic control of inventory. As part of this effort we decided it was time to move away from our custom binary protocol to Apache Avro for our data serialization needs. Avro provides a large feature set out of the box, however, we settled on the use of a subset of its functionality and then made a few adjustments to best suit our needs in mobile. 

Why Avro?
 
There are many good alternatives for serialization frameworks and just as many good articles that compare them to one another across several dimensions. Given that our focus will be on what made Avro a good fit for us instead of our research into other frameworks.

Avro relies on schemas. When Avro data is read, the schema used when writing it is always present. This permits each datum to be written with no per-value overheads, making serialization both fast and small. This also facilitates use with dynamic, scripting languages, since data, together with its schema, is fully self-describing.  

We maintain the invariant that the schema is always present when reading Avro data, however, unlike Avro’s RPC mechanism we never transport the schema with the data. While the Avro docs note this can be optimized to reduce the need to transport the schema in a handshake, such optimization could prove difficult and would likely not provide much added benefit for us. In mobile, we deal with many small requests from millions of devices that have transient connections. Fast, reliable transmissions with as little overhead as possible are our driving factors.

Paste that text as a definition into your header file, then get the avro schema from the provided method avro_schema_from_json. From that point write/read from the fields in the schema using the provided Avro-C methods as done in AdNetworking.m.

We recently upgraded all of our REST APIs to support Cross-Origin Resource Sharing (CORS). If you're not familiar with CORS, this means that you can now access Flurry APIs via Javascript on web pages other than flurry.com. Get started now or read on to learn more about CORS.

Why do we need CORS?

Modern Browsers implement something called a same origin policy when processing and rendering the data associated with a website. This policy says that the web page can only load resources that come from the same host (origin) as the web page itself. For example, if you load flurry.com your browser will only let the Javascript in the page load resources from flurry.com. 

Why is a same origin policy important? Cross-site scripting attacks, which take advantage of cross-origin interactions, are one of the more common methods of personal information stealing these days. A basic cross-site scripting attack would show a user a webpage which looks just like the login page for your website. When they user enters their credentials into the malicious website, believing it to be yours, the malicious site takes the credentials and uses javascript to log them into your website using AJAX. After being logged in through javascript, they can steal data, manipulate the account and change the password - all using AJAX behind the scenes without the user being aware.

Such attacks appear to the attacked service as legitimate traffic since they originate from a normal computer browser - complete with the cookies you have set. By preventing access to resources not hosted on the origin, and hence preventing AJAX from reaching another host, the browser is protecting you from this kind of attack. 

However, with the rise of HTML5, more and more web content is loaded dynamically through javascript and rendered in the browser. There are now very legitimate uses for cross-origin resource access in javascript, including widgets, applications and content management.

What is CORS?

CORS bridges the gap between security and flexibility by allowing a host to specify which resources are available from non-origin domains. This allows you to make REST APIs available for access from other domains in the browser, but not your login page. 

Adding CORS support is as simple as adding an extra HTTP response header that specifies what origins can access a given resource. To allow any domain to access a resource, you would include the following HTTP header in responses to requests for that resource:

Access-Control-Allow-Origin: *

Or, to only allow access from Flurry's website domain you would use the following:

Access-Control-Allow-Origin: http://flurry.com

Note that since the CORS header is in the response of the HTTP request, the request has already been made before your browser evaluates whether to allow access to the result. It's important to keep that in mind since even if the browser detects a CORS violation, the request will have already been processed on your servers. 

Not all browsers support CORS right now but most modern browsers do. You can read more on the CORS Wikipedia page.

Here at Flurry we make extensive use of HBase, a distributed column-store database system built on top of Apache Hadoop and HDFS. For those used to relational database systems, HBase may seem quite limited in the ways you can query data. Rows in HBase can have unlimited columns, but can only be accessed by a single row key. This is kind of like declaring a primary key on a SQL table, except you can only query the table by this key. So if you wish to look up a row by more than one column, as in SQL where you would perform a select * from table where col1 = val1 and col2 = val2, you would have to read the entire table and then filter it. As you can probably tell, reading an entire table can be an extremely costly operation, especially on the data sizes that HBase usually handles. One of the solutions to this is to use a composite key, combining multiple pieces of data into a single block.

What’s a composite key?

HBase uses basic byte arrays to store its data, in both row keys and columns. However it would be tedious to try and manipulate those in our code, so we store our keys in a container class that knows how to serialize itself. For example:

Here we have two fields forming a single key with a method to serialize it to a byte array. Now any time we want to look up or save a row, we can use this class to do so. 

While this allows us to instantly get a row if we know both the userId and applicationId, what if we want to look up multiple rows using just a subset of the fields in our row key? Because HBase stores rows sorted lexicographically by comparing the key byte arrays, we can perform a scan to effectively select on one of our index fields instead of requiring both. Using the above index as an example, if we wanted to get all rows with a specific userId:

Since we’ve set the start and stop rows of the Scanner, HBase can give us all the rows with the specified userId without having to read the whole table. However there is a limitation here. Since we write out the userId first and the applicationId second, our data is organized such that all rows with the same userId are adjacent to each other, and rows with the same applicationId but different userIds are not. Thus if we want to query by just the applicationId in this example, we need to scan the entire table. 

There are two “gotchas” to this serialization approach that makes it work as we expect it. First, we assume here that userId is greater than 0. The binary representation of a negative 2’s Complement Integer would be lexicographically after the largest positive number. So if we intend to have negative userIds, we would need to change our serialization method to preserve ordering. Secondly, DataOutput.writeUTF specifically serializes a string by first writing a short (2 bytes) for its length, then all the characters in the string. Using this serialization method, the empty string is naturally first lexicographically. If it we serialized it using a different method, such that the empty string were not first, then our scan would stop somewhere in the next userId. 

As you can see, just putting our index data in our row key is not enough. The serialization format of our components determines how well we can exploit the properties of HBase lexicographic key ordering, and the order of the fields in our serialization determines what queries we are able to make. Deciding what order to write our row key components in needs to be heuristically driven by the reads we need to perform quickly. 

How to organize your key

The primary limitation of composite keys is that you can only query efficiently by known components of the composite key in the order they are serialized. Because of this limitation I find it easiest to think of your key like a funnel. Start with the piece of data you always need to partition on, and narrow it down to the more specific data that you don’t often need to distinguish. Using the above example, if we almost always partition your data by userId, putting it in our index first is a good idea. That way we can easily select all the applicationIds for a userId, and also select a specific applicationId for a userId when we need to. However, if we are often looking up data by the applicationId for all users, then we would probably want to put the applicationId first.

As a caveat to this process, keep in mind that HBase partitions its data across region servers based on the same lexicographic ordering that gets us the behavior we’re exploiting. If your reads/writes are heavily concentrated into a few values for the first (or first few) components of your key, you will end up with poorly distributed load across region servers. HBase functions best when the distribution of reads/writes is uniform across all potential row key values. While a perfectly uniform distribution might be impossible, this should still be a consideration when constructing a composite key.

When your key gets too complex

If your queries go beyond the simple funnel model, then it’s time to consider adding another table. Those used to heavily normalized relational databases will instinctively shy away from repeating data in multiple tables, but HBase is designed to handle large amounts of data, so we need to make use of that to overcome the limitations of a single index. Adding another table that stores the same cells with a differently organized key can reduce your need to perform full table scans, which are extremely expensive time-wise, at the cost of space, which is significantly cheaper if you’re running at Hadoop scale. Using the above example, if we routinely needed to query by just userId or applicationId, we would probably want to create a second table. Assuming we still want to be able to query by both userId and applicationId, the field we use as a key for the second table would depend on the distribution of the relationship of user to application and vice versa. If a user has more applications than an application has users, then that would mean scanning a composite key of (userId, applicationId) would take longer than scanning it in (applicationId, userId) order, and vice versa.

The downside to this approach is the added complexity of ensuring that the data in both of your tables is the same. You have to ensure that whenever you write data to one table, you write it to both tables simultaneously.  It helps to have all of your HBase reading and writing encapsulated so that individual producers or consumers are not accessing the HBase client code directly, and the use of multiple tables to represent a single HBase-backed entity is opaque to its users.

If you’ve got a lot of rows

Sometimes storing large parts of your data in the key can be hazardous. Scanning multiple rows is usually more costly than reading a single row, even if that single row has many columns. So if performing a scan on the first part of your composite key often returns many rows, then you might be better off reorganizing your table to convert some of the parts of your composite key to column names. The tradeoff there is that HBase reads entire rows at a time, so while you can instruct the HBase client to only return certain columns, this will not speed up your queries. In order to do that, you need to use column families. 

Getting it right the first time

Finally, always take extreme care to pick the right row key structure for your current and future querying needs. While most developers are probably used to throwing out most of the code they write, and refactoring many times as necessary for evolving business requirements, changing the structure of an HBase table is very costly, and should be avoided at all costs. Unlike with most relational databases, where adding a new index is a simple operation with a mild performance hit, changing an HBase table’s structure requires a full rewrite of all the data in the table. Operating on the scale that you’d normally use Hadoop and HBase for, this can be extremely time consuming. Furthermore, if you can’t afford your table to be offline during the process, you need to handle the migration process while still allowing reads from the old and new tables, and writes to the new table, all while maintaining data integrity.

HBase or not HBase

All of this can make using HBase intimidating to first time users. Don't be afraid! These techniques are common to most NoSQL systems which are the future of large scale data storage. Mastering this new world allows you to unlock a massively powerful system for large data sets and perform analysis never possible before. While you need to spend more time up front on your data schema, you gain the ability to easily work with Petabytes of data and tables containing hundreds of Billions of rows. That's big data. We'll be writing more about HBase in the future, let us know if there's something you'd like us to cover. 

Here at Flurry, we constantly working to evolve our big data handling strategies. If that sounds interesting, we are hiring engineers! Please check out http://flurry.com/jobs for more information.

Working in web operations can be quite exciting when you get paged in the middle of the night to debug a problem, then burn away the night hours formulating a solution on the fly using only the resources you have at hand. It’s thrilling to make that sort of heroic save, but the business flows much more smoothly when you can prepare for the problems before they even exist. At Flurry we watch our systems closely, find patterns in traffic and systems’ behavioral changes, and generally put solutions in place before we encounter any capacity-related problems.

Doubling of outbound bandwidth from Nov-Dec 2011, Dec 24th yellow

Outside of annual events, we also keep a constant eye on our daily peak traffic rates. For example, we watch bandwidth to ensure we aren’t hitting limits on any networking choke points, and requests-per-second is a valuable metric since it helps us determine scaling requirements like overall disk capacity (each request taking up a small amount of space in our data stores) and the processing throughput our systems can achieve overall. The overall throughput includes elements like networking devices (switches, routers, load balancers) or CPU time spent handling and analyzing the incoming data.

Java is an established platform for building applications that run in a server context. Writing fast, efficient, and stable code requires effective use of algorithms, data structures, and memory management, all of which are well supported and documented by the Java developer community. However, some applications need to leverage a core feature of the JVM whose nuances are not as accessible: the Java ClassLoader.

How Does Class Loading Work?

When a class is first needed, either through access to a static field or method or a call to a constructor, the JVM attempts to load its Class instance from the ClassLoader that loaded the referencing Class instance (see note 1). ClassLoaders can be chained together hierarchically, and the default strategy is to delegate to the parent ClassLoader before attempting to load the class from the child. After being loaded, the Class instance is tracked by the ClassLoader that loaded it; it persists in memory for the life of the loader. Any attempt to load the same class again from the same loader or its children will produce the same Class instance; however, attempts to load the class from another ClassLoader (that does not delegate to the first one) can produce a second instance of a Class with the same fully qualified name. This has the potential to cause confusing ClassCastExceptions, as shown below.

Why Not Use The Default ClassLoader?

The default ClassLoader created by the Java launcher is usually sufficient for applications that are either short-lived or have a relatively small, static set of classes needed at runtime (see note 2). Applications with a large or dynamic set of dependencies can, over time, fill up the memory space allocated for Class instances – the so-called “permanent generation.” This manifests as OutOfMemory errors in the PermGen space. Increasing the PermGen allocation may work temporarily; leaking Class memory will eventually require a restart. Fortunately, there are ways to solve this problem.

Class Unloading: Managing The Not-So-Permanent Generation

The first step to using Class memory efficiently is a modular application design. This should be familiar to anyone who has investigated object memory leaks on the heap. With heap memory, object references must be partitioned so the garbage collector can free clusters of inter-related objects when they are no longer referenced from the rest of the application. Similarly, Class memory in the so-called “permanent” generation can also be reclaimed by the garbage collector when it finds clusters of inter-related Class instances with no incoming Class references (see note 3).

To demonstrate, let's consider two Java projects with one class each: a container, and a module (see note 4).

For extremely modular applications where no communication is required between the container and its modules, there is an apparently easy solution: load the module in a new ClassLoader, release references to the ClassLoader when the module is no longer needed, and let the garbage collector do its thing. The following test demonstrates this approach:

Success!

When Third Party Code Attacks

So you've been diligent about modularizing your own code, and each module runs in its own ClassLoader with no communication with the container. How could you have a Class memory leak? The answer could lie in third party code used by both the container and module:

Loading a module in its own ClassLoader is not enough to prevent Class memory leaks when using a ClassLoader with the default delegation strategy. In this case, the module's ResourceLibrary Class instance is the same as the Container's, so the ResourceLibrary's HashMap holds a reference to the module's Class instance – which references its ClassLoader, which references all other Class instances in the module. The following test demonstrates the problem, and a possible solution:

The result:

Although the test with the default ClassLoader fails due to a memory leak, the test with a stand-alone ClassLoader succeeds. Creating a ClassLoader with no parent forces it to load all Class instances itself – even for classes already loaded by another ClassLoader. The (leaky) ResourceLibrary Class instance referenced by the module is different from the one used by the container, so it gets garbage collected when its ClassLoader is released by the container – along with the rest of the module's Class instances. This fixes the Class memory leak; but what happens if you need some communication between the container and the module?

The stand-alone ClassLoader approach won't work now, because the IModule Class instance loaded in the container is different from the IModule Class instance loaded by the module. The result is a confusing ClassCastException when casting a Module to an IModule, as seen in this test:

The result:

This test introduces two new strategies for loading classes: post-delegation and conditional delegation. Post-delegation simply loads classes from the child first, then from the parent if not found. Unfortunately, that doesn't work if the module's classpath includes any of the classes shared with the container, as the test shows. This makes configuring the classpath a chore, especially if shared classes have dependencies on utility classes that both the container and module require. In this test, conditional delegation works best: it allows shared classes to be loaded from the parent ClassLoader, but sandboxes all other classes (like potentially leaky third-party code) to the child ClassLoader.

Conclusion

When using custom ClassLoaders in a Java application, there are many pitfalls to watch out for.  Dealing with ClassCastExceptions and ClassNotFoundExceptions can be frustrating if the overall design is not well planned.  Even worse, memory leaks and PermGen errors can be very difficult to reproduce and fix, especially if they only happen in long-running production systems.  Although there's not one right answer to using custom ClassLoaders, the techniques here can address most of the issues one might encounter (see note 5).

Creating a clear design and strategy for loading classes is not easy, but the reward is efficient, stable, robust server code.  The time spent planning is a small price for the time saved debugging, looking through memory dumps, and dealing with outages.  Please check out my examples on GitHub, find other ways to leak PermGen space, and post comments!

Note 1: in this article, “class” refers to a Java type; the term “Class instance” refers to a runtime object that models a class and is an instance of the type java.lang.Class. The difference is important, as one class can have many Class instances at runtime, if the application uses multiple ClassLoaders.

Note 2: There are other reasons to use custom ClassLoaders. Some common uses are in applications that dynamically update their code without restarting, applications that load classes from non-standard sources like encrypted jars, and applications that need to sandbox untrusted 3^rd party code. This article focuses on a more general scenario, but the lessons are applicable to those cases as well.

Note 3: PermGen Garbage collection was a bit tricky in older versions of the JRE. By default it was never reclaimed (hence the name). Sun used to provide an alternate garbage collection strategy (concurrent mark-sweep) that could be configured to reclaim PermGen space; support from other vendors varied. However, in recent versions of Java, PermGen collection works quite well using the default configuration.

Note 4: the complete source code for these examples is posted in the public git repository here.

Note 5: for more in-depth analysis on how to find and fix PermGen leaks, see Frank Kieviet's excellent blog series on the topic: here, here, and here.

Welcome to the Flurry Technology Blog!

We here at Flurry are lucky to be at the center of the mobile application revolution. Flurry Analytics is currently used by over 65,000 developers across over 170,000 mobile applications and tracks over 1.2 billion application sessions a day. Flurry AppCircle, our application recommendation network,  currently reaches over 300 million users every month. We spend our days working with the developers of apps and games you probably use everyday. 

Up until now we haven’t shared much about how we have built our platform to handle the vast scale and amazing rate of growth seen by our products and the mobile application ecosystem. Let’s start with some statistics about the platform:

• > 26,000 requests per second
• 2 TB of new data processed per day
• > 20 billion custom events tracked per day

Along the way we’ve had to find solutions to problems that many companies that are growing quickly have to face. This blog is our way of sharing what we’ve learned and opening a conversation with you about how to build Big Data services and mobile applications.

This blog will be technical, focusing on optimizing linux servers, efficiently writing Map Reduce jobs and advanced areas of  mobile platforms like iOS. If you’re interested in learning more about the mobile application market in general, please visit the Flurry Blog where we present market research and examine trends across the entire network of Flurry products.

We’ll be writing regular blog posts in the coming months. If there is something that you’d like to learn about or something that you think we can do better just let us know by posting here in the comments.

Enjoy.