Garbage In, Garbage In, Garbage In

Many projects in the Apache ecosystem run Java.  One of the places developers spend time in when dealing with performance issues is the Java Virtual Machine's (JVM) Garbage Collection options.  When the heap becomes full, garbage is collected.

In this past, I have seen that .NET apps that explicitly call the garbage collector improved performance, especially when dealing with black-box code that doesn't dispose of objects itself nicely or bloats memory due to poor design.  I have also seen where it will destroy performance for every .NET application on the machine.

In .NET 4.6 RC,

• Enhancements to garbage collection (GC)
  The GC class now includes TryStartNoGCRegion and EndNoGCRegion methods that allow you to disallow garbage collection during the execution of a critical path.
  A new overload of the GC.Collect(Int32, GCCollectionMode, Boolean, Boolean) method allows you to control whether both the small object heap and the large object heap are swept and compacted or swept only. 

So it seems people are still trying to trick the garbage truck to show up on the wrong day to pick up that rusty mattress or old toilet, or make sure that the garbage truck doesn't pass by when they're in the middle of running out the door with a million Glad bags.

http://stackoverflow.com/questions/118633/whats-so-wrong-about-using-gc-collect

At this point, suppose that performance plays a fundamental role and the slightest alteration in the program's flow could bring catastrophic consequences. Object creation is then reduced to the minimum possible by using object pools and the such but then, the GC chimes in unexpectedly and throws it all away, and someone dies.

Well that got dark really fast, stackoverflow.

Oracle has a good document around the concepts of the Heap and the Nursery.  When the nursery fills up, the older ones leave to public school.  When public school fills up, the oldest are forced out into the real world.

https://docs.oracle.com/cd/E13150_01/jrockit_jvm/jrockit/geninfo/diagnos/garbage_collect.html

Databricks, the Spark folks, and Intel, recently posted a great article about how GC works with Spark and how to tune Spark instances for optimized JVM garbage collection which inspired (and augmented some content for) this post.

https://databricks.com/blog/2015/05/28/tuning-java-garbage-collection-for-spark-applications.html

$500 in Google Cloud Credit with Free MapR Hadoop Training

What does MapR get from Google?  $110 million in capital financing.

What do you get a Google Cloud Engine $500 free credit with MapR training?  Apparently quite a bit...

Compute Engine

5 x Servers

• 434.524 total hours per month
• VM class: Regular
• Instance type: n1-highmem-16
• Region: United States
• Total Estimated Cost: $438.00

Persistent Disk

• SSD storage: 0 GB
• Storage: 100 GB
• Snapshot storage: 0 GB
• $4.00

GCE Network Bandwidth

• Egress - Americas/EMEA: 200 GB
• Egress - Asia/Pacific: 0 GB
• Egress - Australia: 0 GB
• Egress - China: 0 GB
• Google Cloud Interconnect United States: 0 GB
• Google Cloud Interconnect Europe: 0 GB
• Google Cloud Interconnect Asia/Pacific: 0 GB
• Egress to a different Zone in the same Region: 0 GB
• Egress to a different Region within the US: 0 GB
• $24.00

Monthly total: $466.00

If you don't want 128GB of ram and 5 servers in your cluster, you could be a peon and buy some pre-emptible Instances to go the cheaper route.

https://cloud.google.com/compute/docs/instances/preemptible

Hadoop / HBase / Drill Training Link here...
https://www.mapr.com/company/press-releases/mapr-collaborates-google-cloud-platform-offer-500-credit-resources-mapr-fre-0

Sandbox VM download here
https://www.mapr.com/products/mapr-sandbox-hadoop/download-sandbox-drill

Elements of Scale

Amazing, comprehensive article around relational, NoSQL, and many other approaches to reading and writing information.

http://www.benstopford.com/2015/04/28/elements-of-scale-composing-and-scaling-data-platforms/

If a relational database can't solve a specific problem efficiently and timely, perhaps throwing the kitchen sink, or data platform at it could...

Presentations from the Apache Accumulo Summit 2014

"Up to 10 quadrillion entries in a single table"

That's 10,000,000,000,000,000 rows.

Sounds like a limitation to me...

Presentations from the Accumulo Summit.  Accumulo is the Apache implementation of Google BigTable.  http://www.slideshare.net/AccumuloSummit

Information on Hawq & the Accumulo Connector, Ambari, Slider, YARN, TinkerPop, etc.

The TinkerPop stack with Blueprints is my favourite project suite to read about, if only because of the cartoon mascots in their architecture diagrams.  Every project team needs a graphics designer like Ketrina Yim, improve morale and adoption in the community.  So many projects could benefit from the experiences of a designer rather than a programmer when it comes to building user-friendly applications and branding.  Tech projects often take themselves much too seriously.

Would you rather learn more about Graph Server XI or Rexster?  I thought so...

- Ketrina Yim, TinkerPop stack

Visit for the information, stay for the nice graphs about Accumulo adoption in the community and the 172-slide deck from Aaron Cordova on scaling Accumulo clusters, with lots of examples of truly "Big Data".

• 1 Year of Large Hadron Collider = 15PB
• 1 Year of Twitter = 182 trillion tweets & 483TB
• Netflix master = 3.14 PB (Pie!)
• WoW = 1.3 PB
• InternetArchive = 15 PB

That's not big data, This! is big data....

Hadoop'able Materialized Views

The smart teams working on Apache Optiq are promoting in-memory, discardable, materialized views as a potential source of performance improvements when dealing with large distributed datasets in Hadoop.  Why not use up all that memory sitting in your Hadoop cluster?

A presentation on DMMQ here.
http://www.slideshare.net/julianhyde/discardable-inmemory-materialized-queries-with-hadoop

The DMMQ blog at Hortonworks
http://hortonworks.com/blog/dmmq/

The DDM blog at Hortonworks
http://hortonworks.com/blog/ddm/

Querying Hive, the "Microsoft Way"

Apache Hive is an abstraction tool for generation of MapReduce jobs in Hadoop, and a lightweight data warehousing tool providing schema on read capabilities and storage of metadata in its metastore.  By default, it is stored in MySQL.  In Microsoft Azure HDInsight, it is stored in Azure SQL.

Using a "SQL-like" HiveQL language you can write queries that can access data stored in a Hadoop cluster, either within the Hive warehouse (predefined metadata) or in external files (text or binary).

Microsoft has LINQtoHive support through the Hadoop SDK, for those developers who enjoy using LINQ as an abstraction to their data.

Go get LinqPad and try it out!

If you're lucky enough to already have the LinqPad Premium edition, you can do a NuGet on the assembly required directly from the Query Properties pane.

You'll need the following assemblies for this demo query.  For testing purposes, I just installed them using Nuget in Visual Studio, then browsed to the folder containing the assembly in LinqPad.

install-package Microsoft.Hadoop.Hive 
Install-Package Microsoft.AspNet.WebApi.Client -Version 4.0.20710
Install-Package Newtonsoft.Json

Once you've added the assemblies, you can run this C# statement, after replacing the URL, User ID & Password.  The port is the WebHCat port where Hive / HCatalog is available.

var db = new HiveConnection(
            webHCatUri: new Uri("http://<myhadoopclusterurl>:50111"),
            userName: (string) "<myuserid>", password: (string) "<mypassword>");

var result = db.ExecuteHiveQuery("select * from access_logs");
result.Wait();

LinqPad is awesomeness...

Hadoop Summit 2014 presentations

Slides and presentations from the Hadoop Summit 2014 in San Jose are here.

To me, the most fascinating was Hadoop 2 @Twitter Elephant Scale and the size of the data being worked upon during the migration.