Configs and GitHub Viz

In the case of open-source projects, you may need to dig further into what a particular configuration setting does.  If the documentation does not give you enough detailed information on the implementation, you can also trace the configuration details by searching for the file or getting the project from Github or SVN.

Some of the common Hadoop projects configuration code.

Pig configuration
Hive configuration
Sqoop configuration
Flume configuration
Kafka configuration

Github allows you to scope your searches which is useful for narrowing down your search to specific files.

Searching code is documented here.  In the search box, you can search by filename:<myfile> or <myfile> in:path to track down particular files.    You can also search by language, this searches for Scala files.

At the time of this writing, there's some interesting stats available just by looking at language of repositories in Github

• There are 1.5m Java repos with ElasticSearch, Android Universal Image Loader and Reactive Extensions for the JVM showing up as the top 3 best matches.
• There are nearly 900k Python repos with httpie, Flask, the Django framework and the Awesome Python library coming in the top 4 best matches.
• There are 400k C# repos with the .NET framework, SignalR and Mono in the top 3.
• There are 421k C repos with Linux being the best match.
• There are 60k Scala repos with PredictionIO, the Play Framework and Scala itself showing up as top 3 best matches.

Much cooler than just searching is the GitHub Visualizer created by Artem Zukov using D3js.

Apache's visualization shows an assortment of languages in their repos.

The Hive repo's contributors and file extensions

Finding Meetup Resources and Presentations

The Bay Area Hadoop User Group meetup has over 5.2k members.  Meetups like these are hugely popular and provide great resources, slides, etc.

Searching Google brings back 820+ results for PDFs related to Hadoop from Meetup files.  Lots of great information here.

Google inurl:files.meetup.com Hadoop for Hadoop resources or any other meetup topics you might find interesting.  Over 76,000 resources by searching for just inurl:files.meetup.com.

Filling up the Kindle...

Closer look at U-SQL, MIcrosoft's HiveQL

Microsoft U-SQL is the query language used on Azure Data Lake Analytics services.  Based on SCOPE and Cosmos, which has been around since at least 2008, It combines C# type / expressions functionality, schema-on-read, custom processors and reducers into a SQL-like ETL and output language.

Keywords need to be upper case.  The where clause uses C#-style == syntax.  Rows can contain up to 4MB of data per row.

U-SQL supports SQL.MAP<k,v> and SQL.ARRAY(<T>)

U-SQL supports inline C# expressions, UDFs, UDAs to custom aggregate, UDOs to generate process and consume rowsets.

U-DOs are user-defined operators build with Visual Studio.
https://azure.microsoft.com/pt-pt/documentation/articles/data-lake-analytics-u-sql-develop-user-defined-operators/

It will be interesting to see if this language makes it into SQL Server itself.  Extractors and Outputters would be highly useful to replace some of the functionality of SSIS.

I built a similar tool a few years ago for schema-on-read.  It brought CSV files into BLOB columns in SQL Server (read my article on BLOBs on SQL Server Central)  and allowed you to query them by converting to nvarchar(max), applying a schema, and then outputting to a table.

Kind of felt like a data lake at the time.... though it wasn't massively parallel and didn't have any kind of map-reduce job spinning up.  Then MS introduced the filestream object...

Virtualbox error - Kernel driver after Centos Update

On my Centos7 box, after an update I lost the kernel sources.  Virtualbox would no longer start a VM due to updates requiring a recompile.

Running  usr/sbin/rcvboxdrv setup 
showed some errors in cat /var/log/vbox-install.log
After removing & reinstalling kernel sources and running above command again, Virtualbox recompiled the kernel.

yum remove kernel-devel gcc
yum install kernel-devel gcc

Unfortunately this may remove some dependencies also, backup your environment!

Then had to reboot to avoid the "Creating a process..." message for VirtualBox.

ZSH and Oh-My-Zsh Shell Plugins

I remember a long, long time ago, in a galaxy far, far away, I played around with setting up custom DOS prompts.  Memories of Ansi.sys and custom ANSI art come streaming into my brain...

Forget all that.  On CentOS, these two commands will install the Z Shell and Oh-My-Zsh

yum install zsh
sh -c "$(wget https://raw.github.com/robbyrussell/oh-my-zsh/master/tools/install.sh -O -)"

Tab allows you to visualize potential paths, running processes, ls without hitting enter, and other awesomeness.

There are one or two Themes and Plugins available.

Some laundry lists, tricks and cheat sheets.

If you're running Windows, 720MB of Babun will get you Zsh among other things...

Hue on HDInsight and HDInsight on Linux

Microsoft might have just made Data Lakes a commodity offering.

Convergence with the Linux realm is happening again at Microsoft with the introduction of Hue on HDInsight (a graphical interface for Hadoop/HDP) and HDInsight on Linux. Hue has been around for quite awhile in the Apache realm and in most Hadoop distros, glad do see HDInsight is finally getting a user-friendly GUI.

Another announcement introduces U-SQL (see Michael Rys (@MikeDoesBigData) Introducing  U-SQL).  A SQL-like, Hive/Pig/Grep/Awk combo language to ELT+QE (Extract/Load/Transform + Query/Extract) on top of the HDInsight Big Data Lake.

The biggest announcement is the Azure Data Lake itself...

Shared folders with Virtualbox and Centos 7

Got a build error with VirtualBox add-ins and HDP Sandbox 2.2.

Building the main Guest Additions module   [FAILED]

Fixed by checking the log file for errors,  Missing kernel directory issue.

$ export KERN_DIR=/lib/modules/2.6.32-504.1.3.el6.x86_64/

$ cd /usr/src/kernels

$ ln -s /usr/src/kernels/2.6.32-573.7.1.el6.x86_64/ 2.6.32-504.1.3.el6.x86_64