Closer look at U-SQL, MIcrosoft's HiveQL

Microsoft U-SQL is the query language used on Azure Data Lake Analytics services.  Based on SCOPE and Cosmos, which has been around since at least 2008, It combines C# type / expressions functionality, schema-on-read, custom processors and reducers into a SQL-like ETL and output language.

Keywords need to be upper case.  The where clause uses C#-style == syntax.  Rows can contain up to 4MB of data per row.

U-SQL supports SQL.MAP<k,v> and SQL.ARRAY(<T>)

U-SQL supports inline C# expressions, UDFs, UDAs to custom aggregate, UDOs to generate process and consume rowsets.

U-DOs are user-defined operators build with Visual Studio.
https://azure.microsoft.com/pt-pt/documentation/articles/data-lake-analytics-u-sql-develop-user-defined-operators/

It will be interesting to see if this language makes it into SQL Server itself.  Extractors and Outputters would be highly useful to replace some of the functionality of SSIS.

I built a similar tool a few years ago for schema-on-read.  It brought CSV files into BLOB columns in SQL Server (read my article on BLOBs on SQL Server Central)  and allowed you to query them by converting to nvarchar(max), applying a schema, and then outputting to a table.

Kind of felt like a data lake at the time.... though it wasn't massively parallel and didn't have any kind of map-reduce job spinning up.  Then MS introduced the filestream object...

Virtualbox error - Kernel driver after Centos Update

On my Centos7 box, after an update I lost the kernel sources.  Virtualbox would no longer start a VM due to updates requiring a recompile.

Running  usr/sbin/rcvboxdrv setup 
showed some errors in cat /var/log/vbox-install.log
After removing & reinstalling kernel sources and running above command again, Virtualbox recompiled the kernel.

yum remove kernel-devel gcc
yum install kernel-devel gcc

Unfortunately this may remove some dependencies also, backup your environment!

Then had to reboot to avoid the "Creating a process..." message for VirtualBox.

ZSH and Oh-My-Zsh Shell Plugins

I remember a long, long time ago, in a galaxy far, far away, I played around with setting up custom DOS prompts.  Memories of Ansi.sys and custom ANSI art come streaming into my brain...

Forget all that.  On CentOS, these two commands will install the Z Shell and Oh-My-Zsh

yum install zsh
sh -c "$(wget https://raw.github.com/robbyrussell/oh-my-zsh/master/tools/install.sh -O -)"

Tab allows you to visualize potential paths, running processes, ls without hitting enter, and other awesomeness.

There are one or two Themes and Plugins available.

Some laundry lists, tricks and cheat sheets.

If you're running Windows, 720MB of Babun will get you Zsh among other things...

Hue on HDInsight and HDInsight on Linux

Microsoft might have just made Data Lakes a commodity offering.

Convergence with the Linux realm is happening again at Microsoft with the introduction of Hue on HDInsight (a graphical interface for Hadoop/HDP) and HDInsight on Linux. Hue has been around for quite awhile in the Apache realm and in most Hadoop distros, glad do see HDInsight is finally getting a user-friendly GUI.

Another announcement introduces U-SQL (see Michael Rys (@MikeDoesBigData) Introducing  U-SQL).  A SQL-like, Hive/Pig/Grep/Awk combo language to ELT+QE (Extract/Load/Transform + Query/Extract) on top of the HDInsight Big Data Lake.

The biggest announcement is the Azure Data Lake itself...

Shared folders with Virtualbox and Centos 7

Got a build error with VirtualBox add-ins and HDP Sandbox 2.2.

Building the main Guest Additions module   [FAILED]

Fixed by checking the log file for errors,  Missing kernel directory issue.

$ export KERN_DIR=/lib/modules/2.6.32-504.1.3.el6.x86_64/

$ cd /usr/src/kernels

$ ln -s /usr/src/kernels/2.6.32-573.7.1.el6.x86_64/ 2.6.32-504.1.3.el6.x86_64

Multi tab Putty

Just as it sounds, multiple tabbed putty.
http://ttyplus.com/

And for Windows, there's Clover
http://ejie.me/

Scala kernel for Jupyter notebook and randomness

Some links for Jupyter Notebook setup and randomness...

The Jupyter Scala kernel from Alexandre Archambault
https://github.com/alexarchambault/jupyter-scala

The Jupyter Spark kernel from Brian Schlining
https://github.com/hohonuuli/sparknotebook

Something random from Bryan - convert Genome information to Midi music.
https://github.com/hohonuuli/dna-music/blob/master/README

Something else, video to ascii with akka streams?
https://github.com/hohonuuli/streamerz

Jupyter Server
https://github.com/jupyter/jupyterhub

Setting up public server
http://jupyter-notebook.readthedocs.org/en/latest/public_server.html

Java 9 Kernel for some REPL prototyping
https://github.com/Bachmann1234/java9_kernel
http://blog.takipi.com/5-features-in-java-9-that-will-change-how-you-develop-software-and-2-that-wont/

Adding R to Jupyter
http://ihrke.github.io/jupyter.html