TL;DW for Clojure Data Science

Edmund Jackson talked at the 2012 Clojure/Conj, and you can see his talk here.

I took these notes as I watched it:

1. What is "data science"?
1. "That realm of endeavor that requires, simultaneously, advanced computational and statistical methods."
2. Some people aren't sure whether "data science" is a thing, or just data analysis dressed up with a fancy name. That question amuses me.
2. What's new, such that everybody suddenly cares about data science?
1. widely available computing resources, open source tools such as R, and large amounts of data available in private companies and in public
2. Compares to early days of Linux, when there was a bunch of new stuff that everybody could hack on
3. Interactive tools aren't enough; you're not taking some data, analyzing it, and coming back with the answer. You need platform features like native language speed, data structures, language constructs, connectivity, and QC in order to embed your analysis in business processes.
4. The tools with better analysis features (e.g., R, Mathematica) lack the platform features, and the tools with better platform features (he focuses primarily on C++ as his example here) lack the analysis features.
5. Python is in the sweet spot, with platform features and (via numpy, scipy, and pandas) analysis features. But:
1. It's full of mutable data!
2. The mode of expression in imperative languages poorly matches the content of expression when you're dealing with maths.
6. F#, Scala, and Clojure are all functional, and therefore (immutable data, more natural expression of maths) better alternatives than Python.
7. Clojure yay! points:
1. Native: Incanter, Storm, Cascalog, Datomic
2. JVM: Mahout (ML on Hadoop), jBLAS, Weka (Java lib with many ML algorithms)
3. Interop: Rincanter (call out to R), JNI
8. From here he goes into calculating the entropy of a distribution, and the relative entropy of different distributions.
9. Demonstrates using relative entropy fns in Datomic queries