Category Archives: Big Data

Interactive PCA Scatterplots

This is an HTML interactive plot of the popular iris dataset that is compatible with Jupyter Notebook. When the paintbrush is selected, it allows you to select a subset of data to be highlighted among all of the plots. When the cross-arrow is selected, it allows you to to mouseover the data point and see information about the original data. This functionality is very useful when doing exploratory data analysis.

Here is the code:

Capstone project update

Getting Satellite Data

In short, getting satellite data from the government is a big headache. The trouble includes links that loop you around to the original site, multiple sites for a given satellite/project, weird registration for data access, and verbose and obscure variable names. I guess this is part of being a data scientist.

Research from NREL on the topic

These slides include a list of resources of satellites with cloud information. When I’m done downloading the data, I will do cloud computing in the clouds on the clouds!

(Big) Data Science Happenings

Big Data! Cloud Computing!

So, I’ve been learning quite a bit at Galvanize this past week about Spark and AWS. Today it culminated in deploying Spark on multiple clusters on AWS to process large files. Spark has a growing number of machine learning models available, so you can do machine learning in the cloud!

Earlier this week I deployed a small AWS instance and installed Anaconda on it. When running IPython Notebook from AWS, I used a password to protect it. It’s really freaking cool that you can remotely access IPython Notebook! The only problem that I had was that matplotlib didn’t display plots. This was solved by installing the ubuntu-desktop which loaded the qt backend necessary for matplotlib to make plots.

Capstone Project!

I’ve really got to start buckling down on this capstone project. Thanks to Galvanize instructors Isaac and Clayton for bouncing ideas today!

Capstone project

I just finished my fourth week at Zipfian Academy (Galvanize data science immersive) and I’ve been learning a lot about machine learning algorithms such as random forests, boosting, and support vector machines that are used in data classification. I looked at some Python code again that I was trying to figure out a few months ago that employed machine learning and I understood it better. Practice makes perfect they say. And there’s no shortage of practice at Zipfian!

While the exercises can be instructive, I learn the most when working on projects. And the biggest project at Zipfian is our capstone project. We have a month to work on a dataset (or datasets) for a specific subject. Currently, I have several ideas for my project. I had been working on EEG data before Zipfian, so the analysis uses machine learning but it is not really in the same spirit of “web scraping social data to find interesting insights” demonstration of data science flexing its muscles.

I come from a solar energy background, so it would be interesting to do a project looking at solar panel failure rate and find contributing factors. Given that food shortage is a current and future worldwide issue, I would like to look improve the efficiency of a vertical farm – an indoor, completely controlled environment for farming – by analyzing data collected by sensors such as light level, water level, pH, electricity usage, air flow, etc. This, I believe, would be very interesting and relevant as California continues to experience water shortage. However, there may not be enough information on this yet, so I may have to use data for regular old outside data. I’ll keep you posted as I hone in on a project idea. Leave a comment if you have an idea or have a lead for an interesting dataset!