Here’s a draft of my Galvanize DSI capstone project! See the complete project here!
Getting Satellite Data
In short, getting satellite data from the government is a big headache. The trouble includes links that loop you around to the original site, multiple sites for a given satellite/project, weird registration for data access, and verbose and obscure variable names. I guess this is part of being a data scientist.
Research from NREL on the topic
These slides include a list of resources of satellites with cloud information. When I’m done downloading the data, I will do cloud computing in the clouds on the clouds!
Big Data! Cloud Computing!
So, I’ve been learning quite a bit at Galvanize this past week about Spark and AWS. Today it culminated in deploying Spark on multiple clusters on AWS to process large files. Spark has a growing number of machine learning models available, so you can do machine learning in the cloud!
Earlier this week I deployed a small AWS instance and installed Anaconda on it. When running IPython Notebook from AWS, I used a password to protect it. It’s really freaking cool that you can remotely access IPython Notebook! The only problem that I had was that matplotlib didn’t display plots. This was solved by installing the ubuntu-desktop which loaded the qt backend necessary for matplotlib to make plots.
I’ve really got to start buckling down on this capstone project. Thanks to Galvanize instructors Isaac and Clayton for bouncing ideas today!
I just finished my fourth week at Zipfian Academy (Galvanize data science immersive) and I’ve been learning a lot about machine learning algorithms such as random forests, boosting, and support vector machines that are used in data classification. I looked at some Python code again that I was trying to figure out a few months ago that employed machine learning and I understood it better. Practice makes perfect they say. And there’s no shortage of practice at Zipfian!
While the exercises can be instructive, I learn the most when working on projects. And the biggest project at Zipfian is our capstone project. We have a month to work on a dataset (or datasets) for a specific subject. Currently, I have several ideas for my project. I had been working on EEG data before Zipfian, so the analysis uses machine learning but it is not really in the same spirit of “web scraping social data to find interesting insights” demonstration of data science flexing its muscles.
I come from a solar energy background, so it would be interesting to do a project looking at solar panel failure rate and find contributing factors. Given that food shortage is a current and future worldwide issue, I would like to look improve the efficiency of a vertical farm – an indoor, completely controlled environment for farming – by analyzing data collected by sensors such as light level, water level, pH, electricity usage, air flow, etc. This, I believe, would be very interesting and relevant as California continues to experience water shortage. However, there may not be enough information on this yet, so I may have to use data for regular old outside data. I’ll keep you posted as I hone in on a project idea. Leave a comment if you have an idea or have a lead for an interesting dataset!
In a couple of days, I’ll be leaving Chicago for San Francisco to attend a 12-week data science bootcamp. With any luck, I’ll blog about it here!
Update 5/28/2015 11:00am: I arrived in San Francisco Tuesday night. Today I get to meet the Zipfian Academy team! Yesterday I visited Galvanize and it looked pretty cool with a coffee shop on the first floor, but that’s about all I saw. I did see that they were hosting a Google event. I’m super excited to get started this Monday!
Update 5/28/2015 9:10pm: Today I had a meet-and-greet with fellow students and instructors – everybody was great! Galvanize is an awesome co-work space lined with many interesting startups. Still working on my precourse material (homework) but I expect to be done soon so that I can explore the city a little bit more before the course.