Personal Data

From Nothing to Something (The Beginning)

As part of a personal project, which I’m managing (and actively working) here, I’ve decided to do a little write up on my approach, what I’m learning, and other technical things I’ve encountered. This is as much for my own memory, as it is in the hopes that I can help some others avoid the technical pitfalls that I have encountered.

The Product:

I’ve always been someone extremely interested in data, especially data that no one else is looking at. So, what is the logical place to go? The most accessible data is the data that is already out there for the grabbing. So…scraping.

What has no one else scraped, or at least scraped and aggregated AND displayed well? Game prices across different platforms. There’s aggregators for all different kinds of products (ammo, outdoor gear, etc.) but no one seems to have implemented one for games well, although they have tried.

With that goal in mind we are building a product for people to track game prices, and favorite games so that they no longer have to track news on multiple sites and check multiple web marketplaces for the best prices on games. This means we will be scraping Reddit, Twitter, and other news/social media sites, in addition to game marketplaces like Steam and Sony’s Playstore.

What I Hope to Gain:

At the end of the day, maybe we strike gold by building the coolest website and app that ever existed and people love. More realistically, I want to build a platform with which I can add data as needed for my own wants/needs. I want to become expert level  using certain libraries and frameworks, and be at a point where I’m not just a Business Intelligence and ETL developer but can develop all over the stack as needed with ease.

Also, I want to gain experience in setting up a highly performant, extensible, ETL platform off of which I end up with an app on a marketplace and at least one download. All of which will be done on a shoe-string budget. I can then use that platform to pivot and build any sort of data-centric application for whatever purpose/reason I want.

The Steps:

So, with all this being said, there are three main topics I will be writing about on a broad level.

  1. Writing scrapers with Python’s Scrapy library, which run 24/7 around the clock
  2. Writing ETL’s to a Postgresql database with near real time availability and using a budget AWS instance
  3. Serving up the data to end users using an open source tool

More updates in the coming days!

The Successes (and Issues) with Snapchat

Before I start this, full disclosure, I’m highly skeptical of the current valuations of some of the new social media companies. Of all of these, I find it hard to believe that Snapchat’s valuation is accurate at $16 billion. While it makes sense that owning the medium where people place there eyes for hours a day is worth a large value as a marketing platform, how can that high of a valuation be justified? Looking at the balance sheet of Snapchat, it appears that companies are willing to pay to get advertising in front of Snapchat users, but not nearly enough to justify the valuation. This brings me to the next question. How does a phone app that a $16 billion company was founded on look and perform?

Snapchat Landing Page

In order to answer this question, I started by signing up for an account. All things considered, they do have a nice landing page and signing up was easy. Download the app, enter an email address, and get started. I created the handle mgreis25 and was instantly opening the app on my phone and inviting to connect with friends who already had Snapchat. The user interface is intuitive, with five simple buttons that are used from the main screen to interact with the app. Straight off the bat there are two main things that make this app a cool and innovative experience.

The Good:

Screenshot_2016-02-11-20-38-42.pngThe first is the medium. The fact that people are interacting almost entirely through pictures and videos makes the communications more relatable and engaging. With over 90% of people in the US having smartphones, why don’t consumers and businesses orient pictures and video into more communications? A five second video is much more illuminating than a 140 character Tweet.

The second big feature delivers more than a twitter hashtag will ever be able to. The ability of the Live Feed compilations to show an event that is currently happening and bring you into the moment to see all the craziness, or ordinariness, at certain events is truly unique. This is unlike anything that I have experienced on a social media platform to date.

The Bad:

The big issue I see with Snapchat is the fact that the content is not permanent. After viewing the Snapchat, you cannot save for viewing later. Did you like what you just saw? Well you won’t be able to view it ever again. While it makes sense considering Snapchat’s somewhat scandalous origin, I’m not a fan.

The Future:

Taking into account the humble origins of the app, the founders and creators need to realize that in order to appeal to a wider market they need to remove the auto deleting post feature. Let the people keep their Snapchats! By having the sender choose whether the receiver is allowed to keep the Snapchat, they will be able to retain the original functionality while gaining new users who want to keep the content that is shared and received.
Snapchat DemographicsThe users of this app are primarily between 13-25 and majority female. There has yet to be a foot hold established in the older user demographic. The younger demographics appear to be using this app the best. I have yet to find/hear of a review of the content Screenshot_2016-02-09-18-55-26older users are posting, but older individuals are generally not too fond of Snapchat. Considering the secretive nature of Snapchat content sharing, the entities who are truly using this medium effectively as a marketing platform are the companies that are paying big money to be featured within the app underneath the Discover and Live sections. Snapchat is extremely proud of some of the successes that they have had, an example being the marketing for Fast and Furious. If a user clicks on the bottom right menu, they are lead to feeds for different live events and business entities. Once inside a corporation’s Snapchats, the user has the ability to watch a slew of videos and easily share with friends in one swipe.

Snapchat is a slick application for younger users and can be an effective marketing platform for high paying corporate customers, but I don’t think that valuation of $16 billion is reasonable. With Snapchat retaining no content from the users and only allowing for large companies to broadcast their message, the company is limited on how many paths it can take to generate revenue. Whereas Facebook allows for small businesses to advertise locally by paying on an ad-by-ad basis and gathers vast amounts of personal data, Snapchat does not allow this. At best I see the company rapidly losing value unless the business model changes drastically, becoming a shell of its former self. At worst, Snapchat may die as the user base ages and has less interesting content and time to share with one another, reducing the user base.