DAta Mining 9.1 - Is DA Dying?

Status

Summary

Holiday break was fun yet otherwise uneventful.  I read the 9-11 commission report during the break.  This coming week I'll be taking my second cross-country trip for business.

I looked at how many comments are posted on DeviantArt.  Right now the number of comments posted daily is about 30% less than in 2011 and continues to decline.

After my last journal I noticed the staff did quickly remove the "folder" portion of the friends API call.  Personally I don't care too much about that, I just wish they'd make some form of announcement when they change the API.  From a developer standpoint it's very difficult to tell why a particular piece of code stops working - whether it's something that went wrong on your computer or something DA changed.

Life - Winter Break

My family took a mini vacation during the holidays.  My dad and I walked some nature trails.  I'm not one for nature, so after the first hour it gets a bit boring for me.  My dad also tends to walk twenty paces in front of me, so we don't talk much.  Since this is rather typical, I plan ahead and put a book-on-tape on my mp3 player.  I listened to that for most of the trails

I was looking for some interesting books online and the 9-11 commission is available for free on Librivox. Oddly enough, for an American publication, it has quite a few readers with British accents.  Anyway, the report gives a good insight into what happened that day and what led up to it.

If you're looking for something to keep you up at night, I'd recommend reading chapter 9.  It covers the horrors of what first responders were facing that day.  There were firemen crushed by suicide jumpers flying off the roof of the twin towers.  911 operators told trapped office works to stay put in their offices all the way up until the towers collapsed.  The report gives a detailed view of what people were thinking and how little information some people were working with.  From an organizational point of view, I think it would be extremely challenging to organize an emergency response to deal with a 911-type event with anything less than 100% perfect knowledge.  No one inside the towers or the pentagon knew how big the planes were; no one was planning around the possibility the buildings would collapse.

The report also explains the jargon quite well and the background of the players.  Bin Laden was the (if I recall correctly) 26th child of 58 kids - talk about being a middle child.  I'd equate Al Qaeda to an American defense contractor - it's a large organization but their big product is the tools and personnel for terror attacks and war.  Bin Laden seems to be more like the CEO of a start-up company.  He was responsible for putting together terror plans and getting them funded, either through scams or through direct backers.  He organized with Al Queda to get the suicide bombers for his plot.  JIHad is the extremist view of a particular religious sect, kind of like a company's mission statement that directs a company for long-term planning.

The FBI did research terrorist groups before 911, and several plans did come close to taking out Bin Laden before 9-11.  I think what interests me the most, and wasn't covered much, was the funding structure for Bin Laden.  It seems that the structure of international laws prevented the FBI from quickly and effectively freezing the assets of foreign entities, so they didn't research the flow of money that much.  However, nothing happens without those millions of dollars, so I imagine it would still be useful to understand the background and mindset of such financiers.  I would think the financiers would be a single weak point for such an organization, even if the FBI's primary tool isn't effective.  Incidentally, that's one point I've never quite fully understood about Star Wars - where does all the money come from to finance this grand civil war?  Why wouldn't either side try to attack the funding source of the other side?

Life - Business Travel

I'm writing this while traveling cross country for business.  I'm reminded how big and diverse the climate of America is when I'm wearing shorts while packing an inch-thick jacket before a six hour flight.

They need “warm bodies” to test hardware on the east coast.  It’s ironic they keep sending us to such a cold location (there were two major snow storms during the two weeks I was there).  If I put on a bunny suit one more time I’m going to turn into a rabbit.

DAta Mining

I was talking with a user a while ago about spyed's infamous red pill post back in 2003.  The question was: just how many comments did he post that day?  Certainly it must have been quite a few since you come across them all the time when looking through old profiles.  I did some quick math and came up with an estimate that he sent comments to, if I recall correctly, around 20% of all accounts on DA at the time.  But I was curious about the exact figure so I put together a profile-scraper to get comments and let it run.

I recently pulled up the data and found that there was a noticeable drop in posts over the last few years.  Summer is usually the most active time on DA.  Yet, in 2014, the daily post rate during the summer was lower than during the late Fall of 2011 (typically the slowest time of year for DA).



Due to the fluctuations in activity throughout the year, it's hard to get a good feel for the underlying trend.  I decided to clean up the data.  I broke the data down by week (so around 50 data points per year - dropped the first and last week).  I found the maximum posting rate for each week in any given year (so, for example, the maximum posting rate for the 10th week of the year was 10.1 million comments from 2012).  I scaled all the data to these weekly maximum values and plotted it.  I also used the same procedure for number of new accounts on DA and number of deviations submitted.  I removed any obvious outliers from the data (spam attacks or DA migrating all journals into deviations).



To me this is rather intersting.  There are two distinct trends.  The noise on the comments data set is low because comments are so numerous.  You can clearly see that the number of comments being posted daily has fallen 30% from its peak in 2011-2012 and continues to decline at a steady pace.

Conversely, the number of new accounts being created and artwork being posted appears to be as high as it has ever been, seeming to have hit a steady state.  I'm missing the 2014 data for deviation posting.  It'll take me about a month to collect that data, but I wanted to get this journal posted now before I fly cross-country again.

To me the fact that the commenting rate is not only static but decreasing is rather alarming.  Is DA's community shrinking?  I've heard it said that it's ok for DA to start having competitors like Tumblr, Instagram, Pinterest, etc and that it doesn't hurt DA since users just create multiple accounts on various networks.  I think you can see from this data that competition does have an impact since users spend less time on DA.  Sure users create accounts and submit art, but in the long term they spend less time on DA commenting.

Questions and Comments encouraged

dAta Mining Series
dAta Mining 1 - account type breakdown
dAta Mining 2.1 - point transfers
dAta Mining 2.2 - points followup
dAta Mining 3.1 - comments per day, number of banned/closed accounts
dAta Mining 3.2 - official verification of data mining numbers
dAta Mining 4  - group watchers
dAta Mining 5  - global activity
dAta Mining 5.2 - deviousness genders
dAta Mining 6 - user demographics
dAta Mining 7.1 - art and journal posting rates
dAta Mining 7.2 - deviation title keywords
dAta Mining 7.3 - premium memberships updated
dAta Mining 8.1 - user activity
dAta Mining 8.2 - source of submissions by join date
DAta Mining 8.3 - Friend Folders