This was my RescueTime recorded work-day yesterday. I woke up at 3am and couldn't go back to sleep. I got into a pretty good flow, and didn't stop until 11 to take a shower and head in to work.
I think the smartest Internet startup entrepreneurs model their businesses as a social experiment. They understand the dynamics of where their users are coming from, how long they stick around, and how much money they can make from them. Suprisingly, you can actually predict the outcome (peak user base and peak revenue) based on collecting data on your current users' behavior.
I'm giving the following talk at Startup Weekend tomorrow (8/29/09). I've also attached a spreadsheet from which anyone can plug in the actual data for their Freemium web service and find out how their site will do (assuming the fundementals don't change).
Submitted by zzelinski on Mon, 08/03/2009 - 11:34.
10...9...8...7...6...5...
Mapping technologies and mobile technologies have often been at odds with each other. Mobile technologies require compact, high performance applications, and maps require massive amounts of data and special interfaces. Recently, mapping technologies have been optimized for the increased power of modern mobile devices. This talk will give an overview of what Google offers to mobile developers, including Android MapView and iPhone MapKit, and will demonstrate how to get started with the new Google Maps API version 3, which has been optimized for mobile devices.
Who: Mano Marks, Google When: 6:00 pm, Tuesday September 1st 2009 Where: StartPad offices -- 811 First Avenue, Suite 480, Seattle, WA 98104 Cost: Free! (Pizza and drinks will be served)
About the Presenter: Mano joined Google's Geo API team in 2006. He helps people all over the world develop and deploy their content in KML and Google Maps, working with large companies, small startups, and international aid organizations. Before coming to Google, Mano had an eclectic career that involved getting a Masters in History, a Masters in Information Management and Systems, and working as a data manager in small and mid-sized organizations for over a decade.
**This talk was followed by a Google Geo Hackathon on Thursday, September 3rd at Google Fremont! Find out more.**
Submitted by zzelinski on Mon, 07/20/2009 - 14:14.
10...9...8...7...6...
The ACLU, emerging technology, and business leaders may sound like the setup of a bad joke, but it’s not. Join us to learn more about the ACLU of Northern California’s soon-to-be-announced “Demand Your dotRights” campaign linking online privacy and government surveillance. We will focus in particular on the role that companies play in establishing and safeguarding the privacy and free speech rights of their users – and how companies can benefit from having strong policies that favor these rights, as described in our recent publication, “Privacy and Free Speech: It’s Good for Business”
Who: Chris Conley, ACLU of Northern California When: 6:00 pm, Tuesday August 4th, 2009 Where: StartPad offices -- 811 First Avenue, Suite 480, Seattle, WA 98104 Cost: Free! (Pizza and drinks will be served)
About the Presenter: Chris Conley is the Technology & Civil Liberties Fellow at the ACLU of Northern California where he focuses on launching the organization’s new online privacy campaign, “Demand Your dotRights.” He started out on the technology side, earning a B.S.E. summa cum laude in Electrical Engineering from The University of Michigan and a S.M. in Computer Science from M.I.T., and spending several years as a software developer and consultant. He later received his J.D. cum laude in 2007 from Harvard Law School, where he was Editor-in-Chief of the Harvard Journal of Law & Technology. Prior to joining the ACLU in September 2008, he was a Resident Fellow at the Berkman Center for Internet & Society at Harvard University, where he studied international Internet censorship and surveillance.
I'm a Google AppEngine developer (Go2.me), as well as a user of a friend's AppEngine application (puzzazz.com). On Saturday morning, I noticed that I could not Sign In to any AppEngine application - I was greeted with a very opaque "Server Error". This looked like a very serious problem - so I waited a few hours thinking that Google would be "on it", and tried again ... still broken.
Now I start looking for the official Google report on this bug. There are several ways that Google uses to communicate with developers on AppEngine:
To my dismay, none of these pages had ANYTHING to say about the service outage. Doing some more investigation, I found someone report on the developer group that they could not log in on iPhone. I had been trying to log in on my Android (Google) phone; it had not occurred to me that an error like this would be specific to mobile phone access, but it was. I could log in normally via a browser, but not via a mobile phone.
With no real way to reach a person at Google, I had to just wait and hope that Google would recognize this error and fix it. Finally, on Monday afternoon, a Google engineer responded on the forum that they were looking into the issue. Within hours, the problem was fixed.
Mature, well run, engineering organizations have several mechanisms to combat failures like this one:
The use of automated testing before releasing software that tests the broad range of capabilities (including emulation of Sign In via a mobile web browser, for example).
A well defined error reporting and response system to tell users when problems are identified, and give an estimate of their repair.
While you could argue that AppEngine is a "beta" product, I was surprised and disappointed by this incident for several reasons:
Apparently every AppEngine application lost the ability for mobile (iPhone/Android) users to log in via Google Sign in.
This problem persisted for 60+ hours.
There seemed no clear way for developers to report this to Google (there were two bug reports on the Google AppEngine Group - but that's not guaranteed to be monitored or create an Issue Ticket).
There were no errors logged in our error logs on AppEngine (though 500 errors were clearly being returned and so Google should have "known" about this as soon as they started happening).
At no time was this issue ever acknowledged on the App Engine service status page.
I decided to send my thoughts on this to the engineers who fixed this problem. I was pleasantly surprised to hear back from Chris Beckmann:
Thanks for reaching out to us. I'm one of the product managers for App Engine.
We take the login bug seriously and it was the topic of significant discussion at our teamwide engineering meeting today. As you can imagine, many of Google's external services are actually composed of the efforts of many underlying teams, and in this case a change made by another team affected login for App Engine apps. Generally speaking, there are several methods for catching changes that break another service including static tests, however, in this case, it managed to slip by the other team undetected.
Concretely, we're working with the other Google team to make sure they incorporate additional tests specific to App Engine, as well figuring out some additional monitoring within the App Engine service to discover these kind of errors. From a monitoring perspective, logins are relatively rare compared to overall traffic, and since this outage affected only mobile logins, the increase in 500s didn't immediately raise the alarm.
You also touched on an important question regarding why this wasn't picked up earlier as there was a thread on the group. Just by way of explanation, the problem really emerged over the weekend when there's fewer internal folks monitoring the group, so it's more difficult to separate real problems from typical discussions or other noise. That said, we're working on making improvements to how we react to community input, regardless of the day or hour at which it is received. Our best option right now is to post to the group with a link to the Issue Tracker (http://code.google.com/p/googleappengine/issues/list) so that other developers can verify whether they are also experiencing these problems and escalate in general.
I hope that addresses some of your concerns. Thanks for reaching out to us and feel free to contact me with any other feedback or questions.
What a great response. While recognizing a failure in their systems, they reassured me that they are taking the problem seriously, and are working hard to address it to avoid repeating the mistake in the future. I also get the feeling that they really are the high quality development organization that I would expect from Google, and that they are trying to do things "right". Chris even agreed to let me share his email publicly.
This just goes to show the power of good customer service, and treating customers with respect and openness can go a long way in building trust and loyalty in their brand.