Hi John, as I read through this for the fourth time, I think this is the post of the year. There have been so many hints of what you describe over the past few years, and so many factors working in support - mobile computing, cross domain login, embed code, non programmers doing programmer things, facebook / twitter / friendfeed, the newsfeed / river format, real identity associated with UGC, API culture - but like you, for the first time, I'm starting to feel it come together on a web wide basis. Exciting!
billV23
· 2 months ago
I agree that is very exciting to think about, even if it is a little bit hard to grasp. Being able to wrap your mind around it with the whole picture remaining clear is the trick. I'm going to have to read through it a few more times myself. free money for bills?
fredwilson
· 7 months ago
oh man, you've written way more than a blog post here John
this is a treatise and an investment strategy and a mantra all in one
the question you ask near the end "waht do you do when people swarm" is the same question i asked you when you showed me chartbeat, "this is awesome, but what do i do with this information"
i invested in disqus for a bunch of reasons, but the main one was the email me the comments to my blackberry feature. when the swarm hits, you want to engage with it and the tethered to the computer model just doesn't work for that
the one thing that you really didn't weave into this is the role of mobile.
twitter's shortcode is the most used shortcode in the US. that's a telling piece of data.
this real time stream is with us all the time, and it would not happen without mobility
daryn
· 7 months ago
"twitter's shortcode is the most used shortcode in the US. that's a telling piece of data." -- Wow, this is a very surprising (and interesting) fact. Is shortcode traffic data publicly available? I'm curious what that tail looks like.
John - really excellent post. I need to read it a few more times and digest. Thanks!
fredwilson
· 7 months ago
i heard it second or third hand daryn. i'd love to see the data confirming or not confirming it.
markslater
· 7 months ago
this is good.
Shortcodes are mobile command line interfaces to the next wave of RT social apps.
opentable for instance should shortcode. Restaurants, viewing their bookings can present '2 hour specials' that get agregated and bullhorned. i can type 'on' top SC XXXXX at 6pm and receive a list of my 6 favorites offering immediate incentives.... I can complete my rezzi and have it dynamically shares across my other activity streams and so on.....
the one piece missing is auditable geolocation.
there are way to many use cases with SCs and RT - many of them will become very large businesses.
Johnborthwick
· 7 months ago
Thank you Fred. We are gradually working to figure out the pieces here — mobile is a key piece to the puzzle. Devices shouldnt be considered as separate or distinct from sites — its all data and its becomes more tightly woven, in more interesting ways. Didnt know data on Twitter's short code -- Fascinating times.
I also realized that the prezi thing got disabled this am, its back now — adds a little more data to the post. As an aside — I’m loving prezi — I think the zoom metaphor it uses (think Google earth for data) is a part of this puzzle, I want to be able to zoom in and out of these data streams. See what we did on betaworks.com :)
danrua
· 7 months ago
nice prezi. most of those datapoints involved the shift from google search traffic to facebook/twitter discovery traffic. have you seen similar data on discovery vs. search, but at a higher level -- comparing goog search referrals to social media discovery referrals (including blogs, microblogs, social networks, RSS)? I've got to believe the shift to discovery stream referrals is even larger than represented by tw/fb alone.
fredwilson
· 7 months ago
the prezi thing was not working this morning. i just looked at it. such a cool way to present things. i wish i could have used that yesterday in my talk at google.
but how do you cut and paste stuff out of prezi? i wanted to share one of those on tumblr?
danrua
· 7 months ago
Great post John.
Chartbeat reminds me of another, possibly larger flow in streams I've been investigating: implicit data/feeds. Most streams today are explicitly created by users, either by creating content, making a friend, saving a favorite etc. For every explicit action of a user, there are probably 100+ implicit datapoints from usage; whether that is a page visit, a scroll, a video/shopping abandon etc. I also believe such implicit data is less skewed by the digerati that creates a significant chunk of today's explicit dataset -- providing a more accurate view of online likes/dislikes/activities.
Therefore, I applaud your work on Chartbeat and encourage you to turn that data inside out by connecting with cross-site implicit datapoints. That becomes less about site/page-specific webmaster analytics, and more about surfer/user (anonymous or not, depending upon user choice) implicit activity feeds flowing into a larger stream...keep me posted if u go that route.
Johnborthwick
· 7 months ago
The implicit data point is a really good one. The bit.ly team are storing all this data on each link that is sent through the system -- click data (on the + page), entity data (from calais and other semantic extraction tools), location data etc. etc. I just asked Nate and the hash specific data is approx 20 gig and then there is approx 200 gig of metrics data (real time and roll up's). There is a lot of implicit value here.
theschnaz
· 7 months ago
Interesting post. Traffic as "chunks" was most illuminating for me.
@fredwilson, I think your comment is spot-on for the author of a blog. When the swarm hits, the author can respond.
What about the readers or the people leaving the comments? I found this post from a passed link, so there has to be other people reading as I'm typing. It took me a few mins to read the entire post. From the time I arrived to the time I made it to the comments section, did someone else comment? Can I chat with people who are on this page too? Can real-time conversations spontaneously breakout in a post's comment section?
This is just the beginning, it's very exciting.
(I refreshed the page to check, 3 new comments.)
Johnborthwick
· 7 months ago
Agree -- there are people here right now and figuring out how to interact with them in a meaningful way, other than comment boards, is a good question. It's also recursively fun -- given that the post is partially about this very question. An illustration of this -- real time --- see the alert below I just got from Chartbeat that brought me back to the post.
We did try live chat -- bubble chat on the page (firef.ly) -- but the interactions were thin and fleeting, it was more game like than a forum for conversation -- and many people dont actually want to chat.
-------- what i just got in email ----------
A chartbeat alert has been triggered. The alert is:
I see what you mean. I want to chat with others while they're on the page, but in a structured non-chatroom way. Without going overboard, the easiest way to describe this is real time disqus. If I'm on a page I can see new comments as they appear in real time. If I respond, others see my comments too. This side steps the chat room issue.
dpakman
· 7 months ago
As always, really enjoyed your thoughts and exploration, John.
A few thoughts come to mind: - We should tip our hats to Amazon Web Services (and burstable cloud infrastructure) which is what allows sites built for 40 visitors to suddenly accommodate a swarm. The architecture underneath our communities has quietly evolved to allow exactly this phenomena even before social media created the digital social pathways to create the swarms themselves. - I agree that a "bookmark" or "remember" button for the RT web is needed. Right now we have a binary digital history: the present and the saved (i.e., the way back machine). But just saving the nugget won't be enough to re-create the stream at that moment! Won't what came before it, what inspired it, who commented on it, your responses to them, and what came after it -- isn't that really what you want to remember? Can we ever "bookmark" the stream? I don't think we will be able. We'll pull nuggets out, but the totality of the moment will have moved on.
I am inspired by how much is unclear about the RT future.
Johnborthwick
· 7 months ago
DP -- Agree on AWS -- the platfrom has help drive much of the stream related innovation. On bookmarks not sure if you want to bookmark a stream, rather flag items within a stream
Ted Rheingold
· 7 months ago
Wow. Bravo.
Couple thoughts:
* I think posts (the element behind RSS) would be the first stream. Even on webpage you'd read them as separate but consecutive. But that would be preceded by forums, chat rooms, IRC and mail lists, so maybe I'll leave that up to those that take these things to the mat to decide ;)
* I think we're seeing 2 distinct types of stream users. Those that want to hold the rope and those that watch it go by when they look at it. For lots of people twitter or facebook or flickr doesn't provide enough meaning if they follow too much. FriendFeed seems to be very binary, people love it or don't use, but rarely use just now and then.
* Asynchronous connections are much more (adult) human like. Thanks for that insight.
* You just made me think handheld computers have really blow the doors off the page-based paradigm and that we're a lot closer to an always-on Internet experience then is generally believed ;)
Thanks for the deep thoughts.
Johnborthwick
· 7 months ago
Its interesting that the primary attribute of broadband was not speed (as expected) but always on.
terrycojones
· 7 months ago
When we get our Alpha out (coming soon, I promise), I will give you a remember button. A useful one.
Thanks for writing all those thoughts up. I have too many comments to comment, but you know I agree with much/all of this, and....... well, you know.
William Mougayar
· 7 months ago
Brilliant articulation of where this is going: the web and our web-lives are being re-configured. What gets passed is more important than what gets printed. But we’ll need to occasionally freeze-frame the stream within a given context to allow us to process and digest.
I think an exciting part will be the merging of the social+now+context/structured web (using your terms) together via a “personal API” or semantic lens of sorts, and allowing us to manipulate that knowledge to gain greater insights.
rdlove
· 7 months ago
Thanks for the insightful thoughts; it's encouraging to find someone who thinks like me (occasional summaries on important subjects that run consistently through our daily life). I'm excited to see how society adjusts to these changes as well as to find out what "infrastructure" will be made to help the human mind process them!
Rutger
· 7 months ago
I agree filtering is and will be key. Not only do I want to be able to filter on importance but also on the channel I receive the information in. Cheers.
ericrumsey
· 7 months ago
>> Figuring out where the stream metaphor came from is hard<<
Salman Rushdie saw it coming, in 1990 -- The Stream of Stories that he envisions is : "made up of a thousand thousand thousand and one different currents, each one a different color, weaving in and out of one another like a liquid tapestry of breathtaking complexity." Sounds like he's talking about the story we'll all living every day on the Web (Stream?).
David Sanger
· 7 months ago
>>Figuring out where the stream metaphor came from is hard <<
People used to watch it like hawks, looking for the next uptick or downtick, the pulse of the financial nation.
And in the back room traders were feeding it their messages.
Johnborthwick
· 7 months ago
Interesting. Others have suggested Gelernter and Freeman ie: Lifestreams -- or @ericrumsey (below) suggested Rushdie (http://bit.ly/l2RMp).
David Sanger
· 7 months ago
John - I'd like to see better integration of the OS and the "stuff" I remember. Real time of course. and accessible on my different devices.
• folders on my desktop for people, websites, posts, comments, photos and feeds I like? realtime, clickable not static.
• drag and drop so I can use all these in messaging, posting, commenting and tagging?
• why should I ever have to type <a href= or use the link-chain-thingy? again drag/grab and drop from desktop to WP, email?
• real time means these are always current, synched (like Plaxo) and the real data is stored who knows where.
marianasoffer
· 7 months ago
I loved the second quote of Eno, I think I can also relate it to the idea that the media shapes the message (or becomes) which McLuhlan started talking about.
A Kundera quote, dealing with time acceleration: -Why has the pleasure of slowness disappeared? Ah, where have they gone, the amblers of yesteryear? Where have they gone, those loafing heroes of folk song, those vagabonds who roam from one mill to another and bed down under the stars? Have they vanished along with footpaths, with grasslands and clearings, with nature? There is a Czech proverb that describes their easy indolence by a metaphor: 'they are gazing at God's windows.' A person gazing at God's windows is not bored; he is happy. In our world, indolence has turned into having nothing to do, which is a completely different thing: a person with nothing to do is frustrated, bored, is constantly searching for an activity he lacks.(kundera)
Interesting comparison about both kind of streams. It's important to have these facts clear in order to realize what can be done and where can we go.
About Swarms: Traffic is not smooth is has spikes, like swarms, they move in spikes when something happens given the high speed at which the information is transmitted between its members, spikes are produced by a relevant fact or a mere fake rumor which spreads through the communication channels reaching individuals at an exponential rate of Grouth. check out my post about streams: singyourownlullaby.blogspot.com
Hampers
· 5 months ago
An insight worthy to be read not only once but more than twice. Indeed this hitech "distribution" scheme is going to the point of no longer bothering if the following is true or just a passing note. The absence of "truthful feelings" sometimes can no longer be felt. As what the twittering is all about...we are not 100% sure if our "friends" or followers really are following us...who cares? Hmmm...I'll just do what I think is better for me and for the others - no matter what the circumstances are.
Traveller_Adventure
· 5 months ago
What I can say is very nice and helpful as well as informative post...really help me very much more!! Thanks.. Cheers, Buat Duit Dengan Blog
Deck Tile
· 4 months ago
i agree with you david! I think streams will become de-localized and a lot more structured
Social Bookmarking
· 4 months ago
Network is the most powerful method these days. If you know lots of powerful people, you're powerful.
jual_rumah
· 4 months ago
This is the most heavy post I've ever found but I like it. Very deep insight and sharp analysis. I love the term of distribution, content, real time stream. Thank you for the valuable thought.
Essay Help
· 4 months ago
The functional identity of things is a product of our interaction with them. And our own identities are products of our interaction with everything else. Now a lot of cultures far more “primitive” than ours take this entirely for granted...
Thanks and Regards
Hostgator Review
· 3 months ago
Thanks for the insightful thoughts; it's encouraging to find someone who thinks like me (occasional summaries on important subjects that run consistently through our daily life). I'm excited to see how society adjusts to these changes as well as to find out what "infrastructure" will be made to help the human mind process them!
Thanks and Regards
Crush the Castle
· 3 months ago
I have no real idea whats going to emerge here but cursory thoughts include making sure the author is present to manage comments etc., build in a dynamic mechanism to alert the crowd to other related items? Beyond that its not clear to me but I think its a question that will be answered — since users are asking it.
Thanks and Regards
David Semeria
· 7 months ago
Yup. I think streams will become de-localized and a lot more structured, with embedded metadata, permissions and (optional) usage tolls: http://lmframework.com/page.php?id=vision
All Comments
this is a treatise and an investment strategy and a mantra all in one
the question you ask near the end "waht do you do when people swarm" is the same question i asked you when you showed me chartbeat, "this is awesome, but what do i do with this information"
i invested in disqus for a bunch of reasons, but the main one was the email me the comments to my blackberry feature. when the swarm hits, you want to engage with it and the tethered to the computer model just doesn't work for that
the one thing that you really didn't weave into this is the role of mobile.
twitter's shortcode is the most used shortcode in the US. that's a telling piece of data.
this real time stream is with us all the time, and it would not happen without mobility
John - really excellent post. I need to read it a few more times and digest. Thanks!
Shortcodes are mobile command line interfaces to the next wave of RT social apps.
opentable for instance should shortcode. Restaurants, viewing their bookings can present '2 hour specials' that get agregated and bullhorned. i can type 'on' top SC XXXXX at 6pm and receive a list of my 6 favorites offering immediate incentives.... I can complete my rezzi and have it dynamically shares across my other activity streams and so on.....
the one piece missing is auditable geolocation.
there are way to many use cases with SCs and RT - many of them will become very large businesses.
I also realized that the prezi thing got disabled this am, its back now — adds a little more data to the post. As an aside — I’m loving prezi — I think the zoom metaphor it uses (think Google earth for data) is a part of this puzzle, I want to be able to zoom in and out of these data streams. See what we did on betaworks.com :)
but how do you cut and paste stuff out of prezi? i wanted to share one of those on tumblr?
Chartbeat reminds me of another, possibly larger flow in streams I've been investigating: implicit data/feeds. Most streams today are explicitly created by users, either by creating content, making a friend, saving a favorite etc. For every explicit action of a user, there are probably 100+ implicit datapoints from usage; whether that is a page visit, a scroll, a video/shopping abandon etc. I also believe such implicit data is less skewed by the digerati that creates a significant chunk of today's explicit dataset -- providing a more accurate view of online likes/dislikes/activities.
Therefore, I applaud your work on Chartbeat and encourage you to turn that data inside out by connecting with cross-site implicit datapoints. That becomes less about site/page-specific webmaster analytics, and more about surfer/user (anonymous or not, depending upon user choice) implicit activity feeds flowing into a larger stream...keep me posted if u go that route.
@fredwilson, I think your comment is spot-on for the author of a blog. When the swarm hits, the author can respond.
What about the readers or the people leaving the comments? I found this post from a passed link, so there has to be other people reading as I'm typing. It took me a few mins to read the entire post. From the time I arrived to the time I made it to the comments section, did someone else comment? Can I chat with people who are on this page too? Can real-time conversations spontaneously breakout in a post's comment section?
This is just the beginning, it's very exciting.
(I refreshed the page to check, 3 new comments.)
We did try live chat -- bubble chat on the page (firef.ly) -- but the interactions were thin and fleeting, it was more game like than a forum for conversation -- and many people dont actually want to chat.
-------- what i just got in email ----------
A chartbeat alert has been triggered. The alert is:
total visitors on any goes above monthly max.
The domains that were triggered are:
borthwick.com -- total visitors: 157
A few thoughts come to mind:
- We should tip our hats to Amazon Web Services (and burstable cloud infrastructure) which is what allows sites built for 40 visitors to suddenly accommodate a swarm. The architecture underneath our communities has quietly evolved to allow exactly this phenomena even before social media created the digital social pathways to create the swarms themselves.
- I agree that a "bookmark" or "remember" button for the RT web is needed. Right now we have a binary digital history: the present and the saved (i.e., the way back machine). But just saving the nugget won't be enough to re-create the stream at that moment! Won't what came before it, what inspired it, who commented on it, your responses to them, and what came after it -- isn't that really what you want to remember? Can we ever "bookmark" the stream? I don't think we will be able. We'll pull nuggets out, but the totality of the moment will have moved on.
I am inspired by how much is unclear about the RT future.
Couple thoughts:
* I think posts (the element behind RSS) would be the first stream. Even on webpage you'd read them as separate but consecutive. But that would be preceded by forums, chat rooms, IRC and mail lists, so maybe I'll leave that up to those that take these things to the mat to decide ;)
* I think we're seeing 2 distinct types of stream users. Those that want to hold the rope and those that watch it go by when they look at it. For lots of people twitter or facebook or flickr doesn't provide enough meaning if they follow too much. FriendFeed seems to be very binary, people love it or don't use, but rarely use just now and then.
* Asynchronous connections are much more (adult) human like. Thanks for that insight.
* You just made me think handheld computers have really blow the doors off the page-based paradigm and
that we're a lot closer to an always-on Internet experience then is generally believed ;)
Thanks for the deep thoughts.
Thanks for writing all those thoughts up. I have too many comments to comment, but you know I agree with much/all of this, and....... well, you know.
I think an exciting part will be the merging of the social+now+context/structured web (using your terms) together via a “personal API” or semantic lens of sorts, and allowing us to manipulate that knowledge to gain greater insights.
Cheers.
Salman Rushdie saw it coming, in 1990 -- The Stream of Stories that he envisions is : "made up of a thousand thousand thousand and one different currents, each one a different color, weaving in and out of one another like a liquid tapestry of breathtaking complexity." Sounds like he's talking about the story we'll all living every day on the Web (Stream?).
The image that always comes to mind for me is the Dow Jones Stock Ticker
People used to watch it like hawks, looking for the next uptick or downtick, the pulse of the financial nation.
And in the back room traders were feeding it their messages.
• folders on my desktop for people, websites, posts, comments, photos and feeds I like? realtime, clickable not static.
• drag and drop so I can use all these in messaging, posting, commenting and tagging?
• why should I ever have to type <a href= or use the link-chain-thingy? again drag/grab and drop from desktop to WP, email?
• real time means these are always current, synched (like Plaxo) and the real data is stored who knows where.
A Kundera quote, dealing with time acceleration:
-Why has the pleasure of slowness disappeared? Ah, where have they gone, the amblers of yesteryear? Where have they gone, those loafing heroes of folk song, those vagabonds who roam from one mill to another and bed down under the stars? Have they vanished along with footpaths, with grasslands and clearings, with nature? There is a Czech proverb that describes their easy indolence by a metaphor: 'they are gazing at God's windows.' A person gazing at God's windows is not bored; he is happy. In our world, indolence has turned into having nothing to do, which is a completely different thing: a person with nothing to do is frustrated, bored, is constantly searching for an activity he lacks.(kundera)
Interesting comparison about both kind of streams. It's important to have these facts clear in order to realize what can be done and where can we go.
About Swarms:
Traffic is not smooth is has spikes, like swarms, they move in spikes when something happens given the high speed at which the information
is transmitted between its members, spikes are produced by a relevant fact or a mere fake rumor which spreads through the communication channels reaching individuals at an exponential rate of Grouth.
check out my post about streams: singyourownlullaby.blogspot.com
Cheers,
Buat Duit Dengan Blog
Thanks and Regards
Thanks and Regards
Thanks and Regards