picture of Josh

Hi, I'm Josh, an engineer and entrepreneur living in San Francisco. I write about code, companies, and ideas. I work at Keen IO.
Find me on twitter and github.

weeve - HTML5 Twitter uber-streaming powered by Firebase, Keen IO, and Singly

weeve logo

Ever wish you could trade Twitter timelines with someone for a day? Or just a few minutes?

weeve goes one step further, allowing groups of Twitter users to combine their timelines to create an uber-timeline. Try it out - join the weeve at http://weeve.dzello.com/.

I created weeve in 2 days over the winter holiday as a fun way to get hands-on with some new tech. My goals for weeve were:

  • to play with some APIs I’d been hearing a lot about
  • to build a server-less app using a Backend-as-a-Service
  • to open-source the project for others to benefit from
  • to build something useful that could help people discover new content on Twitter
  • to distract me lest I consume hordes of delicious holiday foods

All but one of those goals was accomplished. :)

In this blog post I’ll talk about how weeve works and APIs that made it possible.

The API’s

These APIs, and the companies behind them, are doing great things. It’s no coincidence I chose them to hack on. Here they are, including a brief summary of what they do in (mostly) my own words.

Firebase - https://firebase.com/

Firebase provides data storage that’s directly and securely accessible from the browser. With Firebase, you lay out your data hierarchically, then bind clients to changes that happen at individual nodes. The unification of messaging locations and data is intuitive and powerful. And the JavaScript SDK is a pleasure to work with. The Security API has a learning curve, but it’s a unique take on a important problem for server-less development - definitely worth spending the time to pick up.

Example code, taken from writing data in the Firebase docs:

1
2
3
4
5
6
7
8
// First we get a reference to the location of the user’s name data:
var nameRef =
  new Firebase(
    'https://SampleChat.firebaseIO-demo.com/users/fred/name');

// And then we write data to his first and last name locations:
nameRef.child('first').set('Fred');
nameRef.child('last').set('Swanson');
Keen IO - https://keen.io/

Keen IO does analytics by API. It’s a developer-friendly solution that solves both problems - collecting event data and looking at event data. Recently, Keen IO released a JavaScript Visualization API that lets you draw charts & graphs with just a few lines of code.

Example code, taken from weeve:

1
2
3
4
5
6
// Draw a pie chart in just 2 function calls!
new Keen.Metric("tweets", {
  analysisType: "count", groupBy: "source_screen_name"
}).draw($("#top-weevers")[0], {
  showLegend: true, title: "Top weevers today"
})
Singly - https://singly.com/

Singly is the intelligent, unifying proxy for social network APIs you asked Santa for this year. Singly already has support for over 35 services. One thing I love about Singly is that their APIs are open source.

And if that wasn’t enough, Singly is also behind The Locker Project, which has the bad-ass mission of making it possible for you to keep track of the mountains of personal data you spread around the nets. It’s like insurance for the next time this happens: “All Online Data Lost After Internet Crash”.

Example code using the profile Singly endpoint:

1
2
3
4
5
6
7
// Retrieve a user's unified profile
$.getJSON("https://api.singly.com/profile"),
  { accessToken: "HOHOHO" },
  function(profile) {
    alert("Would "  + profile.name + " like some fruitcake?")
  }
)

Special mentions

The following services and libraries are also used by weeve.

Twitter - https://twitter.com/

When people say ‘the twitters’ they really just mean Twitter.

jQuery, Bootstrap, Backbone, twitter-text, socket.io, CDNJS & more

Get the full list on the Github README.

How weeve works

weeve has no server. weeve is just an HTML page, a JS file, and a CSS file. You can deploy weeve under to any static web server like Apache and nginx, or to Github Pages.

Despite being flat, weeve has a host of features that traditionally require a server:

  • OAuth Authentication

    Singly proxies the OAuth conversation with Twitter. Then, Singly creates a Firebase authentication token and hands it back to weeve as a location fragment. The weeve Backbone router uses the token to establish a faux ‘session’ backed by localStorage, so the authentication persists across page refreshes. Read more about this flow here - Firebase Authentication Setup and here.

  • Central data storage

    Firebase is used to store data in a way that’s accessible to all clients, yet still secure. User A cannot tamper with User B’s data, etc. See the weeve security rules JSON definition to see what’s defined.

  • Shared messaging layer

    Clients get notifications about what’s going on by binding to Firebase data references. Everything happens in real-time and neatly stays in sync. weeve uses Firebase ‘on’ bindings to reflect the presence of users and to add tweets to the uber-timeline.

  • Analytics

    Application events are logged to Keen IO via their Javascript SDK. weeve uses the combined Keen IO SDK to get event logging and metrics/visualization all in one package. The event logging piece is also available separately. The SDK lets you define metrics, including timeframes, intervals, and group-by, and then makes it easy to represent the metric as a number, pie chart, or line graph.

    weeve uses Keen IO to display who’s contributing the most tweets (pie chart), how many total weevers there are (number), and recent tweet volume as a series (line graph).

  • Twitter’s Streaming API

    weeve uses the user stream endpoint of the Twitter streaming API. Tweets appear instantaneously; as in the instant someone pushes ‘send’. It’s quite thrilling.

    (I did have to write a 40-line node.js twitter streaming proxy to make the streaming API accessible from the browser. In the future, I hope Singly’s experimental push support can replace the proxy!)

Get the code

weeve is open source. You can read the annotated source code and learn how to host your own weeve(s) on Github at dzello/weeve.

If you’re looking to build something with one or more of these APIs, definitely check out the repo - there’s a good chance you might find a relevant example of something you’d like to do.

weeve.js is the main (and only!) JavaScript file. I’ve added lots of comments and taken extra care to lay the code out in a way that’s easy to follow.

Conclusion

I’m very happy with the way things turned out. I started with quite a few unknows and estimated at least a few unsightly hacks would be required to get it all working.

The streaming proxy was the only ‘hack’ required. I think that says a lot about the feasability of these API’s for pure-client-side development and the readiness of backend-as-a-service providers for real-world HTML5 apps.

weeve is only a few hundred lines of JavaScript. And of that maybe 100 lines aren’t ‘boilerplate’. Needing just 100 lines of ‘reasoned’ code to get this feature set is mind-blowing.

And what’s positively mind-altering is that there’s no server to scale, deploys are sub-second (cp baby), and the heavy lifting is distributed across the browsers of your users and your API providers. Quite literally, I’m already thinking about new domains this topology can be applied to.

Overall, I’m excited to see what other apps & patterns emerge as developers consider ‘staying off the server’.

Now then. Why not head over to a weeve and try it out?

Google+ interview streamed from ROFLcopter, future

Today, I’m trying something new on this blog. I’ll take an interview summary I read this morning and see what would happen if I had live-tweeted it under the alias of my product-testing alter-ego @upforadoption.



“Google’s Horowitz: Facebook is social network of the past”.

source CNET

@upforadoption: This is CNET’s first article published with a positive chronological offset. That means it’s from the future. Perhaps an alternate future.

NEW YORK–Facebook is the social network of the past, and the way it implements advertising doesn’t really work, a Google executive overseeing the search giant’s social-networking product said today. Bradley Horowitz, the Google vice president of product for Google+, said Facebook isn’t set up in a way that’s compatible with the real world.

@upforadoption: Facebook was the first major social network built on real-world identities. Doesn’t count?

People should be able to have a conversation with a certain group of friends without involving others.

@upforadoption: Like facebook lists or more like facebook groups or…?

“In designing Google+, we keep thinking about the real world, the way people actually are,” Horowitz said today during a Business Insider conference in New York.

@upforadoption: Skating to where the puck is. Can we look beyond what we already do today?

Meanwhile, Horowitz compared the ads in a Facebook user’s newsfeed to a sandwich board. Such ads don’t really have any context and often aren’t very effective, he said.

@upforadoption: What’s a sandwich board? Is that what Googlers eat on instead of plates? Sounds awesome.

“Jamming ads and agendas into user streams is pissing off users and frustrating brands too,” he said. “That’s not the way the world works. Rather, in the real world, there has to be intent. When a person’s hungry, he or she goes into a restaurant.”

@upforadoption: Hungry people go to restarants, now Google Plus?

But being able to search for a lunch place when hungry and finding recommendations from friends is much more effective.

@upforadoption: Future headlines? - ‘Hungry for growth, Google+ takes a bite out of Yelp”

Horowitz added that Google doesn’t “have to make payroll by jamming users with ads” on Google+.

@upforadoption: No, not on Google+, just in email and on all web pages everywhere.

When asked if Google+ will ever incorporate ads, Horowitz said it would do so if there’s an effective way to add them without upsetting users.

“We aren’t struggling with how to monetize,” he said. “We have real plans.”

@upforadoption: ‘I can’t tell you what they are, but they are saved on Google Drive.’

Google said in September that 100 million people are using Google+ each month, and it said it now has 400 million total users. Horowitz declined to provide updated numbers today but said the figures from September are “stale”.

@upforadoption: Then the question is whether November’s numbers are already ‘stale’.

Horowitz noted that Google+ currently is “foundational layer” for Google’s other products – the “identity, relationship, and interest system for Google.” What it wants to be, he said, is the go-to place for people go to wish their friends happy birthday, much like Facebook is today except “uncluttered”.

@upforadoption: So, Google’s strategy to kill Facebook is part of Facebook’s strategy that killed Myspace? This will end well.


Parting thoughts

To be clear, I’m not trying to hate on G+ or Horowitz. I don’t have a strong preference between G+ and Facebook - I use both, if occasionally. There are parts of both companies I admire and there are parts of each that I just don’t get.

That said, the way Horowitz spoke about the competition (or the way the journalist chose to present it) left a lot to be desired in terms of substance/facts/professionalism. There was just a lot of dissing.

Even in a one-sided pissing match someone’s shoes still get wet.

Google has a lot more amazing technology in store for its users and I’d love to see them spend time on that instead of trying to ‘catch up’ to Facebook.

Make on-demand slideshow GIFs from search terms with gifs.pty.io

Warning: The GIFs in this post change every hour. If you like one and you want to keep it, make sure to save a copy!

Some background

Recently I spent an evening getting to know the awesome guys behind Catalyst Class. Catalyst trains developers “from zero to employable over 12 weeks in San Francisco.” That’s not easy, but it’s hard to imagine a group more prepared or excited than Catalyst’s students and faculty.

Companies like Catalyst and others like LearnUp help address unemployment and underemployment by re-training workers in the skills that today’s employers are looking for. I think that’s fantastic.

The challenge

While at Catalyst I was given an idea for a party.io-related hack from co-founder Shawn Drost. Afterwards Shawn sent me a gentle reminder and that’s when I knew it was on:

trololo.pty.io

Boom. This is just a simple project, but it was fun to work on and maybe you’ll feel like trying it out.

Party Gifs!

Announcing gifs.pty.io - freshly-made, on-demand GIFs for your search terms.

Here’s how it works: gifs.pty.io is actually anything.pty.io!

GIF from anything.pty.io

Or science.pty.io. Or batman.pty.io.

For every subdomain of pty.io you visit you’ll be rewarded with a unique GIF, made on-the-fly you wait from related Google Images. It’s a surprise.pty.io every time!

GIF from surprise.pty.io

In addition to visiting subdomains, you can get the actual GIF file by using a URL like this: http://gifs.pty.io/pancakes.gif.

Pancakes

This works better in chat rooms or where you’re sharing just the image and don’t need the HTML page.

The app

Ruby+Sinatra+Thin. The Google Images API is used to find still (i.e. boring) images and Imagemagick gifs them life.

To reduce the time it takes to make you a GIF, we use eventmachine and em-http-request to retrieve images asynchronously in parallel. And adding in em-synchrony means we get readable, callback-free code like this:

1
2
3
4
5
6
7
8
9
10
multi = EventMachine::Synchrony::Multi.new
10.times { |n|
  multi.
    add(n, EventMachine::HttpRequest.new(api_url(search_term))).
    aget
}
multi.perform

# all 10 requests are complete, make this gif!
gif! multi.requests.values.map(&:response)

We also start the image processing the moment we have enough GIFs for your query, aborting any outstanding, slower requests. We really want that GIF to reach your chat room at the perfect time so you achieve:

Total domination

Caching

GIF’s are cached for 1 hour. That means the GIFs you’ve seen so far in this post are all less than 1 hour old! Anyone who read this post over this hour ago has seen different GIFs that you have.

GIFs for some searches evolve slowly over time, like “Yosemite”, while others change rapidly, like “IMDB opening this week”.

Biggest rocks IMDB opening this week

Extras

Here’s a script for Hubot that lets you summon GIFs to your chat rooms. Use it like this:

1
hubot gif me tequilia

Tequila

Download it now, or look for this script in hubot-scripts soon.

The Singularity

It occurred to me while writing this post that this app could eventually destroy itself and take down the Internet along with it. Here’s how it could happen:

  1. gifs.pty.io gets really popular. You never know, right??
  2. Due to it’s increased page rank, GIFs created from gifs.pty.io become the top search results on Google.
  3. gifs.pty.io’s image search results now start to come from gifs.pty.io.
  4. gifs.pty.io goes into an infinite loop trying to make gifs of gifs [of gifs].
  5. Everyone blogs about how gifs.pty.io can’t scale. Our page rank explodes - all press is good press.
  6. Google tries feverishly to index gifs.pty.io. But a Black Hole Is Black. Google goes into it’s own endless loop and the entire Internet faceplants.

Faceplant

Amirite? But don’t like let that stop you from using it or anything.

When can I use it?

Today! Knock yourself out. Try weird search combinations, or embed a dynamic gif in your blog.

Credits

Imagemagick whisperer @_kerry. Inspiration from bananas.jpg.to.

gifs.pty.io is a service by party.io. If you like collecting GIFs head over and make an account or you can check out my GIFs.

Work with us

Developer? Designer? Community builder? All-around hustler? Come work with us at party.io.

Corporate handshake

We’re small and you’d be huge. Email me. I’m dzello at party dot io.