How To Keep Widgets From Slowing Down Sites: WEDJE

The whole world is going to widgets. This overused, overhyped term refers to third-party code one places on their website or blog in order to display such things as Flickr photos, Twitter status, or iTunes playlists. Everybody and their mom is putting out widgets these days, and although only about 1% of them are useful or interesting, they are an important new distribution mechanism that is changing the way companies think about syndication.

But there is a big problem with widgets: they slow down the sites that use them. In the best case, you have a company like Flickr whose servers are almost always snappy, and in the worst case, you have a young startup that is constantly struggling against increasing demand and occasionally can’t serve up any code at all.

The problem is that in either of these cases, the completion of your site’s loading and rendering depends on someone else’s code living on someone else’s server. Including a fast and reliable Flickr widget still slows your site down by at least a split second and including a less stable one can leave your site hanging indefinitely.

We’ve been developing what we think is going to be a gangbusters widget at Newsvine over the last few months but just as we were getting ready to deploy it, Intern Rob and I hacked together what we think is a method of deploying widgets in such a fashion that they don’t affect the load times of their parent sites whatsoever.

Following is a breakdown:

Root Causes

Most widgets are deployed using one of two methods: raw embedding or javascript embedding.

Raw embedding usually involves providing a chunk of object/embed code to display a Flash widget inline. The advantage of this method is that it works in environments like MySpace which don’t allow the use of javascript. The disadvantages are that a) many people don’t like placing raw Flash code in their pages, b) some widgets aren’t Flash-based, and c) the widget author can’t change much after the fact since the widget’s code is more or less set in stone.

Javascript embedding involves using a script include to remotely deploy code into the user’s site. The advantage is that it’s much cleaner and more flexible. The disadvantage is that it slows sites down a little every time and a lot when there are problems.

The Solution: WEDJE

Providing a raw embed method is still a necessity if your target environment is MySpace, but if you’re going after people with their own sites/blogs, javascript embedding is still a better ideal. We need a catchy acronym for our method in order for people to use it so we’re going with: WEDJE. Or, “Widget Enabled DOM Javascript Embedding”… which is an acronym inside of an acronym, so it’s extra cool.

In a nutshell, here are what the two existing javascript methods do versus what WEDJE does:

Standard document.write

Most javascript embedding simply uses document.write to write code into your site inline. The problem is that if the widget server hangs, the rest of your site can’t draw until the script is done executing. This could be a split second or it could be forever. Many people mistakenly think you can use the defer attribute in your script tag to keep this from happening, but defer effectively kills the document.write function entirely so this is not a solution.

Deferred innerHTML code

The other method of javascript embedding is to write out a div with an ID, load the javascript with the defer parameter added, and then use innerHTML to write to the div whenever the script has loaded. This seems great in theory, but in reality, Internet Explorer is the only browser which seems to properly render the rest of the page before the script is done executing.

WEDJE

WEDJE is similar to the innerHTML method above except it creates what is effectively a cross-platform, cross-browser defer, enabling your script to load and execute asynchronously across all environments. We write out a div with javascript, then we create a script element with javascript, and then we append the script element to the div, again with javascript.

By linking elements together in this way, browsers appear to completely decouple the loading and execution of our attached javascript from the loading and execution of the original document, which is exactly what we’re looking for.

An example of WEDJE in action is below. Reload this page and watch the area below. There is a 7 second delay in the execution of the javascript file in order to simulate a slow-loading script:

This text is before the widget code.

This text is after the widget code.

Notice how everything loads regardless of the state of the widget? Voila.

Caveats

Intern Rob and I are under no illusions that this is rocket science, nor that it hasn’t already been tried by someone else, nor that it works perfectly. To the last point, we’d love your feedback as to what could be improved or where it may not be working. So far, tested browsers are:

  • Safari: Mac and PC
  • Firefox: Mac and PC
  • Internet Explorer: PC
  • Opera: Mac and PC

1. Here is the sample code which goes in the js file:

2. Here is the sample code to embed in your page:

Compact Version (Don’t use if you serve your pages as application/xml):

Application/XML Mime-Type Version (thanks Scott):

UPDATE: There is apparently some random IE crashing behavior that can occur using this method upon subsequent page views (but not full reloads). If anyone has any clues, let me know.
Like this entry? You can follow me on Twitter here, subscribe via email here, or get the RSS feed if that's how you roll.

84 Responses:

  1. This is pretty awesome, google should use this with their js such as adsense and analytics.

  2. Jick says:

    Did anyone try multiple widgets on one page?

    I can’t get it to work.

    Jick

  3. Mike D. says:

    Asbjorn: Uhhhh, an alternative to server-side includes? What? Who said this has anything to do with those? Widgets are *third-party code* which cannot be placed on your server.

    Jick: To use multiple ones, you just change the name of the main div in both the inline JS and the linked JS file. The idea is that this div will be named something unique each time, based on what widget is embedded.

  4. Kyle says:

    This is completely awesome that you’re actually developing something with your readers. I know it started with Mike and Rob, but seeing the progression of such a functional THING with readers is pretty cool!

  5. Deja vu… we came up with essentially the same approach for NewsGator Widgets. We’ve been using it for about six months now and it works quite well, although there are a few gotchas in IE6. For instance, your widget may not load if there’s something else on the page that takes a very long time to load. Also we’ve run into some very odd problems with appending script blocks when someone else on the page is doing document.write()’s. But by and large it’s a great approach.

    If you want to check out our implementation there are number of pages around that have NG widgets on them. A couple good examples are the Discovery Everest Blog(second box down in the right column) and the photo rotators on my blog (top two in the right column).

    We’re giving away free accounts, you can sign up here if you want to (click the “add buzz to your site” link in the upper right)

  6. Mike D. says:

    Brian: Maybe I’m missing something but I checked out the widget code on your blog and I am not seeing the similarities at all. From what I can tell, you are calling a script normally, using document.write, and not guarding against slowness in the script returning. I know I’m missing something, so if you could be so kind, please enlighten me…

  7. There is the one write in there, but it’s just loading static JS so it’s fast. Still, you’re right that it’s not good, so I’ll be taking it out in the next version.

    If you watch the requests we actually do a number of other requests after the page loads (or during, depends on your browser).

  8. tc says:

    So I just read all these comments and I’m wondering what the finished script looks like. Is it the one at the top? Sounds like it been though a few versions all within the life of these comments.

  9. Mike D. says:

    tc: Yeah, I’ve merged all suggestions/improvements into the above code. I wouldn’t call it final (because nothing ever is these days) but it’s lookin’ pretty good.

  10. Steve says:

    For this, I use the object element and a small satellite page with the widget script in

    <object classid=”clsid:25336920-03F9-11CF-8FD0-00AA00686F13″ data=”your URL here” type=”text/html”>

  11. Scott says:

    Mike and Rob

    Excellent work as I know I’ve had the same headaches with embedding widgets as well. I also really like your approach to the solution. Can’t wait to run tests with it in the wild.

    On a side note when I see WEDJE I think of the all to feared wedgy. So I’m off to give my web sites a Wedgy. :-D

  12. Steve — you may have something there… I made a test using your far-more-straightforward object embed method, completely sans-WEDJE:
    http://robgoodlatte.com/dev/WEDJE/objecttest.html

    The only downside I see is that you have to set the height of the object. I also haven’t tested this across all browsers, only Safari. If you’re right this could be a real “D’oh!” moment..

  13. Mike D. says:

    Steve: Yeah, the object method seems like a good alternative to iframes, but that’s basically all it is, right? Just XHTML 1.1’s alternative to the IFRAME tag?

  14. Steve — I took a really close look at embedding using the object tag, and I actually think it’s pretty useful — but there are a handful of limitations:

    • You have to set the height / width of the object inline, otherwise you get scrollbars. This means you can only really embed something that you absolutely know the height of.
    • If the external page fails to load, you’re left with whitespace on your page because you have to specify the height
    • This one is a biggie — you can’t style the embedded code with CSS to my knowledge. If the external script is depositing html tags into your code, your CSS won’t be able to access those elements, and they won’t become part of your page’s DOM. I found this out firsthand when trying to convert my flickr photo embeds to use the object tag.

    Embedding via the object tag is a great option for flash widgets that will always have the same height. But depositing HTML elements doesn’t really work with an object embed due to the styling issues. So count Flickr/del.icio.us/magnolia/newsvine embedded JSS links out. WEDJE still looks like the best option for embedding HTML.

  15. Rust says:

    Just to note, the loading times that this script seems to (mostly) solve is not restricted to widgets, but any third-party content. The biggest site-killer I’ve found so far is actually ads. I don’t know how many times I’ve gone to a site like Ars Technica, Wired, or even Digg and had to wait anywhere from 2 to 20 seconds while a banner ad that I don’t even want to see is loaded from some ad server somewhere – and that server is overloaded for some reason.

    So thanks for solving the widget load-times, but we really need those ads to stop breaking sites! :)

  16. I’m a little bit confused about where I’m supposed to place the code. Does it go in the tags of the header, or in a .js file? And which .js file? (I’m using WordPress and want this code for my sidebars.)

  17. teddybear says:

    How do I insert codes for Flash content? I want to put a video on my website, but it asks for a widget code and I don’t know how to write it.

  18. Bios says:

    This is a very intuitive technique. Iframes could be used as well, and then implement a script that would auto size the iframe to match its contents.

  19. Bill Kay says:

    I agree with Rust, ads are really bad for this sort of thing. Also those tracking gifs that are insisted on by marketing (Hit Box, Revenue Science, etc.). There needs to be some standards set.

    May be we should lobby doubleclick?

  20. Anon says:

    It appears Javascript variables are not available to the widget when using WEDJE in IE6 or IE7.

    <script type=”text/javascript”>
    <!–
    var widget_background_color = “black”;
    var widget_text_color = “white”;
    //–>
    </script>
    <script type=”text/javascript”><!– WEDJE… //–></script>

    Works in Firefox and Opera, but in IE6 or IE7 the widget will appear without styling. Passing style parameters via the widget URL is a workaround, but personally I find it awkward.

    Also, I think using Math.random() in the container name is good practice so users can embed multiple of the same widget on a page (with different parameters for example).

  21. Tomaz says:

    Has anyone figured out how to insert and execute javascript in DIV tag ?
    Like below but of course this doesn’t work :

    myDiv.innerHTML+=’..script tag with src..’;

  22. One thing you could try is:
    div.innerHTML = ”;

  23. Whoops.. seems that the javascript filter block it, here’s a spaced out version that goes inside the above single quotes:

    If you don’t see that either then the trick is that you put some sort break to the “script” part of the script tag so that html doesn’t see either part as an opening or closing tag.

    eg: scr’+’ipt

  24. Joe says:

    Hmm… I posted a comment here, and its gone :S

    I tested this and less than 5 minutes in IE5 had crashed twice.

  25. Joe says:

    I did a little more testing, and it works, and you can refresh the page many times and it works. but, when you click in the address bar and simply press enter (ie. go to same page without refreshing…) it crashes.

    Any fix for this?

  26. Mike D. says:

    Joe. Thanks for discovering this. I had noticed some IE crashing as well and was never able to pin it down to a certain behavior. The fact that it doesn’t happen upon a full reload makes me think it’s some sort of timing issue. I’ll look into it, but these things are pretty hard to debug sometimes…

  27. matt says:

    The widget is not loading in the example?
    Are some “famous” widgets using this technique? Or do the random crashing behaviour happen too often to be used in a real environment?

    thanks!

  28. Mike D. says:

    Hi Matt. Yeah, that’s probably just an artifact of me switching to WordPress… but yeah, I wouldn’t use WEDJE at this point because I never got around to fully sussing out the IE6 issue. It’s unpredictable.

  29. I like the method to isolate widget load from the page load. Thanks.

    We are looking to distribute widgets to clients. Not every page view will use our widget, but only when the consumer directly interacts with the widget. Therefore, to limit each trip to our server for the basic widget draw, we want to have the HTML live in the distributed .js file. This way only when a consumer interacts with the widget will AJAX API calls come to our servers; otherwise it is all on our clients servers?

    So, deployment is providing the snippet _AND_ the .js file which we ask them to host on their system.

    Again, the .js file contains the widget display, defaults, etc and the API calls to our systems.

    Is this how other people are doing things? What are some alternatives to think about.

  30. Wow, I have been looking for this forever. This REALLY helped me out with the horrible twitter badge hanging my site everytime.

    Thank you :)

  31. In its latest version, it crashes on IE7 every single time when I press Enter in the address bar.

  32. Anonymous says:

    Fixing the IE crash:

    It seems like if the script src is added to the script element AFTER the script element is appended to the div, IE doesn’t crash.
    So, the setTimeout first argument should be: “document.getElementById(‘wedje_div_yourcompanyname’).appendChild(s);s.src=’http://www.mikeindustries.com/blog/scripts/sleepywidget.php’;”.

    Don’t forget to remove the original s.src=… before the setTimeout call.

  33. Jason says:

    Sorry to bring an old post back to life, but I have a question I don’t see an obvious answer for…

    Why not just put the tag above the script tag? This way you wouldn’t have to worry out the differences between text/html and application/xml.

    Is there something obvious I’m missing here?

    Thanks for the great informative post/thread btw.

  34. Jason says:

    Sorry, the html was filtered out of that last post… It should say “Why not just put the ‘div’ tag above the script tag.

Shared
The Ocean in 185 Lines of Javascript:

Mesmerizing. Try tweaking some of the variables in the “sea” section of the code.

“"Design had been a vertical stripe in the chain of events in a product’s delivery; at Apple, it became a long horizontal stripe, where design is part of every conversation.””
Why I Just Asked My Students To Put Their Laptops Away:

A great essay about how toxic everyday distractions can be.

Humanity's deep future:

A group of researchers at the Future of Humanity Institute talk about where our race may be going and how artificial intelligence could save or kill us all.

Steve Jobs speaks about the future at the International Design Conference in 1983:

31 years later, it’s safe to say this is one of the most prescient speeches about technology ever delivered. Jobs covers wireless networking, tablets, Google StreetView, Siri, and the App Store (among other things) many years before their proliferation. A fantastic listen.

How to travel around the world for a year:

Great advice for when you finally find the time.

LiveSurface:

A fantastic app for prototyping your design work onto real world objects like billboards, book covers, and coffee cups. This seems like just as great of a tool for people learning design as it does for experts.

50 problems in 50 days:

One man’s attempt to solve 50 problems in 50 days using only great design. Some good startup ideas in here…

How to Do Philosophy:

If you’ve ever suspected that most classical philosophy is a colossal waste of time, Paul Graham tells you why you’re probably right.

TIME: Why Medical Bills Are Killing Us:

Stephen Brill follows the money to uncover the pinnacle of corruption that is the U.S. Health Care system. A must-read article if there ever was one.

DIY Dot Org:

A beautifully designed site full of fun and challenging DIY projects. I could spend months on here.

The Steve Jobs Video Archive:

A collection of over 250 Steve Jobs videos in biographical order

Self-portraits from an artist under the influence of 48 different psychoactive drug combos.

Water Wigs are pretty amazing.

David Pogue proposes to his girlfriend by creating a fake movie trailer about them and then getting a theater to play it before a real movie. Beautiful and totally awesome.