Building the YUI Browser History Manager

By Julien LecomteFebruary 21st, 2007

Julien LecomteAbout the Author: Julien Lecomte is an engineer on Yahoo!’s DHTML Evangelist team, a group that provides architectural assistance to Yahoo! developers on the design and implementation of rich interactions in the browser. He has worked extensively on Yahoo!’s new Mail application. Julien is the author of a new, experimental YUI component, the Browser History Manager, which was released earlier this week. In this article, he explores the broad problems associated with controlling history and state in the browser during rich interactions and he lays out the multifaceted approach he took to building a YUI history manager with support for all A-Grade browsers. -Eric Miraglia

Most Ajax Applications Break Standard Browser Functionality

Our industry has made significant progress in the past few years. Web pages have become vastly more sophisticated in terms of appearance and responsiveness. Some companies have pushed the envelope so far that their web page could easily be mistaken for a native application (check out Scalix Web Access or the new Yahoo! Mail, to name just two examples). In the meantime, that same rush to produce the most stunning web experience has led us to forget some of the most basic principles of the web. Our pages run inside a web browser, and even if modern web browsers can now be used to support very complex applications they are mostly used for what they were originally intended to do: browsing the web.

This dual purpose creates a lot of confusion among users. Some of the basic features of browsing, such as the use of back and forward buttons, or the ability to bookmark a page, rarely achieve the desired result in a dynamic web application because dynamic changes to a page do not naturally register themselves with the browser’s native history engine. Moreover, there is a difference between "backing out changes" in an application — the undo command, ctrl-z — and retracing one’s steps in a navigation context. Many web-application developers find themselves puzzling over what role the back button should play when they make the transition from writing web sites to writing web applications.

Ignorance Is Bliss…

Only recently did I find myself impacted in my own work by the absence of a good solution for managing navigation history and session state in a rich application. At the time, I was working on a very simple web site for my father’s new venture. After “ajaxifying” the site, the navigation became a lot faster. However, a coworker of mine pointed out to me that the back and forward buttons were not working anymore, and that individual sections of the web site could not be bookmarked. In my case, this problem was a deal-breaker; I absolutely had to find a solution. After a bit of time digging around on Yahoo! Search, I discovered a long and impressive library of articles from developers attempting to solve this problem.

Currently Available Solutions

One of the most important articles in this collection is Brad Neuberg’s piece on how to handle bookmarks and the back button in Ajax applications. Neuberg’s Really Simple History (RSH) framework works well on most browsers, but unfortunately it does not work on Safari — considered an A-Grade browser in Yahoo!’s Graded Browser Support approach. Another disadvantage of the RSH framework is the need for a server round-trip every time a new entry is added to the browser history on Internet Explorer. Nevertheless, Neuberg’s work, which pushed back the frontier on this difficult issue, made me believe that my quest was not hopeless.

I then read Mark Schiefelbein’s article on developing Ajax applications that preserve standard browser functionality. His article mainly looks at the Backbase Ajax engine and how it provides support for the back/forward buttons. The Backbase solution works well, but I was once again confronted by the issue of browser support; their demos do not appear to work on Opera and Safari. Moreover, Backbase is a commercial product, and I wanted a solution that would play well with the open-source tools I was already using, and in particular one that could be included as part of YUI.

Creating a Browser History Manager for YUI

Given that background, I decided to tackle the problem on my own. After a little bit of tinkering, I was able to put together a demo that worked on Firefox and Opera using the URL fragment identifier. The idea is very simple: Change the URL fragment identifier, using the window.location.hash property, to add a new entry to the browser history and periodically check the value of the hash to detect history changes. In addition to solving the issue with the back/forward buttons, this also allows the application to be bookmarked in a specific state. This approach provides support for Firefox (and other Gecko-derived browsers), Opera, and for recent nightly builds of WebKit (the open-source rendering engine underlying Apple’s Safari browser). Unfortunately, the window.location.hash approach does not work fully on current versions of Safari — and, more importantly, it does not work at all on Internet Explorer. It might serve as part of the solution, but it obviously couldn’t constitute a unified approach across browsers.

One of my colleagues at Yahoo!, Dav Glass, is an accomplished Mac geek who has a great deal of experience solving hard problems on Safari; Dav volunteered to help tackle browser history in the current versions of Safari. With Safari, there were two main issues to overcome:

  1. We needed to identify when back forward interactions occurred;
  2. When those interactions occurred, we needed to identify where we were in our stack of saved states.

After reading David Bloom’s post on using document.body.scrollTop, we started thinking about what other properties might change during back/forward navigation. So we wrote a script to dump all the window.* variables and found out that, in Safari, window.history.length changed each time the browser history changed — not that it should change, but it clearly did. Armed with this knowledge, we began by setting up our own “history” array to track the changes keyed off of the window.history.length change. This approach gave us the two pieces of information we needed: We could tell when the user hit "back" or "forward", and we could tell where they were when they did so. (Note: The fact that window.history.length changes when the user hits the back button is a bug that has been fixed in WebKit. Therefore, this approach requires “forward compatible code” for Safari by using the URL fragment identifier as the preferred, long-term method of detecting history changes; we only fall back to relying on the history object itself when it’s necessary to do so. Over time, the window.history.length hack will become less and less important.)

All of that still left us in need of a solution for Internet Explorer. On IE, we had to adopt a radically different approach. We create a hidden IFrame that we initially point to an existing document on the web server. Writing to the IFrame using document.write adds a new entry to the browser history. The HTML we write looks like this (“current_state” is replaced by the actual state of the application):

<html>
 <body>
  <div id=”state”>current_state</div>
 </body>
</html>

We then periodically (every 50ms) look at the content of the IFrame — especially the text inside the inner <div> element — to detect history changes. Once we detect a change to the history, we change the URL fragment identifier using top.location.replace(...); this technique allows the application to be bookmarked in a specific state without impacting IE’s history stack.

In sum, the YUI Browser History Manager uses three different techniques to provide support across the A-Grade browsers:

Firefox (and Gecko Browsers), Opera and Nightly WebKit Add history entries and detect history changes by using the URL fragment identifier (window.location.hash)
Safari (current) Add history entries by using the URL fragment identifier (window.location.hash), and detect history changes by checking window.history.length
Internet Explorer Add history entries by writing to an IFrame, detect history changes by checking the content of the IFrame, and add bookmark support by setting the URL fragment identifier using top.location.replace(...)

The Problem Of The Initial State

Another concern in creating the Browser History Manager was to provide a mechanism for identifying the initial state of the application when the page/application loads via the back button (eg, the user has navigated away from the application to another site, then returns to our application using the back button). Indeed, from a programmatic standpoint, returning to a page using the back button or the forward button, or simply loading the page for the first time, are identical scenarios. However, the state of the application may need to be different depending on the situation. For example, if you return to a page using the back button, the state of the application should be the last state visited by the application before you left the page; this will likely be different from the initial state when the page first loaded during this browser session.

We approach this problem by storing state information in a hidden form field because the value of form fields persists across pages during the entire browser session. If we set the value of our form field to the current state of the application every time the application changes state, we can then initialize our page appropriately when the user returns to our application after having interacted with it beyond its initial state.

Intended Use: "Back" vs. "Undo"

The main role of the YUI Browser History Manager is to persist the history of states visited by a web application within a single browser session; in this week’s initial (and experimental) release we believe that it handles this problem well. However, there are limits to what these techniques allow you to do. The simple fact that URLs have a maximum length limits the amount of state data that can be stored (that limit is browser dependent — it is 2083 bytes on Internet Explorer, a bit more on other browsers). Imagine for example a TreeView Control containing thousands of nodes located at several levels of depth. If you wanted to represent the collapsed/expanded state of each node as a string, you would quickly run out of room; the information space provided by the URL doesn’t have the capacity to support that level of detail. More complex web applications can have a theoretically infinite number of states.

Given these limitations, and given the user’s expectation of what a "back button does" as compared to an "undo" command, we recommend thinking about the Browser History Manager as a tool for managing high-level navigational states: what screen is displayed, what tab is selected, what month is being displayed in a dynamically-generated calendar, etc. Because we cannot encode an infinite number of states, the back button cannot act as an “undo” command that takes snapshots of session states at high resolution throughout an interaction — and it shouldn’t try to do so. This sometimes subtle difference should drive the way you plan to make use of the YUI Browser History Manager.

Feedback

Due to the complexity of this problem space as it applies to the A-Grade browser family, we’re releasing the YUI Browser History Manager as an experimental utility at this point. We look forward to your feedback (here, or in our forums), and we hope to improve this implementation over the coming months to make it a genuinely robust, reliable, and cross-browser utility that solves one of the most vexing and problematic challenges involved in building rich internet applications.

41 Comments

  1. This is a great piece of work. I implemented a deep-linking solution only for Talent Show, so that people could link to specific videos that loaded in our viewing application. The x-browser challenges to wire up the Back/Fwd buttons was beyond scope of our time requirements, but I was able to work with an early version of the BHM and was very impressed. I’m eager to unleash the finished BHM on our next application.

  2. Artem Khodush said:
    February 22, 2007 at 9:03 am

    My 1.5 cents – last time I tried IFRAME for keeping history in IE, it worked only if there was only one IFRAME on the page. Adding second hidden IFRAME, for example for issuing server requests, mess up the history in unpredictable way. FWIW, YMMV – that was more than a year ago.

  3. Julien Lecomte said:
    February 22, 2007 at 1:32 pm

    Artem,

    It is true that having several IFrames on a page may mess up the browser history stack. The best way to work around this is to use XHR for your server requests. If you absolutely need an IFrame, try to use location.replace(…) as it won’t change the history stack.

    The YUI Browser History Manager is not meant to solve all possible situations and it is best suited for simple applications.

    Regards,
    Julien

  4. As far as I can tell the Google Web Toolkit has done history the same way since at least last summer. I think it’s funny how YUI and GWT pretend each other don’t exist. There is no harm in each learning from the other and each getting better for it. I use both YUI and GWT for different projects at work and both are great.

  5. This looks great; thanks for the shoutout in the article. It’s good to see the state of the art moving forward in Ajax browser history.

    Best,
    Brad Neuberg

  6. Julien Lecomte said:
    February 22, 2007 at 3:03 pm

    Sandy,

    Before starting my work on the new YUI Browser History Manager, I looked at the solution packaged with GWT. I quickly realized that it had several problems that I solved in the YUI Browser History Manager. For example (you can verify this when using GMail), the forward button is always disabled with GWT.

    Of course, there are many other non-technical issues with using a toolkit made by another company.

    Regards,
    Julien

  7. Julien,

    Two corrections to your response to my post.

    1. GMail is not based on GWT but you’re right about GMail not supporting the forward button.

    2. Backward and Forward buttons work just fine with GWT’s history manager. (At least in Safari, Opera, and Firefox.) Try their Kitchen Sink demo.

    Regardless, keep up the good work.

  8. Julien Lecomte said:
    February 22, 2007 at 4:23 pm

    Sandy,

    You are completely right and I was mistaken. Indeed, the GWT history component works very well on all A-grade browsers. However, it has the same problem as Brad Neuberg’s RSH framework: it requires an unnecessary server round trip on IE to add new entries to the browser history. Nevertheless, it is a very attractive solution.

    Thanks for your encouragement.

    Regards,
    Julien

  9. There is also the unFocus History Keeper.
    http://www.unfocus.com/projects/HistoryKeeper/

    It was the only solution I could find that supports current Safari (until now), using the same technique of watching window.history.length

    I was curious if you’d looked at that and if the YUI solution improves upon it?

  10. [...] Yahoo explains, via Don MacAskill’s blog (my new favorite) [...]

  11. [...] Julien Lecomte, the author of Browser History Manager, talks about some issues that continue to plague Ajax applications and how he went about solving them. Browser History Manager is one of several new additions to the YUI library version 2.2.0 released yesterday. [...]

  12. [...] Julien Lecompte also told me about his browser history manager (though I was just happy to speak French) which I will have to read about tomorrow on the plane. [...]

  13. Julien Lecomte said:
    February 22, 2007 at 8:38 pm

    Hi Scott,

    I had never heard of the unFocus History Keeper. I will definitely look at their approach. Thanks!

    There are so many great solutions out there that there is no more excuse to not handle the back button appropriately when it makes sense to do so. I think this is a great way forward for all web developers who care about their users.

    Regards,
    Julien

  14. Julien Lecomte said:
    February 22, 2007 at 8:44 pm

    Kudos to the folks at Ballhype who are already successfully using the new YUI Browser History Manager (only a day after the 2.2.0 release!) with the TabView control.

  15. [...] Building the YUI Browser History Manager Yahoo! engineer Julien Lecomte blogs about his experience writing the new Browser History Manager in the Yahoo! UI Library (YUI), and the difference between this new solution and previous aproaches to the Back button in Ajax applications. (tags: yahoo! javascript) [...]

  16. I’m glad to see that you found my writeup on Safari history helpful! The YUI Browser History Manager looks really cool and I think it might help bring the fragment identifier history technique to more real-world websites.

    One limitation you mentioned in the YUI Browser History Manager is that the length of the fragment identifier limits the amount of state information that can be stored. I’ve been working on a newer history library, called HistoryState, that resolves this problem by using various tricks (which vary based upon the browser) to store session variables (a “session” meaning a contiguous set of fragment identifier changes on one page in the browser’s history stack). It also uses the scrollTop trick in Mozilla so that two identity URL fragment identifier entries at different indexes of the session history can be uniquely identified. Basically, it allows the developer to keep the URL clean and free of session state information and other noise, while storing a very large amount of session state information transparently (at least several hundred KB!).

    It works very well in IE and Firefox, and I know I can pull it off in Safari and Opera as well but I haven’t gotten around to it. Also, there is a major blocking issue in Firefox (Mozilla bug #348857) that causes Firefox to forget this state information if the user leaves the page then returns to the middle of the page’s session by using the back or forward button dropdown menu.

    If any Yahoo! engineers are contributors to Mozilla’s code and could help with this, I would really appreciate it…

  17. been waiting for this one…been implementing my own based on Brad’s RSH and EXANIMO’s StateManager.

  18. Thanks, Julien … kudos to you for getting the high-quality API into the 2.2 release.

  19. Very cool stuff and well-written, the implementation details and write-up make it all the more interesting.

    I’ve struggled with some of these issues and ultimately never figured out how to get the hash working in Safari, so it’s great to see another new implementation where someone’s solved it. This utility will be incredibly useful in the right hands.

  20. I just wanted to post an updated link for the unFocus.History stuff: http://www.unfocus.com/projects/source/

    The site is terrible, I’m aware of that, and hopefully I’ll get around to fixing it soon. There is a bunch of info that I have stored in my head about all the various problems in the various browsers, that I’d be happy to share with anyone that wants to know (much of it is stored in the forum on my site). I’ve been trying to migrate some of that information from my head into the source comments, but progress is slow.

    BTW, the Safari support has been updated in the unFocus.History script. It now supports Safari 1.2+

    That’s it. Enjoy. :-)

    Kevin N.

  21. [...] If you don’t know what this means, you can read this because it does a really good job of explaining it. [...]

  22. [...] YUI Browser History Manager, Julien Lecomte This is probably the culmination of all previous attempts at solving the back / forward button issue. Without a doubt, and even though it is “experimental”, it is certainly the most robust history manager out there in terms of cross-browser support. Since it’s still pretty new, I highly recommend you take a good look at Lecomte’s YUI post to see how he built it and understand the choices JS developers have right now as well as how he overcame some of the development issues. I had dsHistory almost done at the time this came out, so I can’t say I really used it for much insight since it just wasn’t available. However, as far as the implemtation goes, it’s really incredible. In spite of this though, I still wanted to finish my little pet project since the YUI History Manager still didn’t do exactly what I was hoping to do. [...]

  23. [...] YUI Browser History Manager, Julien Lecomte [...]

  24. [...] YUI Browser History Manager, Julien Lecomte [...]

  25. It is so strange that the paragraph “After reading David Bloom’s post on using document.body.scrollTop … not that it sh” got clip off under IE6. Ok in FF. You may want to know since you are talking about x-browser support here…

  26. [...] Building the YUI Browser History Manager [...]

  27. [...] I wrote recently that this is one of the unsolved problems with applications that use a lot of AJAX. Well, Julien Lecomte from the Yahoo! User Interface team has come up with a possible solution that he’s calling Browser History Manager. [...]

  28. I just read this post and I wondered about this: you are part of the “DHTML Evangelist team, a group that provides architectural assistance to Yahoo! developers on the design and implementation of rich interactions in the browser.”

    Are you seriously evangelizing AJAX for regular navigation of webSITES? If so, I think you and/or your team are on the absolute wrong track, because navigation of websites is already solved, has been for 10+ years.

    It’s for state changes in single page webAPPLICATIONS that we need to have custom history, because this kind of ‘navigation’ is not tracked by the browser.

    It comes down to the difference between websites and webapplication, and I hope your team will not convince developers to create complicated navigation using javascript when the regular navigation is sufficient. The following quote is where I think the mistake is made (the simple website is not a rich application):

    “Only recently did I find myself impacted in my own work by the absence of a good solution for managing navigation history and session state in a rich application. At the time, I was working on a very simple web site for my father’s new venture.”

    So, the history manager is great and needed, but don’t advocate its use for regular navigation, emphasize the use in webapplications.

    Good luck!

  29. Administrator said:
    October 24, 2007 at 4:43 am

    @http://yuiblog.com/blog/2007/02/21/browser-history-manager/#comment-216276>Mike

    My colleague Julien can answer for himself, of course, but I’ll jump in right now and say that I agree with you.

    In fact, the sane guidance you’re offering is what Julien provides to teams across the company. Knowing when *not* to do something is just as important as when to do something.

    Thanks for the clarifying question.
    Nate

  30. [...] support. Since it’s still pretty new, I highly recommend you take a good look at Lecomte’s YUI post to see how he built it and understand the choices JS developers have right now as well as how he [...]

  31. [...] YUI solution to the back button is much more complex than I initially thought (then I found this really great post about what it took to create it). Hopefully I didn’t think about that because it is late. Anyway, I don’t want to [...]

  32. Hi Julien,

    I have a quick question on the use of History Manager:

    It seems that when the Back button is pressed, the manager updates the current state automatically. How do I go about implementing a validation callback so that when ‘back’ button is intercepted I can decide based on that callback’s returned value to block the operation and leave the app is the state it was prior to pressing ‘back’ button?

    Thanks!
    Ran

  33. @Ran

    Unfortunately, you can’t prevent the state of the page to remain the same when the user clicks the navigation buttons.

    – Julien

  34. Thanks so much! Because of the awesome B.H.M. script I was able to make my ajax truly functional. I can hardly wait to see the looks on the faces of other developers when I show them the site :)

  35. Is there any likelihood that the state returned by getCurrentState during initializatiobn be different from the the initial state registered at all?

    Here is my thought about the state issue.
    1. the state is fully determined by bookmark if there is a bookmark, so the current state is the state of bookmark.
    2. the state returned by getCurrentState during startup is actually always the same as the registered state retrieved from bookmark, since
    2.a. the hidden field _stateField’s value is always empty druing startup, since the browser will not set its the value to value set up by script, instead the browser set it to the value downloaded from server, so it’s always empty. if _stateField’s value is empty, then getCurrentState during startup return the state as is regitered initially.
    2.b even if 2.a may not always be true, the value in statefield should corresponds to the bookmark, so the result returned by getCurrentState during startup is still the same as the state by bookmark.

    If my thinking is correct, then I think your pattern of BHM samples should be re-examined.

  36. Hi,
    I’ve been trying to figure out how to use YUI history manager to “update” history rather than just “add” to it (eg: you delete a page, and you’d like to replace the url to point to the next valid page, so that clicking back shouldnt point to the deleted page).

    Would anyone know if YUI history manager allows this? Thanks.

  37. @Riyaz — The best place to ask questions about YUI History (or any other YUI component) is on the forums: http://yuilibrary.com/forum/ . -Eric

  38. Saving state to hidden field is an issue with dynamic pages. The value is always the value set by server when the page load.