Curtis Bartley’s Blog

Using MemTrack to instrument Firefox's memory management

MemTrack is a library for tracking memory allocations in C++ programs.  I had two goals with MemTrack.  The first was a simple demonstration of the classic C/C++ preprocessor-based technique for instrumenting memory allocations, which I'm sure a lot of people have done, but few people have documented.  The second was as a testbed for a rather novel technique for tagging each memory allocation with type information, something that is not normally possible with the preprocessor macro-based approach.

The macro-based approach has some serious limitations, but for some codebases it can work quite well.  I tested against my FH-reader library (a library for reading Macromedia FreeHand files) which was an ideal test case.  All I needed to do was add a '#include "MemTrack.h"' directive in a strategic location and then add one call to the test harness to dump the gathered information after the test run was complete.

I was curious how well the technique would work for a large real world codebase, so I tried it out on the Mozilla Firefox codebase, which is on the order of 2 million lines of C++.  This was kind of a brutal exercise, but I was able to get some potentially useful data out of the effort.  The Firefox team has already used other tools for the substantial memory usage optimizations that were done for Firefox 3.  MemTrack's memory tracking would be a poor substitute for what those tools would provide.  However, MemTrack's ability to annotate allocations with C++ type information might be a useful extension.  My understanding is that type information was already available for XPCom objects but not for other C++ types.

In order to use MemTrack at all on the Firefox codebase, I had to resort to a static pre-processing step.  One purpose of this step is to insert a '#include "MemTrack.h"' directive at the beginning of every .cpp file.  The other major purpose of the static pass is to temporarily suspend the new macro in places where it's going to cause problems.  A common case is any class that declares and defines a custom operator new.  The static preprocessing code is simply a python script which relies on a rather scary regular expression which identifies individual lines in the source code as being problematical.  The script simply undefines the new macro above the problematical line and redefines the macro below it.

For exampe:

#ifdef MEMTRACK_NEW        //:memtrack!
  #undef new               //:memtrack!
#endif                     //:memtrack!
  void* operator new(size_t aSize, nsIPresShell* aHost) {
#ifdef MEMTRACK_NEW        //:memtrack!
  #define new MEMTRACK_NEW //:memtrack!
#endif                     //:memtrack!

There aren't that many files that declare or define operator new but for expediency I ended up wrapping any preprocessor directives that include system headers.  This is probably not a big deal either way, since one would rarely do much debugging or development against an instrumented build.  Nevertheless the amount of explicit instrumentations could probably be reduced, and that would be a good thing.

To sum up the current state of this exercise: I have to statically pre-process the Firefox codebase in order to use MemTrack.  This preprocessing step is pretty much completely automatic on the Mac although it's still pretty crude.  To make it actually useful, MemTrack's tracking layer should be stripped out and the MemTrack instrumentation layer should be piggy-backed on top of the existing memory instrumentation code.  The required static preprocessing step is somewhat of a hassle, but it's not really all that onerous.  If one has to do a special build of Firefox to instrument memory usage anyway, it's probably not a big deal to include a static source preprocessing step as well.  The key question is whether the extra type information that can be gathered by MemTrack is sufficiently useful.  I think that it might be, but I'll need to do some more investigation before I can say for sure.

For future reference, the regular expression I'm using is:

    newIgnorePat = re.compile(r"operator new|include NEW_H|NS_DECL_AND_IMPL_ZEROING_OPERATOR_NEW|::new[ (]|include [<]|[)] *new[ (]|[*] new[ (]|!new[ (]|include.*nsILocalFileMac|include.*nsIInternetConfigService|new PathChar")

This pattern includes common cases like occurrences of "operator new" (not just "new") and include directives for system headers as well as some special cases like "new PathChar", which is specific to Firefox on the Mac.

Firebug UI Notes: Is it a button or is it a menu?

Firebug crams a lot of disparate UI into a relatively small amount of screen real estate, and consequently it can't help but be noisy and confusing.  However, I think there's some obvious areas for improvement, and I wanted to document some of the problems I've been noticing as I go along, and maybe propose some solutions while I'm at it.

First up: Is it a button or is it a menu?

One of the most notable aspects of the Firebug UI is that it wants to stick menus all over the place, including on some of the tabs.  Firebug is not very consistent about how it does this, and so the user is left wondering: Is it a menu, or just a button?  For example, in the screenshot below, you can see two Firebug bug icons -- one on the right end of the Firefox status bar at the bottom of the window and another one (slightly larger but otherwise identical) as the first control on the Firebug toolbar.  The small icon behaves as a simple button -- it toggles Firebug on and off.  The larger icon actually triggers a menu.

Next on the toolbar is the Inspect button, which always appears in the location, regardless of the current tab.  The Inspect button triggers a mode where you can click on elements in an HTML page and have them show up in Firebug.  Continuing to the right, the "all" button actually controls a menu, and there's a helpful arrow next to it to indicate that there's a menu there.  Next is the button that's labeled "slideshow.html".  It's also a menu, only this time there's no helpful arrow to indicate that.

One simple improvement that can be made here is to consistently use the arrow next to all menu items.  I'd also consider replacing the Inspect label with an icon, and the Firebug icon with just "Firebug", so the button is an icon, and the menu has a label.  This sounds more natural to me, although in terms of actual usability, I don't know if it would make a big difference.

This is just scratching the surface, but it seemed like a good place to start.

Firefox weirdness on OS X when launching from the command-line

Mozilla issued me a new MacBook, and I've been trying to get it set up for development.  In particular I wanted to get it set up for Firebug development since that's my primary focus right now.  This should be easy -- Firebug is just an extension written in JavaScript.  However, one thing I wanted to do is set up a build script that will automatically launch Firefox using a specific profile just for testing.  It's not at all hard to launch Firefox from the command-line in OS X, except that I was getting really bizarre behavior: A Firefox window would appear, but there would be no menu bar, and indeed, Firefox didn't show up in the applications list when you hit command-tab.  Even worse, there was no way to get keyboard focus into the window.  You could close the window by clicking on it, but then you'd have a zombie Firefox process that you couldn't quit, and you couldn't Force-Quit it, because it didn't show up in that list either.  It was necessary to kill it from the command-line.

I tried everything I could think of, and a handful of times, everything worked, and then it went back to being broken.  Eventually I discovered that the path to the executable that I typed out differed from the path in the filesystem by the case of a single letter.  I had /Applications/Firefox.app/Contents/MacOs/firefox-bin, when what I really wanted was /Applications/Firefox.app/Contents/MacOS/firefox-bin (MacOs ==> MacOS).  That was it.

Addendum: This is now bug #453913 in Bugzilla.

 

My first day working for Mozilla

Today was my first day working for Mozilla Corporation.  It was pretty uneventful, which is no real surprise since I'm working from home in Seattle rather than from Mountain View or Toronto.  I will be splitting my time between Firefox UI and Firebug, although I'm not sure what that means yet. In the short term I'm looking at Firebug, which has the benefit of being a much smaller project than Firefox, both in terms of source code and number of people working on it.  If that means I can make a useful contribution in weeks rather than months, then I'm all for that.

Deep Search: Extending Google search into the web page

My most recent project has been a Firefox extension called "Deep Search". The idea is to extend Google search one level "deeper", into the returned web pages themselves. This is by no means a new idea. Google already does this for cached pages -- if you view a cached page the search terms will be highlighted. Google can't do this for the real web pages, of course, but the Deep Search Firefox extension can.

Deep Search also attaches decorations ("marks") to the scroll bar to show at a glance where in the web page the search terms were found. One of the limitations of the current Deep Search extension is that the marks appear next to the scroll bar rather than directly on it. This is not ideal but it's acceptable for a proof-of-concept. You can click on a particular mark to immediately scroll to that search term match.

I've attached a screenshot below. I've circled the changes made by Deep Search in red. The search term is highlighted and the "mark" is right next to the scrollbar thumb in the lower right corner. The gray line at the top of the mark shows the precise position of the search term relative to the scrollbar. The red area below it is, not coincidentally, roughly the same size as the thumb. It shows where the thumb needs to be positioned so that the search term corresponding to this mark appears at the top of the window.

Crane City

When I was in Indonesia last year, one of the things I remember is how many buildings were under construction in Jakarta, each one with a tower crane. Then I got back to Seattle. I just shot this picture from my building's rooftop deck. There are three other cranes nearer by that I couldn't capture without a wideangle lens. It's pretty fuzzy, so I think I'm doing something wrong.

Who do I talk to about Firefox UI ideas

I have an idea for a Firefox feature that I think would be useful. I'll get into the idea itself in a later post. In the meantime, I've got a Firefox extension that I've written as a proof of concept, and I wasn't clear on what to do next. So I emailed my contact on the Firefox development team for his advice. Since I expect this advice will be useful for others, I'm repeating it here.

• Upload the extension to the AMO (addons.mozillla.org) sandbox.
• Blog about it, pointing to the AMO page.
• Post to DAF (mozilla.dev.apps.firefox newsgroup or its mirror mailing list).
• Go to #ux on irc.mozilla.org, point to your post, and ask for feedback.

 

First Post(erous)

I need a blog. I've experimented with Blogger in the past and I've been underwhelmed. Wordpress looks like it's just too complicated to run myself, plus I'd need to upgrade my web hosting account to use it. A hosted Wordpress blog at Wordpress.com might be an option, but it looks like if you go that route that you don't easily have access to all those Wordpress plugins. I've also investigated PyBlosxom, but I can't get it to install, although I may take another look at it when I'm more motivated. In the meantime, I'm checking out Posterous. Right now it's pretty minimalist, although in a different way than PyBlosxom and the other Blosxom variants. I really like the idea of posting by email, not least because I don't need to use one of those crummy web-based WYSIWYG editors in order to post.