Insertion of tracing code using the C++ preprocessor

A few months back I got this wild idea that you could insert trace-logging code (not jit-tracing, that's something else) into more or less arbitrary C++ by redefining certain C++ keywords as macros. It turns out that you can, in fact, make this work, at least for certain non-trivial cases. Notably, I've been able to compile an instrumented version of Firefox on OS X using this technique.

I'm not sure whether the technique is generally useful, but given our new focus on crashes and hangs, I thought I'd at least mention it. It's probably not particularly useful for crashes but it may have some utility for debugging hangs. I used an earlier version of this code to debug a Firebug hang (Bug 497028), although in that case I already had a good idea where the problem was and I probably could have used the debugger just as effectively.

Here's the latest tracing code, which I have in a file called "quicktrace.h":

  #ifdef __cplusplus

#include <stdio.h>
#include <string.h>

static void QuickTrace(const char *fileName, int lineNumber)
{
if (lineNumber % 100 != 0) return;
// if (strstr(fileName, "/js/") != NULL) return;
fprintf(stderr, ";;; %s %d\n", fileName, lineNumber);
}

#define if if (QuickTrace(__FILE__, __LINE__), 0) {throw 0;} else if
#define for if (0) {throw 0;} else for
#define switch if (0) {throw 0;} else switch
#define do if (0) {throw 0;} else do

#endif

I inserted this code globally by adding the following two lines to my .mozconfig:

  export CXXFLAGS="-include /Users/cbartley/dev/mozilla-e/src/quicktrace.h"
ac_add_options --enable-cpp-exceptions

Note that the use of exceptions is non-functional -- the only purpose is to suppress warnings about code paths that don't return a value. If you don't care about the warnings you can dispense with the "throw 0" statements in the header file and the --enable-cpp-exceptions in your .mozconfig file.

Notes:

  • This is unquestionably an abuse of the preprocessor. Don't go down this route unless you really think you know what you're doing.
  • If you really want to create an instrumented build, there are better ways to do it, for example Taras Glek's "Pork" tool. Even using shell scripts and sed to create an instrumented source tree might be a better choice.
  • I think the code in this post will probably work on any GCC-based build, but I don't know for sure. As far as I can tell, Visual C++ doesn't have an equivalent of GCC's "-include" flag and I haven't tried this on Windows.
  • To do useful trace logging you really want a global implementation of the "QuickTrace" function. This is somewhat challenging for Firefox because of the way it's linked. I'm leaving this part as an exercise to the reader.
  • The Firefox build actually compiles some tools from source and then uses those tools later on in the build. This means that if you're going to generate logging from the trace function you do not want to send it to stdout.
  • It might make more sense to include the quicktrace.h header file from another very commonly used header file. This might be the only option on Windows anyway.
  • I think it should be pretty easy to build an instrumented Firefox using this approach and the Try server. I haven't actually attempted this yet, though.
  • If you generate output every time the trace function is called, it will be unbearably slow.
  • You can't generate a proper call graph using this approach, although with some post-processing you could probably get a useful approximation of a call graph.
  • There's probably some caveat that I forgot to mention.

 

Comments [2]

404 Error Page Status

Feature Description

We want Firefox to override server-supplied 404 error pages in certain cases.  The Firefox 404 page should provide the user with tools we think are more useful to the user for resolving the situation, than the server supplied page.  This includes alternate ("did you mean") URLs derived from the Places database and some pre-loaded search links/controls.  This is being tracked in bug 482874.

Status

On hold for the week while I worked on higher priority Firefox 3.6 stuff.  There are still some L10N issues to address and some general touch up before I can roll a new patch.

Next Steps

Address remaining issues raised in last round of code review and get another patch out.  This will probably have to wait until after the Firefox 3.6 code freeze that's coming up.

Notes

  • The feature is controlled by a pref so it can be turned off.
  • Webmasters who don't want their 404 error pages to be overridden may have to add padding to their 404 error pages.  However, since IE and Google Chrome are already overriding 404 error pages using a similar size test, webmasters already need to do this.
  • We want to provide a way for the user to see the original 404 error page, but that's not in this patch.

Comments [2]

Troubleshooting Information (AKA about:support) Status

Feature Description

Mozilla's support organization has a longstanding request for a Firefox diagnostic page that can provide information about the user's Firefox installation such as which extensions are installed and what prefs have been modified.  The result is the "Troubleshooting Information" page, which is also accessible by typing "about:support" in the location bar.  This feature is fairly constrained for 3.6 since we didn't start work on it until a few days before string freeze. 

Screenshot: https://bug367596.bugzilla.mozilla.org/attachment.cgi?id=400664
Project Page: Firefox/Projects/about:support
Bug: 367596 -  (about:support) [RFE] Create a support info page/lister.

Status

  • Landed!  Troubleshooting Information is now available in both trunk and Firefox 3.6 nightlies.
Next Steps
  •  Bug 518601 -  Troubleshooting Information page should not allow copy-and-paste of the profile directory.  This is a potential security issue and needs to be addressed before Firefox 3.6 ships.
  •  Bug 516616 -  Add an "Installation History" section to about:support.   Nice to have for 3.6.
  •  Bug 516617 -  Add an "Update History" section to about:support.  Nice to have for 3.6.

Notes
  • This is a starting point, not an ending point.  We can extend the functionality for Firefox 3.7.

Comments [3]

Troubleshooting Information (AKA about:support) Status

Feature Description

Mozilla's support organization has a longstanding request for a Firefox diagnostic page that can provide information about the user's Firefox installation such as which extensions are installed and what prefs have been modified.  The result is the "Troubleshooting Information" page, which is also accessible by typing "about:support" in the location bar.  This feature is fairly constrained for 3.6 since we didn't start work on it until a few days before string freeze. 

Screenshot: https://bug367596.bugzilla.mozilla.org/attachment.cgi?id=400664
Project Page: Firefox/Projects/about:support
Bug: 367596 -  (about:support) [RFE] Create a support info page/lister.

Status

  • The implementation went through seven or so revisions in the days just prior to Firefox 3.6 string freeze.
  • Current page design includes "Application Basics" (App name, version, profile directory, etc.), "Installed Extensions", and "Modified Preferences".
  • Strings for all of the above were landed on trunk and the 1.9.2 (Firefox 3.6) branch in a strings-only patch.  This patch also included strings for "Installation History" and "Update History" sections, in hopes that we can get those features in as well for Firefox 3.6 (Bugs 516616 and 516617).
  • Discovered a performance bug in the FUEL preferences API (Bug 517312).  I've rewritten part of this code to use the lower-level preference API, which avoids the bug and is just much speedier all around.
Next Steps

Address the remaining issues from the last round of code review, and see if we can get the main patch landed.

Notes
  • This is a starting point, not an ending point.  We can extend the functionality for Firefox 3.7.

Comments [1]

404 Error Page Status

Feature Description

We want Firefox to override server-supplied 404 error pages in certain cases.  The Firefox 404 page should provide the user with tools we think are more useful to the user for resolving the situation, than the server supplied page.  This includes alternate ("did you mean") URLs derived from the Places database and some pre-loaded search links/controls.  This is being tracked in bug 482874.

Status

Final work was delayed while I worked on about:support/Troubleshooting Information.  Since then I've rewritten the Places part of the patch in JavaScript as requested in the most recent code review.  I still have some L10N issues to address and some general touch up.

Next Steps

    Pinky: "Gee Brain, what do you want to do next week?"
    The Brain: "The same thing we do every week, Pinky—try to land this patch!"

Notes

  • The feature is controlled by a pref so it can be turned off.
  • Webmasters who don't want their 404 error pages to be overridden may have to add padding to their 404 error pages.  However, since IE and Google Chrome are already overriding 404 error pages using a similar size test, webmasters already need to do this.
  • We want to provide a way for the user to see the original 404 error page, but that's not in this patch.

Comments [4]

404 Error Page Status

Feature Description

We want Firefox to override server-supplied 404 error pages in certain cases.  The Firefox 404 page should provide the user with tools we think are more useful to the user for resolving the situation, than the server supplied page.  This includes alternate ("did you mean") URLs derived from the Places database and some pre-loaded search links/controls.  This is being tracked in bug 482874.

Status

Found and worked around an SQLite bug (Bug 514291).  Did some performance analysisis, suggestions query tweaking, etc.

Latest patch is attached to the bug and review is requested.

A try server build is available

Next Steps
  • If the reviews come back positive, get it landed.
  • If the reviews raise issues, address them ASAP.
Notes
  • The feature is controlled by a pref so it can be turned off.
  • Webmasters who don't want their 404 error pages to be overridden may have to add padding to their 404 error pages.  However, since IE and Google Chrome are already overriding 404 error pages using a similar size test, webmasters already need to do this.
  • We want to provide a way for the user to see the original 404 error page, but that's not in this patch.

Comments [4]

404 Error Page Status

Feature Description

We want Firefox to override server-supplied 404 error pages in certain cases.  The Firefox 404 page should provide the user with tools we think are more useful to the user for resolving the situation, than the server supplied page.  This includes alternate ("did you mean") URLs derived from the Places database and some pre-loaded search links/controls.  This feature is being tracked in bug 482874.

Status

Almost ready to go.  Notable progress this week:

  • The docshell changes have r+ from bz with just a few changes, most of which I've already made.
  • Axel has provided some feedback on localization.
  • Johnath has provided feedback on the XHTML.
  • Marco has provided some feedback about the Places changes.  Notably he points out that query performance could be an issue on large databases.
  • David Dahl has provided me with a "max_places" test database.  The bad news is the URL suggestions query can take ~30 seconds to run against this database.  The good news is that a very simple length-check guard in the query can make a big difference.
Next Steps
  • Address  the localization changes Axel raised, and then get formal L10N review.
  • Address the performance issues in the URL suggestions query.  (Probably just a LIMIT-based approach for the initial patch)
  • Get a formal review of Places changes (Dietrich did an informal review a few weeks ago).
  • Get a final UI review.
  • Make sure all the parts of the patch have r+ from the various parties.
  • Land the thing.

Comments [10]

404 Error Page Status

Feature Description

We want Firefox to override server-supplied 404 error pages in certain cases.  The Firefox 404 page should provide the user with tools we think are more useful to the user for resolving the situation, than the server supplied page.  This includes alternate ("did you mean") URLs derived from the Places database and some pre-loaded search links/controls.

Status

The newest patch is almost ready for review.

This patch is mostly focused on making the code right, but there are a couple of functional changes:

  • The Firefox 404 error page is now pref-controlled.  There's a pref to turn it on and off, and a pref to limit which server-supplied 404 pages we'll override.  Currently we will only override server 404 pages of 512 bytes or less.
  • We're dropping the background image for now.


Next Steps

  • Complete the patch and get it in for review.
  • Blog about it.

Notes

  • This feature is being tracked by Bug 482874 - Provide a friendlier/more useful alternative when the user encounters a 404 error page.

Edit: Added Notes section.

Filed under  //   firefox   mozilla  

Comments [8]

Building Firefox under Eclipse/CDT, or I gotta have my identifier completion

Introduction

I've been using Eclipse as a C++ IDE for Mozilla development, which apparently puts me in a distinct minority among Mozilla developers.  Since I'm neither an Emacs nor a Vim user, I needed an alternative.  Frankly, Eclipse is not all that great as an editor, but having an IDE with halfway decent identifier completion and code browsing goes a long ways towards making up the difference.  It's not all that hard to set up an Eclipse project for mozilla-central (i.e. Firefox), but it's not exactly obvious either.  The purpose of this blog post is to explain one way to do so.

Disclaimer and Caveats

I'm not claiming that this is the optimal way to set up mozilla-central under Eclipse.  In fact, I'm not even claiming that it's right.  All I'm saying is that this approach is relatively simple and seems to actually work.  For expediency, these instructions make some assumptions.  First, they assume that you are developing under OS X.  I believe that they are easily adaptable to Linux and Windows+cygwin (or Windows+mozilla-build), but I don't know for sure.  Second, the instructions assume that you are already able to build mozilla-central from the command-line.  If you don't already know how to do that, then you don't want to start here.

Disclaimer #2

As a final disclaimer, this approach is relatively new to me.  I've mostly been using an older version of the CDT under Eclipse 3.0 and I'm not even going to mention what I had to do to get indexing to work properly.  This new approach is a lot simpler, and I'm reasonably confident that it will work well for sustained use.

Screenshots

I originally wrote these instructions in text-only form.  I've since added some marked-up screenshots for many of the steps.  These screenshots are attached at the end of the post.

Acknowledgements

And last, but not least, thanks to Mike "firetoad" Collins for trying out a slightly earlier version of these instructions.

Now the instructions.

Install Eclipse/CDT

  • Obviously you need Eclipse.  I will offer some advice, but I'm largely assuming that you can figure this part out for yourself.  These instructions assume that you are using Eclipse 3.5, better known as Galileo.  You need Eclipse with the CDT, the "C/C++ Development Tooling".  The easiest way to get this is to go to http://www.eclipse.org/downloads/ and download the pre-packaged Eclipse IDE for C/C++ Developers, which is what I've done.  I have an existing Eclipse installation, but rather than trying to upgrade, I've just got them side-by-side, which doesn't seem to be a problem as long as you set up separate workspaces for them.
  • By default Eclipse runs with a 256M heap, which is too small to reliably index the mozilla-central tree.  I've been running with 512M without any problem, although I've not really pushed it very hard yet.  There ought to be a way to configure the Eclipse heap size through the UI, but if there is, I haven't found it yet.  In the meantime, I've been launching it from the command-line, like so:
    • /Applications/eclipse-galileo/Eclipse-galileo.app/Contents/MacOS/eclipse -vmargs -Xmx512M &
  • Your path will differ, of course.  Also note that the path above is a little bit unusual because I've renamed the actual Eclipse application so it won't conflict with my other (older) Eclipse installation.
  • By default, Eclipse engages "Scalability Mode" for large files, which means it turns off most of the cool features.  I've increased the threshold to 15,000 lines (nsDocShell.cpp is about 11,000 lines of code, for example).  To change this setting, choose Eclipse >> Preferences, then type "scalability" into the search box.

Create a new project

  • Start Eclipse (but see the note above about making sure you're running it with enough memory).
  • Ctrl-click in the Project Explorer, choose New >> C++ Project
    • this will open the C++ Project dialog
  • In the C++ Project dialog,
    • In the Project name text field, enter the project name
      • I'm using firefox-eclipse
    • Uncheck Use default location
    • In the Location text field, enter the full path for the project source code.
      • I'm using /Users/cbartley/Dev/firefox-eclipse/src
    • Under Project type, choose Makefile project >> Empty Project
      • Note that other project options can cause Eclipse to create a default makefile, which you don't want.
    • Click Finish
  • You should now see the project firefox-eclipse in the Project Explorer.

Configure the project

  • Ctrl-click on firefox-eclipse in the Project Explorer, and choose Properties.
    • This will bring up the Properties for firefox-eclipse dialog.
  • In the Properties for firefox-eclipse dialog,
    • Select the Resource panel
      • Note the path after Location in the Resource panel.
        • This is the same path you entered a few steps above, but if you're like me, you've probably forgotten it already.
    • Select the C/C++ Build panel
      • Select Builder Settings inside the C/C++ Build panel
        • Uncheck Use default build command
        • In the Build command text field, enter:
          • bash -l -c 'make -f client.mk $0 $1' -b
            • this hairy looking command invokes a bash shell which in turn invokes the actual make command.  The dash-small-L option tells it to behave like a login shell.  This sidesteps some environment problems that I've run into when running make directly from Eclipse.  I won't elaborate on the $0, $1, or -b, except to tell you to be careful that you put the quotes in the right place!
        • In the Build directory text field, enter the path to the project source
          • I'm using /Users/cbartley/Dev/firefox-eclipse/src
            • Note that this the same path found in the Location text field on the Resource panel. I usually just Cmd-C copy it from there any more, to reduce the possibility of a typo
      • Select the Behavior tab inside the C/C++ Build panel
        • In the Build (Incremental build) text field, delete all and just leave the field empty.
    • Expand the C/C++ Build tab to show the tabs for its subpanels.
      • Select the Settings sub-panel
        • Select the Mach-O parser under Binary Parsers
    • Expand the C/C++ General tab to show the tabs for its sub-panels
      • Select the Index sub-panel
        • Check Enable project specific settings
        • Under Select indexer
          • Choose Full C/C++ Indexer (complete parse)
          • Check Index unused headers as C++ files
    • Click Apply
  • The project is now set up, but there's no source code yet.

Get the Source Code

  • Eclipse has created the directory for the project, but it's empty.  I've created my project at /Users/cbartley/Dev/firefox-eclipse/src.  Following standard convention, I want to check the mozilla-central project out to a directory named src, the same one as above, in fact.  The problem is that Mercurial complains if the directory already exists.  Checking the source out first doesn't help, because then Eclipse complains that the directory already exists.  I'm going to check mozilla-central out to a different location and then manually merge them.  Note that Eclipse has created some files inside the src directory that we want to preserve.
  • cd into the project directory
    • I do cd /Users/cbartley/Dev/firefox-eclipse/src
  • cd out one level
    • cd ..
      • I'm now in /Users/cbartley/Dev/firefox-eclipse
  • Now type the following commands
    • hg clone http://hg.mozilla.org/mozilla-central src-aside  # get the source code
    • mv src-aside/.h* src                                      # move mercurial files into src
    • mv src-aside/* src                                        # move regular files into src
    • rmdir src-aside                                           # remove the now superfluous src-aside directory

Set up the .mozconfig file

  • Create a basic .mozconfig file for the project; I'm assuming you've already got a usable one some place.
    • For example, I do cp ../mozilla/src/.mozconfig src
  • Make sure the "-s" option is NOT on for make
    • Normally make displays the complete command line for building each file.  When invoked with the "-s" option, however, it only displays the name of the file being built.  This is a problem for building under Eclipse, since Eclipse parses the build output to figure out where the header files are.
    • In my existing .mozconfig, I have the following line:
      • mk_add_options MOZ_MAKE_FLAGS="-s -j4"        # before
    •     I delete the "-s", leaving
      • mk_add_options MOZ_MAKE_FLAGS="-j4"           # after

Build the project under Eclipse

  • Make sure that firefox-eclipse is selected in the Project Explorer.
  • Select the Console tab in the bottom pane so you can see the build output
  • Select Project >> Build Project from the menu
  • Wait

Build the project under Eclipse, again

  • We want to do Clean rebuild under Eclipse; This extra step seems to be necessary for Eclipse to find the IDL-generated header files.  Don't ask me why.
  • Make sure that firefox-eclipse is selected in the Project Explorer
  • Select the Console tab in the bottom pane
  • Select Project >> Clean... from the menu
    • Make sure that Start a build immediately is checked
    • Click OK
  • Wait

Make sure indexing is started

  • After the build completes Eclipse should start Indexing automatically.  A progress indicator should appear in the status bar on the lower right.
  • If indexing hasn't started, you can invoke it manually by:
    • Ctrl-click on firefox-eclipse in the Project Explorer, and choose Index >> Rebuild
  • Indexing takes something like four hours on my machine.  Eclipse remains usable in the meantime, you just won't have access to the more advanced code-completion and browsing features.

After Indexing is completed

  • Cmd-Shift-R is kind of like the Awesome Bar for source files.
  • Cmd-Shift-T is kind of like the Awesome Bar for types, functions, etc.
  • Ctrl-Space invokes identifier completion.
  • Just typing identifier. or identifier-> will show a list of member variables and functions if you pause for a second.
  • F3 will take you to the declaration of an identifier
  • F3 over a #include will open the header file.
  • Hovering over a macro will show you its expansion.
  • Actually, many of the features will work in full (e.g. Cmd-Shift-R) or in part (e.g. identifier completion) before indexing is completed.

                     
Click here to download:
Building_Firefox_under_Eclipse.zip (1101 KB)

Comments [4]

How do you surf the firehose?

How do you surf the firehose?

I'm not sure that "drinking from the firehose" is the right metaphor for Mozilla.  I still claim it's more like having people constantly shooting you with water pistols from every different direction.

There's not much method to my madness, but here's how I try to keep up with the firehose:

I'm subscribed to four newsgroups in Thunderbird:

  • mozilla.dev.planning
  • mozilla.dev.apps.firefox
  • mozilla.dev.platform
  • mozilla.dev.tree-management
I don't actually read them any more, with the exception of mozilla.dev.planning which I now have set up to deliver straight to my main Mozilla email account.  (Because Mike Beltzner told me to.  "There's hardly any traffic on that list anyway" I recall him saying.)  Since conventional email is hardly used at Mozilla, this seems to work OK.

I also have bugzilla notifications going to my main email account.  I don't know what I was thinking.

I'm subscribed by email (my personal email) to the Firebug Group on Google Groups.  In hindsight this was a mistake, but I haven't fixed it yet.  I'm mostly skimming the thread subjects at this point, but I at least have some inkling of what's going on there.  I check this a couple of times a day when I check my personal email.

I read planet.mozilla.com every couple of days.  Anymore I mostly skim it, reading just the occasional post.  I really should try to check it every day, since I find it pretty useful to the extent that I do read it.

I peruse reddit and news.ycombinator.com (Hacker News) several times a day, and slashdot every now and then.  If a blog post doesn't make it to reddit or Hacker News, I probably won't see it.  I don't feel like I'm missing a lot of stuff by doing this, but on the other hand I do feel like I have to wade through a lot of crap to find the stuff I do want to read.  (Ha, Sturgeons Law: 90% of everything is crap.  Curtis's corollary to Sturgeons Law: On the Internet, 90% of the rest is crap too.)

I don't do twitter, largely because it seems to scratch an itch I don't have.

I have three hours or so of phone meetings every week, not counting the weekly Mozilla.org meeting.  I take these meetings fairly seriously since I'm working remotely.  I don't take notes, but I do attempt to actually pay attention.  I'm prone to zoning out even in meetings I'm physically in, and it's worse when I'm dialed in remotely.  My secret weapon?  I usually surf pictures on Flickr while I'm on the phone.  This works surprisingly well since it doesn't require any major effort from the verbal part of my brain.  In the future I should probably take notes at least some of the time.  Like any time Mike Beltzner is saying something important.

I don't use a feed reader.  In the past I've used Bloglines and Google Reader.  I abandoned Google Reader because I always hated its infinite scrolling model.  (I don't remember why I hated it; I like infinite scrolling in other cases) .  I do have a master list of blogs I read.  Despite the dozens of blogs on this list, I only read a few regularly these days, and none of them are technical blogs anymore.

And of course Mozilla lives and breathes IRC.  I'm using Colloquy as my IRC client.  It's not wonderful, but it works OK.

Tag, you're it.

Robcee's original post (complete with standard questions which I totally ignored) was designed to be viral.  I can't beg off this part, since I put him up to it.  Seriously, though.  I'm only asking because I think I might learn something.

Comments [2]

About