A user reports that Show Source with the format set to TeX opens a new tab, not a popup window using Firefox 3.6.8 (Linux x86_64). I couldn't repro this on Windows. The tests were conducted on the mathjax.org TeX samples page.
Davide P. Cervone
This is something that is controlled by the browser, not MathJax. There are Firefox plugins that allow users to force pop-ups into tabs, for example, and there are configuration settings that do the same.
Have the user enter "about:config" in the URL and then "browser.link.open_newwindow" in the serach box. He or she should see two entries, one for browser.link.open-newwindow and one for browser.link.open_newwindow.restriction. These should have the values 3 and 2 respectively by default. If the second is not 2, then that could be the source of the problem.