DTrace Probing 0.2

What a way to cut it down to the wire.

After about 2 weeks of finishing my job at the Country Club, and with 1 week left in the project. I get this email from Vladimir Vukicevic from Mozilla.

Whoops, just now save this mail when dave pointed it out. What I can
offer as an immediate take is a patch that jonas and I put together for
basically manual probing of a few things. It's not complete, but it'll
be a start.

http://people.mozilla.com/~vladimir/misc/debug-appshell.patch

Basically look at all the printfs; they time certain things, and we'd
probably want probes at the start and end (basically where the timings
start and end). This covers the main chunks of reflow, parsing, and
event handling.

I'd like to see proposed names for all the probes and their location;
I'd suggest xxxxx.start and xxxxx.end. For example, in nsThread.cpp,
the patch has:

diff --git a/xpcom/threads/nsThread.cpp b/xpcom/threads/nsThread.cpp
--- a/xpcom/threads/nsThread.cpp
+++ b/xpcom/threads/nsThread.cpp
@@ -506,9 +506,27 @@ nsThread::ProcessNextEvent(PRBool mayWai

if (event) {
LOG(("THRD(%p) running [%p]\n", this, event.get()));
+
+ PRBool isMainThread = NS_IsMainThread();
+ if (isMainThread)
+ printf ("{");
+
+ PRIntervalTime t0 = PR_IntervalNow();
+
++mRunningEvent;
event->Run();
--mRunningEvent;
+
+ PRIntervalTime t1 = PR_IntervalNow();
+
+ PRUint32 td = PR_IntervalToMilliseconds(t1 - t0);
+ if (isMainThread) {
+ if (td > 100) {
+ printf ("-E(0x%08x,%d)", (unsigned int) (void*) event.get(), td);
+ }
+ printf ("}");
+ }
+
} else if (mayWait) {
NS_ASSERTION(ShuttingDown(), "This should only happen when
shutting down");
rv = NS_ERROR_UNEXPECTED;


So we should have an nsthread event processing start probe at the t0 =
PR_IntervalNow(); for args, initially let's pass down the thread ID and
whether the thread is the main thread or not (or actually, let's just
pass 0 for the thread id if it's the main thread). A second nsthread
event processing end probe should happen at the t1 = PR_IntervalNow bit.

The patch has a bunch of if (t > 100) { ... }, which is just for sanity
sake when printing output -- the probes should not be conditional on how
long things take.

Args for the various probes should just be kept simple for now, we can
figure out later what data we want to pass down.

The big piece that's missing here is painting; since I'm assuming we're
MacOS X only for now, the right place would be in nsChildView.mm in
drawRect -- basically at the start of drawRect and at the end of
drawRect. (I guess a probe isn't really needed, since it's just one
function, but we should have one anyway.)

Getting this stuff in would be a huge help, thanks for taking this on!
I'd love to ship with the dtrace probes enabled in release builds at
some point.

- Vlad

He has been working on this probing portion for Sayrer for some time. And has created a patch to help with debugging the code base. And basically, he would like me to add start and end probe calls on places where

PRIntervalTime xx = PRIntervalNow( );

The first one gets the start, the next one gets the end.

I think I am able to do this, but with 5 days left and the assignment due at 11:59:59 on Saturday - I will really need to cut it down to the wire.

Thankfully, I am gathering all the resources I need to work on it today. And reviewing the patch to make sure I know where things are needed to go.

Hopefully, the patch will compile with and without the probes.

Adding D-Trace to Code Base 0.1 Release

For those who need a reminder of what I am doing: My Project Page

This post recognizes that I have released my 0.1 to the public. I have released:

• Mac OS X 10.5 (Leopard) and OpenSolaris only
  
• Non-extension based
  
• Instructions to compile Mozilla for D-Tracing
  
• Test script to interact with existing probes

Basic Instructions to compiling this are as follows (more advanced and well-explained instructions are on my 0.1 Release - link is at the bottom of the page:

1. Download the Mozilla source tree. The version must be >= 3.0.3
   
2. Extract the tarball at your home directory.
   
3. Within your .mozconfig file. Add this line: ac_add_options --enable-dtrace along with your other options
   
4. Make sure your $PATH variable has at least:
   export PATH=/usr/include/sys:/usr/include:/opt/local/bin:/opt/local/sbin:$PATH
   
5. Compile the source tree.
   
6. Go to your proper executable and run firefox-bin (mozilla/dist/bin is OpenSolaris, mozilla/dist/Minefield.app/Contents/MacOS/
   
7. On a second terminal, create a file called js_callcount.d and add this code to it (use a 2nd terminal than the one handling Minefield):
   

   
   #pragma D option quiet
   
   dtrace:::BEGIN
   {
          printf("Tracing... Hit Ctrl-C to end.\n");
   }
   
   javascript*:::function-entry
   {
          @funcs[basename(copyinstr(arg0)), copyinstr(arg2)] = count();
   }
   
   dtrace:::END
   {
          printf(" %-32s %-36s %8s\n", "FILE", "FUNC", "CALLS");
          printa(" %-32s %-36s %@8d\n", @funcs);
   }
   

   
   
8. Compile and run this script with the second terminal using:
   [sudo] dtrace -s js_callcount.d
   
9. Perform some actions like go to a different web page, hit some buttons, do a Google search etc. without closing Minefield.
   
10. When you think you are done, go to the 2nd terminal and kill the script using Ctrl+C. The output will tell you how many functions were called in each file the D-Trace script picked up on.
    

You may optinally use the command

[sudo] dtrace -n 'javascript*' -l

to display any currently active probes. This will only work while Mozilla is running.

I did a test run of it myself, doing the following scenario:

1. Ran Minefield on first terminal. Created a profile called DTrace and used it
   
2. Ran my D-Script on my second terminal.
   
3. On Minefield - typed [http://www.wikipedia.org/wiki/Santino_Marella http://www.wikipedia.org/wiki/Santino_Marella] and hit Enter.
   
4. Clicked on the first instance of RAW (should be in basic description field on top of wiki).
   
5. Ctrl+C'd out of my D-Script on second terminal.
   
6. Closed Minefield and Ctrl+C'd out of my first terminal.
   

And got these results:

//DTrace::BEGIN
Tracing... Hit Ctrl-C to end.

//Started picking up on my Minefield interactions

//DTrace::END
 FILE                             FUNC                                    CALLS
 Santino_Marella                  writeln                                     2
 WWE_Raw                          writeln                                     2
 autocomplete.xml                 QueryInterface                              2
 autocomplete.xml                 attachController                            2
 autocomplete.xml                 getBoundingClientRect                       2
 autocomplete.xml                 getComplexValue                             2
 autocomplete.xml                 getIntPref                                  2
 autocomplete.xml                 handleEnter                                 2
 autocomplete.xml                 setConsumeRollupEvent                       2
 autocomplete.xml                 stopSearch                                  2
..continues on
autocomplete.xml                 hasChildNodes                              334
 autocomplete.xml                 substr                                    338
 autocomplete.xml                 setAttribute                              344
 autocomplete.xml                 parseInt                                  508
 autocomplete.xml                 min                                       592
 autocomplete.xml                 push                                      610
 javascript&smaxage=21600&maxage=86400 substr                               712
 autocomplete.xml                 getAttribute                             1506
 autocomplete.xml                 charCodeAt                               3876

The probe that handles function-entry (basically when a function is called) seems to handle more on browser.js (was cut off, but appeared) and the xml files than the other locations. A person would need this information to make sure that thier functions are being called the right amount of times. Which helps if a user needs to write more functions, cut off ones that are least used, or access more that can be used to make sure they are "earning thier keep".

To improve on 0.1 so that I can work with 0.2, I only have one thing. There was a patch released on Bugzilla Bug 370906 in Post 20/23 that renames the Probe conventions from javascript* to trace_mozilla*. This is helpful if you have 2 programs running that use the same naming conventions, and maybe the same ID (the * is a wildcard). I was able to run it on Solaris, but not on Mac. I didn't post the instructions that way because I wanted to make sure that I was cross-platform compliant. I want to try to figure out how to get it running on Mac. But I fear that Sayrer (the man who suggested that this technology would be useful for Mozilla) may not like it, or he knows how to solve my problem.

For 0.2, I plan on actually writing custom probes for other sections of the Mozilla Code base. Hopefully, where Sayrer wants them to go. Then I am gonna remodify the above script to check for these "custom probes".

I have written better instructions and easier to copy code on this 0.1 Release wiki. I apologize if my pre tags cut off the code on the screen. It is the way this page is rendered at 1024x768 that bugs me.

Open Source - Done - Now for D-Trace

After much tedious discussions on IRC (between ted and humph), a power outage in the ORI lab and a little GO bus nap, I have got my Mac compiling to work with D-Trace. And I have accomplish 40% of my 0.1 release by doing this.

It turns out, when I compiled it, there is a difference between running the Firefox exe inside the dist, then there is actually running Minefield a little deeper. The reason this was a problem was because Firefox wasn't accepting keyboard input at all (and other programs were).

Now I have D-Trace to look into. From there I can find a probe and learn to use it.

Luckily, I have the research material printed out from the web. And all this will accomplish is make life easier when I actually sit down on the Mac and continue with my 0.1 release.

For those who need a reminder: I am taking D-Trace probes, and adding them to Mozilla Firefox functions. For 0.1, I am taking an existing probe, and using it on one section of the code, then post the results of it to the people. I won't be writing any probes until 0.2.

I had to be able to compile Mozilla on a Mac (accomplished), look up D-Trace and how it works (doing this weekend), then finding an existing probe (weekend), and utilizing it with functions inside the Mozilla code that have a relation (the week).

I have complete confidence that I should be able to get this all done by Saturday. And this assignment is more about time management now than it is actual "grunt work".

Open Source - Compiling Mozilla on Mac

I remember back when I was in Grade 3. Most of our computers at school were old MACs. And I mean the black and white ones with no CD Drives, with no funky colored monitors - and iPods were a vision of the future.

Back then, I wasn't a computer person like I am now. But now, I began to change....

I started using Windows 98 & XP. Good times. Too bad Service Pack 2 on XP wasn't friendly at all!

And Linux became easy to use for me. I had some trouble with vi at first, but then I came to love it like a brother. Then, it happened....

In order for me to begin working with D-Trace, I needed to use OpenSolaris. Problem: I needed a Mac. And it literally took me 20 minutes and a Google search on Safari just to find the location of the Command Prompt :(.

All I had to do in Windows was Windows Key + R > cmd. Linux I just simply opened it on the front screen. But this is Leopard 10.5 :(.

I got adjusted pretty decently to the atmosphere known as Mac OS. But the compilation instructions on Firefox were extremely fuzzy. And I mean EXTREMELY fuzzy. Especially since I didn't know what XTools or Darwinports were.

So I did the next best thing: Went on IRC (nick Hellwolf36) - and had a conversation between ted and jboston. Ted was helpful - his instructions were useful I wanted to get it on a home machine or remote terminal. But I am working in the Open Source lab.

jboston knew what the problem was - it was my PATH! I had already installed GLib and idlLib, XTools, D-Trace on this machine. I just didn't have my PATH set to find it. I pretty much facepalmed and was about to smash my head on the desk (metaphorically speaking).

Other than that, Mozilla Firefox seems to compile and work correctly. I will now share with you people how to avoid this trouble some fate. First, follow these instructions to the letter, especially the Software Requirements part.

Second, you may need to do this to your PATH and possibly your MANPATH.

Finally, you better find something to do for 30-50 minutes ;). You don't exactly want to sit there and watch your build happen.

My next phase for my 0.1 is to learn D-Trace and get a probe to work for me. One week left is the real nail-biter. But I have discovered, over the many years I work - that sometimes pressure brings out the best in all of us.

Open Source - Compiling Mozilla with D-Trace

I begin my step up to 0.1 glory with the bread and butter - compiling Mozilla with D-Trace. 

It's quite simple: Simply add ac_add_options --enable-dtrace in the .mozconfig file. And pray that nothing will explode.

But now, what am I to do in a lab for 1 hour?

If I was at home, I would probably log onto SCHTHACK and play Phantasy Star Online. Or boot up ZSNES for some Secret of Mana II (Seiken Densetsu 3). But considering I am on a tight schedule for 0.1 (which is due Saturday), and I don't want to let David Humphrey's links on D-Trace go to waste - I will be learning technology that is foriegn to me.

This will be by 1st time on a Mac. And I will openly admit, I am not a Mac person, but I can use Linux - so that has some merit. As long as I know what D-Trace is and how it works, I should be ready to get some things going. I wouldn't be surprised if I got half of this assignment done in 1 day. But it should be exciting, if I manage to pull it off.

Well, hopefully everything works out for the better. And given the fact this course is more time-based than difficulty - Failure is not an option.

Open Source - Time to Start

The time has come to actually do something useful for this course. WOW! Useful!!!

For the first month, I have been basically either working, playing a little PSO here and there, or forced to do other assignments - such as my Game Modeling course.

To help us in the right direction, Professor Humphrey shows us Bugzilla. It is a website that allows people who are working with Mozilla's technology to post bugs, to allow for fixes, or for people to work on them. They can then post thier findings here.

He asks us to take 3 bugs found on this page, and add them to our "Project Wiki Page". Then follow a person to see what kind of findings they create.

It looked simple for some people - cause a vast majority of people who are in my course either got bug fixing, testing, or updating something that exist. But alas, I don't.

I am basically creating probe programs via D-Trace to help Mozilla debug using that software. But this is something a whole lot of people haven't done before. All the sources I have are from my "mystery contact" in IRC (we crash at the freenode or moznet servers @ #seneca). I need to meet him this week after I finish a probing program in C++.

All and all, Bugzilla was fruitless for me. But it isn't all that bad. What if there is a bug I see, and my "probing software" finds the problem for the user. That would save MANY people a lot of time, and a lot of headbangings on the table or wall :).

Open Source - Diving into Source Code

Hooray for more adventures!

Last week, it was learning how to install this source code. This week, it is about actually seeing the components that make Mozilla Firefox, well... Firefox.

David Humphrey (my professor in the OSD course) offered us a "code search engine"

It basically allows you to "Google" things inside Mozilla code for certian functions, definitions, etc.

I decided to search how Mozilla uses Bookmark Page to keep track of your favorite pages. So I began searching the option itself. Which lead me to a UI definition file, and had the name of the function. That was simple, but it needed more.

Things got complicated there, I was basically searching through 7 files, until I found out that BookmarkPage (the name of a function) was an alias to a Javascript function.

And to another surprise the function PCH_bookmarkPage was only about 60 lines long, excluding closing brackets being on seperate lines (I often do that myself).  Not a bad length for saving your webpages.

The function itself basically stores any information about your page in Javascript variables, and then sends you a prompt asking if you want to Bookmark the page. Clicking yes proceeds on to adding your page to the Bookmarks section, no is for ignoring it.

But alas, the "fun" is just beginning. I must now think of a plan for my 0.1 release of the D-Trace program for Mozilla. Because of working at this golf club...

things got out of hand. But in life, we all need to make sacrifices to do something else, just some more than others. The life of a future programmer :(.

Open Source - Playing with Ubiquity

Well, it's time yet for another blog post, fortelling more "nightmares" with... Open source.

This time, I got into a little Mozilla Firefox 3.0 addon called Ubiquity. It is a program that allows you to type commands and Mozilla will do things with them regarding web pages and such.

For instance, typing wikipedia LOL inside of its command-line will open a wikipedia page for LOL.

However, it is pretty limited. It can only do certian things at first, but seeing how Firefox is open-source, other poeple starting making Ubiquity commands to make it do interesting things. You can make it search for NASDAQ quotes, search on Mininova.org for certian torrents, or activate a search engine to search for images or such.

I did something relatively simple. Benny Sheerin created an open command. I took it and rewrote parts of it to make a goto command using Javascript (the base language the Ubiquity commands are written in). It works like a Hash Table in Perl programming. There would be a key (an alias or keyword), and then would be a value associated with it (in this case, a webpage). 

When you have so many favorites pages or you dont have the hard drive space to store these pages on your PC, then this little add-on is right for you. All you need to do to add a new page, is create a new key-value pair inside the Javascript program. And then re-add it to the Ubiquity command list.

The biggest problem I ran into was compiling it. I did read over Benny's existing code well, it's just - I personally SUCK at Javascript. I hate the language personally, it was the sole reason my hopes of getting an A in INT222 went down the tube. So I had to perservere :( . JSLint (JS Compiler) wouldn't cooperate on a couple of lines and was giving me odd errors. After sitting in front of the PC for about 3 hours, and a few rounds of PSO, I managed to get it operational

I am definitly interested in this program and it's capabilities. If I write another program for this, I would make it so you can type in a word, and then a forum (running on vBulletin) and it would open the Search results to said forum. The only problem with this: You may need to log onto said forum, so you can use the Search tool. And I would also have to write it so it either shows results as threads, or posts. To start, I would need to find an existing search program, and rewrite it to search on boards instead of search engines.

If you wish to gain access to the Javascript code and use it for yourself, or you wish to "borrow" it and use it for your diabolical and/or productive schemes, you may find it here, along with other scripts written by my fellow comrades.

Open Source - Cathedrals and Bazaars

Greetings. As you can see, this is my first blog post of the many that will soon follow. This is also my first time writing on a blog based site (I am personally more used to forums as a means of online discussion, assuming the users don't troll and flame in your topics). And what a way to start things off, with a synopsis of a reading.

I am currently in OSD600 at Seneca College. This is a course where a series of students band together and work with Open Source mediums such as Mozilla Firefox and OpenOffice. Hopefully, creating software add-ons to these programs that can be shared across people. But because the code we do is open source, we are required to share it amongst users to  As part of our introduction to the course, one of the professors, David Humphery released an essay.

At first I thought - it's an essay, it won't take long to read something (I am one of those people who assumes an essay is about 4 pages with 5-7 Paragraphs with evidence and/or sources). But then it turned out to be a 15 page (when printed out) story about one man's adventure with Open Source style programming and distribution.

Link to article - Cathedrals and Bazaars

written by Eric Raymond

http://www.firstmonday.org/issues/issue3_3/raymond/

He begins by explaining to the reader what is meant by "Cathedrals" and "Bazaars". Eric uses the environments these locations hold as symbolism to two completely different programming styles. A Cathedral would be a secluded location, where developers would create software, without any interaction to the outside world and deal with problems like bugs and data structures themselves. Only the people who are working within this "Cathedral" have direct access to any progress. While a Bazaar, is pretty much the opposite. Its where people become a necessity, and not only the program is shipped out, but also the source code used to create it. This allows any person to take pre-written information and either improve on it, or turn it into something completely different.

The Bazaar method is believed to be only successful by a wing and a prayer. But it has worked in the past successfully - the development of Linux. The creator of Linux, Linus Torvalds took an existing OS for 386 processor-based PCs (Minux) and created Linux from its foundation. Then shipped the source code for it between a large community of users and hackers, which in return supplied Linus with either requests or code to fix various bugs or for improvements to functionality and performance. Seeing how this went successful for the creator of Linux, Eric decided to give it a test.

Eric started off with 2 things: 1) An ancestral mailing client called FetchPop(allowed a user to retrieve mail using POP) and 2) Inherited code from a programmer named Carl Harris for a  program called popclient, which he had given up on for some time. However, there was a decent memberbase of approx. 250 users using this one program. And his final outcome would be an mail retrieving program that used STMP via MTA/MDA. 

Using the "bazaar" Eric had (the popclient users), he released numerous versions of Fetchmail (the name of Eric's masterpiece) to this localized public. He recieved numerous forms of feedback: Either they were praising him, pointing out bugs that were visible, or supplied code to either improve performance, or to fix the existing bugs. However, there needs to be a limit, such as security, encryption, and in some cases - project scope. This is one advantage to Open Source programming: You are using more than 1 set of "eyes" to assist you in making a more stable product. The more people that are inside your community,  there is a possibility that bugs will become more apparent to at least someone and work can be done to correct them. However, that is assuming these people are dedicated enough to your cause and realize the scope of your program. Also, the members of the community don't rip your code and create a product that could just be as capable as yours. Although that is the worst case scenario - it's just how the world of Open Source works.

Eric also had to release patches often, and as many times as necessary to be able to keep his memberbase from leaving his community. Without a community, he would have been developing Fetchmail without "help". Making multiple releases lets your users know that you are still working on this program and that the people's demands or recommendations are either being considered or met. If you cannot keep up with the patches, then you would either need to name a successor to your program, or get more co-developers to assist you. 

In the end, Eric created a successful FetchMail, a program that allows you to collect mail from email accounts via SMTP fowarding. Although the scope of his original design had to change a large sum (via his own intentions to keep his mind in this program and the feedback from his community of co-developers), he managed to get a stable program starting from segments of pre-written code. Without Open-sourcing his code or retrieving open source code from the get-go, he would have had to write this program completely from scratch, and with a secluded team of developers, or worse - by himself.

I personally know about writing code solo with zero base - it is not fun, and extremely stressful and time consuming. I would rather do something like play a Diablo II mod or some other game than spend 20 hours a day writing code from scratch. But in previous courses, I had no choice - plagarism was a big-time offence in the Academic Policy at Seneca. Every single course in the Computer Programming and Analysis abided by that code - except this one. The world of Open Source recommends AND REQUIRES that the programmer must "steal" existing code to get a base going. But in doing so, you also cannot acquire code that is protected under copyrights or secluded companies, regardless of how useful or useless the code can become. And you will also probably open yourself up to potential "thieves" who may make a clone of what you made. However, it's better that the code is being put to use - whether it would be to further your plans or further someone else's. 

I also hate debugging code solo. I can see why Eric decided to go Open Source and use a community. Depending on the complexity of the code, debugging for me can take between 1 hour, to 1 week (assuming I work straight through an entire day(s)). There are always solo methods to debugging: Like write one block at a time, compile it and test it, or pen and paper approach - where you physically draw out or "walkthrough" a possible, or current solution. Although both are good tactics, it will only be fast if you do it piece by piece - across a possible 500+ lines of code. It will make matters worse if you don't document your code as you write it. People aren't exactly going to help you, open or closed source, if you don't make your code readable and understandable to the person next to you. I do need other people to help me out when I need to debug - but I mostly used professor help over student help. Student help is often flawed and don't see my program eye to eye too well. It's worse when it looks like you might be the more "educated" one with how you see things, which then makes the person next to you look ignorant. Professor has a higher chance of getting something solved - but you are limited to Student Help Time or In-Class. And some professors need to be communicated in a way they can understand it. I can, however, rely on either people to beta test my programs at least. The ones who could possibly understand might be able to help me debug. But not do all the debugging for me, unless that problem is the only one that is happening.

So as you can see, I for one am in support for Open Source development. But I still need to be careful with how I do this and who I do it with. For this will possibly be my first time I give code to others, but definitly not my first time "borrowing" from others. This is also something I need to fix if I am to succeed in this course, and later on in my professonal career as a Software Developer. I will be fixing it by balancing the equation, and being more open to what I do and/or give to other people.