The Story

That's all there is to it ... at least at this point. I've got a few little tests lying around that might be helpful, but I'll start here from zero.

I think I'll try to do this one in a pretty solid test-driven fashion. But there may be a few trouble spots. We'll see. And I'm going to start from the inside out. My plan is to start by counting the words in a line, then in a stream, then in a file, then in all the files. Or something about like that -- as always what we do will be driven by what we discover along the way.

We'll start with a new project, WordCount, and a test, that looks like this:

using System;
using NUnit.Framework;

namespace WordCounter
{
  [TestFixture] public class TestWordCounter: Assertion {
  
    public TestWordCounter() {
    }
    
    [SetUp] public void SetUp() {
    }
    
    [Test] public void SimpleCount() {
      WordCounter wc = new WordCounter();
      AssertEquals(0, wc.CountText(""));
    }
  }
}

using System;

namespace WordCounter
{
  class WordCounter
  {
    [STAThread]
    static void Main(string[] args)
    {
    }
  }
}

Of course, that doesn't compile, because there is no CountText method on WordCounter. So I'll fake that:

using System;

namespace WordCounter
{
  class WordCounter
  {
    public int CountText(String text) {
      return 0;
    }

    [STAThread]
    static void Main(string[] args)
    {
    }
  }
}

That works. I'll test something harder:

    [Test] public void SimpleCount() {
      WordCounter wc = new WordCounter();
      AssertEquals(0, wc.CountText(""));
      AssertEquals(6, wc.CountText("This sentence is six words long"));
    }

Naturally this doesn't work. I'm thinking I'll try to split the string on whitespace and see whether that's useful:

    public int CountText(String text) {
      return text.Split().Length;
    }

I really expected that to work. But it says "Expected 0 but was 1". I wonder which line of test is failing. That's what happens when you don't make a separate test for every case. I could use the debugger, but to train myself to be good, I'll recast the tests:

    WordCounter wc;
	
    [SetUp] public void SetUp() {
      wc = new WordCounter();
    }
    
    [Test] public void SimpleCount() {
      AssertEquals(0, wc.CountText(""));
    }

    [Test] public void SixCount() {
      AssertEquals(6, wc.CountText("This sentence is six words long"));
    }

Well, of course I'm an idiot, since only the first test could have been failing, but at least we have a better testing structure going here. So what we have discovered is that the Split() is returning an empty string. Reading up on Split(), I get the impression that it could do that. Since I want to count words with at least one character in them, I think I'll just count the non-empty strings in the result of Split():

    public int CountText(String text) {
      String[] split = text.Split();
      int count = 0;
      foreach (String s in split) {
        if (s.Length > 0) count++;
      }
      return count;
    }

So far, so good. Now I'll test a line with some HTML in it:

    [Test] public void Html() {
      AssertEquals(6, 
        wc.CountText("<P>This sentence is <a href=\"six.html\">six</a> words long."));
    }

Naturally, this doesn't work. My plan is to rip out all the tags using a Regex. I've played with those in the past, so here's what I'll type in for the code:

    public int CountText(String text) {
      String deTagged = DeTag(text);
      String[] split = deTagged.Split();
      int count = 0;
      foreach (String s in split) {
        if (s.Length > 0) count++;
      }
      return count;
    }

    private String DeTag(String tagged) {
      Regex regex = new Regex("<[^>]+>");
      return regex.Replace(tagged, "");
    }
                                

The DeTag() method just sets up a regular expression that will recognize a less-than (<), anything but a less-than, followed by a greater-than (>). That should grab any tag. The Replace() method on Regex replaces everything that matches in the input string (tagged) with the substitution string, in this case an empty string.

This test actually works. Enough for now, time to take a break.

Moving Towards Files

We haven't forgotten that our real mission here is to word-count files. It's time to write some tests and code addressing that capability. I've found it tricky to find out in Visual Studio's help how to do simple file reading, but Chet and I wrote a few tests relating to it a while back. I'll dig those out (the chapter about that hasn't been written yet), and adapt them to the situation here.

My rough plan is first to process a stream with multiple lines, then to process a file. Then I'll test-write the code to get all the file names. Then I should be done except for printing the final report. Let's see what really happens:

    [Test] public void MultiLineString() {
      String multi = @"<P>Here are four words.</P>
<H2>And Three More</H2>";
      AssertEquals(7, wc.CountReader(new StringReader(multi)));
    }
                                

Of course CountReader isn't implemented yet, so we'll first Fake It:

    public int CountReader(TextReader reader) {
      return 7;
    }
                                

That runs. Now we'll Make It:

    public int CountReader(TextReader reader) {
      int total = 0;
      String line;
      while ( (line = reader.ReadLine()) != null) {
        total += CountText(line);
      }
      return total;
    }

That works as well. This makes me think I should be able to test a file. To do that, however, I'd have to be able to open a file, so I'll look that up as I write the next test:

    [Test] public void ReadFile() {
      StreamReader reader = File.OpenText("eightwords.htm");
      AssertEquals(8, wc.CountReader(reader));
    }
                                

With the associated file, eightwords.htm. Well, almost. The program looks for that file in the runtime directory /bin/Debug/. I want it in the source directory so I'll change the test to look there explicitly:

    [Test] public void ReadFile() {
      StreamReader reader = File.OpenText(@"C:\Data\CSharp\WordCount\eightwords.htm");
      AssertEquals(8, wc.CountReader(reader));
    }

That works just fine. Whee, successfully read a file and the code looks almost decent. Make a note to go back and check the Customer Tests for the Notepad to see how we did it there.

Now it's time to find all the files and count them up. No, wait. It makes more sense to allow the WordCounter to open the file itself. (This might also result in closing the file, which I notice I didn't do. So let's change the test a bit:

    [Test] public void ReadFile() {
      AssertEquals(8, wc.CountFile(@"C:\Data\CSharp\WordCount\eightwords.htm"));
    }

That begs for this method on WordCounter:

    public int CountFile(String fileName) {
      using (StreamReader reader = File.OpenText(fileName)) {
        return CountReader(reader);
      }
    }

The tests still run.

Have You Noticed ...

That the code we're writing is getting better-looking bit by bit. The above file-oriented stuff is, I think, almost presentable C#. Remember: this book is about learning, and there is still less than a week's actual C# programming experience represented here. Look for this same thing to happen in your work as you progress. And keep an eye out for your ugly old ways of doing things, and update them when next you pass by. The end result -- we hope -- will be a pretty good-looking program even if you've started out as a newbie.

We now return you to our regularly-scheduled program.

Finding and Processing Files

We need to find all the files that match "acs*.htm" in the XP Magazine directory on my computer. We've got related code that does that sort of thing in our Customer Tests for the Notepad. I'll take a look there. Ah, here it is:

    [Test] public void TestAllFiles() {
      String[] testFiles = Directory.GetFiles("c:\\data\\csharp\\notepad\\", "*.test");
      AssertEquals(1, testFiles.Length);
      foreach (String testFile in testFiles) {
        InterpretFileInput(testFile);
      }
    }

I'll make a test along those lines for the acs*.htm files:

    [Test] public void AcsFiles() {
      String[] acsFiles = Directory.GetFiles(@"C:\Data\XProgramming\site\xpmag\acs*.htm");
      AssertEquals(2, acsFiles.Length);
    }
                                

Now of course I know there are more than two file. I just want to see the test fail and see if that gives me an idea for a better test. And ... with the message "Illegal characters in path". What's that about? Oh. The GetFiles method wants two strings, the path and the string to match on. I guess I didn't read the example very carefully. Let's fix that:

    [Test] public void AcsFiles() {
      String[] acsFiles = Directory.GetFiles(
        @"C:\Data\XProgramming\site\xpmag\", "acs*.htm");
      AssertEquals(2, acsFiles.Length);
    }

That fails as expected, sort of. "Expected 2 but was 39". I didn't think I had written that many ACS chapters yet, and in fact at this moment the index only shows 35. I'm going to have to print these or something to see what else is being found. First, though, I'll look in the actual directory. I found a few files to remove, but I still think there is one extra. First I'll run the test again to see what it says. It says 36, and there are 35 chapters in the index. Plus the index file itself. I think I'm good to go. Let's see, how can I make this a reasonable test, I can't keep changing that number ...

I think I'll test for size > 30, and then move on. I'm close to getting an answer here.

    [Test] public void AcsFiles() {
      String[] acsFiles = Directory.GetFiles(
        @"C:\Data\XProgramming\site\xpmag\", "acs*.htm");
      Assert("Not enough files", 30 < acsFiles.Length);
    }

That runs, of course. Now I'm going to test the word count for all files, just because I'm interested in that total. I haven't forgotten that I'm really here to get a report.

    [Test] public void CountAcsFiles() {
      Assert(31, wc.CountFilesMatching(@"C:\Data\XProgramming\site\xpmag\", "acs*.htm"));
    }
                                

Of course we'll get a much bigger number than 31. As in the previous test, I've just put that value there to see the test fail. First, of course, it will fail because I don't have the CountFilesMatching method:

    public int CountFilesMatching(String path, String pattern) {
      int total = 0;
      foreach ( String fileName in FilesMatching(path, pattern)) {
        total += CountFile(fileName);
      }
    }
                                

Written by intention to beg for:

    private String[] FilesMatching(String path, String pattern) {
      return Directory.GetFiles(path, pattern);
    }
                                

Now you might not leave that helper method there, but I like the way it expresses itself. So let's go with it and see what happens. Ahem, well. First, I forgot to return the total:

    public int CountFilesMatching(String path, String pattern) {
      int total = 0;
      foreach ( String fileName in FilesMatching(path, pattern)) {
        total += CountFile(fileName);
      }
      return total;
    }

Second, I said Assert in the test above, and should have said AssertEquals. I was looking forward to when I change it again just to say bigger than something. For now, I'll change it to AssertEquals. Forgive me if I don't copy the code for you yet.

Ha! Now the test runs. It says "Expected 31 but was 66330". So my word counter thinks there are 66,330 words in the book so far. Cool.

That's enough work for now. I have a lunch date and need to get on the road. We'll finish up the report in a while. Be right back ...

Not So Right Back After All

It has been three days since I wrote "Be right back" up above. I mention it because there's a valuable hidden lesson in this way of programming. I put the tests into the source in chronological order: the last test at the bottom of the file. When it's time to pick up a program that I've been working on, a quick review of what's at the bottom of the file refreshes my mind on where I was. It helps me remember what I was up to, but I also spot things to do next that I might not have noticed last time through. For example, let's look at the last two tests in the file right now:

    [Test] public void AcsFiles() {
      String[] acsFiles = Directory.GetFiles(@"C:\Data\XProgramming\site\xpmag\", "acs*.htm");
      Assert("Not enough files", 30 < acsFiles.Length);
    }

    [Test] public void CountAcsFiles() {
      AssertEquals(31, wc.CountFilesMatching(@"C:\Data\XProgramming\site\xpmag\", "acs*.htm"));
    }

I haven't looked back at the text above here to see what the narrative says, but those two test tell me that First I tested how to use Directory.GetFiles, and then I jumped right to counting the words in them. Checking the WordCounter.cs source file ...

    public int CountFilesMatching(String path, String pattern) {
      int total = 0;
      foreach ( String fileName in FilesMatching(path, pattern)) {
        total += CountFile(fileName);
      }
      return total;
    }

    private String[] FilesMatching(String path, String pattern) {
      return Directory.GetFiles(path, pattern);
    }

I'm reminded that I programmed the CountFilesMatching() method "by intention", creating the FilesMatching() method to get the actual file names. (It might be that I programmed it inline and then factored it out, but either way, we have that method.

Looking at the tests above, I realize that the AcsFiles test is what we might call a "learning test": I was making sure that I knew how to use something. There will be some more examples of that when we talk about the LittleTests.cs file later on. Now, it might be best to retain the learning nature of that test. But I'd also like to have a test of the FilesMatching() method in WordCounter.

I see two alternatives. I can change the AcsFiles test to use the FilesMatching method that now exists in WordCounter, or I can write a new test to do the same. If I change the AcsFiles test, I'll be erasing the fact that I learned something at that point, kind of disguising the actual flow of how this little program came into being. On many days, I might do that.

Today, however, I'm going to write a new test. Furthermore, because it tells a better story, I'm going to break my rule of putting the tests in chronological order, and insert it between those last two steps. I'd rather have the reader's understanding grow in a smooth way than reflect the somewhat more convoluted thinking of the original implementor. So here goes:

    [Test] public void AcsFiles() {
      String[] acsFiles = Directory.GetFiles(@"C:\Data\XProgramming\site\xpmag\", "acs*.htm");
      Assert("Not enough files", 30 < acsFiles.Length);
    }

    [Test] public void FilesMatching() {
      String[] acsFiles = wc.FilesMatching(@"C:\Data\XProgramming\site\xpmag\", "acs*.htm");
      Assert("Not enough files in wordcounter", 30 < acsFiles.Length);
    }

    [Test] public void CountAcsFiles() {
      AssertEquals(31, wc.CountFilesMatching(@"C:\Data\XProgramming\site\xpmag\", "acs*.htm"));
    }

Well, I expect that test to work already, and it almost does. The FilesMatching method in WordCounter is private. I make it public to allow the test to run, and it does. However, we probably don't want this method to be public. I don't usually fret much over making a method public in order to test it, but there is an alternative in C#. I know this because, feeling a little bad about making the method public, I looked up "public" in the help and found "Accessibility Levels", which told me that I can define the method "internal", which means that it can be accessed by anything in the current project. We'll try that. Sure enough, it works just fine:

    internal String[] FilesMatching(String path, String pattern) {
      return Directory.GetFiles(path, pattern);
    }

However, still there is a red bar, because that last test, for the wordcount, is still failing. Of course it's failing in a good way, since I'm really glad I have written more than 31 words on this book so far, but I left it red to figure out a better test for the feature. And that's why I came back.

Moving On to the Report

Our real objective, remember, is to create a little report on the chapters, saying how many words are in each one. It's time to move on to that. For now, I'll change the CountAcsFiles test to expect more than 60,000 words, so that it will be green. Then we'll move on.

    [Test] public void CountAcsFiles() {
      Assert("Not enough words", 60000 < wc.CountFilesMatching(@"C:\Data\XProgramming\site\xpmag\", "acs*.htm"));
    }

Fine, that works. Let's get to the report.

Now testing reports is always a bit tricky. I'm often tempted to test them by eyeball, and often I succumb to that temptation. However, recently I visited with a client who tested their reports by eyeball, and on one critical day the eyeballs didn't notice an error. The board of directors, to whom the report went, did notice it. I don't have a board of directors watching me, but that's enough to inspire me to do the test. However, as you'll see, my approach is to automate my eyeballs.

Here's my plan. I'll add a couple more test files like that eightwords.htm file, and then do a report from them. I'll write the report test to check for an impossible string, so that it will fail. Then when the report comes out, and the test fails, I'll eyeball the report once, then copy it into the test. First the new files:

tenwords.htm

<P>Here are four words.</P>
<H2>And now enough to make ten.</H2>

ninewords.htm

<P>Here we have five words.</P>
<H2>And four makes nine.</H2>

Now this gives me a chance to write better tests for the Directory.GetFiles, and FilesMatching, so let's do that first. Should have thought of this before, but I didn't. It'll only take a moment, and it leaves a better trail.

    [Test] public void AcsFiles() {
      String[] acsFiles = Directory.GetFiles(@"C:\Data\CSharp\WordCount\", "*.htm");
      AssertEquals("Not enough files", 3,  acsFiles.Length);
    }

    [Test] public void FilesMatching() {
      String[] acsFiles = wc.FilesMatching(@"C:\Data\CSharp\WordCount\", "*.htm");
      AssertEquals("Not enough files in wordcounter", 3, acsFiles.Length);
    }

These run fine. That makes me so happy I'm going to fix the Count test as well:

    [Test] public void CountHtmFiles() {
      AssertEquals("Not enough words", 27, wc.CountFilesMatching(@"C:\Data\CSharp\WordCount\", "*.htm"));
    }

That's really a lot better. Those tests are much more precise, and frankly easier than what I did before. If this were a book about how to be perfect, I'd probably have shown you this solution and not the other. But this is a book about how to learn as we go. For some reason, perhaps lack of a pair, or perhaps just rank mental dullness, I didn't think about setting up a few files and counting them. Having thought of it, it's worth taking the couple of minutes it takes to improve the tests. Someday some poor devil will have to maintain this program, and this is a much more stable and understandable set of tests for him to learn from. And the poor devil might be me, so I'd like to help him out.

Now maybe we can actually write a test for the report. I guess what I'll do is tell the WordCounter to write the report to a file, and then read it back in. Although the report is pretty small, maybe I'll just have him write it to a String. Yes, let's try that.

    [Test] public void Report() {
      StringWriter reportWriter = new StringWriter();
      wc.ReportFiles(reportWriter, @"C:\Data\CSharp\WordCount\", "*.htm");
      String report = reportWriter.ToString();
      Console.WriteLine(report);
      AssertEquals("impossible", report);
    }

With the implementation:

    public void ReportFiles(TextWriter writer, String path, String pattern) {
      int total = 0;
      foreach ( String fileName in FilesMatching(path, pattern)) {
        int count = CountFile(fileName);
        writer.WriteLine("{0} {1}", fileName, CountFile(fileName));
        total += count;
      }
      writer.WriteLine("Total {0}", total);
    }

This produces this report:

C:\Data\CSharp\WordCount\eightwords.htm 8
C:\Data\CSharp\WordCount\ninewords.htm 9
C:\Data\CSharp\WordCount\tenwords.htm 10
Total 27

That's nearly good. I think I'd like to change it a little bit, though. I'd like the path to be printed just once, up at the top of the report, and I'd like the file names to be just the names, not the full path. I could actually type that report in now, but I think I'll just code it and then snap it into the test. Here goes ...

    public void ReportFiles(TextWriter writer, String path, String pattern) {
      int total = 0;
      writer.WriteLine(path);
      writer.WriteLine();
      foreach ( String fileName in FilesMatching(path, pattern)) {
        int count = CountFile(fileName);
        String shortFileName = fileName.Remove(0,path.Length);
        writer.WriteLine("{0} {1}", shortFileName, CountFile(fileName));
        total += count;
      }
      writer.WriteLine();
      writer.WriteLine("Total {0}", total);
    }

This produces a nicer little report:

C:\Data\CSharp\WordCount\

eightwords.htm 8
ninewords.htm 9
tenwords.htm 10

Total 27

So we'll update the test to look for that report:

    [Test] public void Report() {
      StringWriter reportWriter = new StringWriter();
      wc.ReportFiles(reportWriter, @"C:\Data\CSharp\WordCount\", "*.htm");
      String report = reportWriter.ToString();
      String result = 
@"C:\Data\CSharp\WordCount\

eightwords.htm 8
ninewords.htm 9
tenwords.htm 10

Total 27
";
      AssertEquals(result, report);
    }
  }

The test runs. So I'm sure that the real report will run. I need to get a look at it. Since all I want to do is look at it, I think I'll just write a test that sends it to the Console, and look at it there.

    [Test] public void BookReport() {
      StringWriter reportWriter = new StringWriter();
      wc.ReportFiles(reportWriter, @"C:\Data\XProgramming\site\xpmag\", "acs*.htm");
      Console.WriteLine(reportWriter.ToString());
    }
  }

And here's the result:

C:\Data\XProgramming\site\xpmag\

acsAddTestFixture.htm 1688
acsCodeManagementVision.htm 1071
... and so on ...

Total 68796

We could clean that up a bit more for presentation purposes, but for my purposes, it's almost done. Just one more thing: let's estimate page count. My acquisition editor told me that a rule of thumb for book-printed pages is 350 words per page. In the course of that, I've improved the names in the ReportFiles() method a bit:

    public void ReportFiles(TextWriter writer, String path, String pattern) {
      int words = 0;
      int pages = 0;
      writer.WriteLine(path);
      writer.WriteLine();
      foreach ( String fileName in FilesMatching(path, pattern)) {
        int count = CountFile(fileName);
        int pageEstimate = (count + 349) / 350;
        String shortFileName = fileName.Remove(0,path.Length);
        writer.WriteLine("{0} {1} {2}", shortFileName, count, pageEstimate);
        words += count;
        pages += pageEstimate;
      }
      writer.WriteLine();
      writer.WriteLine("Total {0}, pages {1}", words, pages);
    }

And I had to update the little report test to reflect the page count:

    [Test] public void Report() {
      StringWriter reportWriter = new StringWriter();
      wc.ReportFiles(reportWriter, @"C:\Data\CSharp\WordCount\", "*.htm");
      String report = reportWriter.ToString();
      String expected = 
        @"C:\Data\CSharp\WordCount\

eightwords.htm 8 1
ninewords.htm 9 1
tenwords.htm 10 1

Total 27, pages 3
";
      AssertEquals(expected, report);
    }

And your final answer:

C:\Data\XProgramming\site\xpmag\

acsAAcontents.htm 103 1
acsAddTestFixture.htm 1688 5
acsCodeManagementVision.htm 1071 4
acsCodeManager.htm 1302 4
acscsharpnotepad.htm 1724 5
acsDarkClouds.htm 1400 4
acsDontTryThisAtHome.htm 1553 5
... and so on ...

Total 69106, pages 216

Summing Up

Well, in all I think it went pretty well. I rather like the way the bottom-up approach worked in this case. Quite often TDD proceeds in a more top-down fashion, but this one felt good done bottom up. I think the reason was that I wanted to get to the problem of finding the words in the line and getting rid of the HTML tags right away.

As an exercise you might want to try solving the problem top down. What would the first simple test be in that case? A list of only one file? A list of files like eightwords.htm and its cousins?

Recall that early on I made a mistake -- I was confused about which test was failing. That set me off the rails for a while but of course I quickly got back on by splitting the tests. The same thing could also have been learned by setting a breakpoint, or maybe even by backing out my code changes. Sooner or later I had to discover that the Split() operation could return empty strings. And Sooner is better.

My practice is "never" to debug. I've learned how to get NUnit to run under the debugger now, and I'll write about that the first time I actually do it. But my experience is that when I go into the debugger, there's no telling if I'll be there for a minute or an hour. Using the debugger injects such variability into my work that I prefer not to go there. Now that I can do it if I want to, I'll need to develop a good habit. Before, I just couldn't. That was a pretty good approximation to using it perfectly.

I hope you can tell that the C# code we're writing here is getting better. That's really the main point of the book: that we can start as a newbie with a language, develop and ship useful code, and evolve our ability, and the program's code quality, as we go along.

I made a number of little mistakes on my own that a pair would quite likely have found. I misread the meaning of a test; I misread the manual; I didn't think of testing some canned files until quite late in the process. Now these are just little mistakes, and certainly I forgive myself. And why not: entire companies have fallen through mistakes I've made. They're just mistakes.

However, with a pair, I would probably have missed at least two of this mistakes, and quite possibly all three. Of course it means that I have to make mistakes in front of my pair, but since I'm now making them in front of tens of millions of readers (a man has to have a dream), making them in front of one person isn't so bad. Please try pair programming a bit, until you're good at it. If you're much like me, you might find you really like it. And I'm sure you'll find that it improves your code.

I wrote a one-line helper method:

    private String[] FilesMatching(String path, String pattern) {
      return Directory.GetFiles(path, pattern);
    }

You might think that's inefficient. Well, if you can measure the amount of time it takes to send a message, you're a pretty hot programmer: try it. Then see if you mind the inefficiency in this case. I don't mind it, because I want the code to be clear.

In that same area, I hit a couple of unexpected compiler error messages. There's some debate about that. Getting the compiler to find these problems is easy, and of course it is also lazy. Does it have drawbacks? I would say that it might. It might be that if we spent a little more time reading the code, taking out dumb compiler errors like that, we would also find logic errors that are lurking in there. Chet and I keep meaning to try to work that way, but so far we just can't seem to make ourselves do it.

On the other hand, there seem to be very few bugs in the code. So maybe it's not necessary. Or maybe I'm just a lazy slug. You decide.

We learned that the tests provide us context when we're picking up the code after some time off. That's a good thing.

We learned that while we're working, ordering the tests chronologically helps us keep track. But ordering the tests in a different logical order may tell a better story for subsequent maintenance.

We learned that it's easy to revise tests to be better, and often tells a better story.

And we got a look at one way to test a reporting program without doing any very hard work.

This was a very productive session, from a learning viewpoint, and I got the information I wanted in a repeatable way. I feel like a better C# programmer, I feel refreshed about my practices. Definitely a good few hours' work, spread over a few days. I'm pleased, and I hope you can see why.