Purposes of this assignment

• To teach concurrency with Actors
• Give you more experience with Scala

General idea of the assignment

Crawl my website and produce an "index" of the pages, lectures, and images that you find there. Web pages have the extensions .htm and .html, images are one of .gif, .jpg, and .png, and lectures are .ppt or .pptx.

In more detail

When I refer to the "name" of a file, I mean everything after the last slash in the URL. When I refer to "the URL", I mean everything after the http:// -- in the case of a relative URL, this means reconstructing the absolute URL.

The absolute URL can be formed by taking URL of the page you are at, removing everything after the last slash (if there is anything after the last slash), and appending the relative URL to it.

Here's a suggested program structure that I think will work fairly well. Feel free to deviate from it, as long as you are using Actors in a reasonably appropriate fashion.

Main program

• Put the main program in a Manager object
• The Manager should create three initial Actors to process the initial web page--one to handle web pages, one to handle images, and one to handle lectures.
• Start things going, by using another Actor to process the initial page.
• When all the Actors have finished, write the results to a file.
  • The first section should be the names of all the HTML pages, followed by their URL; one name/URL pair per line, in alphabetical order by name.
  • The second section should be the names of all the PowerPoint slides, followed by their URL; one name/URL pair per line, in alphabetical order by name.
  • The third section should be the names of all the images, followed by their URL; one name/URL pair per line, in alphabetical order by name.
  • When a name begins with one or two digits followed by a hyphen (for example, 06-methods.ppt), the digits and hyphen should be omitted from the name, but not from the complete URL.

Actors

Image collector

One actor should collect URLs of all the images that are reported to it. Since images are often reused, only one image with a given name should be kept. You do not need to check if images are the "same."

Lecture collector

One actor should collect URLs of all the lectures that are reported to it. I usually prefix each lecture with one or two digits and a hyphen. Since I rearrange lectures each year, the prefix should be ignored when comparing the names of lectures to see whether they are the same. Since I reuse lectures but also frequently update them, the most recent lecture with a given name should be the one whose URL is kept. (Since most of my URLs contain a year number, you can use this.)

Web page collector

One actor should collect URLs of the web pages that are reported to it. The following URLs should be discarded:

• URLs that are not on my site
• URLs that contain the # character; these are links into the middle of some page.
• URLs that contain the ? character; these are searches.
• URLs that start with dots or a slash (.. or /); these are links to somewhere on this site, but not in this subdirectory.

Web page processors

Every time a new web page is encountered, a new actor should be created to process it.

Processing consists of

• Finding all the references to web pages, images, and PowerPoint presentations.
• Informing the relevant collectors of each thing found.
• Creating new actors to process each new web page found.

You probably need to check for cycles in the links, as these could cause your program to run forever. Since you will be discarding URLs that begin with .., this will prevent many, but not all, cycles.

In less detail

The point of this assignment is to learn to use Actors (and to use them appropriately). The above structure is basically a Map-Reduce style structure, and I think it should work out okay; but if you think there's a better way to use lots of Actors, go ahead and do things differently.

However you structure your program, the output should be one file, containing three clearly labeled sections, for pages, lectures, and images. Each section should be alphabetical by name (not by URL), with no duplicate names.

Possibly useful code bits

I believe the following code bits to be correct, as far as they go; but use at your own risk.

val url = new java.net.URL(from)
val inStream = url.openStream
var byte = inStream read()    

val file = new java.io.File(to)
val outStream = new FileOutputStream(file)
outStream.write(byte)    

val chooser = new FileChooser()
chooser.fileSelectionMode = FileChooser.SelectionMode.DirectoriesOnly
val theFileResult = chooser.showSaveDialog(null)

val newDirectory = new File(fullPathName)
val success = newDirectory.mkdir()

val HtmlPattern =
  """(?i).*<\s*(a)\s+.*href\s*=\s*['"]?([^'" ]+\.html?)['" >].*""".r

val ImgPattern =
      """(?i).*<\s*(img)\s+.*src\s*=\s*['"]?([^'" ]+\.(gif|jpg|png))['" >].*""".r

string match {
  case Pattern(tag, url) =>
    println("tag:  " + tag + ", url: " + url)
  case Pattern(tag, url, ext) =>
    println("tag:  " + tag + ", url: " + url)
  case x => println("*** No match: " + x + "\n")
}

Grading

The program should work as specified.

The first Scala assignment encouraged you to do things "the Scala way" by avoiding vars, not using a loop where a recursion would do, not using a recursion where a call to a higher-order function would do, using pattern matching, etc.

You should continue to try to do things the Scala way. You may use vars, while loops, etc., when they are appropriate, but also use the Scala features. I reserve the right to take off significant points if it looks like you made no effort whatsoever to write in Scala, rather than writing a Java program using Scala syntax. I will probably not ever do this, so there's no need to be paranoid, but I reserve that right.

Due date