CIS 700 Downloader
David Matuszek, Summer II, 2010

Purposes of this assignment

• To teach concurrency with Actors

General idea of the assignment

Given a URL, download all web pages (.htm and .html), and images (.gif, .jpg, and .png) from that site. Recreate the site directory structure on your computer. Do not download files of any other type, or any offsite files.

In more detail

Here's a program structure that I think will work fairly well. Feel free to deviate from it, as long as you are using Actors in a reasonably appropriate fashion.

Main program

• Get from the user the initial URL and the place in which to create the new directory.
• Create and start a DownloadManager actor
• Send the initial URL to the QueueManager

DownloadManager

• Create a queue (java.util.concurrent.LinkedBlockingQueue) of URLs to be processed.
• Create and start some number of Downloaders, and give them access to the queue of URLs to be processed.
• Create a Parser.
• Seed the queue with the first URL (the one obtained from the user).
• Use a while(true) loop to receive messages from the Parser.
  • Messages from the Parser should have the form (URLthatWasParsed, List[newURLs]).
  • New URLs that have not been seen before should be added to the queue; but if they have been seen before, ignore them.

Downloader

For each URL gotten (from the queue of URLs to be processed):

• Create the appropriate directory, if it doesn't already exist.
• Download the page and save it in the directory.
• Send the page, along with its URL, to the Parser.

Parser

For each page received (from the Downloader):

• Search the page and make a List of appropriate URLs to explore.
• Send the URL of the page just parsed, along with a list of appropriate URLs found on it, to the DownloadManager.

You may make the following simplifying (though incorrect) assumptions:

• An element may not be split across lines.
• No line will contain more than one a tag or one img tag.

These assumptions are intended to minimize the amount of string manipulation required, especially for students who don't know regular expressions. Feel free to do a better job handling HTML elements, but you will not get any extra points for doing so.

Warning

You should read and understand About /robots.txt before trying out your program. Following that, you should look at the robots.txt file for the university. If you don't obey the rules there, you may get Penn IP addresses banned, which in turn may get you into trouble with the university.

I will not require your program to read the robots.txt file, but you should read it and not break the rules.

A relatively safe place to try out your robot is any one of my old course pages at Penn. You can find a list of these at HouseOfChaos.com. Any one of these courses is probably a good size to use for testing (except CIS700, which uses a wiki instead), but you probably don't want to download material from all 32 courses.

Useful code bits

val url = new java.net.URL(from)
val inStream = url.openStream
var byte = inStream read()    

val file = new java.io.File(to)
val outStream = new FileOutputStream(file)
outStream.write(byte)    

val chooser = new FileChooser()
chooser.fileSelectionMode = FileChooser.SelectionMode.DirectoriesOnly
val theFileResult = chooser.showSaveDialog(null)

val newDirectory = new File(fullPathName)
val success = newDirectory.mkdir()

val HtmlPattern =
  """(?i).*<\s*(a)\s+.*href\s*=\s*['"]?([^'" ]+\.html?)['" >].*""".r

val ImgPattern =
      """(?i).*<\s*(img)\s+.*src\s*=\s*['"]?([^'" ]+\.(gif|jpg|png))['" >].*""".r

string match {
  case Pattern(tag, url) =>
    println("tag:  " + tag + ", url: " + url)
  case Pattern(tag, url, ext) =>
    println("tag:  " + tag + ", url: " + url)
  case x => println("*** No match: " + x + "\n")
}

Grading

The program should work as specified.

For up to 15 points extra credit, give the user the ability to stop all the actors (by sending them an appropriate message) and quit the program cleanly, if the user types q or Q.

No mechanical deductions or bonuses for vars, etc. However, we reserve the right to give bonuses or deductions according to our subjective estimates of the quality of your code, especially the extent to which it embraces or violates "the Scala way." Please note that, in this context, "subjective" means we liked it or didn't like it, and as such is not open to arguments about points.

Due date