Purposes of this assignment

• To teach concurrency with Actors

General idea of the assignment

Given the URL of a page that contains zero or more images and references zero or more subpages, download all web pages (.htm and .html), and images (.gif, .jpg, and .png) from that site. Recreate the site directory structure on your computer, so that links work. Do not download files of any other type, or any offsite files. Provide a report of files downloaded, offsite files, and broken links.

In more detail

Here's the recommended program structure. Variations are okay, as long as the behavior is essentially the same.

• Ask the user for a URL. You can read this in from the console, or use a dialog box.
• Ask the user for a directory in which to store the downloaded files. Use a FileChooser.
  • If the directory already exists and isn't empty, it might be a good idea to ask the user if that's okay.
• Ask the user how deep to search (0 = just this page, 1 = this page and its "child pages", 2 = this page and its children and "grandchildren", 3 = ...)
• Use the self function to make a proxy actor for "yourself" (this thread).
  • Scala Threads are the same as Java Threads. An Actor is a Thread with special features, such as a "mailbox." Your initial Thread is a plain old Java Thread that doesn't have these features. Think of your self as a "power suit" that you put on to give you these extra features.
• Create an actor with the given URL, the given directory, the depth limit, and your proxy (so the actor knows where to report back to). Each actor should:
  • Download the file at the given URL (if it exists!) and save it in the appropriate location.
  • Search the page for additional URLs, and for each one found:
    • If it's an image, download it and report the fact back to the original Thread.
    • If it's an HTML page, and the current depth limit is greater than zero, create a new actor (at depthLimit-1) to deal with it.
      • Note that the "depth limit," as I've defined it, decreases as you go deeper, and you don't go below 0. This widely-used technique lets you pass around one number instead of two.
  • Report back to the original method (possibly main) the URL of this page, whether the download was successful (remember, the URL may be invalid or refer to a missing page), and how many URLs were discarded as not useful.
• The original Thread should collect the responses, and print out:
  • A list of successful URLs. For brevity, these should be relative to the originally given URL.
  • A list of unsuccessful URLs; again, relative.
  • A count of the total number of URLs that were found but ignored because inappropriate.
  • A count of the total number of URLs that were found but were beyond the depth limit.

You may make the following simplifying (though incorrect) assumptions:

• Although tags and attributes may be in either case, they won't be mixed case: that is, HREF or href, but not HrEf.
• An element may not be split across lines.
• No line will contain more than one a tag or one img tag.
• There will be no whitespace between the opening angle bracket and the tag name.
• All URLs are quoted, with either single or double quotes. No URL contains a quote of the other kind.

These assumptions are intended to minimize the amount of string manipulation required, especially if you don't know regular expressions. Feel free to do a better job handling HTML elements, but you will not get any extra points for doing so.

If you do know regular expressions, be aware that they can take time that is exponential on the length of the string being processed. If your program exhibits exponential time behavior (takes unreasonably long to run), it will be penalized severely. Regular expressions are powerful, but remember the adage: "With great power comes great responsibility."

Useful code bits

val url = new java.net.URL(from)
val inStream = url.openStream
var byte = inStream read()    

val file = new java.io.File(to)
val outStream = new FileOutputStream(file)
outStream.write(byte)    

val chooser = new FileChooser()
chooser.fileSelectionMode = FileChooser.SelectionMode.DirectoriesOnly
val theFileResult = chooser.showSaveDialog(null)

val newDirectory = new File(fullPathName)
val success = newDirectory.mkdir()

Grading

The program should work as specified.

For up to 20 points extra credit, skip the Console I/O and make a Scala GUI for the program.

No mechanical deductions or bonuses for vars, etc. However, we reserve the right to give bonuses or deductions according to our subjective estimates of the quality of your code, especially the extent to which it embraces or violates "the Scala way." Please note that, in this context, "subjective" means we liked it or didn't like it, and as such is not open to arguments about points.

Due date