New blog post: Adventures in Archiving

_posts/2017-06-15-adventures-in-archiving.md

@@ -0,0 +1,83 @@
+---
+layout: post
+title: Adventures in Archiving
+date: 2017-06-15 22:00:00
+permalink: /blog/2017/06/15/
+machines:
+  - type: pcx86
+    id: ibm5150
+    config: /devices/pcx86/machine/5150/cga/64kb/machine.xml
+    autoMount:
+      A:
+        path: /disks/pcx86/dos/ibm/1.00/PCDOS100.json
+      B:
+        path: /apps/pcx86/1983/adventmath/ADVENTMATH100.json
+    autoType: 8-10-81\\rb:\\ra:basica castle\\r
+---
+
+In October 1983, IBM released a game called "Adventures in Math", which apparently tried to make mathematics as
+exciting as walking through a dungeon full of locked doors, treasures, spiders, and more.  I had never heard of this
+game, until I was recently browsing the [Internet Archive](https://archive.org/)'s
+[MS-DOS Showcase](https://archive.org/details/softwarelibrary_msdos_showcase):
+
+> This collection is a hand-picked set of selections from the MS-DOS Software Library of the Internet Archive.
+They've been chosen because they represent major parts of the MS-DOS story, because they are particularly impressive,
+and because they're fun. The intention is to eventually link these games to manuals and other materials as time goes on.
+
+> Curation was done by Jason Scott, Internet Archive's Software Librarian. He can be reached at jscott@archive.org with
+suggestion, questions and inquiries.
+
+For some reason, "Adventures in Math" caught my eye.  You can play it on the
+[Internet Archive](https://archive.org/details/msdos_Adventures_in_Math_1983), or play it below, using PCjs.  In
+fact, I invite you to play it in BOTH places, and let me know what you think.
+
+A word of warning: this is NOT a great game.  I find it interesting only as an example of the challenges we face
+archiving and emulating old software.
+
+{% include machine.html id="ibm5150" %}
+
+Hosting the software on the PCjs website was more tedious than I'd expected, but that wasn't due to technical issues.
+The biggest challenge was isolating and recreating the original distribution software.  You can find all the gory
+details on the page I created for "[Adventures in Math](/apps/pcx86/1983/adventmath/)".
+
+In short, although the software is archived inside multiple ZIP archives at the [Internet Archive](https://archive.org/),
+none of them appeared to contain an un-mingled, un-modified, distribution-only copy of the original software.
+And discovering that the software was archived inside multiple ZIP archives was a purely accidental discovery, not
+something one could easily determine using the Internet Archive's own search engine.
+
+### Emulation Challenges
+
+There are also some fundamental problems with the Internet Archive's approach to emulation.  They load sets of files,
+along with custom batch files, into tightly coupled machine configurations, with no indication which files were part
+of the original software distribution, and no hardware details regarding their "mystery" machine configurations. 
+
+And the Internet Archive's overall "emulation experience" is often not very faithful.  Their rendering of
+"Adventures in Math" gives the impression that the game ran *much* faster than it actually ran on a 4.77Mhz IBM PC.
+While their emulator can be configured operate at different speeds, it's unclear which of their holdings have already
+been "tuned".  As a fallback, they do provide some user controls:
+
+> "While the program is running, press CTRL-F11 to slow the program down. Pressing CTRL-F11 repeatedly will slow it
+down further, until the speed is more reasonable. (Pressing CTRL-F12 repeatedly will attempt to speed things up.)"
+
+but in general, the user has no idea what speed they have currently selected, or what the target speed should be.
+For users who simply want "to play the game," that may be fine, but an historian would probably prefer to see how the
+game originally operated and how each screen was painstakingly rendered, to better understand both the original user
+experience and the challenges that early software developers faced.
+
+The failure to archive software exactly as it was originally distributed, generally as one or more disk image (IMG)
+files along with scanned (PDF) copies of the documentation, is the most serious problem we collectively face.  I don't
+blame the Internet Archive, because in most cases, they're simply archiving what someone else archived before them,
+and those earlier "archivists" often didn't take the time to thoroughly preserve and document everything.
+
+On the other hand, the Internet Archive could do a better job of cataloging its holdings, recording the provenance of
+everything they have archived, maintaining a clear separation of original material from subsequent modifications
+(including any modifications created solely for ease of emulation), providing more control over machine configurations,
+and enabling the use of third-party emulators -- like PCjs -- which may be able to perform a more faithful rendering
+of certain software packages.
+
+In addition, I would hope that anything the Internet Archive "archives" itself (ie, from original distribution media)
+is duplicated and preserved according to a *much* higher standard than any files, documents, etc, it has simply scraped
+from the Internet.
+
+*[@jeffpar](http://twitter.com/jeffpar)*  
+*Jun 15, 2017*

apps/pcx86/1983/adventmath/README.md

@@ -194,47 +194,14 @@ which translates to:
 	a:basica castle     [run BASICA on drive A: and load CASTLE.BAS on drive B:]
 
 This is one of the preferred methods of demonstrating software in PCjs: loading unmodified disk images into machines
-that match the system requirements as closely as possible, powering the machine, and then automatically executing
+that match the software's system requirements as closely as possible, powering the machine, and then automatically
-the same series of commands that an owner of that hardware and software combination might have executed.
+executing a series of commands that an owner of that hardware and software combination would have likely executed.
 
 The other preferred method is to use a machine state file, which restores the machine to a state where the software is
 already running.  This is the fastest approach, and it is useful when the software has an involved installation
 process, but it takes more effort to set up.  The [VisiCalc (1981)](/apps/pcx86/1981/visicalc/) demo operates this way,
 but it does so purely for convenience (and because it predates the "autoType" feature).
 
-An alternative approach, adopted by the Internet Archive, involves modified disk images (eg, with custom batch files)
-loaded into tightly coupled machine configurations.  While this approach offers ease of use, it's difficult for the
-user to download or inspect the software exactly as it was originally distributed, or to load the software into any
-machine configuration other than the "mystery" configuration chosen by the Internet Archive. 
-
-Also, the Internet Archive's overall "emulation experience" is often not very faithful.  Their rendering of "Adventures
-in Math" gives the impression that the game ran *much* faster than it actually ran on a 4.77Mhz IBM PC.
-While the Internet Archive's emulator can be configured operate at different speeds, it's unclear which of their
-holdings have already been "tuned".  As a fallback, they do provide some user controls:
-
-> "While the program is running, press CTRL-F11 to slow the program down. Pressing CTRL-F11 repeatedly will slow it
-down further, until the speed is more reasonable. (Pressing CTRL-F12 repeatedly will attempt to speed things up.)"
-
-but in general, the user has no idea what speed they have currently selected, or what the target speed should be.
-For users who simply want "to play the game," that may be fine, but an historian would probably prefer to see how the
-game originally operated and how each screen was painstakingly rendered, to better understand both the original user
-experience and the challenges that early software developers faced.
-
-The failure to archive software exactly as it was originally distributed, generally as one or more disk image (IMG)
-files along with scanned (PDF) copies of the documentation, is the most serious problem we collectively face.  I don't
-blame the Internet Archive, because in most cases, they're simply archiving what someone else archived before them,
-and those earlier "archivists" often didn't take the time to thoroughly preserve and document everything.
-
-On the other hand, the Internet Archive could do a better job of cataloging its holdings, recording the provenance of
-everything they have archived, maintaining a clear separation of original material from subsequent modifications (including
-any modifications created solely for ease of emulation), providing more control over machine configurations, and
-enabling the use of third-party emulators -- like PCjs -- that may be able to do a better job running and rendering
-some software packages.
-
-In addition, I would hope that anything the Internet Archive "archives" itself (ie, from original distribution media)
-is duplicated and preserved according to a *much* higher standard than any files, documents, etc, it has simply scraped
-from the Internet.
-
 References
 ----------
 

disks/pcx86/fixed/10mb/README.md

@@ -7,12 +7,80 @@ permalink: /disks/pcx86/fixed/10mb/
 10Mb Hard Drive (Fixed Disk) Images
 ---
 
-This folder contains the following 10Mb disk images:
+This folder contains the following 10Mb fixed disk images:
  
+* Empty formatted disk ([PCDOS200-EMPTY](PCDOS200-EMPTY.json))
 * PC-DOS 2.00 with Windows 1.01 for CGA ([PCDOS200-WIN101-CGA](pcdos200-win101-cga.xml))
 * PC-DOS 2.00 with Windows 1.01 for EGA ([PCDOS200-WIN101-EGA](pcdos200-win101-ega.xml))
 
-These disk images are used by various [Model 5160](/devices/pcx86/machine/5160/) machine configurations.
+These disk images are used by various IBM PC XT [Model 5160](/devices/pcx86/machine/5160/) machine configurations.
+
+NOTE: The empty disk is *completely* empty.  It was partitioned with the PC-DOS 2.00 **FDISK** utility, allocating
+the entire disk to a single DOS partition, and then it was formatted with the PC-DOS 2.00 **FORMAT** utility.
+Neither the **FORMAT** "/S" option nor the **SYS** command were used, so no system files were transferred, leaving
+the disk completely empty and non-bootable.
+
+After formatting, PC-DOS 2.00 reports:
+
+	10592256 bytes total disk space
+	10592256 bytes available on disk
+
+As explained in the [DiskDump source code](/modules/diskdump/lib/diskdump.js), in its description of the 10Mb
+BPB, a 10Mb "Type 3" PC XT fixed disk contains:
+
+	306 cylinders
+	4 heads
+	17 sectors/track
+
+for a total of 20808 sectors or 10,653,696 bytes.  However, as p.1-179 of the PC XT Technical Reference Manual
+(April 1983) notes:
+
+	WARNING: The last cylinder on the fixed disk drive is reserved for diagnostic use.
+    Diagnostic write tests will destroy any data on this cylinder.
+
+And this is confirmed by the PC XT BIOS, p.A-94, in the code for the "GET PARAMETERS (AH = 8)" function:
+
+	C800:03B2 268B07        MOV     AX,ES:[BX]      ; MAX NUMBER OF CYLINDERS
+	C800:03B5 2D0200        SUB     AX,2            ; ADJUST FOR 0-N AND RESERVE LAST TRACK
+
+By "RESERVE LAST TRACK", they really meant "RESERVE LAST CYLINDER", because 68 (not 17) sectors are reserved at
+the end of the disk.  In addition, the first sector of the disk is reserved for the Master Boot Record (MBR), so there
+are a total of 69 reserved sectors.  (20808 - 69) = 20739 or 0x5103, which is exactly what's stored in the "total
+sectors" field of the disk's BPB, yielding a total partition size of 10,618,368 bytes.
+
+However, that 69-sector overhead is not the end of the story.  There is also overhead incurred by the FAT file system,
+which, in this case, consists of:
+
+- Boot sector (1)
+- FAT sectors (2 * 8 = 16)
+- Root directory sectors (32)
+
+for a total of 49 sectors, leaving (20739 - 49) = 20690 sectors.  Moreover, free space is measured in clusters,
+not sectors, and the partition uses 8 sectors/cluster, leaving room for 2586.25 clusters.  Since a fractional cluster
+is not allowed, another 2 sectors are lost, bringing the total FAT file system overhead to 51 sectors.
+ 
+Thus, actual free space is (20739 - 51) * 512, or 10,592,256 bytes, which is exactly what DOS reports as the available
+space.
+
+Some sources on the internet (eg, http://www.wikiwand.com/en/Timeline_of_DOS_operating_systems) claim that the (FAT)
+file system overhead for the XT's 10Mb disk is "50 sectors".  As they explain:
+
+	"The fixed disk has 10,618,880 bytes of raw space: 305 cylinders (the equivalent of tracks) × 2 platters
+	× 2 sides or heads per platter × 17 sectors per track = 20,740 sectors × 512 bytes per sector = 10,618,880
+	bytes...."
+
+and:
+
+	"With DOS the only partition, the combined overhead is 50 sectors leaving 10,592,256 bytes for user data:
+	DOS's FAT is eight sectors (16 sectors for two copies) + 32 sectors for the root directory, room for 512
+	directory entries + 2 sectors (one master and one DOS boot sector) = 50 sectors...."
+
+However, that's incorrect.  First, the disk has 306 cylinders, not 305.  Second, there are TWO overhead values:
+the overhead OUTSIDE the partition (69 sectors) and the overhead INSIDE the partition (51 sectors).  They failed
+to account for the reserved cylinder in the first calculation and the lost fractional cluster in the second
+calculation, and then they conflated the two values to produce a single (incorrect) result.
+
+---
 
 To use one of these disks with another machine configuration, include:
 

modules/shared/lib/debugger.js

@@ -905,8 +905,8 @@ class Debugger extends Component
              * And to avoid conflicts with MACRO-10 syntax, I've replaced the original mod operator ('%') with '^/'.
              *
              * The MACRO-10 binary shifting suffix ('B') is a bit more problematic, since a capital B can also appear
-             * inside symbols.  So I pre-scan for that suffix and replace all non-symbolic occurrences with an internal
+             * inside symbols, or inside hex values.  So if the default base is NOT 16, then I pre-scan for that suffix
-             * shift operator ('^_').
+             * and replace all non-symbolic occurrences with an internal shift operator ('^_').
              *
              * Note that Str.parseInt(), which parseValue() relies on, supports both the MACRO-10 base prefix overrides
              * and the binary shifting suffix ('B'), but since that suffix can also be a bracketed expression, we have to
@@ -923,7 +923,9 @@ class Debugger extends Component
              * that to generate an error; if we converted it to "AB", evaluation might inadvertently succeed.
              */
             var regExp = /({|}|\|\||&&|\||\^!|\^B|\^O|\^D|\^L|\^-|~|\^_|_|&|!=|!|==|>=|>>>|>>|>|<=|<<|<|-|\+|\^\/|\/|\*|,,| )/;
-            sExp = sExp.replace(/(^|[^A-Z0-9$%.])([0-9]+)B/, "$1$2^_").replace(/\s+/g, ' ');
+            if (this.nBase != 16) {
+                sExp = sExp.replace(/(^|[^A-Z0-9$%.])([0-9]+)B/, "$1$2^_").replace(/\s+/g, ' ');
+            }
             var asValues = sExp.split(regExp);
             value = this.parseArray(asValues, 0, asValues.length, this.nBase, aUndefined);
             if (value !== undefined && fPrint) {