CS 235 Advanced Operating Systems, Fall 2010

Lab 2: Memory Management and Exceptions

Due 11:59pm Friday, October 22

Introduction

In this lab, you'll write the memory management and initial exception handling code for your operating system.

The first component of memory management is virtual memory, where we set up the PC's Memory Management Unit (MMU) hardware to map the virtual addresses used by software into the physical addresses that map directly to memory chips. You will modify JOS to set up virtual memory mappings according to a specification we provide. In particular, you'll build a page table data structure to match our specification.

The second component is managing the physical memory of the computer so that the kernel can allocate and free physical memory as needed. The x86 divides physical memory into 4096-byte regions called pages. Your task will be to maintain data structures that record which physical pages are free and which are allocated, and how many processes are sharing each allocated page. You will also write the routines to allocate and free pages of memory.

Finally, you'll write initial exception-handling code for JOS, so that it can take an interrupt while in kernel mode.

Getting started

In this and future labs you will progressively build on this same kernel. With each new lab we will update our source to contain additional files and possibly some changes to existing files; you'll need to merge your changes from the previous lab into the new source tree.

To fetch the new source, use Git to commit your lab 1 and save that code in a branch called lab1. Then fetch the latest version of the course repository, and update your local branch based on our lab2 branch, origin/lab2:

% git status
...               # Look over the output of this command.
                  # If you added new files to your source, add them
                  # using "git add".
% git commit -am 'my solution to lab1'
Created commit 254dac5: my solution to lab1
 3 files changed, 31 insertions(+), 6 deletions(-)
% git branch lab1
                  # This creates the lab1 branch.
% git fetch
                  # This fetches our new code.
% git checkout -b lab2 origin/lab2
Branch lab2 set up to track remote branch refs/remotes/origin/lab2.
Switched to a new branch "lab2"

The git checkout -b command shown above actually does two things: it first creates a local branch lab2 that is based on the origin/lab2 branch provided by the course staff, and second, it changes the contents of your lab directory to reflect the files stored on the lab2 branch. Git allows switching between existing branches using git checkout branch-name, so you can recover your Lab 1 code with git checkout lab1. But you should commit any outstanding changes on one branch before switching to a different one.

You will now need to merge the changes you made in your lab1 branch into the lab2 branch, as follows:

% git merge lab1
Merge made by recursive.
 kern/kdebug.c  |   11 +++++++++--
 kern/monitor.c |   19 +++++++++++++++++++
 lib/printfmt.c |    7 +++----
 3 files changed, 31 insertions(+), 6 deletions(-)

In some cases, Git may not be able to figure out how to merge your changes with the new lab assignment (e.g. if you modified some of the code that is changed in the second lab assignment). In that case, the git merge command will tell you which files are conflicted, and you should first resolve the conflict (by editing the relevant files) and then commit the resulting files with git commit -a.

You can download the tarball if you'd like, but we recommend using Git.

You should browse through these source files for Lab 2, most of which are new.

• inc/memlayout.h
• kern/pmap.c
• kern/pmap.h
• kern/kclock.h
• kern/kclock.c
• inc/trap.h
• kern/trap.h
• kern/trap.c
• kern/trapentry.S

memlayout.h describes the layout of the virtual address space that you must implement by modifying pmap.c. pmap.h defines the Page structure that you'll use to keep track of which pages of physical memory are free. kclock.c and kclock.h manipulate the PC's battery-backed clock and CMOS RAM hardware, in which the BIOS records the amount of physical memory the PC contains, among other things. The code in pmap.c needs to read this device hardware in order to figure out how much physical memory there is, but that part of the code is done for you: you do not need to know the details of how the CMOS hardware works. The last four files are used to set up the processor's interrupt descriptor table and handle interrupts and exceptions.

Lab Requirements

In this lab, you'll need to do all of the regular exercises described in the lab and at least one challenge problem. Some challenge problems are more challenging than others, of course! And feel free to design your own challenge problem; just check with me first.

Additionally, write up brief answers to the 8 explicitly-marked Questions posed in the lab and a short (one or two paragraph) description of what you did to solve your chosen challenge problem. If you implement more than one challenge problem, you only need to describe one of them in the write-up, though of course you are welcome to do more. Place the write-up in a file called answers.txt (plain text) or answers.html (HTML format) in the top level of your lab2 directory before handing in your work.

Hand-In Procedure

When you are ready to hand in your lab code and write-up, run make tarball in your jos directory. This will create a file called lab2-yourusername.tar.gz, which you should submit via CourseWeb at 11:59pm on Friday, October 22. If you have problems with CourseWeb, you may also email me the file.

As before, we will be grading your solutions with a grading program. You can run make grade to test your kernel with the grading program. You may change any of the kernel source and header files you need to in order to complete the lab, but we check to make sure you aren't changing or otherwise subverting the grading code.

Part 1: Virtual Memory

Before doing anything else, you will need to familiarize yourself with the x86's protected-mode memory management architecture: namely segmentation and page translation.

In x86 terminology, a virtual address is a "segment:offset"-style address before segment translation is performed; a linear address is what you get after segmentation but before page translation; and a physical address is what you finally get after both segmentation and page translation. Be sure you understand the difference between these three types or "levels" of addresses! (For instance, what type of addresses are stored in page directory and page table entries, virtual or physical?)

The JOS kernel tries to use consistent type names for different kinds of address. In particular, the type uintptr_t represents virtual addresses, and physaddr_t represents physical addresses. Of course, both these types are really just synonyms for 32-bit integers (uint32_t), so the compiler won't stop you from assigning one type to another! Every pointer value in JOS should be a virtual address (once paging is set up), since only virtual addresses can be dereferenced. The kernel runs in protected mode too! To summarize:

The Physical Page Allocator

The operating system must keep track of which parts of physical RAM are free and which are currently in use. JOS manages the PC's physical memory with page granularity so that it can use the MMU to map and protect each piece of allocated memory.

You'll now write the physical page allocator. The kernel maintains an array of struct Page objects, one per physical page. Those Page objects whose physical pages are available for allocation are kept on a linked list, page_free_list. You need to write the physical page allocator before you can write the rest of the virtual memory implementation, because your page table management code will need to allocate physical memory in which to store page tables.

	boot_alloc()
	page_init()
	page_alloc()
	page_free()

Page Table Management

Now you'll write a set of routines to manage page tables: to insert and remove linear-to-physical mappings, and to create page table pages when needed.

In future labs you will often map the same physical page at multiple virtual addresses (or in the address spaces of multiple environments), so you will keep a count of the number of times each physical page is mapped in the corresponding Page's pp_ref. The count should be equal to the number of pointers to the page in the page table(s) below UTOP (the mappings above UTOP are mostly just set up at boot time by the kernel and are not tracked by the reference-counting system). When the count goes to zero, the physical page can be freed.

	pgdir_walk()
	page_lookup()
	page_remove()
	page_insert()

The JOS Kernel Virtual Address Space

Now, mem_init() will set up the kernel's virtual address space -- the part above UTOP. The actual layout is diagrammed in inc/memlayout.h.

In Part 3 of Lab 1 we noted that the boot loader sets up the x86 segmentation hardware so that the kernel appears to run at its link address of 0xf0100000, even though it is actually loaded in physical memory just above the ROM BIOS at 0x00100000. In other words, the kernel's virtual starting address at this point is 0xf0100000, but its linear and physical starting addresses are both 0x00100000. The kernel's linear and physical addresses are the same because we have not yet initialized or enabled page translation.

In the virtual memory layout you are going to set up for JOS, we will stop using the x86 segmentation hardware for anything interesting, and instead start using page translation to accomplish everything we've already done with segmentation and much more. That is, after you finish this lab and the JOS kernel successfully enables paging, linear addresses will be the same as (the offset portion of) the kernel's virtual addresses, rather than being the same as physical addresses as they are when the boot loader first enters the kernel.

In JOS, we divide the processor's 32-bit linear address space into two parts. User environments (processes), which we will begin loading and running in lab 3, will have control over the layout and contents of the lower part, while the kernel always maintains complete control over the upper part. The dividing line is defined somewhat arbitrarily by the symbol ULIM in inc/memlayout.h, reserving approximately 256MB of linear (and therefore virtual) address space for the kernel. This explains why we needed to give the kernel such a high link address in lab 1: otherwise there would not be enough room in the kernel's linear address space to map in a user environment below it at the same time. The JOS kernel also maintains constant complete control over the contents of memory by mapping all of physical memory to its portion of the address space. To change a byte of physical memory, all it needs to do is find a kernel virtual pointer for this address (using the KADDR macro), and change the value stored in that pointer.

Since the kernel and user environment will effectively co-exist in each environment's address space, we will have to use permission bits in our x86 page tables to prevent user code from accessing the kernel's memory: i.e., to enforce isolation. The user environment will have no permission to any of the memory above ULIM, while the kernel will be able to read and write this memory. For the address range [UTOP,ULIM), both the kernel and the user environment have the same permission: they can read but not write this address range. This range of addresses is used to expose certain kernel data structures read-only to the user environment. Lastly, the address space below UTOP is for the user environment to use; the user environment will set permissions for accessing this memory.

It's now time to set up the kernel portion of the address space.

Address Space Layout Alternatives

Many other address space layouts are possible besides the one we chose for JOS; all of this is up to the operating system. It is possible, for example, to map the kernel at low linear addresses while leaving the upper part of the linear address space for user processes. x86 kernels generally do not take this approach, however, because one of the x86's backward-compatibility modes, known as virtual 8086 mode, is "hard-wired" in the processor to use the bottom part of the linear address space, and thus cannot be used at all if the kernel is mapped there.

It is even possible, though much more difficult, to design the kernel so as not to have to reserve any fixed portion of the processor's linear or virtual address space for itself, but instead effectively to allow allow user-level processes unrestricted use of the entire 4GB of virtual address space -- while still fully protecting the kernel from these processes and protecting different processes from each other!

	K> alloc_page
		0x13000
	K> page_status 0x13000
		allocated
	K> free_page 0x13000
	K> page_status 0x13000
		free
	

Part 3: Handling Interrupts and Exceptions

In this part of the lab, you'll add initial support for exception handling to your JOS kernel. This includes processor exceptions, such as divide-by-zero errors; hardware interrupts; and system calls, where user-level programs transfer control to the kernel. (We don't have user-level programs yet, but exceptions like page faults and divide-by-zero can happen in the kernel too.) All of these are types of protected control transfer that, among other things, enable the processor to switch from user to kernel mode cleanly without giving the user-mode code any opportunity to interfere with the functioning of the kernel or other environments.

The first thing you should do is thoroughly familiarize yourself with the x86 interrupt and exception mechanism. In Intel's terminology, an interrupt is a protected control transfer that is caused by an asynchronous event usually external to the processor, such as notification of external device I/O activity. An exception, in contrast, is a protected control transfer caused synchronously by the currently running code, for example due to a divide by zero or an invalid memory access. In this lab we generally follow Intel's terminology, but be aware that terms such as exceptions, traps, interrupts, faults and aborts have no standardized meaning across architectures or operating systems, and often used rather loosely without close regard to the subtle distinctions between them on a particular architecture such as the x86. When you see these terms outside of this lab, the meanings might be slightly different.

Basics of Protected Control Transfer

In order to ensure that protected control transfers are actually protected, the processor's interrupt/exception mechanism is designed so that the code running when an interrupt or exception occurs has strictly limited influence over where and how the kernel is entered. Instead, the processor ensures that the kernel can be entered only under carefully controlled conditions. On the x86, this protection is provided by two particular mechanisms:

1. The Interrupt Descriptor Table. The processor ensures that interrupts and exceptions can only cause the kernel to be entered at a few specific, well-defined entry-points determined by the kernel itself, and not by the code currently running when the interrupt or exception is taken.

   In particular, x86 interrupts and exceptions are differentiated into up to 256 possible "types", each associated with a particular interrupt number (often referred to synonymously as an exception number or trap number). Once the processor identifies a particular interrupt or exception to be taken, it uses the interrupt number as an index into the processor's interrupt descriptor table (IDT), which is a special table that the kernel sets up in kernel-private memory, much like the GDT. From the appropriate entry in this table the processor loads:

   • The value to load into the instruction pointer (EIP) register, which points to the kernel code designated to handle that type of exception.
   • The value to load into the code segment (%cs) register, which includes in bits 0-1 the privilege level at which the exception handler is to run. (In JOS, all exceptions are handled in kernel mode, or privilege level 0.)

2. The Task State Segment. In addition to having a well-defined entrypoint in the kernel for an interrupt or exception handler, the processor also needs a place to save the old processor state before the interrupt or exception occurred, such as the original values of %eip and %cs before the processor invoked the exception handler, so that the exception handler can later restore that old state and resume the interrupted code from where it left off. But this save area for the old processor state must in turn be protected from unprivileged user-mode code; otherwise buggy or malicious user code could easily compromise the kernel.

   For this reason, when an x86 processor takes an interrupt or trap that causes a privilege level change from user to kernel mode, it not only loads new values into %eip and %cs, but also loads new values into the stack pointer (%esp) and stack segment (%ss) registers, effectively switching to a new stack private to the kernel. The processor then pushes the original values of all of these registers, along with the contents of the eflags register, onto this new kernel stack before starting to run the kernel's exception handler code. The new %esp and %ss do not come from the IDT like %eip and %cs, but instead from a separate structure called the task state segment (TSS).

   Although the TSS is a somewhat large and complex data structure that can potentially serve a variety of purposes, in JOS it will only be used to define the kernel stack that the processor should switch to when it transfers from user to kernel mode. Since "kernel mode" in JOS is privilege level 0 on the x86, the processor uses the ESP0 and SS0 fields of the TSS to define the kernel stack when entering kernel mode; none of the other fields in the TSS will ever ever be used in JOS.

Types of Exceptions and Interrupts

All of the synchronous exceptions that the x86 processor can generate internally use interrupt numbers between 0 and 31, and therefore map to IDT entries 0-31. For example, the page fault handler is "hard-wired" by Intel to interrupt number 14. Interrupt numbers greater than 31 are only used by software interrupts, which can be generated by the int instruction, or asynchronous hardware interrupts, caused by external devices when they need attention.

In this section we will extend JOS to handle the internally generated x86 exceptions in the 0-31 that are currently defined by Intel. In the next labs, we'll make JOS handle software interrupt number 48, which it (fairly arbitrarily) uses as its system call interrupt number, and extend it to handle externally generated hardware interrupts such as the clock interrupt.

A Kernel-Mode Exception Example

Let's put these pieces together and trace through an example. Say the processor is executing code in kernel mode (the low 2 bits of the %cs register are 0), and encounters a divide instruction that attempts to divide by zero. Then:

1. The processor pushes the exception parameters on the kernel stack.

                        +-------------------+ <-- old ESP
                        |     old EFLAGS    |        " - 4
                        | 0x0000  | old CS  |        " - 8
                        |      old EIP      | <-- ESP = old ESP - 12
                        +-------------------+  

2. Because we're handling a divide error, which is interrupt number 0 on the x86, the processor reads IDT entry 0 and sets CS:EIP to point to the handler function defined there.
3. The handler function takes control and handles the exception.
4. When it's finished, the handler function can return from the interrupt by executing an iret instruction.

For certain types of x86 exceptions, the processor pushes onto the stack another word containing an error code. The page fault exception, number 14, is an important example. See the 80386 manual to determine for which exception numbers the processor pushes an error code, and what the error code means in that case. (Exceptions 17, 18, and 19 are new since the 80386; see the IA-32 Architecture manual Volume 3, Section 5.) When the processor pushes an error code, the stack would look as follows at the beginning of the exception handler:

                     +-------------------+ <-- old ESP
                     |     old EFLAGS    |         " - 4
                     | 0x0000  | old CS  |         " - 8
                     |      old EIP      |         " - 12
                     |     error code    | <-- ESP = old ESP - 16
                     +-------------------+  

There is one important caveat to the processor's kernel-mode exception capability. If the processor takes an exception while already in kernel mode, and cannot push its old state onto the kernel stack for any reason such as lack of stack space, then there is nothing the processor can do to recover, so it simply resets itself. Needless to say, any decent kernel should be designed so that this will never happen unintentionally.

Of course, user-mode programs can divide by zero too! The processor handles user-mode exceptions slightly differently to provide protection. It would not be safe to simply push interrupt context information onto a user program stack. (For one thing, a user program that runs out of stack space must not cause the processor to reset!) Thus, when a user-mode program takes an exception, the kernel switches to a special kernel stack. Information about the old stack is pushed onto the exception stack first, above the old EFLAGS:

                     +-------------------+ <-- KSTACKTOP
                     | 0x0000  | old SS  |         " - 4
                     |      old ESP      |         " - 8
                     |     old EFLAGS    |         " - 12
                     | 0x0000  | old CS  |         " - 16
                     |      old EIP      | <-- ESP = KSTACKTOP - 20 (maybe)
                     | (optional errcode)| <-- ESP = KSTACKTOP - 24 (maybe)
                     +-------------------+  

You'll handle user-mode exceptions in the next lab; just remember that there are differences in calling convention between kernel-mode and user-mode exceptions.

Setting Up the IDT

You should now have the basic information you need in order to set up the IDT and handle exceptions in JOS. For now, you will set up the IDT to handle all the to handle interrupt numbers 0-31 (the processor exceptions). We'll add additional interrupts later.

The header files inc/trap.h and kern/trap.h contain important definitions related to interrupts and exceptions that you will need to become familiar with. The file kern/trap.h contains trap-related definitions that will remain strictly private to the kernel, while the companion header file inc/trap.h contains general definitions that may also be useful to user-level programs and libraries in the system.

Note: Some of the exceptions in the range 0-31 are defined by Intel to be reserved. Since they will never be generated by the processor, it doesn't really matter how you handle them. Do whatever you think is cleanest.

The overall flow of control that you should achieve is depicted below:

      IDT                   trapentry.S              trap.c
   
+----------------+                        
|   &handler1    |---------> handler1:         +---> void trap(struct Trapframe *tf)
|                |             // do stuff     |     {
|                |             call trap  -----+         // handle the exception/interrupt
|                |                        <----+         return;
|                |             // undo stuff   +---- }
+----------------+
|   &handler2    |--------> handler2:
|                |            // do stuff
|                |            call trap
|                |            // undo stuff
+----------------+
       .
       .
       .
+----------------+
|   &handlerX    |--------> handlerX:
|                |             // do stuff
|                |             call trap
|                |             // undo stuff
+----------------+

Each exception or interrupt should have its own handler in trapentry.S and idt_init() should initialize the IDT with the addresses of these handlers. Each of the handlers should build a struct Trapframe (see inc/trap.h) on the stack and call into trap() (in trap.c) with a pointer to the Trapframe.

After control is passed to trap(), that function handles the exception/interrupt or dispatches the exception/interrupt to a specific handler function. If and when the trap() function returns, the code in trapentry.S restores the old CPU state saved in the Trapframe and then uses the iret instruction to return from the exception.

The Breakpoint Exception

Now that your kernel has basic exception handling capabilities, you'll refine its response to one particular exception. Much more of this will happen in the next lab once we have user processes.

The breakpoint exception, interrupt number 3 (T_BRKPT), is normally used to allow debuggers to insert breakpoints in a program's code by temporarily replacing the relevant program instruction with the special 1-byte int3 software interrupt instruction. In JOS we will abuse this exception slightly by turning it into a primitive pseudo-system call that anyone can use to invoke the JOS kernel monitor, inside or outside the kernel. This usage is actually somewhat appropriate if we think of the JOS kernel monitor as a primitive debugger.

Stack backtrace:
  0: eip f0100047
         kern/init.c:46: _ZL22test_kernel_breakpointv+ac7 (0 arg)
  1: ebp f0117fd8  eip f0100145  args f0104169 00001aac 00000000 00000000
         kern/init.c:215: i386_init+4c (0 arg)
  2: ebp f0117ff8  eip f010003d  args 00000000 00000000 0000ffff 10cf9a00
         kern/entry.S:79: <unknown>+0 (0 arg)

This completes the lab.

Back to CS 235 Advanced Operating Systems