CS 261 Research Topics in Operating Systems, Fall 2011

Lab 2: Memory Management and Exceptions

Due 11:59pm Tuesday, September 27: Submit Here

Introduction

In this lab, you'll write the memory management and initial exception handling code for your operating system.

The first component is a physical memory allocator for the kernel, so that the kernel can allocate memory and later free it. Your allocator will operate in units of 4096 bytes called pages. Your task will be to maintain data structures that record which physical pages are free and which are allocated, and how many processes are sharing each allocated page. You will also write the routines to allocate and free pages of memory.

The second component of memory management is virtual memory, which maps the virtual addresses used by kernel and user software to addresses in physical memory. The x86 hardware's memory management unit (MMU) performs this mapping by consulting a set of page tables which software provides. You will modify JOS to set up page tables that match our specification.

Finally, you'll write initial exception-handling code for JOS, so that it can take an interrupt while in kernel mode.

Getting started

In this and future labs you will progressively build on this same kernel. With each new lab we will update our source to contain additional files and possibly some changes to existing files; you'll need to merge your changes from the previous lab into the new source tree.

% git remote show origin
* remote origin
  Fetch URL: git://read.seas.harvard.edu/git/jos2011.git
  Push  URL: git://read.seas.harvard.edu/git/jos2011.git
  HEAD branch: master
...

% git remote set-url origin git://read.seas.harvard.edu/git/jos2011.git

To fetch the new source, use Git to commit your lab 1 and save that code in a branch called lab1. Then fetch the latest version of the course repository, and update your local branch based on our lab2 branch, origin/lab2:

% git status
...               # Look over the output of this command.
                  # If you added new files to your source, add them
                  # using "git add".
% git commit -am 'my solution to lab1'
Created commit 254dac5: my solution to lab1
 3 files changed, 31 insertions(+), 6 deletions(-)
% git branch lab1
                  # This creates the lab1 branch.
% git fetch
                  # This fetches our new code.
% git checkout -b lab2 origin/lab2
Branch lab2 set up to track remote branch refs/remotes/origin/lab2.
Switched to a new branch "lab2"

The git checkout -b command shown above actually does two things: it first creates a local branch lab2 that is based on the origin/lab2 branch provided by the course staff, and second, it changes the contents of your lab directory to reflect the files stored on the lab2 branch. Git allows switching between existing branches using git checkout branch-name, so you can recover your Lab 1 code with git checkout lab1. But you should commit any outstanding changes on one branch before switching to a different one.

You will now need to merge the changes you made in your lab1 branch into the lab2 branch, as follows:

% git merge lab1
Merge made by recursive.
 kern/kdebug.c  |   11 +++++++++--
 kern/monitor.c |   19 +++++++++++++++++++
 lib/printfmt.c |    7 +++----
 3 files changed, 31 insertions(+), 6 deletions(-)

In some cases, Git may not be able to figure out how to merge your changes with the new lab assignment (e.g. if you modified some of the code that is changed in the second lab assignment). In that case, the git merge command will tell you which files are conflicted, and you should first resolve the conflict (by editing the relevant files) and then commit the resulting files with git commit -a.

You should browse through these source files for Lab 2, most of which are new.

• inc/memlayout.h
• kern/pmap.c
• kern/pmap.h
• kern/kclock.h
• kern/kclock.c
• inc/trap.h
• kern/trap.h
• kern/trap.c
• kern/trapentry.S

memlayout.h describes the layout of the virtual address space that you must implement by modifying pmap.c. pmap.h defines the Page structure that you'll use to keep track of which pages of physical memory are free. kclock.c and kclock.h manipulate the PC's battery-backed clock and CMOS RAM hardware, in which the BIOS records the amount of physical memory the PC contains, among other things. The code in pmap.c needs to read this device hardware in order to figure out how much physical memory there is, but that part of the code is done for you: you do not need to know the details of how the CMOS hardware works. The last four files are used to set up the processor's interrupt descriptor table and handle interrupts and exceptions.

Lab Requirements

In this lab, you'll need to do all of the regular exercises described in the lab and at least one challenge problem. Some challenge problems are more challenging than others, of course! And feel free to design your own challenge problem; just check with me first.

You don't need to write up answers to the 7 explicitly-marked Questions, but do them anyway, to check your understanding.

Additionally, write up a short (one or two paragraph) description of what you did to solve your chosen challenge problem. If you implement more than one challenge problem, you only need to describe one of them in the write-up, though of course you are welcome to do more. Place the write-up in a file called answers.txt (plain text) or answers.html (HTML format) in the top level of your lab2 directory before handing in your work.

Hand-In Procedure

When you are ready to hand in your lab code and write-up, run make tarball in your jos directory. This will create a file called lab2-yourusername.tar.gz, which you should submit via Web form (preferred) or email by 11:59pm on Tuesday, September 27.

As before, we will be grading your solutions with a grading program. You can run make grade to test your kernel with the grading program. You may change any of the kernel source and header files you need to in order to complete the lab, but we check to make sure you aren't changing or otherwise subverting the grading code.

Part 1: Memory Management

Before doing anything else, you will need to familiarize yourself with the x86's protected-mode memory management architecture: namely segmentation and page translation.

Memory addresses in the IA-32 architecture can be virtual, linear, or physical. A virtual address is a "segment:offset"-style address before segment translation is performed; a linear address is what you get after segmentation but before page translation; and a physical address is what you finally get after both segmentation and page translation. In a picture:

           Selector  +--------------+         +-----------+
          ---------->|              |         |           |
                     | Segmentation | Linear  |  Paging   | Physical
Software             |              |-------->|           |---------->  RAM
            Offset   |  Mechanism   |         | Mechanism |
          ---------->|              |         |           |
                     +--------------+         +-----------+
            Virtual

Once we're in protected mode (which we entered first thing in boot/boot.S), there's no way to directly use a linear or physical address. All memory references are interpreted as virtual addresses and translated by the MMU, which means all pointers in C are virtual addresses. But you need to understand the translation mechanism both to help the OS set up the correct translation, and when debugging.

In boot/boot.S, we installed a Global Descriptor Table (GDT) that effectively disabled segment translation by setting all segment base addresses to 0 and limits to 0xffffffff. In lab 3, we'll have to interact a little more with segmentation to set up privilege levels, but as for virtual address translation, we can ignore segmentation throughout the JOS labs. The boot loader also installed a simple page table so that the kernel can run at its link address of 0xf0100000, even though it is actually loaded in physical memory just above the ROM BIOS at 0x00100000. This page table mapped only a 16MB portion of available memory. In this lab, you'll expand this to map the first 256MB of physical memory starting at virtual address 0xf0000000, and to map a number of other regions of virtual memory.

Be sure you understand the difference between these three types or "levels" of addresses! (For instance, what type of addresses are stored in page directory and page table entries, virtual or physical?)

The JOS kernel tries to use consistent type names for different kinds of address. In particular, the type uintptr_t represents virtual addresses, and physaddr_t represents physical addresses. Of course, both these types are really just synonyms for 32-bit integers (uint32_t), so the compiler won't stop you from assigning one type to another! Since they are integer types (not pointers), the compiler will complain if you try to dereference them.

The JOS kernel can dereference a uintptr_t by first casting it to a pointer type. In contrast, the kernel can't sensibly dereference a physical address, since the MMU translates all memory references. If you cast a physaddr_t to a pointer and dereference it, you may be able to load and store to the resulting address (the hardware will interpret it as a virtual address), but you probably won't get the memory location you intended.

To summarize:

While GDB can only access QEMU's memory by virtual address, it's often useful to be able to inspect physical memory while setting up virtual memory. The QEMU monitor has several commands, especially xp, which let you inspect physical memory. To access the QEMU monitor, press Ctrl-a c in the terminal, or press Ctrl-Alt-2 on the QEMU window. Use the xp command in the QEMU monitor and the x command in GDB to inspect memory at corresponding physical and virtual addresses and make sure you see the same data. QEMU's info mem command may also prove useful in the lab: it shows which ranges of virtual memory are mapped and with what permissions. This summary is useful, but lacks a lot of detail, so we've also added an info pg command to our patched version of QEMU that prints out the current page table.

The Physical Page Allocator

The operating system must keep track of which parts of physical RAM are free and which are currently in use. JOS manages the PC's physical memory with page granularity so that it can use the MMU to map and protect each piece of allocated memory.

You'll now write the physical page allocator. The kernel maintains an array of struct Page objects, one per physical page. Those Page objects whose physical pages are available for allocation are kept on a linked list, page_free_list. You need to write the physical page allocator before you can write the rest of the virtual memory implementation, because your page table management code will allocate physical memory in which to store page tables.

	boot_alloc()
	page_init()
	page_alloc()
	page_free()

Page Table Management

Now you'll write a set of routines to manage page tables: to insert and remove linear-to-physical mappings, and to create page table pages when needed.

In future labs you will often map the same physical page at multiple virtual addresses (or in the address spaces of multiple environments), so you will keep a count of the number of times each physical page is mapped in the corresponding Page's pp_ref. The count should be equal to the number of pointers to the page in the page table(s) below UTOP (the mappings above UTOP are not tracked by the reference-counting system). When the count goes to zero, the physical page can be freed.

	pgdir_walk()
	page_lookup()
	page_remove()
	page_insert()

The JOS Kernel Virtual Address Space

Now, mem_init() will set up the kernel's virtual address space -- the part above UTOP. The actual layout is diagrammed in inc/memlayout.h.

JOS divides the processor's 32-bit linear address space into two parts. User environments (processes), which we will begin loading and running in lab 3, will have control over the layout and contents of the lower part, while the kernel always maintains complete control over the upper part. The dividing line is defined somewhat arbitrarily by the symbol ULIM in inc/memlayout.h, reserving approximately 256MB of linear (and therefore virtual) address space for the kernel. This explains why we needed to give the kernel such a high link address in lab 1: otherwise there would not be enough room in the kernel's linear address space to map in a user environment below it at the same time.

The JOS kernel also maintains constant complete control over the contents of memory by mapping all of physical memory to its portion of the address space. This is because it sometimes needs to read or modify memory for which it only knows the physical address (for instance, second-level page tables are accessed via physical addresses from the top-level page directory). Since the kernel, like any other software, must use virtual memory translation and thus cannot directly load and store to physical addresses, JOS remaps of all of physical memory starting from physical address 0 at virtual address 0xf0000000. In order to translate a physical address into a virtual address that the kernel can actually read and write, the kernel must add 0xf0000000 to the physical address to find its corresponding virtual address in the remapped region. You should use KADDR(pa) to do that addition.

The kernel also sometimes needs to be able to find a physical address given the virtual address of the memory in which a kernel data structure is stored. Kernel global variables and memory allocated by boot_alloc() are in the region where the kernel was loaded, starting at 0xf0000000, the very region where we mapped all of physical memory. Thus, to turn a virtual address in this region into a physical address, the kernel can simply subtract 0xf0000000. You should use PADDR(va) to do that subtraction.

Since the kernel and user environment will effectively co-exist in each environment's address space, we must set permission bits in our x86 page tables that prevent user code from accessing the kernel's memory: i.e., that enforce isolation. The user environment will have no permission to any of the memory above ULIM, while the kernel will be able to read and write this memory. For the address range [UTOP,ULIM), both the kernel and the user environment have the same permission: they can read but not write this address range. This range of addresses is used to expose certain kernel data structures read-only to the user environment. Lastly, the address space below UTOP is for the user environment to use; the user environment will set permissions for accessing this memory.

It's now time to set up the kernel portion of the address space.

Address Space Layout Alternatives

Many other address space layouts are possible besides the one we chose for JOS; all of this is up to the operating system. It is possible, for example, to map the kernel at low linear addresses while leaving the upper part of the linear address space for user processes. x86 kernels generally do not take this approach, however, because one of the x86's backward-compatibility modes, known as virtual 8086 mode, is "hard-wired" in the processor to use the bottom part of the linear address space, and thus cannot be used at all if the kernel is mapped there.

It is even possible, though much more difficult, to design the kernel so as not to have to reserve any fixed portion of the virtual linear address space for itself, but instead effectively to allow allow user-level processes unrestricted use of the entire 4GB of virtual address space -- while still fully protecting the kernel from these processes and protecting different processes from each other!

	K> alloc_page
		0x13000
	K> page_status 0x13000
		allocated
	K> free_page 0x13000
	K> page_status 0x13000
		free
	

Part 2: Handling Interrupts and Exceptions

In this part of the lab, you'll add initial support for exception handling to your JOS kernel. This includes processor exceptions, such as divide-by-zero errors; hardware interrupts; and system calls, where user-level programs transfer control to the kernel. (We don't have user-level programs yet, but exceptions like page faults and divide-by-zero can happen in the kernel too.) All of these are types of protected control transfer that, among other things, enable the processor to switch from user to kernel mode cleanly without giving the user-mode code any opportunity to interfere with the functioning of the kernel or other environments.

The first thing you should do is thoroughly familiarize yourself with the x86 interrupt and exception mechanism. In Intel's terminology, an interrupt is a protected control transfer that is caused by an asynchronous event usually external to the processor, such as notification of external device I/O activity. An exception, in contrast, is a protected control transfer caused synchronously by the currently running code, for example due to a divide by zero or an invalid memory access. In this lab we generally follow Intel's terminology, but be aware that terms such as exceptions, traps, interrupts, faults and aborts have no standardized meaning across architectures or operating systems, and often used rather loosely without close regard to the subtle distinctions between them on a particular architecture such as the x86. When you see these terms outside of this lab, the meanings might be slightly different.

Basics of Protected Control Transfer

In order to ensure that protected control transfers are actually protected, the processor's interrupt/exception mechanism is designed so that the code running when an interrupt or exception occurs has strictly limited influence over where and how the kernel is entered. Instead, the processor ensures that the kernel can be entered only under carefully controlled conditions. On the x86, this protection is provided by two particular mechanisms:

1. The Interrupt Descriptor Table. The processor ensures that interrupts and exceptions can only cause the kernel to be entered at a few specific, well-defined entry-points determined by the kernel itself, and not by the code currently running when the interrupt or exception is taken.

   In particular, x86 interrupts and exceptions are differentiated into up to 256 possible "types", each associated with a particular interrupt number (often referred to synonymously as an exception number or trap number). Once the processor identifies a particular interrupt or exception to be taken, it uses the interrupt number as an index into the processor's interrupt descriptor table (IDT), which is a special table that the kernel sets up in kernel-private memory, much like the GDT. From the appropriate entry in this table the processor loads:

   • The value to load into the instruction pointer (EIP) register, which points to the kernel code designated to handle that type of exception.
   • The value to load into the code segment (%cs) register, which includes in bits 0-1 the privilege level at which the exception handler is to run. (In JOS, all exceptions are handled in kernel mode, or privilege level 0.)

2. The Task State Segment. In addition to having a well-defined entrypoint in the kernel for an interrupt or exception handler, the processor also needs a place to save the old processor state before the interrupt or exception occurred, such as the original values of %eip and %cs before the processor invoked the exception handler, so that the exception handler can later restore that old state and resume the interrupted code from where it left off. But this save area for the old processor state must in turn be protected from unprivileged user-mode code; otherwise buggy or malicious user code could easily compromise the kernel.

   For this reason, when an x86 processor takes an interrupt or trap that causes a privilege level change from user to kernel mode, it not only loads new values into %eip and %cs, but also loads new values into the stack pointer (%esp) and stack segment (%ss) registers, effectively switching to a new stack private to the kernel. The processor then pushes the original values of all of these registers, along with the contents of the eflags register, onto this new kernel stack before starting to run the kernel's exception handler code. The new %esp and %ss do not come from the IDT like %eip and %cs, but instead from a separate structure called the task state segment (TSS).

   Although the TSS is a somewhat large and complex data structure that can potentially serve a variety of purposes, in JOS it will only be used to define the kernel stack that the processor should switch to when it transfers from user to kernel mode. Since "kernel mode" in JOS is privilege level 0 on the x86, the processor uses the ESP0 and SS0 fields of the TSS to define the kernel stack when entering kernel mode; none of the other fields in the TSS will ever ever be used in JOS.

Types of Exceptions and Interrupts

All of the synchronous exceptions that the x86 processor can generate internally use interrupt numbers between 0 and 31, and therefore map to IDT entries 0-31. For example, the page fault handler is "hard-wired" by Intel to interrupt number 14. Interrupt numbers greater than 31 are only used by software interrupts, which can be generated by the int instruction, or asynchronous hardware interrupts, caused by external devices when they need attention.

In this section we will extend JOS to handle the internally generated x86 exceptions in the 0-31 that are currently defined by Intel. In the next labs, we'll make JOS handle software interrupt number 48, which it (fairly arbitrarily) uses as its system call interrupt number, and extend it to handle externally generated hardware interrupts such as the clock interrupt.

A Kernel-Mode Exception Example

Let's put these pieces together and trace through an example. Say the processor is executing code in kernel mode (the low 2 bits of the %cs register are 0), and encounters a divide instruction that attempts to divide by zero. Then:

1. The processor pushes the exception parameters on the kernel stack.

                        +-------------------+
               0(%esp)  |      old EIP      | <-- new ESP = (old ESP - 12)
               4(%esp)  | 0x0000  | old CS  |
               8(%esp)  |     old EFLAGS    |
                        + . . . . . . . . . + <-- old ESP 

   (Note: We are drawing addresses increasing down the page, to match how C structures are written, so this stack grows up the page. Many x86 manuals use a different convention.)
2. Because we're handling a divide error, which is interrupt number 0 on the x86, the processor reads IDT entry 0 and sets CS:EIP to point to the handler function defined there.
3. The handler function takes control and handles the exception.
4. When it's finished, the handler function can return from the interrupt by executing an iret instruction.

For certain types of x86 exceptions, the processor pushes onto the stack another word containing an error code. The page fault exception, number 14, is an important example. See the 80386 manual to determine for which exception numbers the processor pushes an error code, and what the error code means in that case. (Exceptions 17, 18, and 19 are new since the 80386; see the IA-32 Architecture manual Volume 3, Section 5.) When the processor pushes an error code, the stack would look as follows at the beginning of the exception handler:

                     +-------------------+
            0(%esp)  |     error code    | <-- new ESP = (old ESP - 16)
            4(%esp)  |      old EIP      |
            8(%esp)  | 0x0000  | old CS  |
           12(%esp)  |     old EFLAGS    |
                     + . . . . . . . . . + <-- old ESP 

There is one important caveat to the processor's kernel-mode exception capability. If the processor takes an exception while already in kernel mode, and cannot push its old state onto the kernel stack for any reason such as lack of stack space, then there is nothing the processor can do to recover, so it simply resets itself. Needless to say, any decent kernel should be designed so that this will never happen unintentionally.

Of course, user-mode programs can divide by zero too! The processor handles user-mode exceptions slightly differently to provide protection. It would not be safe to simply push interrupt context information onto a user program stack. (For one thing, a user program that runs out of stack space must not cause the processor to reset!) Thus, when a user-mode program takes an exception, the kernel switches to a special kernel stack. Information about the old stack is pushed onto the exception stack first, above the old EFLAGS:

                     +-------------------+
                     | (optional errcode)| <-- ESP = KSTACKTOP - 24 (maybe)
                     |      old EIP      | <-- ESP = KSTACKTOP - 20 (maybe)
                     | 0x0000  | old CS  |
                     |     old EFLAGS    |
                     |      old ESP      |
                     | 0x0000  | old SS  |
                     + . . . . . . . . . + <-- KSTACKTOP 

You'll handle user-mode exceptions in the next lab; just remember that there are differences in calling convention between kernel-mode and user-mode exceptions.

Setting Up the IDT

You should now have the basic information you need in order to set up the IDT and handle exceptions in JOS. For now, you will set up the IDT to handle all the to handle interrupt numbers 0-31 (the processor exceptions). We'll add additional interrupts later.

The header files inc/trap.h and kern/trap.h contain important definitions related to interrupts and exceptions that you will need to become familiar with. The file kern/trap.h contains trap-related definitions that will remain strictly private to the kernel, while the companion header file inc/trap.h contains general definitions that may also be useful to user-level programs and libraries in the system.

Note: Some of the exceptions in the range 0-31 are defined by Intel to be reserved. Since they will never be generated by the processor, it doesn't really matter how you handle them. Do whatever you think is cleanest.

The overall flow of control that you should achieve is depicted below:

      IDT                   trapentry.S              trap.c
   
+----------------+                        
|   &handler1    |---------> handler1:         +---> void trap(struct Trapframe *tf)
|                |             // do stuff     |     {
|                |             call trap  -----+         // handle the exception/interrupt
|                |                        <----+         return;
|                |             // undo stuff   +---- }
+----------------+
|   &handler2    |--------> handler2:
|                |            // do stuff
|                |            call trap
|                |            // undo stuff
+----------------+
       .
       .
       .
+----------------+
|   &handlerX    |--------> handlerX:
|                |             // do stuff
|                |             call trap
|                |             // undo stuff
+----------------+

Each exception or interrupt should have its own handler in trapentry.S and idt_init() should initialize the IDT with the addresses of these handlers. Each of the handlers should build a struct Trapframe (see inc/trap.h) on the stack and call into trap() (in trap.c) with a pointer to the Trapframe.

After control is passed to trap(), that function handles the exception/interrupt or dispatches the exception/interrupt to a specific handler function. If and when the trap() function returns, the code in trapentry.S restores the old CPU state saved in the Trapframe and then uses the iret instruction to return from the exception.

The Breakpoint Exception

Now that your kernel has basic exception handling capabilities, you'll refine its response to one particular exception. Much more of this will happen in the next lab once we have user processes.

The breakpoint exception, interrupt number 3 (T_BRKPT), is normally used to allow debuggers to insert breakpoints in a program's code by temporarily replacing the relevant program instruction with the special 1-byte int3 software interrupt instruction. In JOS we will abuse this exception slightly by turning it into a primitive pseudo-system call that anyone can use to invoke the JOS kernel monitor, inside or outside the kernel. This usage is actually somewhat appropriate if we think of the JOS kernel monitor as a primitive debugger.

Stack backtrace:
  0: eip f0100047
         kern/init.c:46: _ZL22test_kernel_breakpointv+ac7 (0 arg)
  1: ebp f0117fd8  eip f0100145  args f0104169 00001aac 00000000 00000000
         kern/init.c:215: i386_init+4c (0 arg)
  2: ebp f0117ff8  eip f010003d  args 00000000 00000000 0000ffff 10cf9a00
         kern/entry.S:79: <unknown>+0 (0 arg)

This completes the lab.

Back to CS 261 Research Topics in Operating Systems