Reducing hit time

Reducing hit time

On many machines, cache access time limits the clock cycle rate !

Therefore, cache design affects more than average memory access time, it affects everything.

Small & simple caches

The less hardware that is necessary to implement a cache, the shorter the critical path through the hardware.

Direct-mapped is faster than set associative for both reads and writes.

In particular, tag checking can overlap data transmission (there is only one piece of data for each index).

Fitting the cache on the chip with the CPU is also very important for fast access times.

Therefore, fast clock cycle time encourages small direct-mapped caches.

Reducing hit time

Avoid address translation during indexing

The CPU uses virtual addresses that must be mapped to a physical address.

The cache may either use virtual or physical addresses.

A cache that indexes by virtual addresses is called a virtual cache , as opposed to a physical cache .

A virtual cache reduces hit time since a translation from a virtual address to a physical address is not necessary on hits.

Also, address translation can be done in parallel with cache access, so penalties for misses are reduced as well.

So why doesn't anyone use them ?

Reducing hit time

Avoid address translation during indexing

Virtual cache difficulties include:

Process switches require cache purging

In virtual caches, different processes share the same virtual addresses even though they map to different physical addresses.

When a process is swapped out, the cache must be purged of all entries to make sure that the new process gets the correct data.

One solution: PID tags

Increase the width of the cache address tags to include a process ID (instead of purging the cache.)

The current process PID is specified by a register.

If the PID does not match, it is not a hit even if the address matches.

Reducing hit time

Virtual cache difficulties include:
Aliasing

Two different virtual addresses may have the same physical address.

In such a case, it is possible to end up with two copies of the same block !

Anti-aliasing hardware
A hardware solution called anti-aliasing guarantees every cache block a unique physical address.

Every virtual address maps to the same location in the cache.

Page coloring
This software technique forces aliases to share some address bits.

Therefore, the virtual address and physical address match over these bits.

A direct-mapped cache that is 2 ^k bytes (where k is the number of matching bits) or smaller can never have duplicate physical addresses for blocks.

Reducing hit time

Virtual cache difficulties include:
Aliasing

Using the page offset
An alternative to get the best of both virtual and physical caches.

If we use the page offset to index the cache, then we can overlap the virtual address translation process with the time required to read the tags.

Note that the page offset is unaffected by address translation.

However, this restriction forces the cache size to be smaller than the page size.

Since the index comes only from the "physical" portion of the virtual address (the page offset).

After doing both in parallel, the tag is checked against the physical address stored in the cache.
High associativity allows for larger cache sizes.

Reducing hit time

Pipelined writes

Write hits take longer than read hits because tag checking is required before the data is written.

One solution is to pipeline the writes (Alpha AXP 21064):

The second stage of the write (cache is updated with new data) occurs during the first stage of the next write.
Allows tag checking and data writing to occur simultaneously.

Cache Optimization Summary

Technique	Miss Rate	Miss Pen.	Hit time	Hardware Complexity
Larger Block Size	+	-		0
Higher Associativity	+		-	1
Victim Caches	+			2
Pseudo-associative	+			2
Hardware Prefetching	+			2
Compiler-controlled Pre	+			3
Compiler Techniques	+			0
Giving Read Misses Priority		+		1
Subblock Placement		+		1
Early Restart/Crit Wd First		+		2
Nonblocking Caches		+		3
Second-Level Caches		+		2
Small and Simple Caches	-		+	0
Avoiding Address Trans.			+	2
Pipelining Writes			+	1

Main Memory

Main memory is usually made from DRAM while caches use SRAM.

SRAM is faster (by almost an order of magnitude).

However, it's also more expensive per bit and 1/4 to 1/8 as dense as DRAM (1 transistor versus 6 transistors).

We now turn to optimizing DRAM performance.

Performance measures include:
Latency

Important for caches.

Bandwidth

Important for I/O.
Also for cache with second-level and larger block sizes.

Improving Main Memory Performance

Latency measures:
Access time

Time between when a read is requested and when the desired word arrives.

Cycle time

This is the minimum time between the starts of two accesses to memory.
This is at least as long as access time, and is usually longer.

Refresh time

DRAMs must occasionally refresh their data.

This is done by reading all of the cells in a row and writing them back.

This must be done every few milliseconds.

However, this operation consumes less than 5% of total time.

This is true because the time necessary to refresh is proportional to the square root of the size of the DRAM.

Improving Main Memory Performance

Amdahl suggested that memory capacity should grow linearly with CPU speed.

Memory capacity grows four-fold every three years to supply this demand.

The CPU-DRAM performance gap is a problem, however, since DRAM performance improvement is only about 7% per year.

Cache innovations have addressed this problem to some degree.

We will now look at innovations in main memory organizations that are more cost effective.

Improving Main Memory Performance

Wider main memory

Improving Main Memory Performance

Wider main memory

DRAM chips are typically 1-8 bits wide.

Any number of them can be accessed in parallel without extra delay.

By increasing the width of memory, the CPU can get more bits in a single cycle.

This increases bandwidth between cache and memory.

For example, consider a cache with 4 word blocks.

Main memory might require:

4 cycles to send the address.
40 cycles to access memory.
4 cycles to transfer over the bus.

If the memory is only one word wide, a miss would require 4 x (4 + 40 + 4) = 192 cycles!
If the memory is enlarged to 4 words wide , miss time is only 48 cycles.

Improving Main Memory Performance

Interleaved memory

Banks are often one word wide, so bus width need not be changed.

However, several independent areas of memory can be accessed simultaneously.

Improving Main Memory Performance

Interleaved memory

For example, we could fetch a block by

Sending 1 address.
Waiting for a single memory cycle.
Transferring 4 words for a total time of 4 + 40 + (4 x 4) = 60 cycles.

Which is a little slower than wider memory (due to bus limitations) but it has several advantages.

Individual writes can also be overlapped if they are addressed to different banks.

One possible interleaving strategy: Word interleaving :

Improving Main Memory Performance

Interleaved memory

Read access optimization is possible if, for example, cache block size is four words since parallel access is possible (no conflicts).

Also, write-back caches make writes sequential as well as reads, improving efficiency even further.

How many banks are sufficient ?

One rule might be `# of banks >= # of clocks to access a word in a bank'.
This allows up to 1 word per clock cycle in best case.

Improving Main Memory Performance

Independent memory banks

The interleaved memory concept can be extended to remove all restrictions on memory access.

We assumed for interleaved memory that only a single memory controller was present.
This allowed the interleaving of sequential access patterns.
Address line sharing among the banks is possible in this scheme.

We can also use multiple independent controllers , e.g. one for I/O devices, one for cache reads and one for cache writes.

Banks are still accessed in parallel, but now there may be multiple independent requests serviced simultaneously.

This can be particularly useful with nonblocking caches (caches that allow multiple outstanding reads misses).

And multiprocessors.

Improving Main Memory Performance

Avoiding memory bank conflicts

As with caches, programs can be modified to improve memory performance.

The most important is to keep all the banks running.

Programs that access all banks evenly will perform best.

However, data memory references are not random and may end up going to the same bank.

Using a prime number of memory banks makes this work well.

However, using a prime number makes the division operation expensive:

Improving Main Memory Performance

Avoiding memory bank conflicts

There are schemes that use a prime number of banks and fast modulo arithmetic to distribute memory accesses to many banks of memory.

For example, the following can be used:

This avoids the use of an expensive `non power of 2' division operation shown previously.

There is a proof that guarantees that the above mapping provides a unique mapping between an address and a memory location.

For numbers of the form 2 ^N -1, there is fast hardware to implement the operation.

Improving Main Memory Performance

The previous methods work with any memory technology.

We now look at techniques that take advantage of the nature of DRAMs.

The first three take advantage of the individual row access and column access operations that occur on a memory access.

DRAMs buffer a row of bits inside the DRAM for column access.

The size of the buffer is usually the square root of the DRAM size, e.g. 16Kbits for 64MBits.

In order to improve performance, DRAMs are designed to allow multiple accesses to this buffer, eliminating the row access time.

Improving Main Memory Performance

DRAM-specific interleaving
Nibble mode

The DRAM can supply three extra bits from locations sequential to the one just accessed.
This can be done after each RAS (Row Access Strobe).

Page mode

The DRAM can act as an SRAM once a row has been selected.
For example, random bits from the row can be selected by changing just the column address.
This can occur until the next RAS or refresh.

Static column mode (Extended Data Out [ EDO ] RAM)

Very similar to page mode, except that it is not necessary to toggle (clock) the column access strobe line every time the column address changes.

These optimizations can improve bandwidth by a factor of four .

Improving Main Memory Performance

Synchronous DRAM (SDRAM)

In this type of DRAM, the clock is supplied to the RAM chip, and all signals are synchronized to it.

This allows the RAM to run at higher speeds.

Similarly, sequential data can be retrieved faster, at the rate of one bit per clock cycle.

VRAM

Video RAM is used to drive displays.

It can be written or read using a normal interface or a special interface that outputs rows one bit at a time (good for video displays!).