Synchronization

• Processors within a parallel system must have some method of synchronization.

 

• Software routines are usually built on top of hardware-supplied synchronization instructions.

 

• For shared-memory machines, the key element is an uninterruptible instruction that can atomically retrieve and change a value.

 

• Atomic exchange (exchanges a register and memory location atomically) or test-and-set are common.

 

• However, implementation is challenging since it requires both a memory read and write to execute atomically.

 

• This complicates coherence and does not scale well.

Synchronization

• Load-linked and store-conditional
• Alternatively, we can use a pair of instructions.

• If the memory location specified by the load linked instruction is changed before the store conditional instruction (to the same address), the store conditional fails.

 

• If it fails, the sequence is executed again.

 

• ll is implemented by storing the address given in the instruction in a link register.
• If an interrupt occurs or if the cache block containing the address is invalidated, the ll register is set to 0 and sc fails.

Implementing Locks using Coherence

• In machines that support cache coherence, we want to cache the locks.
• The spin-lock operation is performed on a local cached copy, reducing memory bandwidth.
• Since there is often locality in lock access, caching reduces time to acquire the lock.

 

• Let's assume that we have an exchange instruction.

 

• To implement a spin-lock , we could use the following where 0 indicates success:

 

• Problem:
• Each exch operation requires a write, most of which result in a write miss (since writing requires exclusive access).

Implementing Locks using Coherence

• In the loop, let's read instead and write only when the lock becomes free.

• For example:

#	P0	P1	P2	Coherence state	Bus/directory activity
1	Has lock	Spins, testing if lock=0	Spins, testing if lock=0	Shared	None
2	Set lock to 0	Invalidate received	Invalidate received	Exclusive	Write invalidate of lock var from P0
3		Cache miss	Cache miss	Shared	Bus/directory services P2 cache miss; writeback from P0.

Implementing Locks using Coherence

#	P0	P1	P2	Coherence state	Bus/directory activity
4		Waits while bus/directory busy	Lock = 0	Shared	Cache miss for P2 satisfied.
5		Lock=0	Executes swap, gets cache miss	Shared	Cache miss for P1 satisfied.
6		Executes swap, gets cache miss	Completes swap: returns 0 and sets Lock=1.	Exclusive	Bus/directory services P2 cache miss; generates invalidate.
7		Swap completes and returns 1	Enter critical section.	Shared	Bus/directory services P2 cache miss; generates writeback.
8		Spins, testing if lock=0.			None.

Implementing Locks using Coherence

• A load linked/store conditional version need not cause any bus traffic during the testing operation:

 

• However, when the lock is released, a lot of traffic is generated.

 

• This makes it difficult to scale this implementation to many processors.

Barrier synchronization

• Another common synchronization operation in programs with parallel loops.

 

• This allows multiple processes on multiple CPUs to wait until a certain number of processes have reached a "barrier."
• When sufficient processes arrive, all waiting processes may continue.

Barrier synchronization

• The above method can be fail if one process `races' ahead and resets release before the last process has been scheduled again and exits spin(release == 1) .

 

• Sense-reversing barrier
• A fix which uses private per-process variables.
• local_sense is initialized to 1 for each process.

• Other techniques are discussed in the text.

Memory Consistency Models

• Deciding when a change to memory should be propagated to other CPUs is a difficult problem.

 

• Message-passing machines require explicit propagation, so the decision is left entirely to software.

 

• Shared memory in hardware has to support hardware mechanisms for making this decision.

 

• The future of Multi-Processors (MPs)
• We have only scratched the surface talking about multiprocessors in this class.

 

• There is more than enough material to spend an entire semester discussing MP hardware design.

 

• Predicting the future of MPs is even harder than understanding how today's MPs work.

The Future of Multiprocessors

• Large scale machines that scale up naturally.
• These would be built from commodity elements that can be added in small numbers to build a a big system.

 

• The interconnect is typically proprietary, but the processor is commodity.

 

• The SGI Origin 200 series and Cray T3E work this way.

 

• Large-scale machines built of clusters of mid-range machines.
• These require faster uniprocessors but can be built from fewer machines.
• Examples include the SGI Challenge.

 

• Off-the-shelf nodes with custom interconnect.
• This uses standard processor boards (like PowerPC boards in RS/6000 workstations) with a high-speed custom-built interconnect.

 

• All off-the-shelf components.
• Workstation clusters are a good example of this.

The Future of Computer Architecture

• What to do with all those transistors?

 

• Many have predicted that chips will soon have a billion transistors on them.

 

• Even if it's only 100 million transistors , what can we build with all that space?

 

• Will it just be ever-larger L1 caches, or will there be different designs?
• Multiple threads on a chip?
• Larger register files?
• Vector units?
• Display integrated onto the CPU?

The Future of Computer Architecture

• Ever-faster CPUs
• The Dept. of Energy wants 10 TFLOP machines within about ten years.

 

• This is a factor of 1000 faster than current small-scale 10 GFLOP machines.

 

• Programming these machines is hard enough.
• How can we scale up systems to 10 TFLOPs?

 

• Ubiquitous computing
• It's been said that a technology has truly arrived when we are not really conscious of using it.

 

• Examples include the telephone, television, and automobile.

 

• When will computing be so cheap that it is fully integrated into our lives such that we cannot imagine living without it?

The Future of Computer Architecture

• Large-scale storage
• All of this computing power must be accompanied by ways to store all the data.

 

• As computing becomes more decentralized, storage may become more centralized .

 

• This may happen because of the difficulties of storing data for the long term.
• Computing cycles are fleeting, but data is forever.

 

• Losing a few cycles is not a big problem, but losing data can cause lots of trouble.