COGS 107B SYSTEMS NEUROBIOLOGY
TA: Flavia Filimon, Mon
NEURAL
NETWORKS
Three types of networks reviewed here:
1) Feedforward supervised learning (with backprop)
2) Attractor/recurrent networks (non-learning)
3) Hebbian feed-forward unsupervised learning
NOTE: “supervised”
simply means you know the correct answer or output.
E.g. you know what pattern of
activation on an output layer corresponds to, say, “dog”.
1). Feedforward
models
Calculation of output:
à 
All this means is that to calculate the output to unit O, you
take each input unit O receives, multiply it by its respective weight, and add
them all up together.
Know how to calculate output of units in a feedforward network!
Calculation of error:
à
Error =
desired – actual (activations on output layer)
Knowing the error allows you to train the network, i.e. to make the network learn to produce a correct output.
For three or more layers:
Backpropagation rule:
à 
Q: Give an example of a kind of problem feedforward models are good at solving.
(linear categorization/classification: e.g., dog vs. cat.)
Calculate Oj:
-1 1
2
-3
ANSWER: Oj = 2*(-1) + (-3)*1 = -2 -3 = -5
Q: What kinds of problems cannot be solved
by (linear) feedforward models?
(- nonlinear
classification problems, e.g. XOR problem)
Why? (because
this involves a non-linear bending or warping of the unit activation space –
see Marty’s diagram in class)
The XOR problem involves giving a certain output (0 OR 1) for certain combinations of two input numbers (1 or 0).
XOR: Input: Output:
1 1 - 0
0 1 - 1
1 0 - 1
0 0 - 0
Q: what do you need to be able to solve the
XOR problem?
1)
a third layer
2)
nonlinear units in one layer
2) Recurrent/attractor models:
- no learning because weights are fixed
- every unit is connected to every other unit
- the goal here is to reduce the energy of the network, which corresponds to finding a stable state, through updating.
Q. What is an attractor? àstable state
Q. Define stable state à when the activation value of all units in a network stop changing when you update them. Example: a typical “dog” corresponds to a stable state.
Q. Give a few examples of problems attractor networks can be useful for.
(e.g. classification – dog vs. cat
– but in a novel way of solving the problem) pattern
reconstruction/completion, e.g. if you see a dog’s tail, you can still recognize as dog in most cases
Update rule for attractor networks:
à
Q. Update unit 1 on the following network:
2
0
3
Oj
-1 1 4
i=1 i=2 i=3
Updating: (-1)*2 + 1*0 + 4*3 = 10 10 > 0 à therefore, unit 1 goes to +1, i.e. it changes
Calculating an attractor
network’s energy:
à Energy = 
Q. What is the energy of the following network?
4
-2
3
2
1
-1
ANSWER:
E = -[1*4*1 +
1*(-2)*(-1) + (-1)*2*(-1) + (-1)*3*1] = -[4 + 2 + 2 -3] = -5
Notice that the “output” is the activation value of the unit the
current unit projects to.
In other
words – take the unit which gives some “input” to another unit (or itself), multiply it by its connection strength to the output unit,
and by the activation value of the output unit to which it connects.
- Updating an attractor/Hopfield network will lead to the energy of the network going down ( = which results in finding a stable state, or an attractor), if:
· weights are symmetric or both positive or both negative
· self-weights (from one unit to itself) are zero
3). Hebbian unsupervised feed-forward learning
Q. Give examples of when in real life Hebbian learning might occur
-. face recognition, orientation
selectivity
Q. Determine whether the following pairs of numbers (i.e. the two columns) are positively, negatively, or un-correlated; What is the correlation?:
-3 -5 4 -2
2 4 -1 3
1 0 0 1
-2 -3 1 -6
5 3 0 7
Q1 = 1/5
[ (-3)*(-5) + 2*4 + 1*0 + (-2)*(-3) + 5*3] = 44/5 =
8.8 à positive correlation
Q2 = 1/5
[4*(-2) + (-1)*3 + 0*1 + 1*(-6) + 0*7] = -17/5 = -3.4 à negative correlation
à Hebbian learning rule:
this translates into a 2-point correlation between two inputs – if the weights of the two inputs are correlated, they get turned up.