Week 7

Sunday, 20-03-01

  • Why does our accurate die?
  • We think it isn't training correctly.

On MicroBotNet:

from functools import reduce

[
    (p.shape, reduce(lambda x, y: x*y, p.shape)) for n, p in model.features[1:11].named_parameters()
    if 'conv' in n and 'weight' in n and 'lastBN' not in n and 'fc' not in n
]
# is
'''
[(torch.Size([24, 8, 1, 1]), 192),
 (torch.Size([24]), 24),
 (torch.Size([24, 1, 3, 3]), 216),
 (torch.Size([24]), 24),
 (torch.Size([8, 24, 1, 1]), 192),
 (torch.Size([8]), 8),
 (torch.Size([32, 8, 1, 1]), 256),
 (torch.Size([32]), 32),
 (torch.Size([32, 1, 5, 5]), 800),
 (torch.Size([32]), 32),
 (torch.Size([16, 32, 1, 1]), 512),
 (torch.Size([16]), 16),
 (torch.Size([80, 16, 1, 1]), 1280),
 (torch.Size([80]), 80),
 (torch.Size([80, 1, 5, 5]), 2000),
 (torch.Size([80]), 80),
 (torch.Size([16, 80, 1, 1]), 1280),
 (torch.Size([40, 16, 1, 1]), 640),
 (torch.Size([40]), 40),
 (torch.Size([40, 1, 5, 5]), 1000),
 (torch.Size([40]), 40),
 (torch.Size([16, 40, 1, 1]), 640),
 (torch.Size([48, 16, 1, 1]), 768),
 (torch.Size([48]), 48),
 (torch.Size([48, 1, 5, 5]), 1200),
 (torch.Size([48]), 48),
 (torch.Size([16, 48, 1, 1]), 768),
 (torch.Size([96, 16, 1, 1]), 1536),
 (torch.Size([96]), 96),
 (torch.Size([96, 1, 5, 5]), 2400),
 (torch.Size([96]), 96),
 (torch.Size([32, 96, 1, 1]), 3072),
 (torch.Size([32]), 32),
 (torch.Size([184, 32, 1, 1]), 5888),
 (torch.Size([184]), 184),
 (torch.Size([184, 1, 5, 5]), 4600),
 (torch.Size([184]), 184),
 (torch.Size([32, 184, 1, 1]), 5888),
 (torch.Size([184, 32, 1, 1]), 5888),
 (torch.Size([184]), 184),
 (torch.Size([184, 1, 5, 5]), 4600),
 (torch.Size([184]), 184),
 (torch.Size([32, 184, 1, 1]), 5888),
 (torch.Size([184, 32, 1, 1]), 5888),
 (torch.Size([184]), 184),
 (torch.Size([184, 1, 5, 5]), 4600),
 (torch.Size([184]), 184),
 (torch.Size([32, 184, 1, 1]), 5888),
 (torch.Size([184, 32, 1, 1]), 5888),
 (torch.Size([184]), 184),
 (torch.Size([184, 1, 5, 5]), 4600),
 (torch.Size([184]), 184),
 (torch.Size([32, 184, 1, 1]), 5888)]
'''

This does not include fully connected layers embedded within our layers

SqueezeNet for comparison

from functools import reduce

[
    (p.shape, reduce(lambda x, y: x*y, p.shape)) for n, p in all_conv_weights
    if not ('classifier' in n or 'features.0.' in n)
]
# is
'''
[(torch.Size([16, 64, 1, 1]), 1024),
 (torch.Size([64, 16, 1, 1]), 1024),
 (torch.Size([64, 16, 3, 3]), 9216),
 (torch.Size([16, 128, 1, 1]), 2048),
 (torch.Size([64, 16, 1, 1]), 1024),
 (torch.Size([64, 16, 3, 3]), 9216),
 (torch.Size([32, 128, 1, 1]), 4096),
 (torch.Size([128, 32, 1, 1]), 4096),
 (torch.Size([128, 32, 3, 3]), 36864),
 (torch.Size([32, 256, 1, 1]), 8192),
 (torch.Size([128, 32, 1, 1]), 4096),
 (torch.Size([128, 32, 3, 3]), 36864),
 (torch.Size([48, 256, 1, 1]), 12288),
 (torch.Size([192, 48, 1, 1]), 9216),
 (torch.Size([192, 48, 3, 3]), 82944),
 (torch.Size([48, 384, 1, 1]), 18432),
 (torch.Size([192, 48, 1, 1]), 9216),
 (torch.Size([192, 48, 3, 3]), 82944),
 (torch.Size([64, 384, 1, 1]), 24576),
 (torch.Size([256, 64, 1, 1]), 16384),
 (torch.Size([256, 64, 3, 3]), 147456),
 (torch.Size([64, 512, 1, 1]), 32768),
 (torch.Size([256, 64, 1, 1]), 16384),
 (torch.Size([256, 64, 3, 3]), 147456)]

Now do MicroBotNet, only convs bigger than 1000:

from functools import reduce

[
    (p.shape, reduce(lambda x, y: x*y, p.shape)) for n, p in model.features[1:11].named_parameters()
    if 'conv' in n and 'weight' in n
    and 'lastBN' not in n and 'fc' not in n
    and reduce(lambda x, y: x*y, p.shape) > 1000
]
# is
'''
[(torch.Size([80, 16, 1, 1]), 1280),
 (torch.Size([80, 1, 5, 5]), 2000),
 (torch.Size([16, 80, 1, 1]), 1280),
 (torch.Size([48, 1, 5, 5]), 1200),
 (torch.Size([96, 16, 1, 1]), 1536),
 (torch.Size([96, 1, 5, 5]), 2400),
 (torch.Size([32, 96, 1, 1]), 3072),
 (torch.Size([184, 32, 1, 1]), 5888),
 (torch.Size([184, 1, 5, 5]), 4600),
 (torch.Size([32, 184, 1, 1]), 5888),
 (torch.Size([184, 32, 1, 1]), 5888),
 (torch.Size([184, 1, 5, 5]), 4600),
 (torch.Size([32, 184, 1, 1]), 5888),
 (torch.Size([184, 32, 1, 1]), 5888),
 (torch.Size([184, 1, 5, 5]), 4600),
 (torch.Size([32, 184, 1, 1]), 5888),
 (torch.Size([184, 32, 1, 1]), 5888),
 (torch.Size([184, 1, 5, 5]), 4600),
 (torch.Size([32, 184, 1, 1]), 5888)] 
'''

Got to 73.5% Accuracy quantizing only conv above 1000 Got to 73.2% Accuracy quantizing conv and bias above 1000

Monday, 20-03-02

  • 65% quantized to zero, one, negative one, 72.3% accuracy
  • TTQ is 72.8% accuracy.
  • 31 layers quantized. 86 layers not quantized
    • 153,792 param quantized
      • 12,222 quantized to one.
      • 134,124 quantized to zero.
      • 7,446 quantized to neg one.
    • 82,866 param not quantized

Thursday, 20-03-05

  • Make a visualization of the network for Pister

Google Collab with the quantization