MNIST::MLP< Loss, ActivationFunction, Constraint > Class Template Reference

The standard densely connected multilayer Perceptron. Template arguments provide the loss function, the activation function of neurons (experimental) and a possible constraint for the weights. More...

#include <mnist_mlp.hpp>

Inheritance diagram for MNIST::MLP< Loss, ActivationFunction, Constraint >:

Public Member Functions
	MLP (std::vector< size_t > layer_sizes, size_t epochs=20, size_t batchsize=100, Real learn_rate=0.01)
	Constructor for random init. More...
	MLP (Json &data, size_t epochs=20, size_t batchsize=100, Real learn_rate=0.01, bool random=false, Constraint constraint=Constraint())
	Constructs the network from json file. The repo provides python scripts to create those from a keras network. More...
Real	max_weight () const override
	Return the largest weight in the network. More...
Real	conv_max_weight (size_t layer_id) const override
Real	min_weight () const override
	Return the smallest weight in the network. More...
Real	max_weight_abs () const override
	Return the largest absolute weight in the network. More...
const size_t &	epochs () const override
const size_t &	batchsize () const override
const Real &	learnrate () const override
const mnist_helper::MNIST_DATA &	mnist_train_set () override
	Returns reference to the train data. More...
const mnist_helper::MNIST_DATA &	mnist_test_set () override
	Returns reference to the test data. More...
const std::vector< cypress::Matrix< Real > > &	get_weights () override
	Return all weights in the form of weights[layer](src,tar) More...
const std::vector< mnist_helper::CONVOLUTION_LAYER > &	get_conv_layers () override
	Return all filter weights in the form of weights[x][y][depth][filter]. More...
const std::vector< mnist_helper::POOLING_LAYER > &	get_pooling_layers () override
const std::vector< size_t > &	get_layer_sizes () override
	Return the number of neurons per layer. More...
const std::vector< mnist_helper::LAYER_TYPE > &	get_layer_types () override
void	scale_down_images (size_t pooling_size=3) override
	Scale down the whole data set, reduces the image by a given factor in every dimension. More...
bool	correct (const uint16_t label, const std::vector< Real > &output) const override
	Checks if the output of the network was correct. More...
virtual std::vector< std::vector< std::vector< Real > > >	forward_path (const std::vector< size_t > &indices, const size_t start) const override
	Forward path of the network (–> inference) More...
virtual Real	forward_path_test () const override
	Forward path of test data. More...
virtual void	backward_path (const std::vector< size_t > &indices, const size_t start, const std::vector< std::vector< std::vector< Real >>> &activations, bool last_only=false) override
	implementation of backprop More...
virtual void	backward_path_2 (const std::vector< uint16_t > &labels, const std::vector< std::vector< std::vector< Real >>> &activations, bool last_only=false) override
	Implementation of backprop, adapted for usage in SNNs. More...
size_t	accuracy (const std::vector< std::vector< std::vector< Real >>> &activations, const std::vector< size_t > &indices, const size_t start) override
	Calculate the overall accuracy from the given neural network output. More...
void	train (unsigned seed=0) override
	Starts the full training process. More...

Static Public Member Functions
static std::vector< Real >	mat_X_vec (const Matrix< Real > &mat, const std::vector< Real > &vec)
	Implements matrix vector multiplication. More...
static std::vector< Real >	mat_trans_X_vec (const Matrix< Real > &mat, const std::vector< Real > &vec)
	Implements transposed matrix vector multiplication. More...
static std::vector< Real >	vec_X_vec_comp (const std::vector< Real > &vec1, const std::vector< Real > &vec2)
	Vector vector multiplication, component-wise. More...
static void	update_mat (Matrix< Real > &mat, const std::vector< Real > &errors, const std::vector< Real > &pre_output, const size_t sample_num, const Real learn_rate)
	Updates the weight matrix based on the error in this layer and the output of the previous layer. More...

Protected Member Functions
void	load_data (std::string path)

Protected Attributes
std::vector< cypress::Matrix< Real > >	m_layers
std::vector< size_t >	m_layer_sizes
std::vector< mnist_helper::CONVOLUTION_LAYER >	m_filters
std::vector< mnist_helper::POOLING_LAYER >	m_pools
std::vector< mnist_helper::LAYER_TYPE >	m_layer_types
size_t	m_epochs = 20
size_t	m_batchsize = 100
Real	learn_rate = 0.01
mnist_helper::MNIST_DATA	m_mnist
mnist_helper::MNIST_DATA	m_mnist_test
Constraint	m_constraint

Detailed Description

template<typename Loss = MSE, typename ActivationFunction = ReLU, typename Constraint = NoConstraint>
class MNIST::MLP< Loss, ActivationFunction, Constraint >

The standard densely connected multilayer Perceptron. Template arguments provide the loss function, the activation function of neurons (experimental) and a possible constraint for the weights.

Definition at line 241 of file mnist_mlp.hpp.

Constructor & Destructor Documentation

template<typename Loss = MSE, typename ActivationFunction = ReLU, typename Constraint = NoConstraint>

MNIST::MLP< Loss, ActivationFunction, Constraint >::MLP ( std::vector< size_t >  layer_sizes, size_t  epochs = 20, size_t  batchsize = 100, Real  learn_rate = 0.01  )

inline

Constructor for random init.

Parameters

layer_sizes	list of #neurons beginning with input and ending with output layer
epochs	number of epochs to train
batchsize	mini batchsize before updating the weights
learn_rate	gradients are multiplied with this rate
constrain	constrains the weights during training, defaults to no constraint

Definition at line 274 of file mnist_mlp.hpp.

        : m_layer_sizes(layer_sizes),
          m_epochs(epochs),
          m_batchsize(batchsize),
          learn_rate(learn_rate)
    {
        for (size_t i = 0; i < layer_sizes.size() - 1; i++) {
            m_layers.emplace_back(
                Matrix<Real>(layer_sizes[i], layer_sizes[i + 1]));
        }

        int seed = std::chrono::system_clock::now().time_since_epoch().count();
        auto rng = std::default_random_engine(seed);
        std::normal_distribution<Real> distribution(0.0, 1.0);
        for (auto &layer : m_layers) {
            // Kaiming init, best suited for ReLU activation functions
            auto scale = std::sqrt(2.0 / double(layer.rows()));
            for (size_t i = 0; i < layer.size(); i++) {
                layer[i] = distribution(rng) * scale;
            }
        }

        // Glorot uniform
        /*for (auto &layer : m_layers) {
            auto limit = std::sqrt(6.0 / Real(layer.rows()+ layer.cols()));
            std::uniform_real_distribution<Real> distribution(0, limit);
            for (size_t i = 0; i < layer.size(); i++) {
                layer[i] = distribution(rng);
            }
        }*/
        try {
            load_data("");
        }
        catch (...) {
            load_data("../");
        }
        m_constraint.setup(m_layers);
    }

template<typename Loss = MSE, typename ActivationFunction = ReLU, typename Constraint = NoConstraint>

MNIST::MLP< Loss, ActivationFunction, Constraint >::MLP ( Json &  data, size_t  epochs = 20, size_t  batchsize = 100, Real  learn_rate = 0.01, bool  random = false, Constraint  constraint = Constraint()  )

inline

Constructs the network from json file. The repo provides python scripts to create those from a keras network.

Parameters

data	json object containing the network information
epochs	number of epochs to train
batchsize	mini batchsize before updating the weights
learn_rate	gradients are multiplied with this rate
random	Use structure from Json, initialize weights random if true
constrain	constrains the weights during training, defaults to no constraint

Definition at line 326 of file mnist_mlp.hpp.

        : m_epochs(epochs),
          m_batchsize(batchsize),
          learn_rate(learn_rate),
          m_constraint(constraint)
    {
        int seed = std::chrono::system_clock::now().time_since_epoch().count();
        auto rng = std::default_random_engine(seed);
        std::normal_distribution<Real> distribution(0.0, 1.0);
        for (auto &layer : data["netw"]) {
            if (layer["class_name"].get<std::string>() == "Dense") {
                auto &json = layer["weights"];
                m_layers.emplace_back(
                    Matrix<Real>(json.size(), json[0].size()));
                auto &weights = m_layers.back();
                auto scale = std::sqrt(2.0 / double(weights.rows()));
                for (size_t i = 0; i < json.size(); i++) {
                    for (size_t j = 0; j < json[i].size(); j++) {
                        if (!random) {
                            weights(i, j) = json[i][j].get<Real>();
                        }
                        else {
                            weights(i, j) = distribution(rng) * scale;
                        }
                    }
                }
                m_layer_sizes.emplace_back(m_layers.back().rows());
                m_layer_types.push_back(mnist_helper::LAYER_TYPE::Dense);
                cypress::global_logger().debug(
                    "MNIST", "Dense layer detected with size " +
                                 std::to_string(weights.rows()) + " times " +
                                 std::to_string(weights.cols()));
            }
            else if (layer["class_name"].get<std::string>() == "Conv2D") {
                auto &json = layer["weights"];
                size_t kernel_x = json.size();
                size_t kernel_y = json[0].size();
                size_t kernel_z = json[0][0].size();
                size_t output = json[0][0][0].size();
                size_t stride = layer["stride"];
                size_t padding = layer["padding"] == "valid" ? 0 : 1;
                std::vector<size_t> input_sizes;
                std::vector<size_t> output_sizes;
                if (!layer["input_shape_x"].empty()){
                    input_sizes.push_back(layer["input_shape_x"]);
                    input_sizes.push_back(layer["input_shape_y"]);
                    input_sizes.push_back(layer["input_shape_z"]);
                } else {
                    if (m_layer_types.back() == mnist_helper::LAYER_TYPE::Conv) {
                        input_sizes.push_back(m_filters.back().output_sizes[0]);
                        input_sizes.push_back(m_filters.back().output_sizes[1]);
                        input_sizes.push_back(m_filters.back().output_sizes[2]);
                    } else if (m_layer_types.back() == mnist_helper::LAYER_TYPE::Pooling) {
                        input_sizes.push_back(m_pools.back().output_sizes[0]);
                        input_sizes.push_back(m_pools.back().output_sizes[1]);
                        input_sizes.push_back(m_pools.back().output_sizes[2]);
                    } else if (m_layer_types.back() == mnist_helper::LAYER_TYPE::Dense) {
                        throw std::runtime_error("Conv after Dense layer not implemented!");
                    }
                }
                output_sizes.push_back((input_sizes[0] - kernel_x + 2*padding)/stride+1);
                output_sizes.push_back((input_sizes[1] - kernel_x + 2*padding)/stride+1);
                output_sizes.push_back(output);
                mnist_helper::CONVOLUTION_FILTER conv_filter(
                    kernel_x,
                    std::vector<std::vector<std::vector<Real>>>(kernel_y,
                    std::vector<std::vector<Real>>(kernel_z,
                    std::vector<Real>(output)))
                    );
                mnist_helper::CONVOLUTION_LAYER conv = {
                    conv_filter,
                    input_sizes,
                    output_sizes,
                    stride,
                    padding};
                m_filters.emplace_back(conv);
                auto &weights = m_filters.back().filter;
                //auto scale = std::sqrt(2.0 / double(weights.rows()));
                for (size_t i = 0; i < json.size(); i++){
                    for (size_t j = 0; j < json[i].size(); j++){
                        for (size_t k = 0; k < json[i][j].size(); k++){
                            for (size_t l = 0 ; l < json[i][j][k].size(); l++){
                                weights[i][j][k][l] = json[i][j][k][l].get<Real>();
                            }
                        }
                    }
                }
                m_layer_sizes.emplace_back(input_sizes[0] * input_sizes[1] * input_sizes[2]);
                m_layer_types.push_back(mnist_helper::LAYER_TYPE::Conv);
                cypress::global_logger().debug(
                    "MNIST", "Conv layer detected with size ("+
                        std::to_string(json.size())+","+std::to_string(json[0].size())+
                        ","+std::to_string(json[0][0].size())+","+std::to_string(json[0][0][0].size())+")");
            } else if(layer["class_name"].get<std::string>() == "MaxPooling2D"){
                std::vector<size_t> size = layer["size"];
                size_t stride = layer["stride"];
                std::vector<size_t> input_sizes;
                std::vector<size_t> output_sizes;
                if (m_layer_types.empty()){
                    throw std::runtime_error("Pooling layer must not be the first layer!");
                }
                if (m_layer_types.back() == mnist_helper::LAYER_TYPE::Conv){
                    input_sizes.push_back(m_filters.back().output_sizes[0]);
                    input_sizes.push_back(m_filters.back().output_sizes[1]);
                    input_sizes.push_back(m_filters.back().output_sizes[2]);
                } else if (m_layer_types.back() == mnist_helper::LAYER_TYPE::Pooling){
                    input_sizes.push_back(m_pools.back().output_sizes[0]);
                    input_sizes.push_back(m_pools.back().output_sizes[1]);
                    input_sizes.push_back(m_pools.back().output_sizes[2]);
                } else if (m_layer_types.back() == mnist_helper::LAYER_TYPE::Dense){
                    throw std::runtime_error("Pooling after Dense not implemented!");
                }
                output_sizes.push_back((input_sizes[0] - size[0] + 2*0)/stride+1);
                output_sizes.push_back((input_sizes[1] - size[1] + 2*0)/stride+1);
                output_sizes.push_back(input_sizes[2]);
                mnist_helper::POOLING_LAYER pool = {input_sizes, output_sizes, size, stride};
                m_pools.emplace_back(pool);
                m_layer_sizes.emplace_back(input_sizes[0] * input_sizes[1] * input_sizes[2]);
                m_layer_types.emplace_back(mnist_helper::LAYER_TYPE::Pooling);
                cypress::global_logger().debug(
                    "MNIST", "Pooling layer detected with size (" +
                                 std::to_string(size[0]) + ", " + std::to_string(size[1]) +
                                ") and stride " + std::to_string(stride));
            }
            else {
                throw std::runtime_error("Unknown layer type");
            }
        }
        m_layer_sizes.push_back(m_layers.back().cols());
//      for (auto &layer : m_layers) {
//          m_layer_sizes.emplace_back(layer.cols());
//      }

        m_mnist = mnist_helper::loadMnistData(60000, "train");
        m_mnist_test = mnist_helper::loadMnistData(10000, "t10k");
        m_constraint.setup(m_layers);
    }

Member Function Documentation

template<typename Loss = MSE, typename ActivationFunction = ReLU, typename Constraint = NoConstraint>

size_t MNIST::MLP< Loss, ActivationFunction, Constraint >::accuracy ( const std::vector< std::vector< std::vector< Real >>> &  activations, const std::vector< size_t > &  indices, const size_t  start  )

inlineoverridevirtual

Calculate the overall accuracy from the given neural network output.

Parameters

activations	output of forward path
start	the start index, uses images indices[start] until indices[start +batchsize -1]
activations	result of forward path

Returns

the number of correctly classified images

Implements MNIST::MLPBase.

Definition at line 900 of file mnist_mlp.hpp.

    {
#ifndef NDEBUG
        assert(activations.size() == m_batchsize);
#endif

        auto &labels = std::get<1>(m_mnist);
        size_t sum = 0;

        for (size_t sample = 0; sample < m_batchsize; sample++) {
            if (start + sample >= indices.size())
                break;
            if (correct(labels[indices[start + sample]],
                        activations[sample].back()))
                sum++;
        }
        return sum;
    }

template<typename Loss = MSE, typename ActivationFunction = ReLU, typename Constraint = NoConstraint>

virtual void MNIST::MLP< Loss, ActivationFunction, Constraint >::backward_path ( const std::vector< size_t > &  indices, const size_t  start, const std::vector< std::vector< std::vector< Real >>> &  activations, bool  last_only = false  )

inlineoverridevirtual

implementation of backprop

Parameters

indices	list of shuffled (?) indices
start	the start index, uses images indices[start] until indices[start +batchsize -1]
activations	result of forward path
last_only	true for last layer only training (Perceptron learn rule)

Implements MNIST::MLPBase.

Definition at line 798 of file mnist_mlp.hpp.

    {
#ifndef NDEBUG
        assert(m_batchsize == activations.size());
#endif
        if(!m_filters.empty()){
            throw std::runtime_error("Conv layer not supported in forward_path function!");
        }
        if(!m_pools.empty()){
            throw std::runtime_error("Pooling layer layer not supported in forward_path function!");
        }
        const auto &labels = std::get<1>(m_mnist);
        const std::vector<cypress::Matrix<cypress::Real>> orig_weights =
            m_layers;
        for (size_t sample = 0; sample < m_batchsize; sample++) {
            if (start + sample >= indices.size())
                break;
            const auto &activ = activations[sample];
            auto error = vec_X_vec_comp(
                Loss::calc_error(labels[indices[start + sample]], activ.back()),
                ActivationFunction::derivative(activ.back()));
            // TODO  works for ReLU only
            update_mat(m_layers.back(), error, activ[activ.size() - 2],
                       m_batchsize, learn_rate);
            if (!last_only) {
                for (size_t inv_layer = 1; inv_layer < m_layers.size();
                     inv_layer++) {
                    size_t layer_id = m_layers.size() - inv_layer - 1;

                    error = vec_X_vec_comp(
                        mat_X_vec(orig_weights[layer_id + 1], error),
                        ActivationFunction::derivative(activ[layer_id + 1]));
                    update_mat(m_layers[layer_id], error, activ[layer_id],
                               m_batchsize, learn_rate);
                }
            }
        }
        m_constraint.constrain_weights(m_layers);
    }

template<typename Loss = MSE, typename ActivationFunction = ReLU, typename Constraint = NoConstraint>

virtual void MNIST::MLP< Loss, ActivationFunction, Constraint >::backward_path_2 ( const std::vector< uint16_t > &  labels, const std::vector< std::vector< std::vector< Real >>> &  activations, bool  last_only = false  )

inlineoverridevirtual

Implementation of backprop, adapted for usage in SNNs.

Parameters

labels	vector containing labels of the given batch
activations	activations in the form of [layer][sample][neuron]
last_only	true for last layer only training (Perceptron learn rule)

Implements MNIST::MLPBase.

Definition at line 848 of file mnist_mlp.hpp.

    {
#ifndef NDEBUG
        assert(m_batchsize == activations.back().size());
#endif
        if(!m_filters.empty()){
            throw std::runtime_error("Conv layer not supported in forward_path function!");
        }
        if(!m_pools.empty()){
            throw std::runtime_error("Pooling layer layer not supported in forward_path function!");
        }
        const auto orig_weights = m_layers;
        for (size_t sample = 0; sample < m_batchsize; sample++) {

            auto error = vec_X_vec_comp(
                Loss::calc_error(labels[sample], activations.back()[sample]),
                ActivationFunction::derivative(activations.back()[sample]));
            // TODO  works for ReLU only
            update_mat(m_layers.back(), error,
                       activations[activations.size() - 2][sample], m_batchsize,
                       learn_rate);
            if (!last_only) {
                for (size_t inv_layer = 1; inv_layer < m_layers.size();
                     inv_layer++) {
                    size_t layer_id = m_layers.size() - inv_layer - 1;

                    error = vec_X_vec_comp(
                        mat_X_vec(orig_weights[layer_id + 1], error),
                        ActivationFunction::derivative(
                            activations[layer_id + 1][sample]));
                    update_mat(m_layers[layer_id], error,
                               activations[layer_id][sample], m_batchsize,
                               learn_rate);
                }
            }
            m_constraint.constrain_weights(m_layers);
        }
    }

template<typename Loss = MSE, typename ActivationFunction = ReLU, typename Constraint = NoConstraint>

const size_t& MNIST::MLP< Loss, ActivationFunction, Constraint >::batchsize ( ) const

inlineoverridevirtual

Implements MNIST::MLPBase.

Definition at line 532 of file mnist_mlp.hpp.

{ return m_batchsize; }

template<typename Loss = MSE, typename ActivationFunction = ReLU, typename Constraint = NoConstraint>

Real MNIST::MLP< Loss, ActivationFunction, Constraint >::conv_max_weight ( size_t  layer_id ) const

inlineoverridevirtual

Implements MNIST::MLPBase.

Definition at line 482 of file mnist_mlp.hpp.

    {
        Real max = 0.0;
        auto layer = m_filters[layer_id];
        auto filter = layer.filter;
        for (size_t f = 0; f < layer.output_sizes[2]; f++) {
            for (size_t x = 0; x < filter.size(); x++) {
                for (size_t y = 0; y < filter[0].size(); y++) {
                    for (size_t z = 0; z < filter[0][0].size(); z++) {
                        max = filter[x][y][z][f] > max ? filter[x][y][z][f] : max;
                    }
                }
            }
        }
        return max;
    }

template<typename Loss = MSE, typename ActivationFunction = ReLU, typename Constraint = NoConstraint>

bool MNIST::MLP< Loss, ActivationFunction, Constraint >::correct ( const uint16_t  label, const std::vector< Real > &  output  ) const

inlineoverridevirtual

Checks if the output of the network was correct.

Parameters

label	the correct neuron
output	the output of the layer

Returns

true for correct classification

Implements MNIST::MLPBase.

Definition at line 678 of file mnist_mlp.hpp.

    {
        auto it = std::max_element(output.begin(), output.end());
        auto out = std::distance(output.begin(), it);
        return out == label;
    }

template<typename Loss = MSE, typename ActivationFunction = ReLU, typename Constraint = NoConstraint>

const size_t& MNIST::MLP< Loss, ActivationFunction, Constraint >::epochs ( ) const

inlineoverridevirtual

Implements MNIST::MLPBase.

Definition at line 531 of file mnist_mlp.hpp.

{ return m_epochs; }

template<typename Loss = MSE, typename ActivationFunction = ReLU, typename Constraint = NoConstraint>

virtual std::vector<std::vector<std::vector<Real> > > MNIST::MLP< Loss, ActivationFunction, Constraint >::forward_path ( const std::vector< size_t > &  indices, const size_t  start  ) const

inlineoverridevirtual

Forward path of the network (–> inference)

Parameters

indices	list of shuffled (?) indices
start	the start index, uses images indices[start] until indices[start +batchsize -1]

Returns

std::vector< std::vector< std::vector< cypress::Real > > > the outputs of all layers, given in output[sample][layer][neuron]

Implements MNIST::MLPBase.

Definition at line 724 of file mnist_mlp.hpp.

    {
        if(!m_filters.empty()){
            throw std::runtime_error("Conv layer not supported in forward_path function!");
        }
        if(!m_pools.empty()){
            throw std::runtime_error("Pooling layer layer not supported in forward_path function!");
        }
        auto &input = std::get<0>(m_mnist);
        std::vector<std::vector<std::vector<Real>>> res;
        std::vector<std::vector<Real>> activations;
        for (auto size : m_layer_sizes) {
            activations.emplace_back(std::vector<Real>(size, 0.0));
        }
        for (size_t sample = 0; sample < m_batchsize; sample++) {
            res.emplace_back(activations);
        }

        for (size_t sample = 0; sample < m_batchsize; sample++) {
            if (start + sample >= indices.size())
                break;
            res[sample][0] = input[indices[start + sample]];
            for (size_t layer = 0; layer < m_layers.size(); layer++) {
                res[sample][layer + 1] = ActivationFunction::function(
                    mat_trans_X_vec(m_layers[layer], res[sample][layer]));
            }
        }
        return res;
    }

template<typename Loss = MSE, typename ActivationFunction = ReLU, typename Constraint = NoConstraint>

virtual Real MNIST::MLP< Loss, ActivationFunction, Constraint >::forward_path_test ( ) const

inlineoverridevirtual

Forward path of test data.

Returns

cypress::Real the accuracy on test data

Implements MNIST::MLPBase.

Definition at line 760 of file mnist_mlp.hpp.

    {
        if(!m_filters.empty()){
            throw std::runtime_error("Conv layer not supported in forward_path function!");
        }
        if(!m_pools.empty()){
            throw std::runtime_error("Pooling layer layer not supported in forward_path function!");
        }
        auto &input = std::get<0>(m_mnist_test);
        auto &labels = std::get<1>(m_mnist_test);
        std::vector<std::vector<Real>> activations;
        for (auto size : m_layer_sizes) {
            activations.emplace_back(std::vector<Real>(size, 0.0));
        }
        size_t sum = 0;
        for (size_t sample = 0; sample < input.size(); sample++) {
            activations[0] = input[sample];
            for (size_t layer = 0; layer < m_layers.size(); layer++) {
                activations[layer + 1] = ActivationFunction::function(
                    mat_trans_X_vec(m_layers[layer], activations[layer]));
            }
            if (correct(labels[sample], activations.back()))
                sum++;
        }

        return Real(sum) / Real(labels.size());
    }

template<typename Loss = MSE, typename ActivationFunction = ReLU, typename Constraint = NoConstraint>

const std::vector<mnist_helper::CONVOLUTION_LAYER>& MNIST::MLP< Loss, ActivationFunction, Constraint >::get_conv_layers ( )

inlineoverridevirtual

Return all filter weights in the form of weights[x][y][depth][filter].

Returns

const mnist_helper::Conv_TYPE &

Implements MNIST::MLPBase.

Definition at line 570 of file mnist_mlp.hpp.

    {
        return m_filters;
    }

template<typename Loss = MSE, typename ActivationFunction = ReLU, typename Constraint = NoConstraint>

const std::vector<size_t>& MNIST::MLP< Loss, ActivationFunction, Constraint >::get_layer_sizes ( )

inlineoverridevirtual

Return the number of neurons per layer.

Returns

const std::vector< size_t >&

Implements MNIST::MLPBase.

Definition at line 585 of file mnist_mlp.hpp.

    {
        return m_layer_sizes;
    }

template<typename Loss = MSE, typename ActivationFunction = ReLU, typename Constraint = NoConstraint>

const std::vector<mnist_helper::LAYER_TYPE>& MNIST::MLP< Loss, ActivationFunction, Constraint >::get_layer_types ( )

inlineoverridevirtual

Implements MNIST::MLPBase.

Definition at line 590 of file mnist_mlp.hpp.

    {
        return m_layer_types;
    }

template<typename Loss = MSE, typename ActivationFunction = ReLU, typename Constraint = NoConstraint>

const std::vector<mnist_helper::POOLING_LAYER>& MNIST::MLP< Loss, ActivationFunction, Constraint >::get_pooling_layers ( )

inlineoverridevirtual

Implements MNIST::MLPBase.

Definition at line 575 of file mnist_mlp.hpp.

    {
        return m_pools;
    }

template<typename Loss = MSE, typename ActivationFunction = ReLU, typename Constraint = NoConstraint>

const std::vector<cypress::Matrix<Real> >& MNIST::MLP< Loss, ActivationFunction, Constraint >::get_weights ( )

inlineoverridevirtual

Return all weights in the form of weights[layer](src,tar)

Returns

const std::vector< cypress::Matrix< cypress::Real > >&

Implements MNIST::MLPBase.

Definition at line 559 of file mnist_mlp.hpp.

    {
        return m_layers;
    }

template<typename Loss = MSE, typename ActivationFunction = ReLU, typename Constraint = NoConstraint>

const Real& MNIST::MLP< Loss, ActivationFunction, Constraint >::learnrate ( ) const

inlineoverridevirtual

Implements MNIST::MLPBase.

Definition at line 533 of file mnist_mlp.hpp.

{ return learn_rate; }

template<typename Loss = MSE, typename ActivationFunction = ReLU, typename Constraint = NoConstraint>

void MNIST::MLP< Loss, ActivationFunction, Constraint >::load_data ( std::string  path )

inlineprotected

Definition at line 254 of file mnist_mlp.hpp.

    {
        m_mnist = mnist_helper::loadMnistData(60000, path + "train");
        m_mnist_test = mnist_helper::loadMnistData(10000, path + "t10k");
    }

template<typename Loss = MSE, typename ActivationFunction = ReLU, typename Constraint = NoConstraint>

static std::vector<Real> MNIST::MLP< Loss, ActivationFunction, Constraint >::mat_trans_X_vec ( const Matrix< Real > &  mat, const std::vector< Real > &  vec  )

inlinestatic

Implements transposed matrix vector multiplication.

Parameters

mat	the matrix to transpose: mat.rows() ==! vec.size()
vec	the vector

Returns

std::vector< cypress::Real > resulting vector

Definition at line 636 of file mnist_mlp.hpp.

    {
#ifndef NDEBUG
        assert(mat.rows() == vec.size());
#endif
        std::vector<Real> res(mat.cols(), 0.0);
        for (size_t i = 0; i < mat.cols(); i++) {
            for (size_t j = 0; j < mat.rows(); j++) {
                res[i] += mat(j, i) * vec[j];
            }
        }
        return res;
    }

template<typename Loss = MSE, typename ActivationFunction = ReLU, typename Constraint = NoConstraint>

static std::vector<Real> MNIST::MLP< Loss, ActivationFunction, Constraint >::mat_X_vec ( const Matrix< Real > &  mat, const std::vector< Real > &  vec  )

inlinestatic

Implements matrix vector multiplication.

Parameters

mat	the matrix: mat.cols() ==! vec.size()
vec	the vector

Returns

std::vector< cypress::Real >

Definition at line 614 of file mnist_mlp.hpp.

    {
#ifndef NDEBUG
        assert(mat.cols() == vec.size());
#endif
        std::vector<Real> res(mat.rows(), 0.0);
        for (size_t i = 0; i < mat.rows(); i++) {
            for (size_t j = 0; j < mat.cols(); j++) {
                res[i] += mat(i, j) * vec[j];
            }
        }
        return res;
    }

template<typename Loss = MSE, typename ActivationFunction = ReLU, typename Constraint = NoConstraint>

Real MNIST::MLP< Loss, ActivationFunction, Constraint >::max_weight ( ) const

inlineoverridevirtual

Return the largest weight in the network.

Returns

value of the weight

Implements MNIST::MLPBase.

Definition at line 471 of file mnist_mlp.hpp.

    {
        Real max = 0.0;
        for (auto &layer : m_layers) {
            auto w = mnist_helper::max_weight(layer);
            if (w > max)
                max = w;
        }
        return max;
    }

template<typename Loss = MSE, typename ActivationFunction = ReLU, typename Constraint = NoConstraint>

Real MNIST::MLP< Loss, ActivationFunction, Constraint >::max_weight_abs ( ) const

inlineoverridevirtual

Return the largest absolute weight in the network.

Returns

value of the weight

Implements MNIST::MLPBase.

Definition at line 520 of file mnist_mlp.hpp.

    {
        Real max = 0.0;
        for (auto &layer : m_layers) {
            auto w = mnist_helper::max_weight_abs(layer);
            if (w > max)
                max = w;
        }
        return max;
    }

template<typename Loss = MSE, typename ActivationFunction = ReLU, typename Constraint = NoConstraint>

Real MNIST::MLP< Loss, ActivationFunction, Constraint >::min_weight ( ) const

inlineoverridevirtual

Return the smallest weight in the network.

Returns

value of the weight

Implements MNIST::MLPBase.

Definition at line 504 of file mnist_mlp.hpp.

    {
        Real min = 0.0;
        for (auto &layer : m_layers) {
            auto w = mnist_helper::min_weight(layer);
            if (w < min)
                min = w;
        }
        return min;
    }

template<typename Loss = MSE, typename ActivationFunction = ReLU, typename Constraint = NoConstraint>

const mnist_helper::MNIST_DATA& MNIST::MLP< Loss, ActivationFunction, Constraint >::mnist_test_set ( )

inlineoverridevirtual

Returns reference to the test data.

Returns

const mnist_helper::MNIST_DATA&

Implements MNIST::MLPBase.

Definition at line 549 of file mnist_mlp.hpp.

    {
        return m_mnist_test;
    }

template<typename Loss = MSE, typename ActivationFunction = ReLU, typename Constraint = NoConstraint>

const mnist_helper::MNIST_DATA& MNIST::MLP< Loss, ActivationFunction, Constraint >::mnist_train_set ( )

inlineoverridevirtual

Returns reference to the train data.

Returns

const mnist_helper::MNIST_DATA&

Implements MNIST::MLPBase.

Definition at line 540 of file mnist_mlp.hpp.

    {
        return m_mnist;
    }

template<typename Loss = MSE, typename ActivationFunction = ReLU, typename Constraint = NoConstraint>

void MNIST::MLP< Loss, ActivationFunction, Constraint >::scale_down_images ( size_t  pooling_size = 3 )

inlineoverridevirtual

Scale down the whole data set, reduces the image by a given factor in every dimension.

Parameters

pooling_size

the factor reducing the image

Implements MNIST::MLPBase.

Definition at line 601 of file mnist_mlp.hpp.

    {
        m_mnist = mnist_helper::scale_mnist(m_mnist, pooling_size);
        m_mnist_test = mnist_helper::scale_mnist(m_mnist_test, pooling_size);
    }

template<typename Loss = MSE, typename ActivationFunction = ReLU, typename Constraint = NoConstraint>

void MNIST::MLP< Loss, ActivationFunction, Constraint >::train ( unsigned  seed = 0 )

inlineoverridevirtual

Starts the full training process.

Parameters

seed	sets up the random numbers for image shuffling

Implements MNIST::MLPBase.

Definition at line 926 of file mnist_mlp.hpp.

    {
        std::vector<size_t> indices(std::get<0>(m_mnist).size());
        m_constraint.constrain_weights(m_layers);
        for (size_t i = 0; i < indices.size(); i++) {
            indices[i] = i;
        }
        if (seed == 0) {
            seed = std::chrono::system_clock::now().time_since_epoch().count();
        }
        auto rng = std::default_random_engine{seed};

        for (size_t epoch = 0; epoch < m_epochs; epoch++) {
            size_t correct = 0;
            std::shuffle(indices.begin(), indices.end(), rng);
            for (size_t current_idx = 0;
                 current_idx < std::get<1>(m_mnist).size();
                 current_idx += m_batchsize) {
                auto activations = forward_path(indices, current_idx);
                correct += accuracy(activations, indices, current_idx);
                backward_path(indices, current_idx, activations);
                m_constraint.constrain_weights(m_layers);
            }
            cypress::global_logger().info(
                "MLP", "Accuracy of epoch " + std::to_string(epoch) + ": " +
                           std::to_string(Real(correct) /
                                          Real(std::get<1>(m_mnist).size())));
        }
    }

template<typename Loss = MSE, typename ActivationFunction = ReLU, typename Constraint = NoConstraint>

static void MNIST::MLP< Loss, ActivationFunction, Constraint >::update_mat ( Matrix< Real > &  mat, const std::vector< Real > &  errors, const std::vector< Real > &  pre_output, const size_t  sample_num, const Real  learn_rate  )

inlinestatic

Updates the weight matrix based on the error in this layer and the output of the previous layer.

Parameters

mat	weight matrix.
errors	error vector in this layer
pre_output	output rates of previous layer
sample_num	number of samples in this batch == mini batchsize
learn_rate	the learn rate multiplied with the gradient

Definition at line 696 of file mnist_mlp.hpp.

    {
#ifndef NDEBUG
        assert(mat.rows() == pre_output.size());
        assert(mat.cols() == errors.size());
#endif
        Real sample_num_r(sample_num);
        for (size_t i = 0; i < mat.rows(); i++) {
            for (size_t j = 0; j < mat.cols(); j++) {
                mat(i, j) = mat(i, j) - learn_rate * pre_output[i] * errors[j] /
                                            sample_num_r;
            }
        }
    }

template<typename Loss = MSE, typename ActivationFunction = ReLU, typename Constraint = NoConstraint>

static std::vector<Real> MNIST::MLP< Loss, ActivationFunction, Constraint >::vec_X_vec_comp ( const std::vector< Real > &  vec1, const std::vector< Real > &  vec2  )

inlinestatic

Vector vector multiplication, component-wise.

Parameters

vec1	first vector
vec2	second vector of size vec1.size()

Returns

std::vector< cypress::Real > resulting vector

Definition at line 658 of file mnist_mlp.hpp.

    {
#ifndef NDEBUG
        assert(vec1.size() == vec2.size());
#endif
        std::vector<Real> res(vec1.size());
        for (size_t i = 0; i < vec1.size(); i++) {
            res[i] = vec1[i] * vec2[i];
        }
        return res;
    }

Member Data Documentation

template<typename Loss = MSE, typename ActivationFunction = ReLU, typename Constraint = NoConstraint>

Real MNIST::MLP< Loss, ActivationFunction, Constraint >::learn_rate = 0.01

protected

Definition at line 250 of file mnist_mlp.hpp.

template<typename Loss = MSE, typename ActivationFunction = ReLU, typename Constraint = NoConstraint>

size_t MNIST::MLP< Loss, ActivationFunction, Constraint >::m_batchsize = 100

protected

Definition at line 249 of file mnist_mlp.hpp.

template<typename Loss = MSE, typename ActivationFunction = ReLU, typename Constraint = NoConstraint>

Constraint MNIST::MLP< Loss, ActivationFunction, Constraint >::m_constraint

protected

Definition at line 260 of file mnist_mlp.hpp.

template<typename Loss = MSE, typename ActivationFunction = ReLU, typename Constraint = NoConstraint>

size_t MNIST::MLP< Loss, ActivationFunction, Constraint >::m_epochs = 20

protected

Definition at line 248 of file mnist_mlp.hpp.

template<typename Loss = MSE, typename ActivationFunction = ReLU, typename Constraint = NoConstraint>

std::vector<mnist_helper::CONVOLUTION_LAYER> MNIST::MLP< Loss, ActivationFunction, Constraint >::m_filters

protected

Definition at line 245 of file mnist_mlp.hpp.

template<typename Loss = MSE, typename ActivationFunction = ReLU, typename Constraint = NoConstraint>

std::vector<size_t> MNIST::MLP< Loss, ActivationFunction, Constraint >::m_layer_sizes

protected

Definition at line 244 of file mnist_mlp.hpp.

template<typename Loss = MSE, typename ActivationFunction = ReLU, typename Constraint = NoConstraint>

std::vector<mnist_helper::LAYER_TYPE> MNIST::MLP< Loss, ActivationFunction, Constraint >::m_layer_types

protected

Definition at line 247 of file mnist_mlp.hpp.

template<typename Loss = MSE, typename ActivationFunction = ReLU, typename Constraint = NoConstraint>

std::vector<cypress::Matrix<Real> > MNIST::MLP< Loss, ActivationFunction, Constraint >::m_layers

protected

Definition at line 243 of file mnist_mlp.hpp.

template<typename Loss = MSE, typename ActivationFunction = ReLU, typename Constraint = NoConstraint>

mnist_helper::MNIST_DATA MNIST::MLP< Loss, ActivationFunction, Constraint >::m_mnist

protected

Definition at line 251 of file mnist_mlp.hpp.

template<typename Loss = MSE, typename ActivationFunction = ReLU, typename Constraint = NoConstraint>

mnist_helper::MNIST_DATA MNIST::MLP< Loss, ActivationFunction, Constraint >::m_mnist_test

protected

Definition at line 252 of file mnist_mlp.hpp.

template<typename Loss = MSE, typename ActivationFunction = ReLU, typename Constraint = NoConstraint>

std::vector<mnist_helper::POOLING_LAYER> MNIST::MLP< Loss, ActivationFunction, Constraint >::m_pools

protected

Definition at line 246 of file mnist_mlp.hpp.

---

The documentation for this class was generated from the following file:

• source/SNABs/mnist/mnist_mlp.hpp