Search Space¶

For hyper-parameter optimization, the search space might be a closed subset of Euclidean space composed of discrete or continuous values. For neural architecture search(NAS), The network architecture is composed of graphs generated by various layers through connections, therefore all feasible graphs constitute the search space.

Hypernets introduced an abstract search space representation and taken into account both of the above requirements, so that it can use an unified primitive to define various search spaces.

illustration of the search space in Hypernets

HyperSpace¶

HyperSpace is an abstract representation of the search space composed of Parameter Space, Connection Space, and Module Space. The general form of HyperSpace is a directed acyclic graph (DAG), consisting of connecting nodes and edges. Each node contains a set of hyperparameters. In some cases, the objective function for optimatization has only one node, like the lightGBM model, which we only need to optiizea a few hyper-parameters.

Parameter Space¶

ParameterSpace can be divided into two categories: mutable and immutable. The mutable parameter space is always followed with user defined values. For example, Real(0.1, 0.9), Choice(['relu','tanh']), etc. The immutable includes Constant,Dynamic, and Cascade.

Mutable: Int, Real, Bool, Choice, MultipleChoice
Immutable: Constant, Dynamic, Cascade

Connection Space¶

ConnectionSpace is used to represent possible connections between modules.

Optional: Whether to insert a module between two modules.

from hypernets.frameworks.keras.layers import Dense, Input, Dropout
from hypernets.core.ops import Optional

module_a = Input()
module_b = Dropout()
optional_dropout = Optional(module_b, keep_link=True)(module_a)
module_c = Dense()(optional_dropout)

ModuleChoice: There are several optional modules on a node, choose one of them.

from hypernets.frameworks.keras.layers import Input, BatchNormalization
from hypernets.core.ops import ModuleChoice

module_a = Input()
or_conv_pool = ModuleChoice([sepconv5x5(),sepconv3x3(),avgpooling3x3()])(module_a)
module_c = BatchNormalization()(or_conv_pool)

Sequential: A series of modules connected in order.

from hypernets.frameworks.keras.layers import Dense, Input, BatchNormalization
from hypernets.core.ops import Sequential

module_a = Input()
module_b = Dense()
module_c = BatchNormalization()
Sequential([module_a,module_b,module_c])

Permutation: Choose one of the possible permutations and connect the modules in order.

from hypernets.frameworks.keras.layers import Dense, BatchNormalization, Dropout, Activation
from hypernets.core.search_space import Choice
from hypernets.core.ops import Permutation, Sequential, Optional

dense = Dense(units=Choice([100, 300, 500, 1000]))
act = Activation(activation=Choice(['relu', 'tanh']))
optional_bn = Optional(BatchNormalization(), keep_link=True)
dropout = Dropout(rate=Choice([0, 0.1, 0.2, 0.3, 0.5]))

# Use `Permutation` to try different arrangements of act, optional_bn, dropout
# optional_bn is optional module and will be skipped when hp_use_bn is False
perm_act_bn_dropout = Permutation([act, optional_bn, dropout])

# Use `Sequential` to connect dense and perm_act_bn_dropout in order
seq = Sequential([dense, perm_act_bn_dropout])

Repeat: The module is repeated multiple times and connected in order.

from hypernets.frameworks.keras.layers import Dense, BatchNormalization, Dropout, Activation
from hypernets.core.search_space import Choice
from hypernets.core.ops import Permutation, Sequential, Optional, Repeat

dense = Dense(units=Choice([100, 300, 500, 1000]))
act = Activation(activation=Choice(['relu', 'tanh']))
optional_bn = Optional(BatchNormalization(), keep_link=True)
dropout = Dropout(rate=Choice([0, 0.1, 0.2, 0.3, 0.5]))

# Use `Permutation` to try different arrangements of act, optional_bn, dropout
# optional_bn is optional module and will be skipped when hp_use_bn is False
perm_act_bn_dropout = Permutation([act, optional_bn, dropout])

repeat_seq = Repeat(module_fn=lambda :Sequential([dense, perm_act_bn_dropout]), repeat_times=Choice([2,3,4]))

InputChoice: A modules has multiple upstream modules, choose one or more connections.

from hypernets.frameworks.keras.layers import Input,Dense, BatchNormalization, Dropout, Activation
from hypernets.core.search_space import Choice
from hypernets.core.ops import Permutation, Sequential, Optional, Repeat, InputChoice

module_a = Input()
module_b = Input()
module_c = Input()
ic = InputChoice(inputs=[module_a,module_b,module_a], num_chosen_most=2)([module_a,module_b,module_c])
add = Add()(ic)

Module Space¶

ModuleSpace represents an optimizable function with a set of hyper-parameters. Unlike ParameterSpace and ConnectionSpace, ModuleSpace is related to frameworks or libraries. For different HyperModel, it is necessary to define a corresponding series of ModuleSpace.