Search Space

For hyper-parameter optimization, the search space might be a closed subset of Euclidean space composed of discrete or continuous values. For neural architecture search(NAS), The network architecture is composed of graphs generated by various layers through connections, and all feasible graphs constitute the search space.

Hypernets introduced an abstract search space representation and taken into account both of the above requirements, so that it can use a unified primitive to define various search spaces.

  • illustration of the search space in Hypernets

HyperSpace

The general form of HyperSpace is a graph formed by connecting nodes and edges, each node can contain a set of hyper-parameters. HyperModel can represent a pipeline, or a more complex directed acyclic graph(DAG), in some cases The objective function to be optimized has only one node, for example when we only need to optimize a few hyper-parameters of a lightgbm model.

Parameter Space

ParameterSpace is diveded into two types: mutable and immutable. The mutable is created with a set of possible values, for example, Real(0.1, 0.9), Choice(['relu','tanh']), etc. The immutable includes Constant,Dynamic, and Cascade.

  • Mutable: Int, Real, Bool, Choice, MultipleChoice
  • Immutable: Constant, Dynamic, Cascade

Connection Space

ConnectionSpace is used to represent possible connections between modules.

  • Optional: Whether to insert a module between two modules.

_images/connection_space_optional.png

from hypernets.frameworks.keras.layers import Dense, Input, Dropout
from hypernets.core.ops import Optional

module_a = Input()
module_b = Dropout()
optional_dropout = Optional(module_b, keep_link=True)(module_a)
module_c = Dense()(optional_dropout)
  • ModuleChoice: There are several optional modules on a node, choose one of them.

_images/connection_space_or.png

from hypernets.frameworks.keras.layers import Input, BatchNormalization
from hypernets.core.ops import ModuleChoice

module_a = Input()
or_conv_pool = ModuleChoice([sepconv5x5(),sepconv3x3(),avgpooling3x3()])(module_a)
module_c = BatchNormalization()(or_conv_pool)
  • Sequential: A series of modules connected in order.

_images/connection_space_sequential.png

from hypernets.frameworks.keras.layers import Dense, Input, BatchNormalization
from hypernets.core.ops import Sequential

module_a = Input()
module_b = Dense()
module_c = BatchNormalization()
Sequential([module_a,module_b,module_c])
  • Permutation: Choose one of permutations of a module list and connected in order.

_images/connection_space_permuation.png

from hypernets.frameworks.keras.layers import Dense, BatchNormalization, Dropout, Activation
from hypernets.core.search_space import Choice
from hypernets.core.ops import Permutation, Sequential, Optional

dense = Dense(units=Choice([100, 300, 500, 1000]))
act = Activation(activation=Choice(['relu', 'tanh']))
optional_bn = Optional(BatchNormalization(), keep_link=True)
dropout = Dropout(rate=Choice([0, 0.1, 0.2, 0.3, 0.5]))

# Use `Permutation` to try different arrangements of act, optional_bn, dropout
# optional_bn is optional module and will be skipped when hp_use_bn is False
perm_act_bn_dropout = Permutation([act, optional_bn, dropout])

# Use `Sequential` to connect dense and perm_act_bn_dropout in order
seq = Sequential([dense, perm_act_bn_dropout])
  • Repeat: The module is repeated n times and connected in order.

_images/connection_space_repeat.png

from hypernets.frameworks.keras.layers import Dense, BatchNormalization, Dropout, Activation
from hypernets.core.search_space import Choice
from hypernets.core.ops import Permutation, Sequential, Optional, Repeat

dense = Dense(units=Choice([100, 300, 500, 1000]))
act = Activation(activation=Choice(['relu', 'tanh']))
optional_bn = Optional(BatchNormalization(), keep_link=True)
dropout = Dropout(rate=Choice([0, 0.1, 0.2, 0.3, 0.5]))

# Use `Permutation` to try different arrangements of act, optional_bn, dropout
# optional_bn is optional module and will be skipped when hp_use_bn is False
perm_act_bn_dropout = Permutation([act, optional_bn, dropout])

repeat_seq = Repeat(module_fn=lambda :Sequential([dense, perm_act_bn_dropout]), repeat_times=Choice([2,3,4]))
  • InputChoice: A modules has n upstream modules, choose one or more connections.

_images/connection_space_inputchoice.png

from hypernets.frameworks.keras.layers import Input,Dense, BatchNormalization, Dropout, Activation
from hypernets.core.search_space import Choice
from hypernets.core.ops import Permutation, Sequential, Optional, Repeat, InputChoice

module_a = Input()
module_b = Input()
module_c = Input()
ic = InputChoice(inputs=[module_a,module_b,module_a], num_chosen_most=2)([module_a,module_b,module_c])
add = Add()(ic)

Module Space

ModuleSpace represents an optimizable function with a set of hyper-parameters. Unlike ParameterSpace and ConnectionSpace, ModuleSpace is related to frameworks or libraries. In different HyperModel, it is necessary to define a corresponding series of ModuleSpace.