hypernets.experiment package

Submodules

hypernets.experiment.cfg module

hypernets.experiment.compete module

class hypernets.experiment.compete.CompeteExperiment(hyper_model, X_train, y_train, X_eval=None, y_eval=None, X_test=None, eval_size=0.3, train_test_split_strategy=None, cv=None, num_folds=3, task=None, id=None, callbacks=None, random_state=None, scorer=None, data_adaption=None, data_adaption_target=None, data_adaption_memory_limit=0.05, data_adaption_min_cols=0.3, data_cleaner_args=None, feature_generation=False, feature_generation_trans_primitives=None, feature_generation_max_depth=1, feature_generation_categories_cols=None, feature_generation_continuous_cols=None, feature_generation_datetime_cols=None, feature_generation_latlong_cols=None, feature_generation_text_cols=None, collinearity_detection=False, drift_detection=True, drift_detection_remove_shift_variable=True, drift_detection_variable_shift_threshold=0.7, drift_detection_threshold=0.7, drift_detection_remove_size=0.1, drift_detection_min_features=10, drift_detection_num_folds=5, feature_selection=False, feature_selection_strategy=None, feature_selection_threshold=None, feature_selection_quantile=None, feature_selection_number=None, down_sample_search=None, down_sample_search_size=None, down_sample_search_time_limit=None, down_sample_search_max_trials=None, ensemble_size=20, feature_reselection=False, feature_reselection_estimator_size=10, feature_reselection_strategy=None, feature_reselection_threshold=1e-05, feature_reselection_quantile=None, feature_reselection_number=None, pseudo_labeling=False, pseudo_labeling_strategy=None, pseudo_labeling_proba_threshold=None, pseudo_labeling_proba_quantile=None, pseudo_labeling_sample_number=None, pseudo_labeling_resplit=False, retrain_on_wholedata=False, log_level=None, **kwargs)[source]

Bases: hypernets.experiment.compete.SteppedExperiment

A powerful experiment strategy for AutoML with a set of advanced features.

There are still many challenges in the machine learning modeling process for tabular data, such as imbalanced data, data drift, poor generalization ability, etc. This challenges cannot be completely solved by pipeline search, so we introduced in HyperNets a more powerful tool is CompeteExperiment. CompeteExperiment is composed of a series of steps and Pipeline Search is just one step. It also includes advanced steps such as data cleaning, data drift handling, two-stage search, ensemble etc.

static get_creators(hyper_model, X_train, y_train, X_test=None, X_eval=None, y_eval=None, down_sample_search=False)[source]
get_data_character()[source]
run(**kwargs)[source]
to_estimator(X_train, y_train, X_test, X_eval, y_eval, steps)[source]
class hypernets.experiment.compete.DaskEnsembleStep(experiment, name, scorer=None, ensemble_size=7)[source]

Bases: hypernets.experiment.compete.EnsembleStep

get_ensemble_predictions(trials, ensemble)[source]
class hypernets.experiment.compete.DaskPseudoLabelStep(experiment, name, estimator_builder_name, strategy=None, proba_threshold=None, proba_quantile=None, sample_number=None, resplit=False)[source]

Bases: hypernets.experiment.compete.PseudoLabelStep

merge_pseudo_label(X_train, y_train, X_eval, y_eval, X_pseudo, y_pseudo, **kwargs)[source]
class hypernets.experiment.compete.DataAdaptionStep(experiment, name, target=None, memory_limit=0.05, min_cols=0.3)[source]

Bases: hypernets.experiment.compete.FeatureSelectStep

compact_by_columns(X_train, y_train, X_test, X_eval, y_eval, frac)[source]
compact_by_rows(X_train, y_train, X_test, X_eval, y_eval, frac)[source]
fit_transform(hyper_model, X_train, y_train, X_test=None, X_eval=None, y_eval=None, **kwargs)[source]
get_tool_box_with_target(*data)[source]
sample(X, y, frac)[source]
class hypernets.experiment.compete.DataCleanStep(experiment, name, data_cleaner_args=None, cv=False, train_test_split_strategy=None)[source]

Bases: hypernets.experiment.compete.FeatureSelectStep

as_local()[source]
cache_transform(hyper_model, X_train, y_train, X_test=None, X_eval=None, y_eval=None, **kwargs)[source]
fit_transform(**kwargs)
get_fitted_params()[source]
get_params(deep=True)[source]

Get parameters for this estimator.

Parameters:deep (bool, default=True) – If True, will return the parameters for this estimator and contained subobjects that are estimators.
Returns:params – Parameter names mapped to their values.
Return type:dict
transform(X, y=None, **kwargs)[source]
class hypernets.experiment.compete.DriftDetectStep(experiment, name, remove_shift_variable, variable_shift_threshold, threshold, remove_size, min_features, num_folds)[source]

Bases: hypernets.experiment.compete.FeatureSelectStep

fit_transform(**kwargs)
get_fitted_params()[source]
class hypernets.experiment.compete.EnsembleStep(experiment, name, scorer=None, ensemble_size=7)[source]

Bases: hypernets.experiment.compete.EstimatorBuilderStep

build_estimator(hyper_model, X_train, y_train, X_eval=None, y_eval=None, **kwargs)[source]
get_ensemble(estimators, X_train, y_train)[source]
get_ensemble_predictions(trials, ensemble)[source]
select_trials(hyper_model)[source]

select trials to ensemble from hyper_model (and it’s history)

class hypernets.experiment.compete.EstimatorBuilderStep(experiment, name)[source]

Bases: hypernets.experiment.compete.ExperimentStep

build_estimator(hyper_model, X_train, y_train, X_test=None, X_eval=None, y_eval=None, **kwargs)[source]
fit_transform(hyper_model, X_train, y_train, X_test=None, X_eval=None, y_eval=None, **kwargs)[source]
get_fitted_params()[source]
is_transform_skipped()[source]
transform(X, y=None, **kwargs)[source]
class hypernets.experiment.compete.ExperimentStep(experiment, name)[source]

Bases: sklearn.base.BaseEstimator

STATUS_FAILED = 1
STATUS_NONE = -1
STATUS_RUNNING = 10
STATUS_SKIPPED = 2
STATUS_SUCCESS = 0
elapsed_seconds
fit_transform(hyper_model, X_train, y_train, X_test=None, X_eval=None, y_eval=None, **kwargs)[source]
get_fitted_params()[source]
is_transform_skipped()[source]
step_progress(*args, **kwargs)[source]
task
transform(X, y=None, **kwargs)[source]
class hypernets.experiment.compete.FeatureGenerationStep(experiment, name, trans_primitives=None, continuous_cols=None, datetime_cols=None, categories_cols=None, latlong_cols=None, text_cols=None, max_depth=1, feature_selection_args=None)[source]

Bases: hypernets.experiment.compete.TransformerAdaptorStep

get_fitted_params()[source]
is_transform_skipped()[source]
class hypernets.experiment.compete.FeatureImportanceSelectionStep(experiment, name, strategy, threshold, quantile, number)[source]

Bases: hypernets.experiment.compete.FeatureSelectStep

fit_transform(**kwargs)
get_fitted_params()[source]
class hypernets.experiment.compete.FeatureSelectStep(experiment, name)[source]

Bases: hypernets.experiment.compete.ExperimentStep

cache_transform(hyper_model, X_train, y_train, X_test=None, X_eval=None, y_eval=None, **kwargs)[source]
get_fitted_params()[source]
is_transform_skipped()[source]
selected_features
transform(X, y=None, **kwargs)[source]
unselected_features
class hypernets.experiment.compete.FinalTrainStep(experiment, name, retrain_on_wholedata=False)[source]

Bases: hypernets.experiment.compete.EstimatorBuilderStep

build_estimator(hyper_model, X_train, y_train, X_test=None, X_eval=None, y_eval=None, **kwargs)[source]
class hypernets.experiment.compete.MOOFinalStep(experiment, name)[source]

Bases: hypernets.experiment.compete.EstimatorBuilderStep

build_estimator(hyper_model, X_train, y_train, X_test=None, X_eval=None, y_eval=None, **kwargs)[source]
class hypernets.experiment.compete.MulticollinearityDetectStep(experiment, name)[source]

Bases: hypernets.experiment.compete.FeatureSelectStep

fit_transform(**kwargs)
get_fitted_params()[source]
class hypernets.experiment.compete.PermutationImportanceSelectionStep(experiment, name, scorer, estimator_size, strategy, threshold, quantile, number)[source]

Bases: hypernets.experiment.compete.FeatureSelectStep

fit_transform(hyper_model, X_train, y_train, X_test=None, X_eval=None, y_eval=None, **kwargs)[source]
get_fitted_params()[source]
class hypernets.experiment.compete.PseudoLabelStep(experiment, name, estimator_builder_name, strategy=None, proba_threshold=None, proba_quantile=None, sample_number=None, resplit=False)[source]

Bases: hypernets.experiment.compete.ExperimentStep

fit_transform(hyper_model, X_train, y_train, X_test=None, X_eval=None, y_eval=None, **kwargs)[source]
get_fitted_params()[source]
is_transform_skipped()[source]
merge_pseudo_label(X_train, y_train, X_eval, y_eval, X_pseudo, y_pseudo, **kwargs)[source]
static stat_pseudo_label(y_pseudo, classes)[source]
transform(X, y=None, **kwargs)[source]
class hypernets.experiment.compete.SpaceSearchStep(experiment, name, cv=False, num_folds=3)[source]

Bases: hypernets.experiment.compete.ExperimentStep

estimate_time_limit(total_time_limit)[source]
static find_early_stopping_callback(cbs)[source]
fit_transform(hyper_model, X_train, y_train, X_test=None, X_eval=None, y_eval=None, **kwargs)[source]
from_fitted_step(fitted_step)[source]
get_fitted_params()[source]
is_transform_skipped()[source]
search(X_train, y_train, X_test=None, X_eval=None, y_eval=None, **kwargs)[source]
transform(X, y=None, **kwargs)[source]
class hypernets.experiment.compete.SpaceSearchWithDownSampleStep(experiment, name, cv=False, num_folds=3, size=None, max_trials=None, time_limit=None)[source]

Bases: hypernets.experiment.compete.SpaceSearchStep

static create_playback_searcher(history)[source]
down_sample(X_train, y_train, X_eval, y_eval)[source]
from_fitted_step(fitted_step)[source]
search(X_train, y_train, X_test=None, X_eval=None, y_eval=None, **kwargs)[source]
class hypernets.experiment.compete.StepNames[source]

Bases: object

DATA_ADAPTION = 'data_adaption'
DATA_CLEAN = 'data_clean'
DRIFT_DETECTION = 'drift_detection'
ENSEMBLE = 'ensemble'
FEATURE_GENERATION = 'feature_generation'
FEATURE_IMPORTANCE_SELECTION = 'feature_selection'
FEATURE_RESELECTION = 'feature_reselection'
FINAL_ENSEMBLE = 'final_ensemble'
FINAL_MOO = 'final_moo'
FINAL_SEARCHING = 'two_stage_searching'
FINAL_TRAINING = 'final_train'
MULITICOLLINEARITY_DETECTION = 'multicollinearity_detection'
PSEUDO_LABELING = 'pseudo_labeling'
SPACE_SEARCHING = 'space_searching'
TRAINING = 'training'
class hypernets.experiment.compete.SteppedExperiment(steps, *args, **kwargs)[source]

Bases: hypernets.experiment._experiment.Experiment

find_step(fn, until_step_name=None, index=False)[source]
get_step(name)[source]
get_step_index(name_or_index, default)[source]
static to_estimator(X_train, y_train, X_test, X_eval, y_eval, steps)[source]
train(hyper_model, X_train, y_train, X_test, X_eval=None, y_eval=None, **kwargs)[source]

Run an experiment

Parameters:
  • hyper_model (HyperModel) –
  • X_train –
  • y_train –
  • X_test –
  • X_eval –
  • y_eval –
class hypernets.experiment.compete.TransformerAdaptorStep(experiment, name, transformer_creator, **kwargs)[source]

Bases: hypernets.experiment.compete.ExperimentStep

cache_transform(hyper_model, X_train, y_train, X_test=None, X_eval=None, y_eval=None, **kwargs)[source]
fit_transform(**kwargs)
transform(X, y=None, **kwargs)[source]
hypernets.experiment.compete.evaluate_oofs(hyper_model, ensemble_estimator, y_train, metrics)[source]

hypernets.experiment.general module

class hypernets.experiment.general.GeneralExperiment(hyper_model, X_train, y_train, X_eval=None, y_eval=None, X_test=None, eval_size=0.3, task=None, id=None, callbacks=None, random_state=9527)[source]

Bases: hypernets.experiment._experiment.Experiment

train(hyper_model, X_train, y_train, X_test, X_eval=None, y_eval=None, **kwargs)[source]

Run an experiment

hypernets.experiment.job module

class hypernets.experiment.job.CompeteExperimentJobCreator[source]

Bases: hypernets.experiment.job.ExperimentJobCreator

create_and_run_experiment()[source]
create_experiment_with_params(make_kwargs, job_working_dir)[source]
static set_default_eval_dir(job_working_dir, make_kwargs)[source]
static set_default_render_path(job_working_dir, make_kwargs)[source]
class hypernets.experiment.job.ExperimentJobCreator[source]

Bases: object

create_and_run_experiment()[source]

hypernets.experiment.report module

class hypernets.experiment.report.ExcelReportRender(file_path: str = './report.xlsx', theme='default')[source]

Bases: hypernets.experiment.report.ReportRender

MAX_CELL_LENGTH = 50
static log_skip_sheet(name)[source]
render(experiment_meta: hypernets.experiment._extractor.ExperimentMeta, **kwargs)[source]

Render report data into a excel file

Parameters:
  • experiment_meta – if part of {experiment_report} is empty maybe skip to create sheet.
  • kwargs –
class hypernets.experiment.report.FeatureTrans(feature, method, stage, reason, remark)

Bases: tuple

feature

Alias for field number 0

method

Alias for field number 1

reason

Alias for field number 3

remark

Alias for field number 4

stage

Alias for field number 2

class hypernets.experiment.report.FeatureTransCollector(steps: List[hypernets.experiment._extractor.StepMeta])[source]

Bases: object

METHOD_ADD = 'add'
METHOD_DROP = 'drop'
collect()[source]
get_handler(step_class_name)[source]
class hypernets.experiment.report.ReportRender(**kwargs)[source]

Bases: object

render(experiment_meta: hypernets.experiment._extractor.ExperimentMeta, **kwargs)[source]
class hypernets.experiment.report.Theme(theme_name)[source]

Bases: object

get_header_style()[source]
get_row_diff_style()[source]
hypernets.experiment.report.get_render(name)[source]

Module contents