PyTorch Autotuning API

class TorchApp(app_name, module, tune_dataloader, test_dataloader, knobs, tensor_to_qos, combine_qos=<function mean>, target_device=None, torch_device='cpu', model_storage_folder=None)[source]

Bases: predtuner.modeledapp.ModeledApp, abc.ABC

Adaptor for approximable PyTorch Modules with tensor output.

A TorchApp stores the PyTorch Module, datasets for tuning and calibration, set of available TorchApproxKnob each of which may be applied to some layer in the Module, and the quality of service (QoS) metric of application (e.g., accuracy). It provides empirical tuning and predictive tuning capability, automatically supporting modeledapp.LinearCostModel, modeledapp.QoSModelP1, and modeledapp.QoSModelP2.

In contrast to approxapp.ApproxApp and modeledapp.ModeledApp, there should be no need to inherit from TorchApp in most use cases.

Parameters

  • app_name – Name of the application, which is used as an identifier in tuning sessions, etc.

  • module – The PyTorch module to tune.

  • tune_dataloader – A torch.utils.data.Dataset dataset to use as inputs to module during tuning.

  • test_dataloader – A torch.utils.data.Dataset dataset used for QoS testing (see test_configs parameter of ApproxModeledTuner.tune).

  • knobs – A set of TorchApproxKnob to be considered. Each knob has an is_applicable() method which is used to determine which layer it can apply to. approxes.get_knobs_from_file returns a set of builtin knobs that will exactly fit here.

  • tensor_to_qos – QoS metric function which computes QoS from the module’s output. torchutil.accuracy computes the classification accuracy which can be applied here.

  • combine_qos – A function to combine each batch’s QoS into one value. When QoS is Classification Accuracy, this will most likely be numpy.mean (which is the default value).

  • target_device – The target device that this application should be tuned on.

  • torch_device – The PyTorch device where the model inference is run on. This device should be able to run the implementations of the knobs available for this app on target_device.

  • model_storage_folder – A folder to store the serialized QoS models into. QoSModelP1 will be serialized into model_storage_folder / "p1.pkl", and QoSModelP2 into model_storage_folder / "p2.json".

get_tuner()

Sets up an ApproxTuner instance which the user can directly call tune() on with opentuner parameters.

This returns an ApproxModeledTuner, different from approxapp.ApproxApp.get_tuner which returns an ApproxTuner.

Return type

ApproxModeledTuner

get_knobs_from_file(filepath=default_knob_file, extra_name_to_class=None)[source]

Constructs and returns a set of TorchApproxKnob from a knob declaration file. default_knob_file points to a file that is contained in the predtuner package, so just calling get_knobs_from_file() should provide a set of predefined knobs already.

Parameters

  • filepath (Union[pathlib.Path, str]) – the knob declaration file (JSON) to read from.

  • extra_name_to_class (Dict[str, Type[TorchApproxKnob]]) – a mapping from the name of the approximation to the class (implementation) of the approximation. If not given, only the builtin approximations will be considered when parsing the declaration file.

Return type

Set[TorchApproxKnob]

accuracy(output, target)[source]

The “classification accuracy” metric (return value is between 0 and 100).

Parameters

  • output (torch.Tensor) – Probability distribution output from the model.

  • target (torch.Tensor) – A 1d-tensor of labels, one for each input image, from the dataset.

Return type

float

Defining New Approximation Knobs

class TorchApproxKnob(name, devices=None, baseline_priority=None)[source]

Bases: predtuner.approxapp.ApproxKnob

Defines an approximation knob that knows its own expected speedup ratio and what Modules it can apply to, and can be applied to a torch.nn.Module to return an approximated Module.

abstract apply(op)[source]

Applies knob to a Module and returns an approximated Module.

Parameters

op (torch.nn.Module) – the module to apply approximation on.

Return type

torch.nn.Module

abstract property deterministic

Returns true if approx knob does not contain randomness.

abstract property expected_speedup

The speedup this knob is expected to provide. Used for cost prediction.

abstract is_applicable(op)[source]

Returns True if this knob can be applied to this Module.

Parameters

op (torch.nn.Module) – the module to check availability for.

Return type

torch.nn.Module