- Feature Name: API interface - Start Date: 2018-10-30) - RFC PR: - Salt Issue: # Summary [summary]: #summary Salt Module Interface (SMI) concept introduction for virtual modules. # Motivation [motivation]: #motivation Any salt module has couple of specific properties we dealing every day with: - Fixed set of functions - Known functions signatures - Known structure of return This is true but for virtual modules. The virtual module covering several "fixed" or "physical" modules and behaves like that would be one module. But differences between those physical modules on different platforms makes such virtual module a moving target and unpredictable. Virtual modules concept is missing crucial part in the design: interfaces. The interface should define how module looks like and what APIs can be called to it. Interface should move module that is _called_ differently on heterogeneous environments to a module that _reports_ differently on heterogeneous environments. # Design [design]: #detailed-design *DISCLAIMER*: The SMI is not that classic understanding of typical interface one may find in languages like Java. It is also not as same as Zope Interface package or Python Abstract Base Classes (ABC). The SMI should describe the following properties of the module: - Functions - Signatures - Lowest common denominator of the function output format (or minimum required default output structure) SMI only describes functions of the module and is there to make sure that any virtual module is always called exactly the same way, regardless what operating system minion is running on. ## Declaration [declaration]: #declaration SMI are declared just as regular Python classes. Salt's "module to functions" map is Salt's "interface to methods" map. Therefore `self` parameter in the SMI class is not a part of a function signature. Example of SMI definition for module `pkg`: ```python from salt.interfaces import Interface class PkgInterface(Interface): __modulename__ = 'pkg' def list_installed(self, *names, **kwargs): ''' List installed packages. ''' return {} def upgrade_available(self, name, **kwargs): ''' List available upgrades. ''' return {} @Interface.supported(os=['weirdlinux', 'beos', 'frogbsd'], os_family=['linux']) def salute_fireworks(self, name): ''' Launch some fireworks ''' return {} ``` In above incomplete interface example, the list of methods should reflect exact names and signatures as in the module, except `self` parameter. Rules apply: - If a method is not in the SMI class, but function is implemented in the module, then such function is marked as "deprecated". - If a method is in the SMI class but not in the module, then such function is marked as "not implemented". - If a method has a decorator `@Interface.supported`, only on specified systems unimplemented method will be reported as "not implemented", otherwise "not supported". This decorator accepts any grains possible. It then matches them if _any_ specified grain is in proposed lists. From the example above, missing `salute_fireworks` will be reported as "not implemented" if `os_family` grain equals `linux` **or** `os` grain equals `weirdlinux` or `beos` or `frogbsd`. ## Usage [usage]: #usage Once SMI class defined, the usage should be very simple: ```python from salt.interfaces.pkg_module import PkgInterface __virtualname__ = PkgInterface(__name__)() ``` The code above does the following: - Ensures that the `__virtualname__` is properly set according to the interface. - Performs check for the entire module and automatically unifies it to the rules in the "Declaration" section above by adding stub functions that would raise corresponding exceptions or wrap/decorate existing "illegal" functions as "deprecated". ## Effect [effect]: #effect Essentially, the SMI works as automatic checker/corrector for the module on the moment it is lazy-loaded. What PkgInterface does in the example above, it takes the current module and examines if the exported functions are there. Once nothing found, a stub is placed. That means, if module `pkg` requires, e.g. function `lock` and there is implemented `hold`, then function `lock`will be _also_ added as "not implemented" (or "unsupported", depends on decorator in the Interface declaration). SMI will also mark existing functions that are not inside the interface as subject to retirement, by automatically placing a warning decorator to them. That said, if an interface class does not describes `hold` function, but that function is still physically implemented, calling that function will also raise a warning in the log file that this function is deprecated and is subject to be removed in a future. ## Not applicable functions [notapplicable]: #notapplicable On some operating systems certain functions aren't applicable. In this case they should be decorated with the proposed function decorator: ```python class SomeModuleInterface(Interface): @Interface.not_applicable(osfamily=['Windows', 'NetBSD']) def foo(self, name, *args): return {} ``` The decorator would support _any_ kind of grains keys with any of the values to compare with. Once certain grain matches in the list of the given values, decorator is triggered. In this case method `foo` will be still added on Windows and NetBSD minions, despite the fact that the code below adds it only on RedHat Linux. However it will only return specified structure and debug log will inform that not applicable function has been called. Such decorator deals with the cases, where function is being added to the module only on certain conditions, e.g.: ```python if __grains__['osfamily'] == 'RedHat': def foo(name, *args): return {} ``` ## Return Structure Definition [returnstruct]: #returnstruct Return structure in virtual modules is another pitfall. Dynamically replaced module suddenly renders virtual module to return "something else" than is usually expected. This is widely affects API and integration. To the only way to avoid this, is to know what kind of platform minion is dealing with. In this case integration code usually looks like this (pseudo-code): ``` if this_is_debian { function_call({'disabled': False}) else { function_call({'enabled': True}) } ``` There is a catch: some operating systems/platforms _must_ return specific properties that aren't available on other systems. Therefore return structure should be always defined from two blocks: - *Minimal common data.* This comes from every platform, even if this is only one value. This data should be available on _all Salt supported_ platforms. This group must be defined in the Interface. - *Extra specific data.* This comes from a specific platform that is not be available on _all other_ platforms, even if this data might be _also available_ on other platforms. This group is always coming additionally to the basic one and is _not_ part of the interface. SMI class should define return structure from the defined method. This structure is very similar to `config/__init__.py::_validate_opts()` function. SMI also should take care of return structure definition so all virtual modules returns by default the same structure. However, the migration and adoption of the same structure from different physical modules is not easy. Modules are also called through the states and there is already specific structure is used. The usage would not change, but the implementation would be to wrap all functions with a decorator, which would validate the default output. This RFC is not to cover the detailed output structure part, but only foresee a placeholder for it the in current design of the Interface concept. ## Unresolved questions and known possible solutions [unresolved]: #unresolved-questions - Should be confugrable function deprecation while aligning module with the interface? If some function happens to be an alien to the interface, question is how to react on this. Muting and do not report function is obsolete is still asking for a problem. Because if we know that in N years/releases function is going to be retired, simply just do not use it or move away from it. But if this is configured and can be muted, such option will bring more harm than help. - Which path do we choose here to make sure interface is used all the time? One of the possibility is to expect Interface class instance in `__virtualname__` variable, instead of a string. In this case `__call__` is not performed right in the module, but LazyLoader instead gets the `__modulename__` variable content. Another possibility is to adjust PyLint to it and make sure each `__virtualname__` has Interface assigned instead of a string. Alternatively, not to force Interface usage. But this has drawback of setting the interface overall optional, which will eventually be optional everywhere, unfortunately. ## Hints [hints]: #hints To generate an interface out of the signatures of some package, it is just enough to take a reference package and do something like this: cat zypper.py | grep '^def [a-z]' | sed -e 's/(/(self, /g' | sed -e 's/def/ def/g' It will create ready to copy signatures, based on `zypper.py` as a reference. # Strategy [strategy]: #strategy Implementation of this concept must be done in two phases: 1. Implementation of the very mechanism. 2. Migrating module by module in a transparent way. On the second phrase corner cases might force the implementation details to be minor changed. The result, however should be the same: modules should just work as they worked before while used in real systems. The structure definition and migration should be done as well gradually. This should be covered in a separate RFC.