chore: import upstream snapshot with attribution

docs/source/core/adapters/adapter_bib.bib

@@ -0,0 +1,21 @@
+
+
+@inproceedings{houlsby2019adapter,
+  title={Parameter-efficient transfer learning for NLP},
+  author={Houlsby, Neil and Giurgiu, Andrei and Jastrzebski, Stanislaw and Morrone, Bruna and De Laroussilhe, Quentin and Gesmundo, Andrea and Attariyan, Mona and Gelly, Sylvain},
+  booktitle={International Conference on Machine Learning},
+  pages={2790--2799},
+  year={2019},
+  organization={PMLR}
+}
+
+@misc{Junxian2021unified,
+  doi = {10.48550/ARXIV.2110.04366},
+  url = {https://arxiv.org/abs/2110.04366},
+  author = {He, Junxian and Zhou, Chunting and Ma, Xuezhe and Berg-Kirkpatrick, Taylor and Neubig, Graham},
+  keywords = {Computation and Language (cs.CL), Machine Learning (cs.LG), FOS: Computer and information sciences, FOS: Computer and information sciences},
+  title = {Towards a Unified View of Parameter-Efficient Transfer Learning},
+  publisher = {arXiv},
+  year = {2021},
+  copyright = {arXiv.org perpetual, non-exclusive license}
+}

docs/source/core/adapters/api.rst

@@ -0,0 +1,66 @@
+Adapters API
+============
+
+Core
+----
+
+.. autoclass:: nemo.core.adapter_mixins.AdapterModuleMixin
+    :show-inheritance:
+    :members:
+    :member-order: bysource
+    :undoc-members: adapter_module_names
+    :noindex:
+
+
+.. autoclass:: nemo.core.adapter_mixins.AdapterModelPTMixin
+    :show-inheritance:
+    :members:
+    :member-order: bysource
+    :undoc-members: adapter_module_names
+    :noindex:
+
+
+Adapter Networks
+----------------
+
+
+.. autoclass:: nemo.collections.common.parts.adapter_modules.AdapterModuleUtil
+    :show-inheritance:
+    :members:
+    :member-order: bysource
+    :noindex:
+
+
+.. autoclass:: nemo.collections.common.parts.adapter_modules.LinearAdapter
+    :show-inheritance:
+    :members:
+    :member-order: bysource
+    :noindex:
+
+
+Adapter Strategies
+------------------
+
+
+.. autoclass:: nemo.core.classes.mixins.adapter_mixin_strategies.AbstractAdapterStrategy
+    :show-inheritance:
+    :members:
+    :member-order: bysource
+    :undoc-members: adapter_module_names
+    :noindex:
+
+
+.. autoclass:: nemo.core.classes.mixins.adapter_mixin_strategies.ReturnResultAdapterStrategy
+    :show-inheritance:
+    :members:
+    :member-order: bysource
+    :undoc-members: adapter_module_names
+    :noindex:
+
+
+.. autoclass:: nemo.core.classes.mixins.adapter_mixin_strategies.ResidualAddAdapterStrategy
+    :show-inheritance:
+    :members:
+    :member-order: bysource
+    :undoc-members: adapter_module_names
+    :noindex:

docs/source/core/adapters/components.rst

@@ -0,0 +1,94 @@
+Adapter Components
+==================
+
+Adapters can be considered as any set of parameters that are added to a pre-existing module/model. In our case, we currently support the standard adapter in literature, more advanced adapter modules are being researched and can potentially be supported by NeMo.
+
+An adapter module can be any pytorch module, but it must follow certain straightforward requirements -
+
+1) The model accepts an input of some input dimension, and its output must match this dimension.
+2) Ideally, the module is initialized such that the output of the adapter when initialized is such that it does not modify the original input. This allows the model to produce the same output results, even when additional parameters have been added.
+
+According to Junxian et al :cite:`adapters-components-Junxian2021unified`, we can consider an adapter being represented as three components -
+
+1) Functional form - the trainable parameters that will modify the input
+2) Insertion form - Where the adapter outputs are integrated with the original input. The input to the adapters can be the last output of the layer, the input to some attention layer, or even the original input to the module itself (before even the modules forward pass).
+3) Composition function - How the adapters outputs are integrated with the inputs. It can be as simple as residual addition connection, or concatenation, or point-wise multiplication etc.
+
+Functional Form - Adapter Networks
+==================================
+
+Adapter modules represent the functional form of the adapter. We discuss an example of a most commonly used adapter module found in literature, titled the ``LinearAdapter`` (or Houlsby Adapter) :cite:`adapters-components-houlsby2019adapter`.
+
+.. note::
+
+    All adapter modules must extend :class:`~nemo.collections.common.parts.adapter_modules.AdapterModuleUtil` and should ideally have an equivalent DataClass config for easy instantiation !
+
+
+.. autoclass:: nemo.collections.common.parts.adapter_modules.AdapterModuleUtil
+    :show-inheritance:
+    :members:
+    :member-order: bysource
+    :noindex:
+
+-----
+
+.. autoclass:: nemo.collections.common.parts.adapter_modules.LinearAdapter
+    :show-inheritance:
+    :members:
+    :member-order: bysource
+    :noindex:
+
+
+Insertion Form - Module Adapters
+--------------------------------
+
+Adapter modules can be integrated into many different locations of a given module. For example, it is possible to have an adapter that affects only the outputs of the final layer in each module. We can also have a ``Parallel Adapter`` :cite:`adapters-components-Junxian2021unified` that operates at the input of the module itself, in parallel to the forward pass of the module. Yet another insertion location is inside the Multi Head Attention Layers.
+
+On top of this, while adapters are commonly used only in the layers containing the most parameters (say the Encoder of a network), some models can support adapters in multiple locations (Encoder-Decoder architecture for Language Models, Machine Translation, or even Encoder-Decoder-Joint for ASR with Transducer Loss). As such, NeMo utilizes the concept of ``Module Adapters``.
+
+``Module Adapters`` are very simply defined when adding an adapter - by specifying the module that the adapter should be inserted into.
+
+.. code-block:: python
+
+    # Get the list of supported modules / locations in a adapter compatible Model
+    print(model.adapter_module_names)  # assume ['', 'encoder', 'decoder']
+
+    # When calling add_adapter, specify the module name in the left of the colon symbol, and the adapter name afterwords.
+    # The adapter is then directed to the decoder module instead of the default / encoder module.
+    model.add_adapter("decoder:first_adapter", cfg=...)
+
+You might note that ``model.adapter_module_names`` can sometimes return ``''`` as one of the supported module names - this refers to the "default module". Generally we try to provide the default as the most commonly used adapter in literature - for example, Encoder adapters in NLP/NMT/ASR.
+
+Composition Function - Adapter Strategies
+-----------------------------------------
+
+Finally, we discuss how to compose the input and output of adapter modules. In order to generalize this step, we construct ``Adapter Strategies``.
+A strategy is any class (not torch.nn.Module!) that extends :class:`~nemo.core.classes.mixins.adapter_mixin_strategies.AbstractAdapterStrategy`, and provides a ``forward()`` method that accepts a specific signature of the inputs and produces an output tensor which combines the input and output with some specific method.
+
+We discuss a simple residual additional connection strategy below - that accepts an input to the adapter and an adapters output and simply adds them together. It also supports ``stochastic_depth`` which enables adapters to be dynamically switched off during training, making training more robust.
+
+.. autoclass:: nemo.core.classes.mixins.adapter_mixin_strategies.AbstractAdapterStrategy
+    :show-inheritance:
+    :members:
+    :member-order: bysource
+    :undoc-members: adapter_module_names
+    :noindex:
+
+-----
+
+.. autoclass:: nemo.core.classes.mixins.adapter_mixin_strategies.ResidualAddAdapterStrategy
+    :show-inheritance:
+    :members:
+    :member-order: bysource
+    :undoc-members: adapter_module_names
+    :noindex:
+
+-----
+
+
+References
+----------
+
+.. bibliography:: ./adapter_bib.bib
+    :style: plain
+    :keyprefix: adapters-components-

docs/source/core/adapters/intro.rst

@@ -0,0 +1,149 @@
+Adapters
+========
+
+In NeMo, we often train models and fine-tune them for a specific task. This is a reasonable approach when the models are just a few million parameters. However, this approach quickly becomes infeasible when approaching hundreds of millions or even billions of parameters. As a potential solution to such a scenario, where fine-tuning a massive model is no longer feasible, we look to `Adapters <https://arxiv.org/abs/1902.00751>`_ :cite:`adapters-houlsby2019adapter` to specialize our model on a specific domain or task. Adapters require a fraction of the total number of parameters as the original model and are much more efficient to fine-tune.
+
+.. note::
+
+    For a detailed tutorial on adding ``Adapter`` support to any PyTorch module, please refer to the NeMo Adapter :doc:`Tutorials <../../starthere/tutorials>`.
+
+
+What are Adapters?
+------------------
+
+Adapters are a straightforward concept - one formulation can be shown by the diagram below. At their simplest, they are residual Feedforward layers that compress the input dimension (:math:`D`) to a small bottleneck dimension (:math:`H`), such that :math:`R^D \text{->} R^H`, compute an activation (such as ReLU), finally mapping :math:`R^H \text{->} R^D` with another Feedforward layer. This output is then added to the input via a simple residual connection.
+
+.. raw:: html
+
+    <div align="center">
+      <img src="https://mermaid.ink/img/pako:eNptkLFqwzAQhl9F3ORAPDSjA4EUx6RgXEjbycpwWOdG1JaMfEoakrx7ZcfpUKrlxH_fz4d0gcoqggTqxp6qAzoW76k0Ipx1-WI6z3sRxyuRF1GOZ3KisK6d3YG8GFdZ9hRJeLbMDRmqvkRGpDLrTuiUiEWUigBtlyIVqzBnEqZ66I39dcX6iKytKXeUf-wn-286QoFeBMvmu0PTD-EfyXaQpP9JFmP_1XN4S3kfD8W4ue6o18pjc52gYQlzaMm1qFX4msuQSOADtSQhCdfaOupZgjS3QPpOIdNGabYOkhqbnuaAnu3b2VSQsPP0gFKNnw7bibr9AJkZdXU" height=100% />
+    </div>
+
+Adapter modules such as this are usually initialized such that the initial output of the adapter will always be zeros so as to prevent degradation of the original model's performance due to addition of such modules.
+
+``torch.nn.Module`` with Adapters
+---------------------------------
+
+In NeMo, Adapters are supported via a ``Mixin`` class that can be attached to any ``torch.nn.Module``. Such a module will
+have multiple additional methods which will enable adapter capabilities in that module.
+
+.. code-block:: python
+
+    # Import the adapter mixin from NeMo
+    from nemo.core import adapter_mixins
+
+    # NOTE: See the *two* classes being inherited here !
+    class MyModule(torch.nn.Module, adapter_mixins.AdapterModuleMixin):
+        pass
+
+
+AdapterModuleMixin
+------------------
+Let's look into what :class:`~nemo.core.adapter_mixins.AdapterModuleMixin` adds to the general PyTorch module. Some of the most important methods that are required are listed below :
+
+1) ``add_adapter``: Used to add an adapter with a unique name to the module.
+2) ``get_enabled_adapters``: Returns a list of names of all enabled adapter modules.
+3) ``set_enabled_adapters``: Sets whether a single (or all) adapters are enabled or disabled.
+4) ``is_adapter_available``: Check if any adapter is available and enabled or not.
+
+Modules that extend this mixin usually can directly use these methods without extending them, but we will cover a case below
+where you may wish to extend these methods.
+
+.. autoclass:: nemo.core.adapter_mixins.AdapterModuleMixin
+    :show-inheritance:
+    :members:
+    :member-order: bysource
+    :undoc-members: adapter_module_names
+
+
+Using the Adapter Module
+------------------------
+
+Now that ``MyModule`` supports adapters, we can easily add adapters, set their state, check if they are available and
+perform their forward pass. Note that if multiple adapters are enabled, they are called in a chain, the output of the previous adapter is passed as input to the next adapter and so on.
+
+.. code-block:: python
+
+    # Import the adapter mixin and modules from NeMo
+    import torch
+    from nemo.core import adapter_mixins
+    from nemo.collections.common.parts import adapter_modules
+
+    class MyModule(torch.nn.Module, adapter_mixins.AdapterModuleMixin):
+
+        def forward(self, x: torch.Tensor) -> torch.Tensor:
+            output = self.layers(x)  # assume self.layers is some Sequential() module
+
+            if self.is_adapter_available():  # check if adapters were added or not
+                output = self.forward_enabled_adapters()  # perform the forward of all enabled adapters in a chain
+
+            return output
+
+    # Now let us create a module, add an adapter and do a forward pass with some random inputs
+    module = MyModule(dim)  # assume dim is some input and output dimension of the module.
+
+    # Add an adapter
+    module.add_adapter("first_adapter", cfg=adapter_modules.LinearAdapter(in_features=dim, dim=5))
+
+    # Check if adapter is available
+    module.is_adapter_available()  # returns True
+
+    # Check the name(s) of the enabled adapters
+    module.get_enabled_adapters()  # returns ['first_adapter']
+
+    # Set the state of the adapter (by name)
+    module.set_enabled_adapters(name="first_adapter", enabled=True)
+
+    # Freeze all the weights of the original module  (equivalent to calling module.freeze() for a NeuralModule)
+    for param in module.parameters():
+        param.requires_grad = False
+
+    # Unfreeze only the adapter weights (so that we finetune only the adapters and not the original weights !)
+    module.unfreeze_enabled_adapters()
+
+    # Now you can train this model's adapters !
+    input_data = torch.randn(4, dim)  # assume dim is the input-output dim of the module
+    outputs_with_adapter = module(input_data)
+
+    # Compute loss and backward ...
+
+
+Adapter Compatible Models
+-------------------------
+
+If the goal was to support adapters in a single module, then the goal has been accomplished. In the real world however, we
+build large composite models out of multiple modules and combine them to build a final model that we then train. We do this using the
+:class:`~nemo.core.adapter_mixins.AdapterModelPTMixin`.
+
+.. note::
+
+    For an in-depth guide to supporting hierarchical adapter modules, please refer to the NeMo Adapter :doc:`Tutorials <../../starthere/tutorials>`.
+
+
+.. autoclass:: nemo.core.adapter_mixins.AdapterModelPTMixin
+    :show-inheritance:
+    :members:
+    :member-order: bysource
+    :undoc-members: adapter_module_names
+
+Below, we will discuss some useful functionality of Adapter Compatible Models.
+
+1) ``Save and restore a Model with adapter capability``: Any NeMo model that implements this class correctly can save and restore NeMo models with adapter capabilities, thereby allowing sharing of adapters.
+2) ``save_adapters`` and ``load_adapters``: Adapters are usually a very small number of parameters, there is no need for the entire model to be duplicated for each adapter. This method allows storing just the adapter module(s) separately from the Model, so that you can use the same "base" model, and share just the Adapter modules.
+
+
+.. toctree::
+   :maxdepth: 8
+   :caption: Adapters
+
+   components
+   api
+
+
+References
+----------
+
+.. bibliography:: ./adapter_bib.bib
+    :style: plain
+    :labelprefix: adapters
+    :keyprefix: adapters-