wehub/tensorflow--tensorflow
1
0
Fork 0
Code Issues Pull Requests Actions 3 Deployments Models Agent Notes Packages Projects Releases Wiki Activity

chore: import upstream snapshot with attribution
cffconvert / validate (push) Has been skipped
Details
License Check / license-check (push) Failing after 2s
Details

Browse Source
This commit is contained in:
wehub-resource-sync
2026-07-13 12:14:16 +08:00
commit 8a852e4b4e
36502 changed files with 9277225 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tensorflow/python/autograph/g3doc/pyct_tutorial.ipynb
+482
View File
@@ -0,0 +1,482 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "fWfkYsCgPvqR"
},
"source": [
"# Short intro to the SCT library of AutoGraph\n",
"\n",
"**Work in progress, use with care and expect changes.**\n",
"\n",
"The `pyct` module packages the source code transformation APIs used by AutoGraph.\n",
"\n",
"This tutorial is just a preview - there is no PIP package yet, and the API has not been finalized, although most of those shown here are quite stable.\n",
"\n",
"[Run in Colab](https://colab.research.google.com/github/tensorflow/tensorflow/blob/master/tensorflow/python/autograph/g3doc/pyct_tutorial.ipynb)\n",
"\n",
"Requires `tf-nightly`:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "wq1DRamRlqoB"
},
"outputs": [],
"source": [
"!pip install tf-nightly"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "r7Q78WIKe2cu"
},
"source": [
"### Writing a custom code translator\n",
"\n",
"[transformer.CodeGenerator](https://github.com/tensorflow/tensorflow/blob/40802bcdb5c8a4379da2145441f51051402bd29b/tensorflow/python/autograph/pyct/transformer.py#L480) is an AST visitor that outputs a string. This makes it useful in the final stage of translating Python to another language."
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "HHaCMFOpuoVx"
},
"source": [
"Here's a toy C++ code generator written using a `transformer.CodeGenerator`, which is just a fancy subclass of [ast.NodeVisitor](https://docs.python.org/3/library/ast.html#ast.NodeVisitor):"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "PJlTIbJlurpm"
},
"outputs": [],
"source": [
"import gast\n",
"from tensorflow.python.autograph.pyct import transformer\n",
"\n",
"class BasicCppCodegen(transformer.CodeGenerator):\n",
"\n",
" def visit_Name(self, node):\n",
" self.emit(node.id)\n",
"\n",
" def visit_arguments(self, node):\n",
" self.visit(node.args[0])\n",
" for arg in node.args[1:]:\n",
" self.emit(', ')\n",
" self.visit(arg)\n",
"\n",
" def visit_FunctionDef(self, node):\n",
" self.emit('void {}'.format(node.name))\n",
" self.emit('(')\n",
" self.visit(node.args)\n",
" self.emit(') {\\n')\n",
" self.visit_block(node.body)\n",
" self.emit('\\n}')\n",
"\n",
" def visit_Call(self, node):\n",
" self.emit(node.func.id)\n",
" self.emit('(')\n",
" self.visit(node.args[0])\n",
" for arg in node.args[1:]:\n",
" self.emit(', ')\n",
" self.visit(arg)\n",
" self.emit(');')\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "chCX1A3rA9Pn"
},
"source": [
"Another helpful API is [transpiler.GenericTranspiler](https://github.com/tensorflow/tensorflow/blob/ee7172a929cb0c3d94a094fafc60bbaa175c085d/tensorflow/python/autograph/pyct/transpiler.py#L227) which takes care of parsing:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "LmwWewU1Bw0B"
},
"outputs": [],
"source": [
"import gast\n",
"from tensorflow.python.autograph.pyct import transpiler\n",
"\n",
"class PyToBasicCpp(transpiler.GenericTranspiler):\n",
"\n",
" def transform_ast(self, node, ctx):\n",
" codegen = BasicCppCodegen(ctx)\n",
" codegen.visit(node)\n",
" return codegen.code_buffer"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "nUhlScyOjlYM"
},
"source": [
"Try it on a simple function:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "ty9q853QvUqo"
},
"outputs": [],
"source": [
"def f(x, y):\n",
" print(x, y)\n",
"\n",
"code, _ = PyToBasicCpp().transform(f, None)\n",
"print(code)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "rmRI9dG_ydE_"
},
"source": [
"### Helpful static analysis passes\n",
"\n",
"The `static_analysis` module contains various helper passes for dataflow analysis.\n",
"\n",
"All these passes annotate the AST. These annotations can be extracted using [anno.getanno](https://github.com/tensorflow/tensorflow/blob/40802bcdb5c8a4379da2145441f51051402bd29b/tensorflow/python/autograph/pyct/anno.py#L111). Most of them rely on the `qual_names` annotations, which just simplify the way more complex identifiers like `a.b.c` are accessed.\n",
"\n",
"The most useful is the activity analysis which just inventories symbols read, modified, etc.:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "GEJ30Wea4Xfy"
},
"outputs": [],
"source": [
"def get_node_and_ctx(f):\n",
" node, source = parser.parse_entity(f, ())\n",
" f_info = transformer.EntityInfo(\n",
" name='f',\n",
" source_code=source,\n",
" source_file=None,\n",
" future_features=(),\n",
" namespace=None)\n",
" ctx = transformer.Context(f_info, None, None)\n",
" return node, ctx"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "BiwPJrDd0aAX"
},
"outputs": [],
"source": [
"from tensorflow.python.autograph.pyct import anno\n",
"from tensorflow.python.autograph.pyct import parser\n",
"from tensorflow.python.autograph.pyct import qual_names\n",
"from tensorflow.python.autograph.pyct.static_analysis import annos\n",
"from tensorflow.python.autograph.pyct.static_analysis import activity\n",
"\n",
"\n",
"def f(a):\n",
" b = a + 1\n",
" return b\n",
"\n",
"\n",
"node, ctx = get_node_and_ctx(f)\n",
"\n",
"node = qual_names.resolve(node)\n",
"node = activity.resolve(node, ctx)\n",
"\n",
"fn_scope = anno.getanno(node, annos.NodeAnno.BODY_SCOPE) # Note: tag will be changed soon.\n",
"\n",
"\n",
"print('read:', fn_scope.read)\n",
"print('modified:', fn_scope.modified)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "w8dBRlKkFNIP"
},
"source": [
"Another useful utility is the control flow graph builder.\n",
"\n",
"Of course, a CFG that fully accounts for all effects is impractical to build in a late-bound language like Python without creating an almost fully-connected graph. However, one can be reasonably built if we ignore the potential for functions to raise arbitrary exceptions."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "KvLe9lWnFg7N"
},
"outputs": [],
"source": [
"from tensorflow.python.autograph.pyct import cfg\n",
"\n",
"\n",
"def f(a):\n",
" if a > 0:\n",
" return a\n",
" b = -a\n",
"\n",
"node, ctx = get_node_and_ctx(f)\n",
"\n",
"node = qual_names.resolve(node)\n",
"cfgs = cfg.build(node)\n",
"cfgs[node]"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "Cro-jfPA2oxR"
},
"source": [
"Other useful analyses include liveness analysis. Note that these make simplifying assumptions, because in general the CFG of a Python program is a graph that's almost complete. The only robust assumption is that execution can't jump backwards."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "73dARy4_2oAI"
},
"outputs": [],
"source": [
"from tensorflow.python.autograph.pyct import anno\n",
"from tensorflow.python.autograph.pyct import cfg\n",
"from tensorflow.python.autograph.pyct import qual_names\n",
"from tensorflow.python.autograph.pyct.static_analysis import annos\n",
"from tensorflow.python.autograph.pyct.static_analysis import reaching_definitions\n",
"from tensorflow.python.autograph.pyct.static_analysis import reaching_fndefs\n",
"from tensorflow.python.autograph.pyct.static_analysis import liveness\n",
"\n",
"\n",
"def f(a):\n",
" b = a + 1\n",
" return b\n",
"\n",
"\n",
"node, ctx = get_node_and_ctx(f)\n",
"\n",
"node = qual_names.resolve(node)\n",
"cfgs = cfg.build(node)\n",
"node = activity.resolve(node, ctx)\n",
"node = reaching_definitions.resolve(node, ctx, cfgs)\n",
"node = reaching_fndefs.resolve(node, ctx, cfgs)\n",
"node = liveness.resolve(node, ctx, cfgs)\n",
"\n",
"print('live into `b = a + 1`:', anno.getanno(node.body[0], anno.Static.LIVE_VARS_IN))\n",
"print('live into `return b`:', anno.getanno(node.body[1], anno.Static.LIVE_VARS_IN))"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "GKSaqLbKQI_v"
},
"source": [
"### Writing a custom Python-to-Python transpiler\n",
"\n",
"`transpiler.Py2Py` is a generic class for a Python [source-to-source compiler](https://en.wikipedia.org/wiki/Source-to-source_compiler). It operates on Python ASTs. Subclasses override its [transform_ast](https://github.com/tensorflow/tensorflow/blob/95ea3404528afcb1a74dd5f0946ea8d17beda28b/tensorflow/python/autograph/pyct/transpiler.py#L261) method.\n",
"\n",
"Unlike the `transformer` module, which have an AST as input/output, the `transpiler` APIs accept and return actual Python objects, handling the tasks associated with parsing, unparsing and loading of code."
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "eicHoYlzRhnc"
},
"source": [
"Here's a transpiler that does nothing:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "edaG6dWEPvUI"
},
"outputs": [],
"source": [
"from tensorflow.python.autograph.pyct import transpiler\n",
"\n",
"\n",
"class NoopTranspiler(transpiler.PyToPy):\n",
"\n",
" def get_caching_key(self, ctx):\n",
" # You may return different caching keys if the transformation may generate\n",
" # code versions.\n",
" return 0\n",
"\n",
" def get_extra_locals(self):\n",
" # No locals needed for now; see below.\n",
" return {}\n",
"\n",
" def transform_ast(self, ast, transformer_context):\n",
" return ast\n",
"\n",
"tr = NoopTranspiler()"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "hKxmlWeQSQyN"
},
"source": [
"The main entry point is the [transform](https://github.com/tensorflow/tensorflow/blob/95ea3404528afcb1a74dd5f0946ea8d17beda28b/tensorflow/python/autograph/pyct/transpiler.py#L384) method returns the transformed version of the input."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "HXTIYsunSVr1"
},
"outputs": [],
"source": [
"def f(x, y):\n",
" return x + y\n",
"\n",
"\n",
"new_f, module, source_map = tr.transform(f, None)\n",
"\n",
"new_f(1, 1)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "aKO42LBXw3SD"
},
"source": [
"### Adding new variables to the transformed code\n",
"\n",
"The transformed function has the same global and local variables as the original function. You can of course generate local imports to add any new references into the generated code, but an easier method is to use the `get_extra_locals` method:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "_Wl0n5I_1NJZ"
},
"outputs": [],
"source": [
"from tensorflow.python.autograph.pyct import parser\n",
"\n",
"\n",
"class HelloTranspiler(transpiler.PyToPy):\n",
"\n",
" def get_caching_key(self, ctx):\n",
" return 0\n",
"\n",
" def get_extra_locals(self):\n",
" return {'name': 'you'}\n",
"\n",
" def transform_ast(self, ast, transformer_context):\n",
" print_code = parser.parse('print(\"Hello\", name)')\n",
" ast.body = [print_code] + ast.body\n",
" return ast\n",
"\n",
"\n",
"def f(x, y):\n",
" pass\n",
"\n",
"new_f, _, _ = HelloTranspiler().transform(f, None)\n",
"\n",
"_ = new_f(1, 1)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "JcMSHJXK6pO2"
},
"outputs": [],
"source": [
"import inspect\n",
"\n",
"print(inspect.getsource(new_f))"
]
}
],
"metadata": {
"colab": {
"collapsed_sections": [],
"last_runtime": {
"build_target": "",
"kind": "local"
},
"name": "pyctr_tutorial.ipynb",
"provenance": [
{
"file_id": "1dT93XRkt7vUpVp7GZech8LB0u1OytKff",
"timestamp": 1586205976756
}
]
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
tensorflow/python/autograph/g3doc/reference/_control_flow_tutorial.ipynb
+862
View File
@@ -0,0 +1,862 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "-vLwpT31YOJk"
},
"source": [
"TODO(b/138297412): This colab retains some useful code snippets and demonstrations that used to be in the tf.function/AutoGraph customization tutorial, and should be rolled into the existing docs as part of a broader markdown->colab conversion."
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "otIdN1TS8N7S"
},
"outputs": [],
"source": [
"import tensorflow as tf"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "I0xDjO4SHLUD"
},
"source": [
"Define a helper function to demonstrate the kinds of errors you might encounter:"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "D25apou9IOXa"
},
"outputs": [],
"source": [
"import traceback\n",
"import contextlib\n",
"\n",
"# Some helper code to demonstrate the kinds of errors you might encounter.\n",
"@contextlib.contextmanager\n",
"def assert_raises(error_class):\n",
" try:\n",
" yield\n",
" except error_class as e:\n",
" print('Caught expected exception \\n {}:'.format(error_class))\n",
" traceback.print_exc(limit=2)\n",
" except Exception as e:\n",
" raise e\n",
" else:\n",
" raise Exception('Expected {} to be raised but no error was raised!'.format(\n",
" error_class))"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "5f05Vr_YBUCz"
},
"source": [
"## Using AutoGraph\n",
"\n",
"The [autograph](https://www.tensorflow.org/guide/function) library is fully integrated with `tf.function`, and it will rewrite conditionals and loops which depend on Tensors to run dynamically in the graph.\n",
"\n",
"`tf.cond` and `tf.while_loop` continue to work with `tf.function`, but code with control flow is often easier to write and understand when written in imperative style."
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "xgKmkrNTZSyz"
},
"source": [
"## AutoGraph: Conditionals\n",
"\n",
"AutoGraph will convert `if` statements into the equivalent `tf.cond` calls.\n",
"\n",
"This substitution is made if the condition is a Tensor. Otherwise, the conditional is executed during tracing."
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "20WlM9T2I9EV"
},
"source": [
"Here is a function that checks if the resulting graph uses `tf.cond`:"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "E-7KllizZYsy"
},
"outputs": [],
"source": [
"def test_tf_cond(f, *args):\n",
" g = f.get_concrete_function(*args).graph\n",
" if any(node.name == 'cond' for node in g.as_graph_def().node):\n",
" print(\"{}({}) uses tf.cond.\".format(\n",
" f.__name__, ', '.join(map(str, args))))\n",
" else:\n",
" print(\"{}({}) executes normally.\".format(\n",
" f.__name__, ', '.join(map(str, args))))\n",
"\n",
" print(\" result: \",f(*args).numpy())"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "DlqiutEEJHOe"
},
"source": [
"This substitution is made if the condition is a Tensor. Otherwise, the conditional is executed during tracing.\n",
"\n",
"Passing a python `True` executes the conditional normally:"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "fCMywOXwJLIQ"
},
"outputs": [],
"source": [
"@tf.function\n",
"def dropout(x, training=True):\n",
" if training:\n",
" x = tf.nn.dropout(x, rate=0.5)\n",
" return x"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "68D2RZ17JM8u"
},
"outputs": [],
"source": [
"test_tf_cond(dropout, tf.ones([10], dtype=tf.float32), True)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "WEz0QYucJPBa"
},
"source": [
"But passing a tensor replaces the python `if` with a `tf.cond`:"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "o86paGR-Zadi"
},
"outputs": [],
"source": [
"test_tf_cond(dropout, tf.ones([10], dtype=tf.float32), tf.constant(True))"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "5xFLfdApZh8q"
},
"source": [
"`tf.cond` has a number of subtleties.\n",
"\n",
"it works by tracing both sides of the conditional, and then choosing the appropriate branch at runtime, depending on the condition. Tracing both sides can result in unexpected execution of Python code."
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "VTMoZEVaZiwk"
},
"outputs": [],
"source": [
"@tf.function\n",
"def f(x):\n",
" if x > 0:\n",
" x = x + 1.\n",
" print(\"Tracing `then` branch\")\n",
" else:\n",
" x = x - 1.\n",
" print(\"Tracing `else` branch\")\n",
" return x"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "HqBVIZWb0Qzn"
},
"outputs": [],
"source": [
"f(-1.0).numpy()"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "BIMfbXlW0QdP"
},
"outputs": [],
"source": [
"f(1.0).numpy()"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "2nBnJ42v0Pvq"
},
"outputs": [],
"source": [
"f(tf.constant(1.0)).numpy()"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "zyzzvtN5Jfpb"
},
"source": [
"It requires that if one branch creates a tensor used downstream, the other branch must also create that tensor."
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "k_dxWHeFZlaQ"
},
"outputs": [],
"source": [
"@tf.function\n",
"def f():\n",
" if tf.constant(True):\n",
" x = tf.ones([3, 3])\n",
" return x\n",
"\n",
"# Throws an error because both branches need to define `x`.\n",
"with assert_raises(ValueError):\n",
" f()"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "wP-LZP6cztnu"
},
"source": [
"If you want to be sure that a particular section of control flow is never converted by autograph, then explicitly convert the object to a python type so an error is raised instead: "
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "iG_VDavjzrzV"
},
"outputs": [],
"source": [
"@tf.function\n",
"def f(x, y):\n",
" if bool(x):\n",
" y = y + 1.\n",
" print(\"Tracing `then` branch\")\n",
" else:\n",
" y = y - 1.\n",
" print(\"Tracing `else` branch\")\n",
" return y"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "kQ4CRP9T0rH2"
},
"outputs": [],
"source": [
"f(True, 0).numpy()"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "ww9tCzHy0rkv"
},
"outputs": [],
"source": [
"f(False, 0).numpy()"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "ppuV7iug0r7i"
},
"outputs": [],
"source": [
"with assert_raises(TypeError):\n",
" f(tf.constant(True), 0.0)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "yho4J0a0ZkQS"
},
"source": [
"## AutoGraph and loops\n",
"\n",
"AutoGraph has a few simple rules for converting loops.\n",
"\n",
"- `for`: Convert if the iterable is a tensor\n",
"- `while`: Convert if the while condition depends on a tensor\n",
"\n",
"If a loop is converted, it will be dynamically unrolled with `tf.while_loop`, or in the special case of a `for x in tf.data.Dataset`, transformed into `tf.data.Dataset.reduce`.\n",
"\n",
"If a loop is _not_ converted, it will be statically unrolled "
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "OyzGNQAuZsky"
},
"outputs": [],
"source": [
"def test_dynamically_unrolled(f, *args):\n",
" g = f.get_concrete_function(*args).graph\n",
" if any(node.name == 'while' for node in g.as_graph_def().node):\n",
" print(\"{}({}) uses tf.while_loop.\".format(\n",
" f.__name__, ', '.join(map(str, args))))\n",
" elif any(node.name == 'ReduceDataset' for node in g.as_graph_def().node):\n",
" print(\"{}({}) uses tf.data.Dataset.reduce.\".format(\n",
" f.__name__, ', '.join(map(str, args))))\n",
" else:\n",
" print(\"{}({}) gets unrolled.\".format(\n",
" f.__name__, ', '.join(map(str, args))))\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "KFO1BSN9JkRP"
},
"source": [
"### For loops\n",
"\n",
"Here is a `tf.function` that demonstrates static unrolling:"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "frecgTco_00V"
},
"outputs": [],
"source": [
"@tf.function\n",
"def for_in_range():\n",
" x = 0\n",
" for i in range(5):\n",
" x += i\n",
" return x\n",
"\n",
"test_dynamically_unrolled(for_in_range)"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "PMdl0azc_5d4"
},
"outputs": [],
"source": [
"@tf.function\n",
"def for_in_tfrange():\n",
" x = tf.constant(0, dtype=tf.int32)\n",
" for i in tf.range(5):\n",
" x += i\n",
" return x\n",
"\n",
"test_dynamically_unrolled(for_in_tfrange)"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "Q7tmncQTZt6_"
},
"outputs": [],
"source": [
"@tf.function\n",
"def for_in_tfdataset():\n",
" x = tf.constant(0, dtype=tf.int64)\n",
" for i in tf.data.Dataset.range(5):\n",
" x += i\n",
" return x\n",
"\n",
"test_dynamically_unrolled(for_in_tfdataset)"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "eyPzDYiJAC8f"
},
"outputs": [],
"source": [
"@tf.function\n",
"def while_py_cond():\n",
" x = 5\n",
" while x > 0:\n",
" x -= 1\n",
" return x\n",
"\n",
"test_dynamically_unrolled(while_py_cond)"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "l6s7aU-padY5"
},
"outputs": [],
"source": [
"@tf.function\n",
"def while_tf_cond():\n",
" x = tf.constant(5)\n",
" while x > 0:\n",
" x -= 1\n",
" return x\n",
"\n",
"test_dynamically_unrolled(while_tf_cond)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "dSr64Xn6ap-S"
},
"source": [
" If you have a `break` or early `return` clause that depends on a tensor, the top-level condition or iterable should also be a tensor.\n",
"\n",
"Compare the following examples:"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "hG2Fe_OEAwpY"
},
"outputs": [],
"source": [
"@tf.function\n",
"def while_py_true_py_break(x):\n",
" while True: # py true\n",
" if x == 0: # py break\n",
" break\n",
" x -= 1\n",
" return x\n",
"\n",
"test_dynamically_unrolled(while_py_true_py_break, 5)"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "Sr2cn5bY_E_9"
},
"outputs": [],
"source": [
"@tf.function\n",
"def buggy_while_py_true_tf_break(x):\n",
" while True: # py true\n",
" if tf.equal(x, 0): # tf break\n",
" break\n",
" x -= 1\n",
" return x\n",
"\n",
"with assert_raises(TypeError):\n",
" test_dynamically_unrolled(buggy_while_py_true_tf_break, 5)"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "Q-VirD-5avdZ"
},
"outputs": [],
"source": [
"@tf.function\n",
"def while_tf_true_tf_break(x):\n",
" while tf.constant(True): # tf true\n",
" if x == 0: # py break\n",
" break\n",
" x -= 1\n",
" return x\n",
"\n",
"test_dynamically_unrolled(while_tf_true_tf_break, 5)"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "Upx5J0j8_Ldu"
},
"outputs": [],
"source": [
"@tf.function\n",
"def buggy_py_for_tf_break():\n",
" x = 0\n",
" for i in range(5): # py for\n",
" if tf.equal(i, 3): # tf break\n",
" break\n",
" x += i\n",
" return x\n",
"\n",
"with assert_raises(TypeError):\n",
" test_dynamically_unrolled(buggy_py_for_tf_break)"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "GQHbodav_QMt"
},
"outputs": [],
"source": [
"@tf.function\n",
"def tf_for_py_break():\n",
" x = 0\n",
" for i in tf.range(5): # tf for\n",
" if i == 3: # py break\n",
" break\n",
" x += i\n",
" return x\n",
"\n",
"test_dynamically_unrolled(tf_for_py_break)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "hyksHW9TCukR"
},
"source": [
"In order to accumulate results from a dynamically unrolled loop, you'll want to use `tf.TensorArray`.\n"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "HJ3Vb3dXfefN"
},
"outputs": [],
"source": [
"batch_size = 2\n",
"seq_len = 3\n",
"feature_size = 4\n",
"\n",
"def rnn_step(inp, state):\n",
" return inp + state\n",
"\n",
"@tf.function\n",
"def dynamic_rnn(rnn_step, input_data, initial_state):\n",
" # [batch, time, features] -> [time, batch, features]\n",
" input_data = tf.transpose(input_data, [1, 0, 2])\n",
" max_seq_len = input_data.shape[0]\n",
"\n",
" states = tf.TensorArray(tf.float32, size=max_seq_len)\n",
" state = initial_state\n",
" for i in tf.range(max_seq_len):\n",
" state = rnn_step(input_data[i], state)\n",
" states = states.write(i, state)\n",
" return tf.transpose(states.stack(), [1, 0, 2])\n",
" \n",
"dynamic_rnn(rnn_step,\n",
" tf.random.uniform([batch_size, seq_len, feature_size]),\n",
" tf.zeros([batch_size, feature_size]))"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "9gmLpHY-bkly"
},
"source": [
"### Gotcha's\n",
"\n",
"As with `tf.cond`, `tf.while_loop` also comes with a number of subtleties.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "FJdfznhhKO7D"
},
"source": [
"#### Zero iterations\n",
"\n",
"Since a loop can execute 0 times, all tensors used downstream of the while_loop must be initialized above the loop.\n",
"\n",
"Here is an example of incorrect code:"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "CocT5RHwblrQ"
},
"outputs": [],
"source": [
"@tf.function\n",
"def buggy_loop_var_uninitialized():\n",
" for i in tf.range(3):\n",
" x = i\n",
" return x\n",
"\n",
"with assert_raises(ValueError):\n",
" buggy_loop_var_uninitialized()"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "ncr7tRZ1KWh9"
},
"source": [
"And the correct version:"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "Wm7wIKXcCDGf"
},
"outputs": [],
"source": [
"@tf.function\n",
"def f():\n",
" x = tf.constant(0)\n",
" for i in tf.range(3):\n",
" x = i\n",
" return x\n",
"\n",
"f()"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "CM7qXVY0KZHB"
},
"source": [
"#### Consistent shapes and types\n",
"\n",
"The shape/dtypes of all loop variables must stay consistent with each iteration.\n",
"\n",
"Here is an incorrect example that attempts to change a tensor's type:"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "FSftc9cCbpAo"
},
"outputs": [],
"source": [
"@tf.function\n",
"def buggy_loop_type_changes():\n",
" x = tf.constant(0, dtype=tf.float32)\n",
" for i in tf.range(3): # Yields tensors of type tf.int32...\n",
" x = i\n",
" return x\n",
"\n",
"with assert_raises(TypeError):\n",
" buggy_loop_type_changes()"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "M5l90NAHKsUM"
},
"source": [
"Here is an incorrect example that attempts to change a Tensor's shape while iterating:"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "kWF189prbuK0"
},
"outputs": [],
"source": [
"@tf.function\n",
"def buggy_concat():\n",
" x = tf.ones([0, 10])\n",
" for i in tf.range(5):\n",
" x = tf.concat([x, tf.ones([1, 10])], axis=0)\n",
" return x\n",
"\n",
"with assert_raises(ValueError):\n",
" buggy_concat()"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "miYnYcznCHeV"
},
"outputs": [],
"source": [
"@tf.function\n",
"def concat_with_padding():\n",
" x = tf.zeros([5, 10])\n",
" for i in tf.range(5):\n",
" x = tf.concat([x[:i], tf.ones([1, 10]), tf.zeros([4-i, 10])], axis=0)\n",
" x.set_shape([5, 10])\n",
" return x\n",
"\n",
"concat_with_padding()\n"
]
}
],
"metadata": {
"colab": {
"collapsed_sections": [],
"name": "performance.ipynb",
"private_outputs": true,
"provenance": [],
"toc_visible": true
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
tensorflow/python/autograph/g3doc/reference/common_errors.md
+163
View File
@@ -0,0 +1,163 @@
# AutoGraph reference
[Index](index.md)
## Common AutoGraph errors
### "WARNING: AutoGraph could not transform `<name>`"
This warning is output when AutoGraph could not convert a function, for an
unexpected reason. The error message contains the reason why the function could
not be converted, as well as guidance on how to proceed next.
The exact error message may vary from version to version but in general, the
cause of the failure appears somewhere in the text, for example as
"Cause: could not get source code" or "Original error: could not get source
code".
Note: AutoGraph does not always output a warning. For example, constructors
are silently called without conversion.
When this warning is printed, the code returned by AutoGraph still executes, but
the functions indicated in the warning will be executed as they are, without
conversion. If the functions contain pure Python or graph code (for example,
they have no Tensor-dependent control flow), then the code is likely to still
run without error. However, if it contains any constructs that are only
supported in AutoGraph, expect subsequent exceptions.
Note: the warning is output to the [abseil](https://github.com/abseil/abseil-py)
logger, with `WARNING` severity. To direct these warnings to `stdout`, use
`tf.autograph.set_verbosity(0, True)`.
### "Unable to locate the source code" or "Source not found" errors
Newer versions of AutoGraph raise a `ConversionError`. Older versions print a
warning. In both cases, a similar message about finding the source code is
included.
These errors are raised when AutoGraph is unable to find the source code of
functions it needs to transform. See [Limitations](limitations.md) for more
details.
### "WARNING: Large unrolled loop detected"
This warning is output when AutoGraph detects a `for` or `while` loop that
creates TensorFlow ops and which has a large number of iterations and creates.
This usually indicates a loop that was intended to run as a `tf.while_loop`, but
instead runs as a Python loop.
For example, a training loop might mistakenly iterate over a Python `range`,
instead of `tf.range`:
```
num_steps = 10000
step = tf.constant(0)
for i in range(num_steps):
step += 1
train_step(model)
```
Another example is when using custom generators which AutoGraph does not
support, even if they wrap over supported iterators like Datasets:
```
def my_iterator(ds):
for data in ds:
yield data
# Custom iterators always dispatch to a Python for loop.
for x in my_iterator(tf.data.Dataset.range(10)):
tf.print(x)
```
Note: This verification is only performed when `__debug__` is `True`.
Note: the warning is output to the [abseil](https://github.com/abseil/abseil-py)
logger, with `WARNING` severity. To direct these warnings to `stdout`, use
`tf.autograph.set_verbosity(0, True)`.
### "OperatorNotAllowedInGraphError: using a `tf.Tensor` as a Python `bool`"
This exception is raised whenever a `tf.Tensor` is type-cast as a Python `bool`,
in a context where eager execution is not active. The exception is only raised
when graph execution is active, for example inside a `@tf.function` with
AutoGraph turned off.
**When AutoGraph is on**, it can be caused by: * placing a Tensor-dependent
`break`, `continue` or `return` inside a Python loop (see example below) *
attempting to use a `tf.Tensor` in a list comprehension, by iterating over it or
using it in a condition
A typical example of mixing Python and TF control flow in an incompatible way
is:
```
for i in range(3): # Python loop
if i > tf.constant(0): # TF conditional
break # raises OperatorNotAllowedInGraphError
```
The way these errors are typically fixed is by ensuring all control flow is
TF control flow:
```
for i in tf.range(3): # TF loop
if i > tf.constant(0): # TF conditional
break # works
```
**When AutoGraph is off**, it can be caused by using a `tf.Tensor` value as:
* the condition of an `if` or `while` statement: `if <tensor>:`
* the argument in a logical expression: `tensor and another_tensor`
* the argument to the `bool` built-in: `bool(tensor)`
Note: These operations are allowed when executing eagerly.
When encountering this error, make sure that the function is either decorated
with `@tf.function`, or called from another function decorated in this way. Also
look at the console and logging output for conversion warnings (see the section
above).
### "OperatorNotAllowedInGraphError: iterating over `tf.Tensor`"
This exception is raised whenever you try to iterate over a `tf.Tensor`,
in a context where eager execution is not active. The exception is only raised
when graph execution is active, for example inside a `@tf.function` with
AutoGraph turned off. It can be caused by using a `tf.Tensor` value as:
* the iterated of a `for` statement: `for i in tensor:`
* the argument to the `iter` built-in: `iter(tensor)`
Note: These operations are allowed when executing eagerly.
This exception is similar to the previous example, and has similar causes and
remedies.
### "InaccessibleTensorError: The tensor `<name>` is defined in another function or code block"
This exception is common to code which attempts to obtain values calculated
within a `tf.cond`, `tf.while_loop`, or another `@tf.function` without using
functional style or through mutable collections. See
[Capturing External Symbolic Tensors](https://www.tensorflow.org/guide/function#all_outputs_of_a_tffunction_must_be_return_values)
and [Limitations](limitations.md) for more details.
### "StagingError: in converted code"
This exception is used by AutoGraph to wrap exceptions with custom constructors
that it cannot re-raise with the original type. See
[Error handling](error_handling.md) for more details. If your code uses custom
exceptions, expect them to be wrapped by this exception.
### "Unable to identify source code of lambda function"
This error usually appears in the context of a conversion warning. It indicates
that a lambda function could not be parsed (see [Limitations](limitations.md)).
This type of error can usually be avoided by creating lambda functions in
separate simple assignments, for example:
```
l = lambda <args>: <body>
```
tensorflow/python/autograph/g3doc/reference/control_flow.md
+541
View File
@@ -0,0 +1,541 @@
# AutoGraph reference
[Index](index.md)
## Control flow
AutoGraph rewrites all control flow statements with specialized AutoGraph
function calls. These function calls are capable of executing the corresponding
control flow statement using Python semantics for effects outside the Python
interpreter itself (see the [Introduction](intro.md)).
### Dispatch rules
Key Point: Only statements that are conditioned on, or iterate over, a
TensorFlow object such as `tf.Tensor`, are converted into TensorFlow ops.
As described in the [Introduction](intro.md), AutoGraph aims to preserve the
semantics of valid Python code. If a control flow statement runs in graph
execution without raising an error, then AutoGraph will also execute it as
normal Python control flow. Statements which would normally raise an error, for
example an `if` statement using a `bool` `Tensor` as condition, are converted to
TensorFlow control flow ops.
#### Analogy with compile-time constants and code optimization
From the perspective of a TensorFlow graph, non-Tensor values, for example an
integer or a NumPy array, are _constants_: they do not change value while the
graph executes.
For example, in the graph below, the condition is always `True` (it is
invariant):
```
x = 1
y = tf.cond(x > 0, lambda: 3 * x, lambda 5 * x)
```
That is equivalent to the code below:
```
x = 1
y = 3 * x
```
In the example above, we've optimized away the conditional on a constant
condition. The AutoGraph dispatch rules have the same effect: anything that is
not a TensorFlow object is a compile-time constant for TensorFlow, and can be
optimized away. For this reason, you can usually mix Python and TensorFlow
computation, and it will transparently have the expected result even when only
some computations are executed in the graph.
<!-- TODO(mdan): This is actually a limitation (a very subtle one) -->
Caution: The assumption of invariant code made above is not true if the
TensorFlow graph had callbacks into the Python code. If you modify data
from within a `tf.py_function`, then the code outside a `tf.py_function`
will have unpredictable behavior if it depends on the same data.
For example, the `tf.cond` that runs as part of the `if` statement below will
miss the update made by `f`:
```
n = [10]
def f():
n[0] = 20
return 0
tf.py_function(f, (), (tf.int32,))
if tf.equal(n[0], 10):
tf.print('n is 10')
```
```
n is 10
```
### Compound symbols
AutoGraph usually handles basic symbols:
```
if a < 0:
a = -a
```
```
a = tf.cond(a < 0, lambda: -a, lambda: a)
```
But it can also handle complex symbols in many cases. For example, if we treat
`a.b` as a symbol in the code below, then we can use it as if it were a basic
symbol name:
```
if a.b < 0
a.b = -a.b
```
```
a.b = tf.cond(a.b < 0, lambda: -a.b, lambda: a.b)
```
This is useful in methods, which can operate on properties of `self`, as well as
working directly on more complex object structures or collections.
Caution: There are certain [limitations](limitations.md) around using Python
collections and object mutation. When in doubt, place the values you work
with into local variables and operate on those.
### Effects of the tracing process
#### All Python code paths are executed during tracing
When constructing a graph, TensorFlow _traces_ the code. The tracing of control
flow requires visiting _every possible code path_ (usually once).
Note: In rare cases, the runtime may decide to trace some code paths several
times. For example, the condition of a `while` statement may be executed twice,
first with a temporary graph, to determine whether it evaluates to a
`tf.Tensor`, then if it is a `tf.Tensor`, it's executed a second time in the
proper graph.
In other words, tracing executes both branches of an if statement. Similarly,
the body of loops is executed once (even if the loop would otherwise not iterate
at all).
This explains why inserting `print` statements in an `if` statement produces
this output:
```
print('before if')
if tf.constant(True):
print('true branch')
else:
print('false branch')
print('after if')
```
```
before if
true branch
false branch
after if
```
Note: Control flow that is not executed as a TensorFlow graph is not traced. Its
body will execute as expected.
Example of code that runs as regular Python code:
```
print('before if')
if True: # Condition not a Tensor, running normally
print('true branch')
else:
print('false branch')
print('after if')
```
```
before if
true branch
after if
```
#### Python values modified in TensorFlow control flow become Tensors
If a symbol is modified in a TensorFlow control flow statement, then it becomes
a `tf.Tensor`, even if it started off as a Python primitive value.
For example, the conditional below will run as a `tf.cond` (its condition is a
`tf.Tensor`), which in turn will cause `i` to become a `tf.Tensor`.
```
i = 0
if tf.greater(i, 0):
i = 1
# i is now a Tensor
```
### `if` statements
`if` statements whose condition is a `tf.Tensor` are executed as TensorFlow
conditionals by converting them to `tf.cond`:
```
if tf.random.uniform(()) > 0.5:
x = 1
else:
x = 2
```
`if` statements whose condition is not a `tf.Tensor` are executed as normal
Python:
```
if np.random.uniform() > 0.5:
x = 1
else:
x = 2
```
`if` statements executed as TensorFlow conditionals are subject to restrictions
(see [limitations](limitations.md)). All symbols affected by the statement and
used thereafter must be:
* of a data type understood by TensorFlow
* defined in both branches
* of consistent dtypes in both branches, for TensorFlow entities
* of consistent structure in both branches, for static collections (such as
lists or tuples)
### `while` statements
`while` statements whose condition is a `tf.Tensor` are executed as TensorFlow
loops by converting them to `tf.while_loop`:
```
x = 0
while tf.random.uniform(()) > 0.5:
x = x + 1
```
`while` statements whose condition is not a `tf.Tensor` are executed as normal
Python:
```
x = 0
while np.random.uniform() > 0.5:
x = x + 1
```
`while` statements executed as TensorFlow loops are subject to restrictions
(see [limitations](limitations.md)). All symbols affected by the statement and
used thereafter must be:
* of a data type understood by TensorFlow
* defined before the loop
* of consistent dtype at the beginning and the end of the loop,
for TensorFlow entities
* either of consistent shape at the beginning and the end of the loop,
for TensorFlow entities, or declared in `shape_invariants`
* of consistent structure at the beginning and the end of the loop, for
static collections (such as lists or tuples)
Caution: A `while` loop whose condition is a Python scalar will execute as
normal Python. If you intended to run the loop as a TensorFlow loop, the loop
will replicate its body in the graph (it is unrolled). To avoid that, make sure
its condition is converted to a `tf.Tensor`, using for instance `tf.constant`.
For example, the following loop is unrolled, even though the list contains
`tf.Tensor` values, because the type of `l` is a Python `list`:
```
l = [tf.constant(1), tf.constant(2), tf.constant(3)]
for i in l:
tf.print(i) # This is unrolled - three `tf.print`s are built in the graph.
```
If you wish for the loop to run as a TensorFlow loop, stack the loop:
```
l = [tf.constant(1), tf.constant(2), tf.constant(3)]
for i in tf.stack(l):
tf.print(i) # This runs as a TensorFlow loop.
```
<!-- TODO(mdan): List this under limitations -->
Caution: A loop in which the type of the condition changes across iterations, in
a way that would influence the way the loop is executed, is not allowed in
AutoGraph.
For example, the loop below will generate an error, because after the first
iteration, `i` becomes a tf.Tensor:
```
i = 0
while i < 10: # `i < 10` is a Python bool - run as normal while loop
i = tf.constant(1) # Error -- `i < 10` would now be a `tf.Tensor`
```
### `for` statements
`for` statements that iterate over a `tf.Tensor` are executed as TensorFlow
loops by converting them to a `tf.while_loop` which iterates over the first
dimension (equivalent to NumPy):
```
for i in tf.constant(((1, 2), (3, 4))):
tf.print('iteration:', i)
```
```
iteration: [1, 2]
iteration: [3, 4]
```
Note: If possible, AutoGraph will also set the `maximum_iteration` parameter
of the `tf.while_loop`.
`for` statements that iterate over the output of a `tf.range` are executed as
TensorFlow loops by converting them to a `tf.while_loop` which uses the
arguments passed to the `tf.range`:
```
for i in tf.range(3):
tf.print('iteration:', i)
```
`for` statements that iterate over a `tf.data.Dataset` and which do not contain
`break` or `return` statements are executed as TensorFlow loops by converting
them to `tf.data.Dataset.reduce` ops:
```
for i in tf.data.Dataset.range(3):
tf.print('iteration:', i)
```
`for` statements that iterate over a _distributed dataset_ and which do not
contain `break` or `return` statements are executed as TensorFlow loops by
converting them to the dataset's `reduce` ops:
```
for i in tf.distribute.OneDeviceStrategy('cpu').experimental_distribute_dataset(
tf.data.Dataset.range(3)):
tf.print('iteration:', i)
```
`for` statements that iterate over a `tf.data.Dataset` and which contain
`break` or `return` statements are executed as TensorFlow loops by converting
them to a combination of `tf.data.Dataset.scan`, `tf.data.Dataset.take_while`
and `tf.data.Dataset.reduce` ops:
```
for i in tf.data.Dataset.range(3):
tf.print('iteration:', i)
break
```
```
iteration: 1
```
`for` statements that iterate over a `tf.data.Dataset` _iterator_ are executed
as TensorFlow loops by converting them to a combination of `tf.while_loop`,
and `tf.cond` ops:
```
for i in iter(tf.data.Dataset.range(3)):
tf.print('iteration:', i)
```
`for` statements that iterate over a type different from any of the above are
executed as normal Python:
```
for i in [1, 2, 3]:
print('iteration:', i)
```
Caution: A `for` loop over a `list` or `tuple` of `tf.Tensor` is considered to
iterate over a Python `list` (or respectively `tuple`), therefore will be
executed as normal Python. If you intended to run it as a TensorFlow loop,
use `tf.stack` or `tf.concat`.
Caution: A `for` loop over a Python `range` will execute as normal Python.
If you intended to run it as a TensorFlow loop, use `tf.range`.
Note: AutoGraph may output a warning when it believes that you are unrolling
a loop inefficiently. However, the warning thresholds are very conservative.
The warning is only printed when
[__debug__](https://docs.python.org/3/library/constants.html#__debug__) is
`True`.
Note: If `__debug__` is `True`, AutoGraph limits the number of iterations in
normal Python loops to prevent infinite loops and raise an error if the limits
are exceeded. However, the iteration limits are very large and may take a while
to trigger an error.
### `break` statements
Code blocks in which `break` statements are used are rewritten with equivalent
code that uses extra control booleans and conditionals. The control booleans are
used directly in `while` loops. In the case of `for` loops, the AutoGraph
corresponding operator accepts an `extra_test` argument which is similar to
the conditional of a while loop, and which contains the control boolean.
For example, the `while` loop below is rewritten as (showing the output of the
`break` transformation only):
```
while i < 10:
if i > 3:
break
i += 1
```
```
break_ = False
while i < 10 and not break_:
if i > 3:
break_ = True
continue # The continue statement is also rewritten in a subsequent pass
i += 1
```
Another example shows how the control boolean is used in the overload of a `for`
loop (showing portions of the final output):
```
for i in range(10):
if i > 3:
break
```
```
break_ = False
...
def extra_test(break_):
return ag__.not_(break_)
# break_ becomes a loop variable.
break_, = ag__.for_stmt(range(10), extra_test, ..., (break_,))
```
Mixing Tensor-dependent `break` and Python-dependent loops is disallowed:
```
@tf.function
def buggy_while_py_true_tf_break(x):
while True: # python conditional
if tf.equal(x, 0): # tensor break
break
x -= 1
return x
# Raises OperatorNotAllowedInGraphError: using a `tf.Tensor` as a Python `bool` is not allowed
# buggy_while_true_tf_break(5)
```
### `continue` statements
Code blocks in which `continue` statements are used are rewritten with
equivalent code that uses extra control booleans and conditionals, similar to
how `break` is handled.
For example, the `for` loop below is rewritten as (showing the output of the
`continue` transformation only):
```
for i in range(10):
if i > 3:
continue
```
```
for i in range(10):
continue_ = False
if i > 3:
continue_ = True
if not continue_:
i += 1
```
Notice that unlike `break`, `continue` statements are local to the loop and do
not influence the number of iterations.
### `return` statements
`return` statements are also rewritten using control symbols, in a manner
similar to how `break` is converted. In the case of `return` statements, an
additional symbol keeps track of the return value.
Depending on the structure of the code, the return value might be undefined
in parts of the code (for example on code paths in which no return statement
has executed). AutoGraph keeps track of this by using a special value.
This special value is converted to `None` (the default return value) upon
exiting the function.
Caution: TensorFlow control flow does not support undefined values, and an
undefined return value is no exception. Therefore, AutoGraph will raise an
error for TensorFlow control flow in which the return value is not known for
all code paths.
For example, the following code raises an error because the return value would
be undefined when the random number would be less than 0.5:
```
if tf.random.uniform(()) > 0.5:
return 1
```
```
ValueError: A value must also be returned from the else branch.
```
An example of rewriting a `while` (showing the output of the `return`
transformation only):
```
def f():
while i < 10:
if i > 3:
return 1
i += 1
```
```
def f():
do_return = False
retval_ = ag__.UndefinedReturnValue()
while i < 10 and not do_return:
if i > 3:
do_return = True
retval_ = 1
if not do_return:
i += 1
return ag__.retval(retval_) # Transforms any UndefinedReturnValue to None
```
Note: AutoGraph performs an additional code normalization in which an `if`
statement with no `else` branch contains a `return` statement it is rewritten as
an `if-else` statement in which the code that follows the statement is moved
under the `else` branch.
Example (showing the normalization only):
```
def f():
if i > 3:
return 1
i += 1
```
```
def f():
if i > 3:
return 1
else:
i += 1
```
tensorflow/python/autograph/g3doc/reference/debugging.md
+180
View File
@@ -0,0 +1,180 @@
# AutoGraph reference
[Index](index.md)
## Debugging AutoGraph code
The recommended way to debug AutoGraph code is to run it eagerly (see below).
AutoGraph generates a new function, rather than directly executing the input
function. Non-code elements, such as breakpoints, do not transfer to the
generated code.
You can step through the generated code and set breakpoints while debugging.
The converted function is cached, and breakpoints should persist for the
lifetime of the Python runtime.
Note: The code generated by AutoGraph code is more complex than the input code,
and is interspersed with AutoGraph boilerplate.
Note: Python debugging can only be used to step through the code during graph
construction time (or tracing time in the case of `tf.function`). To debug
TensorFlow execution, use Eager execution.
### Debugging `tf.function`: `tf.config.experimental_run_functions_eagerly`
When using `@tf.function`, you can temporarily toggle graph execution by using
`tf.config.experimental_run_functions_eagerly`. This will effectively run the
annotated code eagerly, without transformation. Since AutoGraph has semantics
consistent with Eager, it's an effective way to debug the code step-by-step.
Note: AutoGraph is compatible with Eager, but the converse is not always
true, so exercise care when making modifications to the code while debugging.
Consider the following code:
```
@tf.function
def f(a):
pdb.set_trace()
if a > 0:
tf.print(a, 'is positive')
```
Executing the line below will land the debugger in generated code, when the
function is traced:
```
f(1)
```
```
>l
10 def tf__f(a):
11 pdb.set_trace()
---> 12 ag__.converted_call('print', tf, ag__.STD, (a,), None)
13
14 ...
```
Adding a call to `tf.config.experimental_run_functions_eagerly` before executing
the function will land the debugger in the original code instead:
```
tf.config.run_functions_eagerly(True)
f(1)
```
```
>l
8 def f(a):
9 pdb.set_trace()
---> 10 tf.print(a)
11 if a > 0:
12 tf.print('is positive')
```
### Using `print` and `tf.print`
The `print` function is not converted by AutoGraph, and can be used to inspect
the values of variables at graph construction time.
Mixing `print` with `tf.print` can be confusing at first because they run at
different stages. In general:
* all `print`s run when the TensorFlow graph is constructed
* all `tf.print`s run when the TensorFlow graph is executed
#### Example: `print`
To see the difference between `print` and `tf.print`.
```
@tf.function
def f(a):
print(a)
if a > 0:
a = -a
```
When `a` is a `tf.Tensor` object, it is printed without an actual value:
```
f(tf.constant(1))
```
```
Tensor("a:0", shape=(), dtype=int32)
```
Similarly, when `a` is just a Python value, it is printed directly:
```
f(1)
```
```
1
```
#### Example: `print` followed by `tf.print`
To see the difference between `print` and `tf.print`, let's run them together:
```
@tf.function
def f(a):
print(a)
tf.print(a)
```
For non-`Tensor` values, they produce similar results:
```
f(1)
```
```
1
1
```
For Tensor values, only `tf.print` outputs the actual value:
```
f(tf.constant(1))
```
```
Tensor("a:0", shape=(), dtype=int32)
1
```
#### Example: `tf.print` followed by `print`
Remember that, in general, *all `print`s run before all `tf.prints`*.
What's more, since graphs are usually built once and executed multiple times,
`print` usually runs just once when the function is first called.
So in the example below, even though `tf.print` appears above `print`, it will
run after it, because the graph is executed after it is built:
```
@tf.function
def f(a):
tf.print('At graph execution:', a)
print('At graph construction:', a)
```
```
f(tf.constant(1))
```
```
At graph construction: Tensor("a:0", shape=(), dtype=int32)
At graph execution: 1
```
Calling the function again will re-use the graph in this case:
```
f(tf.constant(1))
```
```
At graph execution: 1
```
tensorflow/python/autograph/g3doc/reference/error_handling.md
+217
View File
@@ -0,0 +1,217 @@
# AutoGraph reference
[Index](index.md)
## Error handling
When an exception occurs in code generated by AutoGraph, the error message
is augmented with information about the location in the original code,
before conversion.
When an error occurs in a TensorFlow graph constructed using AutoGraph code,
the stack trace which points to where the failing op was created is modified
to point to the original code, before conversion.
### Python execution errors
Python execution (or tracing) exceptions that are raised in AutoGraph code are
caught and re-raised with an extended error message that contains references
to the original code.
These functions are re-raised by `@tf.function`. If you use a `try/catch` the
exception inside `tf.function`, you will obtain the original exception.
The exception traceback still contains the entire call stack, including frames
corresponding to generated code.
AutoGraph tries to re-raise an exception of the same type as the original
exception. This is usually possible for subclasses of
`Exception` that do not define a custom `__init__`. For more complex
exception types which define a custom constructor, AutoGraph raises a
`StagingError` instead.
Among the distinctive features of the re-raised exception:
* the exception traceback indicates the call stack of the exception, up to the
first @tf.function
* the error message includes references to the original code within
the `@tf.function`
* the references corresponding to converted code are marked with an
asterisk (`*`)
* the references corresponding to code which AutoGraph reached, but decided not
to convert, are marked with a double asterisk (`**`)
* the references corresponding to code that AutoGraph didn't reach at all have
no marking
For example, the code below triggers an exception in the Python runtime, at
graph construction time:
```
@tf.function
def f():
tf.constant(1) + tf.constant(1.0)
f()
```
An excerpt of the exception that is raised is shown below:
```
Traceback (most recent call last):
File "<ipython-input-10-1938a51c970d>", line 11, in <module>
f()
File "tensorflow/python/eager/def_function.py", line 417, in __call__
self._initialize(args, kwds, add_initializers_to=initializer_map)
... more TensorFlow internal frames ...
TypeError: in converted code:
<ipython-input-9-002fa22f79df>:8 f *
tf.constant(1) + tf.constant(1.0)
tensorflow/python/ops/math_ops.py:900 binary_op_wrapper **
return func(x, y, name=name)
... more TensorFlow internal frames ...
TypeError: Input 'y' of 'AddV2' Op has type float32 that does not match type int32 of argument 'x'.
```
Note: the exact appearance of the various parts in the error message may change
in the future.
Let's look at the individual components of this exception.
The traceback of the exception indicates the location until the call to
`@tf.function`, including any frames internal to TensorFlow:
```
Traceback (most recent call last):
File "<ipython-input-10-1938a51c970d>", line 11, in <module>
f()
File "tensorflow/python/eager/def_function.py", line 417, in __call__
self._initialize(args, kwds, add_initializers_to=initializer_map)
File "tensorflow/python/eager/def_function.py", line 360, in _initialize
*args, **kwds))
File "tensorflow/python/eager/function.py", line 1688, in _get_concrete_function_internal_garbage_collected
graph_function, _, _ = self._maybe_define_function(args, kwargs)
File "tensorflow/python/eager/function.py", line 1992, in _maybe_define_function
graph_function = self._create_graph_function(args, kwargs)
File "tensorflow/python/eager/function.py", line 1878, in _create_graph_function
capture_by_value=self._capture_by_value),
File "tensorflow/python/framework/func_graph.py", line 791, in func_graph_from_py_func
func_outputs = python_func(*func_args, **func_kwargs)
File "tensorflow/python/eager/def_function.py", line 310, in wrapped_fn
return weak_wrapped_fn().__wrapped__(*args, **kwds)
File "tensorflow/python/framework/func_graph.py", line 781, in wrapper
raise e.ag_error_metadata.to_exception(type(e))
```
The exception message includes the location inside the converted function `f`:
```
TypeError: in converted code:
<ipython-input-9-002fa22f79df>:8 f *
tf.constant(1) + tf.constant(1.0)
tensorflow/python/ops/math_ops.py:900 binary_op_wrapper
return func(x, y, name=name)
tensorflow/python/ops/math_ops.py:1198 _add_dispatch
return gen_math_ops.add_v2(x, y, name=name)
tensorflow/python/ops/gen_math_ops.py:549 add_v2
"AddV2", x=x, y=y, name=name)
tensorflow/python/framework/op_def_library.py:564 _apply_op_helper
inferred_from[input_arg.type_attr]))
```
Notice the frame corresponding to the call of `f`. The function is converted,
which is being indicated by the asterisk `*` character displayed next to
`f`:
```
<ipython-input-9-002fa22f79df>:8 f *
tf.constant(1) + tf.constant(1.0)
```
Lastly, the lower part includes the message that the exception originally
reported:
```
TypeError: Input 'y' of 'AddV2' Op has type float32 that does not match type int32 of argument 'x'.
```
Note: Typically, error messages raised by code internal to TensorFlow refers
to arguments of the internal API that failed. Error messages raised by code
internal to AutoGraph (that is, 'tensorflow/python/autograph') usually
refer to symbols used in your code.
### TensorFlow execution errors
TensorFlow execution errors are displayed normally, but the portions of the
error message which correspond to user code contain references to the original
code.
For example, the code below triggers an error in the TensorFlow runtime, at
graph execution time:
```
@tf.function
def my_function():
tf.Assert(tf.random.uniform(()) > 1.0, ['example error'])
my_function()
```
An excerpt of the exception that is subsequently raised is shown below:
```
Traceback (most recent call last):
File "<ipython-input-16-af656fb445f0>", line 11, in <module>
my_function()
File "tensorflow/python/eager/def_function.py", line 435, in __call__
return self._concrete_stateful_fn._filtered_call(canon_args, canon_kwds)
File "tensorflow/python/eager/function.py", line 636, in _filtered_call
self.captured_inputs)
File "tensorflow/python/eager/function.py", line 734, in _call_flat
outputs = self._inference_function.call(ctx, args)
File "tensorflow/python/eager/function.py", line 460, in call
ctx=ctx)
File "tensorflow/python/eager/execute.py", line 68, in quick_execute
six.raise_from(core._status_to_exception(e.code, message), None)
File "<string>", line 3, in raise_from
InvalidArgumentError: assertion failed: [example error]
[[node Assert/Assert (defined at <ipython-input-16-af656fb445f0>:8) ]] [Op:__inference_my_function_79]
```
Notice the error message containing references to the location where the failing
op was defined in the code (`<ipython-input-16-af656fb445f0>:8`):
```
InvalidArgumentError: assertion failed: [example error]
[[node Assert/Assert (defined at <ipython-input-16-af656fb445f0>:8) ]] [Op:__inference_my_function_79]
```
### AutoGraph conversion exceptions
Within `@tf.function`, when AutoGraph fails to convert a function, it displays
a warning message and attempts to run the function without conversion.
For example, the code below make a call to a Python
[generator](https://wiki.python.org/moin/Generators) function, which is not
supported by AutoGraph:
```
def example_generator():
yield 1
@tf.function
def f():
for i in example_generator():
print(i)
```
Calling `f()` will still run the code. AutoGraph will convert the function `f`,
but skips the function `example_generator`. In addition, AutoGraph prints a
warning to the console indicating that the function is called without being
converted.
```
WARNING: Entity <function example_generator at 0x7f951b67f158> appears to be
a generator function. It will not be converted by AutoGraph.
```
tensorflow/python/autograph/g3doc/reference/functions.md
+67
View File
@@ -0,0 +1,67 @@
# AutoGraph reference
[Index](index.md)
## Functions and function calls
Typically, AutoGraph converts one function at a time. If a function calls other
functions, the called function will be converted recursively, as described
below.
### Function calls
AutoGraph rewrites all function calls with a special wrapper that may convert
the called function at runtime.
For example, the function call below:
```
f(x, y, z=1)
```
Is converted to code that schematically looks like this:
```
ag__.converted_call(f, ..., (x, y), {'z': 1}, ...)
```
All calls are rewritten, including calls to other types of callables, builtin
functions, etc.
If the originally called function is not converted, AutoGraph simply
forwards the call to it, so that the wrapper is functionally equivalent with
the original function call.
If the originally called function is converted, then the conversion is performed
first and the converted function is called instead.
Note: a caching mechanism prevents the same function from being converted
multiple times. This mechanism ensures that functions calls made with different
[global or free variables](https://docs.python.org/3/reference/executionmodel.html#binding-of-names)
are handled correctly.
#### Function conversion rules
The following types of functions are not converted:
* functions already converted
* functions defined in an allowlisted module (see autograph/core/config.py)
* non-Python functions (such as native bindings)
* `print`, `pdb.set_trace`, `ipdb.set_trace`
* most built-in functions (exceptions are listed in
autograph/operators/py_builtins.py)
* constructors
* functions without source code attached (prints a warning)(see
[limitations](limitations.md))
* generator functions (prints a warning)
* iterator protocol methods (`__next__`, `__iter__`)
* context manager methods (`__enter__`, `__exit__`)
When AutoGraph encounters a function that it cannot convert outside of this
list, it prints a warning.
### Nested functions
Functions nested inside a function converted by AutoGraph are converted
at the same time as the function containing them. If the nested function is
returned, a converted version of it is returned.
tensorflow/python/autograph/g3doc/reference/generated_code.md
+89
View File
@@ -0,0 +1,89 @@
# AutoGraph reference
[Index](index.md)
## Generated code
For each converted function, AutoGraph creates a new function. The
loading mechanism is an implementation detail and may change, but the
generated function is generally a regular
[Python function](https://docs.python.org/3/reference/compound_stmts.html#function).
This function is typically executed internally by `@tf.function` to construct a
TensorFlow graph.
### Transformations made to the generated code
The generated code is a transformation of the input code. The transformations
are listed below. Any other elements are left unchanged.
Summary of transformations:
* function calls are replaced with a wrapper:
* `foo(args)` -> `ag__.converted_call(foo, args)`
* `if`, `while` and `for` statements are replaced with function calls:
* `if` -> `ag__.if_stmt`
* `while` -> `ag__.while_stmt`
* `for` -> `ag__.for_stmt`
* `break`, `return`, and `continue` statements are replaced with equivalent
`if` statements.
* `and`, `or` and `not` operators are replaced with function calls:
* `and` -> `ag__.and_`
* `or` -> `ag__.or_`
* `not` -> `ag__.not_`
The functions replacing control flow statements are very similar in form with
the corresponding control flow ops in TensorFlow.
### AutoGraph generates normal Python code
You can interact normally with the generated code. For example, you can use
the `inspect.getsourcefile` and `inspect.getsource`:
```
def f(a):
...
converted_f = tf.autograph.to_graph(f)
print(inspect.getsourcefile(converted_f))
```
```
/tmp/tmpm562wlj7.py
```
When using `@tf.function`, you can repeat the same steps using the function's
`python_function` attribute:
```
@tf.function
def f(a):
...
converted_f = tf.autograph.to_graph(f.python_function)
print(inspect.getsourcefile(converted_f))
```
```
/tmp/tmpm562wlj7.py
```
`tf.autograph.to_code(f)` is a shortcut to obtain the generated code, and it's
equivalent with calling `inspect.getsource(tf.autograph.to_graph(f))`.
#### Recording diagnostic information: `tf.autograph.set_verbosity`
AutoGraph can log additional debug information. This is mostly used for filing
bugs, but can also be used to get an indication of whether a function is
converted successfully or not.
You can enable logging by calling `tf.autograph.set_verbosity(level)`. The
`level` argument varies from 0 to 10:
* 0 - no logging
* 3 - includes the generated code
* 4 and above - extremely verbose logging
Caution: The information being logged includes source code as well as
data. Before sharing AutoGraph logs, make sure they don't contain any sensitive
information.
Alternatively, you can control the verbosity level using the environment
variable `AUTOGRAPH_VERBOSITY`.
tensorflow/python/autograph/g3doc/reference/index.md
+23
View File
@@ -0,0 +1,23 @@
# AutoGraph reference
This reference document describes the semantics of AutoGraph transformations.
In `@tf.function`, AutoGraph allows running Eager-style code as a TensorFlow
graph.
* [Introduction](intro.md)
* [Interacting with the generated code](generated_code.md)
* [Debugging AutoGraph code](debugging.md)
* [Control flow](control_flow.md)
* [Functions and function calls](functions.md)
* [Error handling](error_handling.md)
* [Operator semantics](operators.md)
* [Limitations](limitations.md)
* [Common errors](common_errors.md)
Also see:
* [AutoGraph blog post](https://medium.com/tensorflow/autograph-converts-python-into-tensorflow-graphs-b2a871f87ec7)
* [AutoGraph deep dive](https://www.youtube.com/watch?v=NIEgzljyDyI)
* [tf.function guide](https://www.tensorflow.org/guide/function)
* [Intro to TF graphs](https://www.tensorflow.org/guide/intro_to_graphs)
tensorflow/python/autograph/g3doc/reference/intro.md
+161
View File
@@ -0,0 +1,161 @@
# AutoGraph reference
[Index](index.md)
## Introduction
### Terminology
Typically, AutoGraph operates by converting a function into a new function with
new semantics.
```
def f(...):
...
converted_f = tf.autograph.to_graph(f)
```
In TensorFlow 2, AutoGraph is used to convert functions before they are being
traced, when using the `tf.function` API. For example, the following code:
```
def f(...):
...
graph_f = tf.function(f)
```
is roughly equivalent to:
```
converted_f = tf.autograph.to_graph(f)
graph_f = tf.function(autograph=False)(converted_f)
```
For the remainder of this document:
* **converted functions** are functions converted by AutoGraph
* **graph functions** are functions compiled with `tf.function`
Graph functions are usually also converted, unless specified otherwise.
### Safe behavior
The semantics described below can be summarized as:
1. code should either produce the same results as running it in Eager mode, or
fail with error.
2. TF 1.* graph code should produce the same graph as running it directly in
graph mode.
### Python semantics
In general, AutoGraph does not change the semantics of error-free Python.
In special circumstances, AutoGraph may also preserve the semantics of code that
raises error as well. Semantics are preserved in the sense of the
[as-if rule](https://en.wikipedia.org/wiki/As-if_rule).
More specifically, code that would not be legal TensorFlow Graph code may become
legal when converted.
For example, consider the following function and the corresponding converted
function:
```
def f(x):
if x > 0:
return x
return -x
converted_f = tf.autograph.to_graph(f)
```
If `f` executes without error, then the `converted_f` has identical results:
```
>>> f(3) # Runs without error
3
>>> converted_f(3) # Identical result
3
```
### TensorFlow Eager semantics
If a function is called with `Tensor` arguments in Eager execution mode, it has
certain semantics that are different from the Graph execution semantics.
For example, the function below produces different results when executed in
Eager and Graph modes:
```
def f(x):
if x > 0:
return x
return -x
# Use tf.function to run code as a TensorFlow Graph.
unconverted_graph_f = tf.function(f, autograph=False)
graph_f = tf.function(f, autograph=True)
```
```
>>> f(tf.constant(3)) # Valid in Eager
<tf.Tensor ... numpy=3>
>>> unconverted_graph_f(tf.constant(3)) # Error - not legal Graph code
TypeError: Using a `tf.Tensor` ...
```
AutoGraph transfers the Eager semantics to the graph function:
```
>>> graph_f(tf.constant(3)) # Valid in AutoGraph
<tf.Tensor ... numpy=3>
```
### Subset of Eager
AutoGraph currently supports only a subset of Eager. For example, list operation
semantics are not supported:
```
def f(n):
l = []
for i in tf.range(n):
l.append(i)
return l
converted_f = tf.autograph.to_graph(f)
```
```
>>> f(3) # Valid in Eager
[1, 2, 3]
>>> converted_f(3) # Not valid in AutoGraph (by default)
<<error>>
```
### Experimental features
AutoGraph supports additional semantics which are not yet stable. This includes
for example support for list semantics applied to `TensorArray`:
```
def f(n):
l = tf.TensorArray(dtype=tf.int32, size=0, dynamic_size=True)
for i in tf.range(n):
l.append(i)
return l
converted_f = tf.autograph.to_graph(f)
experimental_converted_f = tf.autograph.to_graph(
f, experimental_optional_features=tf.autograph.experimental.Feature.LISTS)
```
```
>>> converted_f(3) # Not valid in AutoGraph (by default)
<<error>>
>>> experimental_converted_f(3) # Valid
<<TensorArray object>>
```
tensorflow/python/autograph/g3doc/reference/limitations.md
+880
View File
@@ -0,0 +1,880 @@
# AutoGraph reference
[Index](index.md)
## Limitations
When AutoGraph is applied to normal Python code, you should expect no change
in functionality.
However, when applied to TensorFlow control flow (for example, an if statement
with a `tf.Tensor` condition), there are certain limitations. This section
describes these limitations and practices that will allow you to avoid them.
Key Term: Python variables refer to Python symbols (or symbols for short) and
should not be confused with TensorFlow variables.
Key Term: A TensorFlow loop variable (or loop variable for short) refers to a
value (typically a `tf.Tensor`) modified by a loop. See `tf.while_loop`.
### Undefined and None values in TensorFlow
TensorFlow does not support undefined or `None` values. All tensors must have
a value.
Example:
```
x = tf.cond(
tf.random.uniform(()) > 0.5,
lambda: tf.constant(1),
lambda: None) # Error -- a Tensor cannot be None
```
The same restriction carries over in AutoGraph. If a variable is created inside
control flow, and used after, then it must be defined before the control flow
statement:
```
if tf.random.uniform(()) > 0.5:
x = tf.constant(1)
else:
x = None
tf.print(x) # Error -- x may be None here
```
For this reason, AutoGraph forbids variables to be defined in only one branch
of a TensorFlow conditional, if the variable is used afterwards:
```
del x
if tf.random.uniform(()) > 0.5:
x = tf.constant(1)
else:
pass
tf.print(x) # Error -- x may be undefined here
```
Note that if the variable is not used after the control flow statement, then it
is considered local to the control flow block, and is not subject to these
restrictions.
```
del x
if tf.random.uniform(()) > 0.5:
x = tf.constant(1) # Okay -- x does not need to be returned from the TF cond
else:
pass
```
Similarly, variables must usually be defined before a TensorFlow loop.
The most common example that is not allowed is a loop which initializes some
accumulator variable in the first iteration:
```
del x
for i in tf.range(100): # Error -- x must be defined before the loop
if i == 0:
x = tf.constant(1)
else:
x = x + 1
tf.print(x)
```
When the variable is only used inside the loop and does not depend on previous
iterations, then it's ok to only be initialized inside the loop.
```
del x
while tf.random.uniform(()) > 0.5: # Okay -- x is not used after the loop
x = tf.constant(1)
```
* New in TF 2.4 *
As long as it doesn't depend on previous iterations, the variable may also be
used after the loop, however in that case the loop must execute at least one
iteration, and will raise a runtime error otherwise.
```
del x
for i in tf.range(10): # Okay -- x does not depend on previous iterations
x = tf.constant(1)
tf.print(x)
```
```
del x
while tf.constant(False): # Error -- loop must initialize x!
x = tf.constant(1)
tf.print(x)
```
Avoid these limitations by defining a default value before the control flow
statement:
```
x = tf.constant()
if tf.random.uniform(()) > 0.5:
x = tf.constant(1)
tf.print(x) # Okay -- x is either 0 or 1
```
Note: `None` values and undefined symbols are allowed in Eager control flow,
because Eager execution uses Python control flow, rather than TensorFlow
control flow ops.
#### Special case: creating Tensors in a loop
* New in TF 2.4 *
A very common use-case is to run a training loop that creates some outputs:
```
for i in tf.range(num_steps):
outputs = train(next(data_iterator))
```
Often times these outputs can be nested structures of Tensors, which makes them
impractical to initialize ahead of the loop.
To help with this use-case, AutoGraph lets you run such loops, under certain
conditions:
* outputs must be a Tensor, Python numeric, or a structure of these
* outputs must not depend on the value from a previous iteration; in other
words, the outputs may only appear to the left of an assignment operation
* the loop must run at least one iteration
If the type of outputs is not recognized, then the usual
"outputs must be defined before the loop" is raised at graph construction.
AutoGraph also inserts a `tf.Assert` statement that raises a runtime error
if the loop did not execute at least one iteration.
### Indirect modifications and hidden side effects in TensorFlow control flow
Key Point: We recommend using a functional programming style, immutable Python
collections, TensorFlow ops and collections. Only TensorFlow objects should be
used for side effects.
#### AutoGraph analyzes code to detect modifications to Python objects
Note: Modifications to TensorFlow objects, such as `tf.Variable`, are tracked
using a different mechanism (automatic control dependencies) which does not
rely on code analysis.
One of the most important functions of AutoGraph is to rewrite Python control
flow statements into equivalent TensorFlow ops. This process requires "wiring"
variables covered by these control flow statements into the respective ops.
The examples below use a `while` loop, but the same notions extend to all
control flow such as `if` and `for` statements.
In the example below, `x` needs to become a loop variable of the corresponding
`tf.while_loop`:
```
while x > 0:
x = x - 1
```
```
x = tf.while_loop(..., loop_vars=(x,)
```
TF control ops support only a limited set of types for loop variables. At the
same time, the efficiency of TensorFlow graphs is influenced by the number of
loop variables, so we don't want to create them unnecessarily. AutoGraph pulls
symbols through loop variables only if necessary to minimize the number of
loop variables.
Note: If a symbol refers to a nested structure, such as a `dict` of `dict`s,
the entire structure is mapped to multiple loop variables - TensorFlow
automatically unpacks it.
For example, the symbol 'y' below is not wired through the `tf.while_loop`'s
`loop_vars` because it is not affected by the `while` loop:
```
y = 0
while x > 0:
x = x - 1
print(y)
```
```
x = tf.while_loop(..., loop_vars=(x,) # y does not need to be a loop variable
```
AutoGraph uses static analysis to determine which symbols are modified by the
code, in order to transform them into control flow variables. Static analysis
is generally performed on single functions - Python's dynamic nature limits its
effectiveness across functions.
#### Modifications of Python objects are not detected across functions
Note: Modifications to TensorFlow objects, such as `tf.Variable`, are tracked
using a different mechanism (automatic control dependencies). Modifications
to `tf.Variable` objects are correctly handled even when called in other
functions.
Because static analysis is limited to single functions, modifications that are
performed in other functions are not visible to AutoGraph:
```
def change_y():
global y
y = y + 1
while x > 0:
change_y() # Problem -- change made to y is not visible here!
```
This can be easily remedied using a functional programming style - writing
functions that use argument for all their inputs and return values for all their
outputs:
```
def change(y):
y = y + 1
return y
while x > 0:
y = change(y) # Okay -- y can now be properly tracked!
```
As noted before, this limitation does not apply to most TensorFlow objects,
although it is still a good idea to use functional programming style for
better code readability:
```
def change(y_var):
y_var.assign_add(1)
y = tf.Variable(1)
while x > 0:
change(y) # This is still okay -- TensorFlow side effects are robust.
```
Keep in mind however that certain types like `tf.TensorArray` don't support
side effects and must have their result assigned, otherwise they may raise an
error:
```
def change(ta):
ta.write(0, 1) # Incorrect use of TensorArray - will raise an error
```
In other words, `tf.TensorArray` must be handled using functional programming
style:
```
def change(ta):
ta = ta.write(0, 1) # Modifications create a new TensorArray efficiently.
return ta
ta = tf.TensorArray(tf.int32, size=0, dynamic_size=True)
while x > 0:
# TensorArray must be handled using functional programming style.
ta = change(ta)
```
#### Modifications of Python objects are not detected in methods
A special case of hidden side effects are methods, which are commonly used
to change the value of objects:
```
class MyClass(object):
def change(self):
self.y += 1
c = MyClass()
while x > 0:
c.change() # Problem -- modification to c.y is not visible here!
```
This can be addressed in a number of ways.
One possibility is to operate directly on the object properties:
```
c = MyClass()
while x > 0:
c.y += 1 # Okay -- c.y can now be properly tracked!
```
Another possibility is to rely on immutable objects with value semantics. This
may lead to many temporary objects when executing eagerly, but their number is
greatly reduced in `@tf.function`:
```
class MyClass(collections.namedtuple('MyClass', ('y',))):
def change(self):
new_y = self.y + 1
return MyClass(new_y)
c = MyClass()
while x > 0:
c = c.change() # Okay -- c is now a loop var.
```
It is also recommended to use a functional programming style with such immutable
objects - that is, all arguments are inputs, all changes are return values:
```
def use_my_class(c: MyClass) -> MyClass:
new_c = c.change()
return new_c
```
Don't worry about creating a few extra objects - they are only used at trace
time, and don't exist at graph execution.
Note: TensorFlow control flow does not currently support arbitrary Python
objects, but it does support basic collection objects such as `list`, `dict`,
`tuple`, `namedtuple` and their subclasses. Design your objects as subclasses
of [namedtuple](https://docs.python.org/3/library/collections.html#collections.namedtuple),
or other types that [tf.nest](https://www.tensorflow.org/api_docs/python/tf/nest/map_structure)
recognizes.
#### Variables closed over by lambda functions
### This limit will be deprecated after 2023-09-23
AutoGraph assumes that variables that local functions close over may be used
anywhere in the parent function, because in general it is possible to hide a
function call in almost any Python statement. For this reason, these variables
are accounted within TensorFlow loops.
For example, the following code correctly captures `a` in the TensorFlow loop
variables:
```
a = 0
def f():
tf.print(a)
for i in tf.range(3):
a = i
f() # Prints 2
```
A consequence is that these variables must be defined before the loop (see
Undefined and None values above). So the following code will raise an error,
even if the variable is never used after the loop:
```
def f():
tf.print(a)
for i in tf.range(3): # Error -- `a` must be defined before the loop.
a = i
```
However, lambda functions are handled differently, for reasons of backward
compatibility. Lambda functions are assumed to be used in the statement where
they are used, or at least in the same block.
```
a = 0
foo(lambda: a) # This lambda is not expected to be called anywhere else.
for i in tf.range(3): # Okay -- `a` is local to the loop.
a = i
```
Due to that reason, the following code will not work as expected for TensorFlow
loops.
```
a = 0
l = lambda: tf.print(a)
for i in tf.range(3):
a = i # `a` is considered local to the loop
l() # Prints 0!
```
Note that none of these restrictions only apply to TensorFlow loops; Python
loops correctly handle closures in all cases.
### Python collections in TensorFlow control flow
Key Point: Use TensorFlow collection classes instead of Python collections.
Python collections are okay to use when they represent a fixed structure (that
is, `list`s don't change length, `dict`s don't add or remove keys).
#### Modifying Python collections in TensorFlow control flow is not allowed
One of the advantages of eager execution is that you may use the usual Python
collections, like `list` or `dict` to hold `tf.Tensor` values. However, these
are generally not compatible with TensorFlow control flow. Specialized
collections like `tf.TensorArray` are required.
Consider the following example:
```
def fn():
l = []
def loop_cond(i):
return i < 10
def loop_body(i):
i = i + 1
l.append(i)
return i,
tf.while_loop(
cond=loop_cond,
body=loop_body,
loop_vars=(0,))
return l
```
This code works in eager execution, which does not use the TensorFlow runtime
for the `tf.while_loop`:
```
fn()
```
However, it does not work in graph execution, because TensorFlow uses special
mechanisms to ensure the computations are correctly sequenced in the dataflow
graph:
```
tf.function(fn)() # Error -- illegal tensor capture!
```
The equivalent AutoGraph code raises the same error:
```
l = []
for i in tf.range(10):
l.append(i) # Error -- illegal tensor capture!
```
Instead, use the specialized `tf.TensorArray` class:
```
l = tf.TensorArray(tf.int32, size=0, dynamic_size=True)
for i in tf.range(10):
l = l.write(l.size(), i) # Okay
```
#### Python collections of fixed structure are allowed TensorFlow control flow
An exception to the previous rule is made by Python collections that are static,
that is, they don't grow in size for the duration of the computation.
Caution: Use functional programming style when manipulating static collections.
Examples:
```
static_list = [tf.constant(3)]
while d.prop > 0:
static_list[0] -= 1 # Okay -- static_list does not change structure
```
```
static_object = MyClass()
static_object.field = tf.constant(3)
while static_object.field > 0:
static_object.field -= 1 # Okay -- static_object does not change structure
```
```
static_dict = {'field': tf.constant(3)}
while static_dict['field'] > 0:
static_dict['field'] -= 1 # Okay -- static_dict does not change structure
```
However, remember to use functional programming style when these collections
are used inside control flow.
#### Python collections of fixed structure with dynamic index
A more subtle error occurs when the collection is static, but is accessed in a
dynamic way, that is with a key that is not constant.
For example:
```
d = {'a': tf.constant(3)}
for i in tf.range(10):
for key in d:
d[key] += i # Problem -- accessing `dict` using non-constant key
```
The code above will raise an "illegal capture" error. To remedy it, write it in
functional programming style:
```
d = {'a': tf.constant(3)}
for i in tf.range(10):
d = {key: value + i for key, value in d.items()} # Okay
```
### Shape and dtype consistency in TensorFlow control flow
Unlike Python, TensorFlow has limited support for dynamic typing. This means
that tensors must maintain consistent shapes and dtypes across control flow
paths.
Note: In general, these restrictions do not apply in control flow in Eager
execution, because Eager execution uses Python control flow, rather than
TensorFlow control flow ops.
#### Mixing dynamic computations and static shapes
Key Point: Use `.shape` on tensors of static shape, and `.shape.rank` on
tensors of static rank; only use `tf.shape` and `tf.rank` when the shape or
rank is dynamic.
TensorFlow has optional static types and shapes: the shape of tensors may be
static (e.g. `my_tensor.shape=(3, 3)` denotes a three by three matrix) or
dynamic (e.g. `my_tensor.shape=(None, 3)`) denotes a matrix with a dynamic
number of rows and three columns. When the shapes are dynamic, you can still
query it at runtime by using the `tf.shape()` function.
Note: `tf.shape` always returns a tensor.
For static shapes, TensorFlow will perform additional shape verifications at
graph construction time, that is, during tracing. These static shape
verifications are useful because they work like a compiler for example, errors
are caught early, before execution even begins.
For example:
```
x = tf.constant([1, 2, 3])
x[4] # Tracing error! 4 is out of bounds.
```
To avoid tracing errors, you can add static shape verifications, which help
make your code more robust:
```
if x.shape[0] > 4:
val = x[4]
else:
val = some_default_value
```
In the snippet above, the code is protected against index-out-of-bounds
errors. The code is also efficient because the verification `x.shape[0] > 4`
will not be included in the graph.
But what happens if you try to perform the index verifications using dynamic
control flow? You might expect that the code works in the same way:
```
val = tf.cond(
x.shape[0] >= 4,
lambda: x[4],
lambda: some_default_value)
```
However, TensorFlow will not let you write code that could result in an error,
even if that code appeared in a branch of a `tf.cond` statement that would
never execute. Remember that the shape of `x` is `(3,)`, so TensorFlow performs
static shape verification.
This can lead to surprising behavior when using `tf.shape` on tensors with
static shape in TensorFlow:
```
x = tf.constant((1, 2, 3))
if tf.shape(x)[0] > 4:
val = x[4] # Error at tracing: 4 is out of bounds!
else:
val = some_default_value
```
Because `tf.shape` always evaluates to a Tensor, the `if` statement above is
converted by AutoGraph into a `tf.cond`, which performs static shape
verification of both branches.
What if you need to write code which can handle both static and dynamic
shapes? There are a few options in this case:
A first option is to always work with dynamic shapes, for instance by
using `input_signature` in `tf.function`. Many shape and shape-related checks
are skipped when the shape is dynamic:
```
@tf.function(input_signature=(tf.TensorSpec(shape=(None,))))
def f(x): # x now has dynamic shape
if tf.shape(x)[0] >= 3: # Builds a tf.cond
val = x[4] # Okay, bounds checks are skipped when the shape is dynamic
else:
val = some_default_value
```
A second option is to first verify whether the shape is static or dynamic.
This can be done at tracing time, allowing to use Python `if` to only trace
the code that is suitable for the situation:
```
if x.shape[0] is None: # Python bool, does not use tf.cond
# ... use x.shape here ...
else:
# ... use tf.shape(x) here ...
```
#### Consistency of dtype
The dtypes across all code paths must be consistent in conditionals and loops.
For example, if a `tf.cond` (and correspondingly, an AutoGraph `if`) sets a
tensor value conditionally, then that tensor must have the same shape and dtype
in both branches of the conditional.
Example of illegal dtype change in a conditional:
```
x = tf.cond(
tf.random.uniform(()) > 0.5,
lambda: tf.constant(1, dtype=tf.int32),
lambda: tf.constant(1, dtype=tf.float32)) # Error -- inconsistent dtypes: int32, float32
```
The same restriction in AutoGraph code:
```
if tf.random.uniform(()) > 0.5:
x = tf.constant(1, dtype=tf.int32)
else:
x = tf.constant(1, dtype=tf.float32) # Error -- inconsistent dtypes: int32, float32
```
Example of illegal dtype change in a loop:
```
# This won't work - "x" changes dtype inside the loop.
x = tf.while_loop(
lambda _: tf.random.uniform(()) > 0.5,
lambda x: tf.constant(1, dtype=tf.float32),
loop_vars=(tf.constant(1, dtype=tf.int32),)) # Error -- inconsistent dtypes: int32, float32
```
The same restriction in AutoGraph code:
```
x = tf.constant(0, dtype=tf.int32)
while tf.random.uniform(()) > 0.5:
x = tf.constant(0, dtype=tf.float32) # Error -- inconsistent dtypes: int32, float32
```
#### Consistency of shape
The shapes across all code paths must be consistent in loops only. When tensors
do need to change shape across iterations, use `shape_invariants`.
Note: Shapes are allowed to be inconsistent in conditionals. The result will be
a partially dynamic shape.
In a `tf.while_loop` (and correspondingly, an AutoGraph `while` or `for` loop)
all loop variables must maintain consistent shape and dtype across iterations.
That is, every loop variable must have the same shape at the end of the loop
body as it had at the beginning of the loop body.
Example of illegal shape change in a loop:
```
def loop_body(x): # x.shape is ()
return tf.constant((1, 2, 3)) # Error -- inconsistent shapes: (), (3,)
x = tf.while_loop(
lambda _: tf.random.uniform(()) > 0.5,
loop_body,
loop_vars=(tf.constant(1,))
```
The same restriction in AutoGraph code:
```
x = tf.constant(1,)
while tf.random.uniform(()) > 0.5:
x = tf.constant((1, 2, 3)) # Error -- inconsistent shapes: (), (3,)
```
### Consistency of control flow types
In AutoGraph, one can write Python control flow like `for i in range(10)`, as
well as TensorFlow control flow like `for i in tf.range(10)`.
However, one could also write (illegal) programs which start as Python control
flow, then turn into TensorFlow control flow. In such cases, an error will be
raised.
Below are a few examples, along with recommendations.
#### Python loop, TF-dependent break or return
Example:
```
for i in range(10):
if tf.greater(i, 3):
break # error - TF break inside Python loop
```
The solution in this case is to change the loop type to a TF loop:
```
for i in tf.range(10):
if tf.greater(i, 3):
break # works
```
#### Python loop that turns into a TensorFlow loop
Example:
```
i = 10
while i > 0:
i = tf.math.subtract(i, 1) # error - loop would turn into a TF loop
```
The solution in this case is to make sure the loop type starts as a TF loop,
typically by making sure the condition is always a Tensor:
```
i = tf.constant(10) # works
while i > 0:
i = tf.math.subtract(i, 1)
```
#### TensorFlow loops never turn into Python loops
Note that this is a legal case, as TensorFlow implicitly converts all Python
values to Tensor:
```
i = tf.constant(10)
while i > 0:
i = 0 # this is ok, will be auto-converted to Tensor
```
### Access to source code
Key point: AutoGraph can only handle functions whose source code can be
accessed at runtime.
Almost all Python functions allow access to their source code. However, a few
exceptions exist:
* functions created in the Python interactive shell
* functions with native bindings (these do not have Python source code)
* functions created dynamically, using `exec` or `eval`
Use
[inspect.findsource](https://docs.python.org/3/library/inspect.html#inspect.findsource)
to quickly diagnose whether the source code is available for a function.
For example:
```
import inspect
def simple_function():
return 1
# If this raises an error, then AutoGraph prints a warning.
# If it returns source code, then AutoGraph should work as well.
inspect.findsource(simple_function)
```
#### Source code of lambda functions
##### TF 2.4 and newer
Key Point: When nesting lambda functions, use distinguishing argument names
to avoid parse errors.
The Python runtime exposes the source code of lambda functions, however it
may omit parts of the actual body, or include surrounding code. This may make it
impossible to parse the exact source code of the lambda function (see
https://github.com/tensorflow/tensorflow/issues/39832).
AutoGraph uses alternate methods to parse the source code more robustly, but
in rare cases it may be unable to distinguish between nested lambda functions
of identical signatures.
Example:
```
l = lambda x: lambda x: x + 1
```
AutoGraph raises an error for the code above because the parser cannot
distinguish between the two function signatures. To work around this limitation,
use distinct argument names:
```
l = lambda outer_x: lambda inner_x: inner_x + 1
```
##### Before TF 2.3 and older
In older versions of TensorFlow, the loading code for lambda functions is not
robust. Follow the guidance below to avoid errors.
Important: Declare lambda functions on single lines to make sure their source
code loads correctly.
The Python runtime exposes the source code of lambda functions, however it
may omit parts of the actual body, or include surrounding code. This may make it
impossible to parse the exact source code of the lambda function.
For example, consider the declaration of a lambda function below:
```
foo = (
lambda y: lambda x: x * y
- y
)
```
The Python runtime will report the following source code for `foo`:
```
>>> inspect.getsource(foo)
' lambda y: lambda x: x*y \n'
```
In other cases, the source code it returns is not valid Python code, resulting
in an error:
```
foo = (
'bar',
lambda: x)
```
The reported source code contains an invalid token `)`:
```
>>> inspect.getsource(foo[1])
' lambda: x)\n'
```
This shortcoming can be avoided by declaring the lambda in a single assignment
or return value, and avoiding placing it inside parentheses which could cause
auto-formatting tools to break it into multiple lines:
```
# Good - single assignment
my_lambda = lambda: x
# Good - single return
return lambda x, y: x*y - y
```
```
# Bad - wrapped in parentheses
my_lambda = (lambda x, y: x * y - y)
# Bad - inlined in another expression
foo(lambda x, y: x + y, bar)
```
tensorflow/python/autograph/g3doc/reference/operators.md
+422
View File
@@ -0,0 +1,422 @@
# AutoGraph reference
[Index](index.md)
## Operator semantics
### Definition
This section describes the semantics of the operators used in code generated by
AutoGraph. Understanding these operators will make it easier to read the
generated code.
AutoGraph operators are Python functions that replace certain Python constructs
in the generated code.
For example, the following statement:
```
if x:
y = 1
else:
y = 2
```
Will result in the following generated code:
```
def get_state():
return (y,)
def set_state(vars_):
nonlocal y
(y,) = vars_
def if_body():
nonlocal y
y = 1
def else_body():
nonlocal y
y = 2
y = ag__.Undefined('y')
ag__.if_stmt(ag__.ld(x), if_body, else_body, get_state, set_state, ('y',), 1)
```
In the example above, `ag__.if_stmt`, `ag__.ld` and `ag__.Undefined` are all
AutoGraph operators.
The source of truth for these operators is the [source code](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/python/autograph/operators)
. All public symbols exported by that module is considered an operator.
### Type-based dispatch
AutoGraph replaces Python statements with operators in order to enable
type-based dispatch. If Python didn't support things like `__add__`, then
AutoGraph would already have an `add` operator.
Dispatch means simply that the operator does different things based on the type
of input.
Generally, the dispatch follows these rules:
* if the input is a type that would execute normally under Python (this is also
referred to as "the default path"), then AutoGraph always reverts to the
corresponding Python operator. For example, `ag__.not(False)` always has the
same result as `not False`.
* if the input is a TensorFlow type, then AutoGraph typically dispatches to an
equivalent TensorFlow API, performs additional checks or just raises an
error. For example, `ag__.eq(tf.constant(1), tf.constant(2))` has the same
result as `tf.math.equal(tf.constant(1), tf.constant(2))`.
The first rule above means that if you convert normal, non-TensorFlow code with
AutoGraph and call it with non-TensorFlow inputs, executing the generated code
should be no different than executing the original.
### Functional form
All AutoGraph operators use pure functional forms. This may sometimes mean that
expressions which normally appear bare in Python, are wrapped inside a function
(also known as thunk). If a Python statement appears as just `foo`, then a
corresponding thunk is `lambda: foo`.
### Operator list
#### Conditional expressions
[Source](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/autograph/operators/conditional_expressions.py)
##### `if_exp`
[Source](https://github.com/tensorflow/tensorflow/blob/bacd16a95d5a6f3d5081e3d56c515671c784d289/tensorflow/python/autograph/operators/conditional_expressions.py#L27)
The Python conditional statement: `foo if bar else baz`.
Args: cond: expression condition; same as `cond` in `_ if cond else _`. if_true:
true value (as thunk); same as `lambda: x` in `x if _ else _`. if_false: false
value (as thunk); same as `lambda: x` in `_ if _ else x`. expr_repr:
human-readable string representing `cond`. Used for error messages.
Example:
```
true_val if cond else false_val
```
```
ag__.if_expr(cond, lambda: true_val, lambda: false_val, 'cond')
```
Dispatch on `cond`:
* default: to Python if-else statement.
* tf.Tensor: to `tf.cond`, checking that `true_val` and `false_val` have
compatible shape and type.
#### Control flow
[Source](https://github.com/tensorflow/tensorflow/blob/bacd16a95d5a6f3d5081e3d56c515671c784d289/tensorflow/python/autograph/operators/control_flow.py)
Unlike Python, AutoGraph control flow operators use explicit control flow
variables, which include all symbols which are modified by the control flow.
For example, the code below has a single loop variable, `x`:
```
while x < 3:
x = x + 1
```
In addition, control flow that is dispatched to non-Python implementation is
subject to restrictions of the respective implementations. For example,
`tf.while_loop` requires that all loop variables have supported types (e.g.
`Tensor` of consistent shape and dtype).
##### `for_stmt`
[Source](https://github.com/tensorflow/tensorflow/blob/bacd16a95d5a6f3d5081e3d56c515671c784d289/tensorflow/python/autograph/operators/control_flow.py#L369)
For loop: `for var in target: body`, extended with a per-iteration
condition to handle early termination (e.g. due to a `break`).
Args:
* iter_: iteration target; same as `n` in `for _ in n`.
* extra_test: optional extra per-iteration condition (as thunk).
* body: loop body (as unary thunk); same as `def body(i): <b>` in `for i in _:
<b>`.
* get_state: returns the current value of the loop variables
* set_state: sets new values into the loop variables
* symbol_names: human-readable string representing each loop variable. Used
for error messages.
* opts: additional, implementation-specific, keyword arguments.
Example:
```
for i in range(3):
j = j + i
```
```
def get_state():
return (j,)
def set_state(vars_):
nonlocal j
(j,) = vars_
def loop_body(itr):
nonlocal j
i = itr
j = j + i
ag__.for_stmt(range(3), None, loop_body, get_state, set_state, ('j',), {})
```
Example (using extra_test):
```
for i in range(3):
if i > 2:
break
j = j + i
```
```
def get_state():
return (j,)
def set_state(vars_):
nonlocal j
(j,) = vars_
def loop_body(itr):
nonlocal j
i = itr
j = j + i
def extra_test():
return not(i <= 2)
ag__.for_stmt(range(3), extra_test, loop_body, get_state, set_state, ('j',), {})
```
Dispatch on `iter_`:
* default: to Python for loop (accounting for `extra_test`).
* `tf.Tensor` produced by `tf.range`: to `tf.while_loop`, removing the
`tf.range`.
* `tf.Tensor`, `tf.RagedTensor`: to `tf.while_loop`, checking the loop vars
for consistency. `opts` forwarded to `tf.while_loop`. Iterates over the
outermost dimension of the tensor (similar to `tf.map_fn`).
* `tf.data.Dataset`: to `tf.data.Dataset.take_while`, checking the loop vars
for consistency.
* `tf.data.Iterator`, `tf.distribute.Iterator`: to `tf.while_loop` called on
the iterator's `get_next_as_optional`, checking the loop vars for
consistency.
* `tf.distribute.Iterable`: to `tf.distribute.Iterable.reduce`.
##### `if_stmt`
[Source](https://github.com/tensorflow/tensorflow/blob/bacd16a95d5a6f3d5081e3d56c515671c784d289/tensorflow/python/autograph/operators/control_flow.py#L1125)
If statement: `if cond: body else: orelse-body`.
Args:
* cond: if condition; same as `cond` in `if cond`.
* body: true branch (as unary thunk); same as `def body(): <b>` in `if _:
<b>`.
* orelse: false branch (as unary thunk); same as `def body(): <b>` in `if _:
<b>`.
* get_state: returns the current value of the conditional variables
* set_state: sets new values into the conditional variables
* symbol_names: human-readable string representing each conditional variable.
Used for error messages.
* nouts: number of output conditional variables. Not all conditional variables
are outputs - some are just inputs. The first nouts values in get_state and
set_state are the conditional outputs.
Example:
```
if k > 1:
j = j + i
```
```
def get_state():
return (j, i)
def set_state(vars_):
nonlocal j, i
(j, i) = vars_
def body():
nonlocal j, i
j = j + i
def orelse():
pass
ag__.if_stmt(k > 1, body, orelse, get_state, set_state, ('j', 'i'), 1)
```
Dispatch on `cond`:
* default: to Python if statement.
* `tf.Tensor`: to `tf.cond`, removing the `tf.range`.
##### `while_stmt`
[Source](https://github.com/tensorflow/tensorflow/blob/bacd16a95d5a6f3d5081e3d56c515671c784d289/tensorflow/python/autograph/operators/control_flow.py#L811)
While loop: `while cond: body`.
Args:
* test: loop condition (as thunk); same as `def test(): cond` in `while cond`.
* body: loop body (as thunk); same as `def body(): <b>` in `while _: <b>`.
* get_state: returns the current value of the loop variables
* set_state: sets new values into the loop variables
* symbol_names: human-readable string representing each loop variable. Used
for error messages.
* opts: additional, implementation-specific, keyword arguments.
Example:
```
while j > 10:
j = j + i
```
```
def get_state():
return (j,)
def set_state(vars_):
nonlocal j
(j,) = vars_
def loop_test():
nonlocal j
return j > 10
def loop_body():
nonlocal j
j = j + i
ag__.while_stmt(loop_test, loop_body, get_state, set_state, ('j',), {})
```
Dispatch on return type of `test`:
* default: to Python while loop.
* `tf.Tensor`: to `tf.while_loop`.
#### Data structures
[Source](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/autograph/operators/data_structures.py)
##### `list_append`
[Source](https://github.com/tensorflow/tensorflow/blob/bacd16a95d5a6f3d5081e3d56c515671c784d289/tensorflow/python/autograph/operators/data_structures.py#L171)
List append operation: `l.append(x)`. Callers should assume that the list
argument is modified, if that is possible.
Args:
* list_: a list-like value.
* x: value to append to list.
Returns:
* same as list_, with an appended value.
Example:
```
l.append(x)
```
```
l = ag__.list_append(l, x)
```
Dispatch on `list_`:
* default: to `list_.append`.
* `tf.Tensor`: to `tf.raw_ops.tensor_list_push_back`.
* `tf.TensorArray`: to `tf.TensorArray.write`.
##### `list_pop`
[Source](https://github.com/tensorflow/tensorflow/blob/bacd16a95d5a6f3d5081e3d56c515671c784d289/tensorflow/python/autograph/operators/data_structures.py#L235)
List pop operation: `l.pop(i)`. Callers should assume that the list
argument is modified, if that is possible.
Args:
* list_: a list-like value.
* i: optional index to remove from.
* opts: optional, implementation-specific arguments.
Returns:
* new_list: same as list_, with the value removed
* x: the value that was removed
Example:
```
x = l.pop()
```
```
l, x = ag__.list_pop(l)
```
Dispatch on `list_`:
* default: to `list_.pop`.
* `tf.Tensor`: to `tf.raw_ops.tensor_list_pop_back`.
##### `list_stack`
##### `ListPopOpts`
##### `ListStackOpts`
##### `new_list`
### Exceptions
##### `assert_stmt`
### Boolean
##### `and_`
##### `eq`
##### `not_`
##### `not_eq`
##### `or_`
### Python built-ins
##### `float_`
##### `int_`
##### `len_`
##### `print_`
##### `range_`
### Slicing
##### `get_item`
##### `GetItemOpts`
##### `set_item`
### Variables
##### `ld`
##### `ldu`
##### `Undefined`
##### `UndefinedReturnValue`