Getting Started with TensorFlow 2.0

TensorFlow has long been a powerful and widely-used framework for building and training neural network models. TensorFlow 2.0’s use of the Keras high-level API makes designing and training neural networks straightforward, while its eager execution mode makes prototyping and debugging models simple. This course first explores the basic features of TensorFlow 2.0 and how its programming model differs from TensorFlow 1.x. It covers the basic working of a neural network and its active learning unit, the neuron. It then compares and contrasts static and dynamic computation graphs and explains the advantages and disadvantages of each. It explains how a neural network is trained using gradient descent optimization, and how the GradientTape library in TensorFlow calculates gradients automatically during training. Finally, it works with the different APIs in Keras and shows how they lend themselves to different use cases — sequential models (layers stacked one on top of another), the functional API, and model subclassing — used to build regression and classification models.

Table of Contents

• Module 1: Exploring the TensorFlow 2.0 Framework
  • Prerequisites and Learning Path
  • TensorFlow 1.x vs. TensorFlow 2.0
  • Introducing Neural Networks
  • Neurons and Activation Functions
  • Demo: Installing and Setting Up TensorFlow
  • Demo: Tensors and Tensor Operations
  • Demo: Variables
  • TensorFlow and Keras
• Module 2: Understanding Dynamic and Static Computation Graphs
  • The Computation Graph
  • Static vs. Dynamic Computation Graphs
  • Demo: TensorFlow v1 Sessions for Static Graphs
  • Demo: Visualizing Graphs with TensorBoard
  • Demo: Eager Execution
  • The tf.function Decorator and AutoGraph
  • Demo: Running in Graph Mode Using @tf.function
  • Demo: Python Side Effects in Graph Mode
  • Demo: Variables in Graph Mode
• Module 3: Computing Gradients for Model Training
  • Gradient Descent Fundamentals
  • Forward and Backward Passes
  • Calculating Gradients: Partial Derivatives and the Gradient Vector
  • Reverse-Mode Automatic Differentiation
  • Demo: GradientTape for Gradient Calculations
  • Demo: Understanding GradientTape Behavior
  • Demo: Simple Linear Regression with Manual Gradient Updates
  • Demo: Simple Regression Using a Sequential Model
• Module 4: Using the Sequential API in Keras
  • Introducing the Sequential API
  • Demo: Exploring and Processing the Life Expectancy Dataset
  • Demo: Building and Training a Sequential Model
  • Demo: Visualizing Training with TensorBoard
  • Demo: Comparing Optimizers and Activation Functions
• Module 5: Using the Functional API and Model Subclassing in Keras
  • The Functional API and Model Subclassing
  • Demo: Exploring the Heart Disease Dataset
  • Demo: Building a Binary Classifier with the Functional API
  • Demo: Exploring the Wine Dataset
  • Demo: Multi-Class Classification with Model Subclassing
• Summary

Module 1: Exploring the TensorFlow 2.0 Framework

Prerequisites and Learning Path

This material assumes a basic understanding of machine learning algorithms and some prior experience building and training simple ML models. A basic understanding of how neural networks work is helpful, though not strictly required — an overview of how neural networks are built and trained is included. Familiarity with TensorFlow 1.x is helpful but not required. Comfort programming in Python is assumed, and all demos use Python running in Jupyter notebooks.

The learning path covers:

1. Exploring the TensorFlow 2.0 framework and comparing it with PyTorch and TensorFlow 1.
2. Working with tensors and variables in TF 2.0.
3. Dynamic and static computation graphs, and how TensorFlow 2 supports both.
4. Training neural network models by computing gradients with the GradientTape.
5. The high-level Keras API: the Sequential API, the Functional API, and model subclassing.

TensorFlow 1.x vs. TensorFlow 2.0

TensorFlow was first launched publicly in November 2015. It is free, open source, and was originally developed by Google, which continues to support it. A competing framework, PyTorch, launched in October 2016. It is also free and open source, developed and supported by Facebook engineers. PyTorch gained widespread adoption thanks to its ease of use and its support for dynamic computation graphs, which makes prototyping ML models simple.

TensorFlow 2.0 is a major new version of TensorFlow, released in September 2019, with several major improvements including support for dynamic computation graphs and greater ease of use. TensorFlow 2.0 is a completely new version that is not backward compatible with TensorFlow 1. If you have never worked with TensorFlow 1, there is no reason to start with it now unless your organization already relies on it. TensorFlow 2.0 is much closer to PyTorch than to TensorFlow 1 — familiarity with PyTorch gives you a head start on TensorFlow 2.

The following table compares TensorFlow 1.x with PyTorch:

The following table compares TensorFlow 1.x with TensorFlow 2.0:

Keras was originally designed as a high-level API capable of running on top of TensorFlow, CNTK, and Theano. With the evolution of both TensorFlow and Keras, Keras is now a central part of the tightly connected TensorFlow 2.0 ecosystem, and it is the recommended way to build, train, and evaluate models in TensorFlow 2. The single biggest change a developer notices in TensorFlow 2 is the default use of eager execution, discussed in detail in Module 2.

Introducing Neural Networks

Building a classification model (for example, a model that classifies reviews as positive or negative) involves two phases:

1. Training — feed in a large, correctly labeled corpus of data that the model learns from.
2. Prediction — use the fully trained model to classify new instances the model has not seen before.

During training, the model makes classifications based on what it has learned so far. This prediction is compared to the true label using a loss function (also called an objective function), and the resulting feedback is used to improve the model’s parameters. This is the basic training loop: feed in data, get an output, compare the output to the truth, and feed that difference back into the model to improve it.

The input fed into a machine learning model is referred to as its features; the output prediction is referred to as the target or label. Representation-based machine learning systems, such as neural networks, learn by themselves which features matter — given a large number of input features, they separate signal from noise.

A neural network is a deep learning algorithm — one of a family of algorithms that learn from input data which features matter. Neural networks are, in fact, the most common class of deep learning algorithm, which is why the terms are often used synonymously. The fundamental building block of any neural network is the neuron, the active learning unit.

A neural network is composed of many layers, each made up of neurons. Each layer learns something slightly different from the underlying data, and the learnings from each layer are combined to produce the final prediction. Layers that accept input into the network or produce its output are called visible layers — you interact with them directly. Any layer between the visible layers is a hidden layer — you don’t interact with it directly, but it still operates on the data flowing through the network and extracts information from it.

flowchart LR
    subgraph InputLayer["Visible Layer: Input"]
        I1((n1))
        I2((n2))
    end
    subgraph HiddenLayer["Hidden Layer"]
        H1((n1))
        H2((n2))
        H3((n3))
    end
    subgraph OutputLayer["Visible Layer: Output"]
        O1((n1))
    end
    I1 --> H1
    I1 --> H2
    I1 --> H3
    I2 --> H1
    I2 --> H2
    I2 --> H3
    H1 --> O1
    H2 --> O1
    H3 --> O1

Neurons and Activation Functions

A neuron is nothing more than a simple mathematical function. It takes several x values as input (x1 through xn) and produces a single output value y, which is then fed onward to neurons in subsequent layers. For a neuron that is actively learning, any change in its inputs should trigger a corresponding change in its output.

Every connection between two neurons in different layers is associated with a weight W, which indicates the strength of that connection — the more sensitive the second neuron is to the output of the first, the higher the value of W. A neural network — its layers, neurons, and interconnections — together makes up a computation graph: the nodes of the graph are neurons, and the edges carry the data the neurons operate on, referred to as tensors. Training a neural network is often described as “finding the model’s parameters” — the weights associated with every edge (connection) in this computation graph.

flowchart LR
    X1[x1] -->|W1| N((Neuron))
    X2[x2] -->|W2| N
    Xn[xn] -->|Wn| N
    B[bias b] --> N
    N -->|affine transform + activation| Y[y]

Each neuron applies exactly two mathematical transformations to its input:

1. Affine transformation — responsible only for learning linear relationships between the inputs and the output. Thought of as a weighted sum of the inputs plus a bias term b: these weights and the bias are the model parameters found during training.
2. Activation function — applied to the output of the affine transformation, responsible for discovering nonlinear relationships between inputs and output.

When the activation function is simply the identity function (it passes through the affine transformation’s output unchanged), the neuron is referred to as a linear neuron, and it can learn only linear relationships in the data. It is the combination of the affine transformation and a nonlinear activation function that allows a neuron to learn an arbitrary relationship between its inputs and output.

Common activation functions:

Every activation function has a gradient or active region — the range over which it is sensitive to changes in input. A neuron that is actively learning operates in its active region; a saturated neuron operates in a region where its output does not change meaningfully as inputs are tweaked, and it stops contributing to learning. Training and adjusting neural network weights requires neurons to remain in their active regions.

Demo: Installing and Setting Up TensorFlow

The demo environment uses the Anaconda distribution of Python with Jupyter notebooks. Requirements verified before installing TensorFlow 2.0:

python --version        # 3.7.6
jupyter --version       # Notebook version 6.0.3
pip --version           # 20.1.1 (TensorFlow 2.0 requires pip >= 19)

# Upgrade pip if needed
python -m pip install --upgrade pip

TensorFlow 2.0 supports Python 3.5–3.8 and is tested on Ubuntu, Windows 7+, macOS, and Raspbian. Standard data-processing and visualization libraries are assumed to be installed (pandas, Matplotlib, Seaborn), along with scikit-learn for preprocessing and train/test splitting:

sudo pip install tensorflow

# Additional libraries used to visualize Keras models
pip install pydot
conda install graphviz

Once installed, the Jupyter Notebook server is launched:

jupyter notebook

The working directory used throughout the course contains two folders: datasets (holding heart.csv and life_expectancy.csv, used in later demos) and my_notebooks (where the course’s Jupyter notebooks live).

Demo: Tensors and Tensor Operations

Tensors are the fundamental data structure in TensorFlow — essentially multidimensional arrays, very similar to NumPy arrays. The key difference is that tensors are designed to be stored and processed across multiple devices (CPUs or GPUs), which is what enables distributed training.

import tensorflow as tf
import numpy as np

tf.debugging.set_log_device_placement(True)

tf.debugging.set_log_device_placement(True) prints additional log messages describing where tensors are located and on which device operations execute. TensorFlow 2.0 computation graphs execute eagerly by default:

tf.executing_eagerly()   # True

A tensor can hold a scalar:

x0 = tf.constant(3)
x0
# <tf.Tensor: shape=(), dtype=int32, numpy=3>

print(x0)
x0.shape    # TensorShape([])  -- scalar, no dimensions
x0.dtype    # tf.int32, inferred automatically
x0.numpy()  # 3 -- NumPy representation of the tensor

Arithmetic operations on tensors produce new tensors (tensors are immutable):

result0 = x0 + 5
result0   # <tf.Tensor: shape=(), dtype=int32, numpy=8>

A one-dimensional tensor (vector), with broadcasting:

x1 = tf.constant([1.1, 2.2, 3.3, 4.4])
x1                          # shape=(4,), dtype=float32

result1 = x1 + 5            # broadcasting a scalar over every element
result1 = x1 + tf.constant(5.0)
result1 = tf.add(x1, tf.constant(5.0))   # tf namespace offers equivalents for arithmetic operators

A two-dimensional tensor, casting, and element-wise multiplication:

x2 = tf.constant([[1, 2, 3, 4], [5, 6, 7, 8]])
x2.shape   # (2, 4)
x2.dtype   # int32

x2 = tf.cast(x2, tf.float32)   # change the dtype of a tensor
x2.dtype   # float32

result3 = tf.multiply(x1, x2)   # element-wise multiplication; shapes must be compatible

Converting between tensors and NumPy arrays:

arr_x1 = x1.numpy()                       # tensor -> NumPy array

arr_x4 = np.array([[10, 20], [30, 40], [50, 60]])
x4 = tf.convert_to_tensor(arr_x4)         # NumPy array -> tensor

np.square(x2)     # NumPy operations work on tensors...
np.sqrt(x2)       # ...but are NOT part of the TensorFlow computation graph,
                  # and can cause problems when training neural networks — avoid where possible

tf.is_tensor(arr_x4)   # False
tf.is_tensor(x4)       # True

Helper methods to initialize and reshape tensors:

t0 = tf.zeros([3, 5], tf.int32)
t1 = tf.ones([2, 3], tf.int32)

t0_reshaped = tf.reshape(t0, (5, 3))   # reshape only works if element counts match

Demo: Variables

Variables are the recommended way to represent shared, persistent, mutable state in a TensorFlow program. Unlike tensors, variables are mutable — a tf.Variable is a mutable container holding a tensor, and the tensor it holds can be changed by running operations on the variable.

v1 = tf.Variable([[1.5, 2, 5], [2, 6, 8]])
# dtype inferred automatically if not specified

v2 = tf.Variable([[1, 2, 3], [4, 5, 6]], dtype=tf.float32)  # explicit dtype

All operations you can perform on tensors can be performed on variables — the operations act on the underlying tensor, and the result of such an operation is a plain tf.Tensor, not a variable:

tf.add(v1, v2)                 # returns a tf.Tensor
tf.convert_to_tensor(v1)       # tensor representation of the variable
v1.numpy()                     # NumPy representation of the underlying tensor

Variables are mutated in place using assign-family methods:

v1.assign([[10, 20, 30], [40, 50, 60]])      # replace the entire tensor
v1[0, 0].assign(100)                          # assign to a specific element
v1.assign_add([[1, 1, 1], [1, 1, 1]])         # add in place
v1.assign_sub([[2, 2, 2], [2, 2, 2]])         # subtract in place

Instantiating one variable from another duplicates the backing tensor — variables never share memory:

var_a = tf.Variable([2.0, 3.0])
var_b = tf.Variable(var_a)     # a full copy of var_a's tensor

var_b.assign([200, 300])       # mutates only var_b

print(var_a.numpy())   # [2. 3.]  -- unchanged
print(var_b.numpy())   # [200. 300.]

TensorFlow and Keras

TensorFlow allows large-scale scientific computation, the most common use of which is building neural network models. Keras is a central part of the tightly connected TensorFlow 2.0 ecosystem — a high-level API for constructing neural networks using layers and models. Keras is not separate from TensorFlow: the TensorFlow 2.0 implementation includes a full implementation of the Keras API spec, exposed through the tf.keras namespace.

Keras is no longer an afterthought — it contains first-class support for TensorFlow-specific functionality, including estimators, pipelines for transforming data, and eager execution of dynamic computation. Working with TensorFlow 2 means using the tf.keras APIs to build, train, and evaluate models, save and restore model parameters, and take advantage of GPU acceleration during training.

Module 2: Understanding Dynamic and Static Computation Graphs

The Computation Graph

A neural network is a computation graph made up of tensors and computation nodes (neurons). All of the edges connecting individual neurons across layers are directed, which makes a neural network a directed acyclic graph (DAG) — this DAG is the computation graph. Every computation in TensorFlow can be thought of as a graph: nodes represent tensors (the data being transformed), and edges represent the functions that mutate that data. Executing the graph transforms input tensors into output results.

Representing computations as a graph is powerful because it lets the TensorFlow framework optimize the operations performed on input data:

• The graphical structure allows TensorFlow to trim and remove common subexpressions.
• The DAG structure lets TensorFlow identify which operations are independent of one another and parallelize them across devices — this is what simplifies distributed training and deployment.

This premise — that neural networks are computation graphs that can be optimized — is not unique to TensorFlow; it applies to all neural network frameworks.

Static vs. Dynamic Computation Graphs

There are two fundamentally different ways to build and execute a computation graph:

• Static graphs are lazily executed — equivalent to symbolic programming.
• Dynamic graphs are eagerly executed — equivalent to imperative programming.

TensorFlow 1 worked directly with static computation graphs. TensorFlow 2.0 supports both dynamic and static computation graphs, which is what makes it so much more powerful than TensorFlow 1. The recommended practice is to develop using dynamic graphs (for easier debugging and prototyping) and deploy using static graphs (for better performance).

sequenceDiagram
    participant Dev as Developer
    participant Static as Static Graph (TensorFlow 1 / tf.compat.v1)
    participant Dynamic as Dynamic Graph (TensorFlow 2 Eager Mode)
    Dev->>Static: Define computation graph using placeholders
    Dev->>Static: Explicitly compile the graph
    Dev->>Static: session.run(graph, feed_dict = real data)
    Static-->>Dev: Result returned only after run()
    Dev->>Dynamic: Write a normal Python statement
    Dynamic-->>Dev: Result computed and returned immediately

The reason TensorFlow 2 supports both kinds of graph is that each is best in a different phase of the ML lifecycle: use eager execution (dynamic graphs) for fast feedback during development, then move to lazy execution (static graphs) in production for optimized performance.

Demo: TensorFlow v1 Sessions for Static Graphs

TensorFlow 1 semantics remain available through the tf.compat.v1 namespace.

import tensorflow as tf

tf.compat.v1.executing_eagerly()   # True by default in TF 2.0

Trying to use a Session while eager execution is enabled raises an error, because eager mode computes results immediately and there is no static graph for a session to execute:

a = tf.constant(5, name="a")
b = tf.constant(7, name="b")
c = tf.add(a, b, name="sum")

sess = tf.compat.v1.Session()
sess.run(c)   # Error: eager execution is enabled, there is no graph to run

To reproduce TensorFlow 1 semantics, eager execution must be explicitly disabled:

tf.compat.v1.disable_eager_execution()
tf.compat.v1.executing_eagerly()   # False

tf.compat.v1.reset_default_graph()

a = tf.constant(5, name="a")
b = tf.constant(7, name="b")
c = tf.add(a, b, name="sum")

c   # tensor of dtype int32 -- value not computed yet!

sess = tf.compat.v1.Session()
sess.run(c)          # 12 -- now the graph is actually executed

d = tf.multiply(a, b, name="product")
sess.run(d)          # 35

sess.close()          # frees all session resources

Variables and placeholders in tf.compat.v1 mode require explicit initialization and a feed dictionary to supply runtime values:

m = tf.Variable([4.0, 5.0, 6.0], tf.float32, name='m')
c = tf.Variable([1.0, 1.0, 1.0], tf.float32, name='c')

x = tf.compat.v1.placeholder(tf.float32, shape=[3], name='x')

y = m * x + c   # y's value is not known until the graph executes

init = tf.compat.v1.global_variables_initializer()

with tf.compat.v1.Session() as sess:
    sess.run(init)

    y_output = sess.run(y, feed_dict={x: [100.0, 100.0, 100.0]})

    print("Final result: mx + c = ", y_output)   # [401. 501. 601.]

    writer = tf.compat.v1.summary.FileWriter('./logs', sess.graph)
    writer.close()

The feed_dict argument to session.run supplies the values for placeholder inputs at execution time. A FileWriter is used here to write out the session graph so it can be visualized in TensorBoard.

Demo: Visualizing Graphs with TensorBoard

TensorBoard can be launched from within a Jupyter notebook:

%load_ext tensorboard
%tensorboard --logdir="./logs" --port 6060

It is generally preferable to open TensorBoard directly in a browser tab (localhost:6060) rather than viewing it embedded in the notebook. The graph shows y = mx + c as a static computation graph: every operation (including the init node that initializes c and m) is a node in the graph, and clicking on a node reveals its dependencies and its inputs/outputs. The Trace inputs toggle highlights the exact path within the graph that leads to a particular node — for example, selecting the multiplication node shows exactly which inputs feed into that computation.

Demo: Eager Execution

Eager execution is TensorFlow 2.0’s default execution mode: every operation is evaluated immediately, with no separate “build” phase followed by an “execute” phase.

import tensorflow as tf

tf.executing_eagerly()   # True

x = [[10.]]
res = tf.matmul(x, x)
res   # <tf.Tensor: shape=(1, 1), numpy=array([[100.]], dtype=float32)>

a = tf.constant([[10, 20], [30, 40]])
b = tf.add(a, 2)
print(b)

print(a * b)   # every operation is evaluated and printed immediately

Variables in eager mode are initialized immediately, and there is no separate initializer operation or concept of a placeholder:

m = tf.Variable([4.0, 5.0, 6.0], tf.float32, name='m')
c = tf.Variable([1.0, 1.0, 1.0], tf.float32, name='c')
x = tf.Variable([100.0, 100.0, 100.0], tf.float32, name='x')

y = m * x + c   # computed immediately

Because eager execution evaluates every operation right away, it is possible to use native Python control-flow constructs — if/elif/else, for loops — directly in your computation. This is something that was very hard to do with static, build-then-run computation graphs:

def tensorflow(max_num):
    counter = tf.constant(0)
    max_num = tf.constant(max_num)

    for num in range(0, max_num.numpy() + 1):
        num = tf.constant(num)

        if int(num % 3) == 0 and int(num % 5) == 0:
            print('Divisible by 3 and 5: ', num.numpy())
        elif int(num % 3) == 0:
            print('Divisible by 3: ', num.numpy())
        elif int(num % 5) == 0:
            print('Divisible by 5: ', num.numpy())
        else:
            print(num.numpy())

        counter += 1

tensorflow(15)

The tf.function Decorator and AutoGraph

Constructing static computation graphs for efficient execution requires metaprogramming — a technique where one program executes while a second program reads, compiles, and analyzes the first during execution, typically to shift computation from runtime to compile-time. TensorFlow 1.0 relied heavily on metaprogramming to build its build-then-run static computation graphs. Metaprogramming is clunky and difficult even for experienced developers, and this was one reason TensorFlow 1.0 was losing ground to PyTorch, whose dynamic computation graphs eliminated the need for it entirely.

TensorFlow 2.0 reduces the need for metaprogramming: for simple, prototyping use cases, you simply write Python functions and they run eagerly by default. But dynamic graphs are not as efficient as static ones, and heavy-duty use cases (distributed training, large models, large training datasets, many small granular operations) still benefit from static graphs. TensorFlow 2.0 bridges this gap with tf.function: prototype your code in eager mode, then get the optimization benefits of a static graph by applying the @tf.function decorator.

tf.function is a decorator applied to Python functions that perform computations on input data. TensorFlow will:

1. Trace the function the first time it is invoked, recording the exact series of operations performed.
2. Rewrite the Python control flow encountered during tracing into TensorFlow equivalents.
3. Construct a static computation graph from that trace — this graph-generating process is often referred to as AutoGraph.
4. Execute the generated graph on subsequent invocations, without re-running the Python code.

Because different input types produce different traces, tf.function generates a separate computation graph for each distinct input signature (a process referred to as tracing). Python statements with side effects (print statements, list mutation, etc.) execute only during tracing — subsequent calls reuse the previously generated graph and skip the Python side effects.

flowchart TD
    A["Call an @tf.function-decorated function"] --> B{"Has a graph already<br/>been traced for this<br/>input signature?"}
    B -->|No| C["Trace the Python code once"]
    C --> D["Python side effects execute<br/>(print, list.append, etc.)"]
    D --> E["AutoGraph converts control flow<br/>and generates a static graph"]
    E --> F["Execute the generated graph"]
    B -->|Yes| F
    F --> G["Return the result"]

Best practices when using tf.function:

• Prototype and develop in eager mode first; only decorate with @tf.function once the code is verified to work correctly.
• Do not rely on Python side effects (mutating objects, appending to Python lists) to behave correctly once decorated — convert such logic to TensorFlow-native equivalents where possible.
• NumPy and plain Python values are converted to constants by tf.function.
• Be careful when decorating stateful functions such as generators and iterators.

Demo: Running in Graph Mode Using @tf.function

Simple helper functions can be decorated directly:

@tf.function
def add(a, b):
    return a + b

@tf.function
def sub(a, b):
    return a - b

@tf.function
def mul(a, b):
    return a * b

@tf.function
def div(a, b):
    return a / b

print(add(tf.constant(5), tf.constant(2)))   # 7
print(sub(tf.constant(5), tf.constant(2)))   # 3
print(mul(tf.constant(5), tf.constant(2)))   # 10
print(div(tf.constant(5), tf.constant(2)))   # 2.5

tf.function-decorated functions can invoke other TensorFlow operations and can be nested:

@tf.function
def matmul(a, b):
    return tf.matmul(a, b)

@tf.function
def linear(m, x, c):
    return add(matmul(m, x), c)   # y = mx + c

m = tf.constant([[4.0, 5.0, 6.0]], tf.float32)
x = tf.Variable([[100.0], [100.0], [100.0]], tf.float32)
c = tf.constant([[1.0]], tf.float32)

linear(m, x, c)

Regular Pythonic dynamic control flow (branching, loops) is automatically converted by AutoGraph into TensorFlow equivalents:

@tf.function
def pos_neg_check(x):
    reduce_sum = tf.reduce_sum(x)

    if reduce_sum > 0:
        return tf.constant(1)
    elif reduce_sum == 0:
        return tf.constant(0)
    else:
        return tf.constant(-1)

pos_neg_check(tf.constant([100, 100]))     # 1
pos_neg_check(tf.constant([100, -100]))    # 0
pos_neg_check(tf.constant([-100, -100]))   # -1

Functions decorated with @tf.function can also perform operations with side effects, and tf.range enables unrolling of for loops within the static computation graph:

num = tf.Variable(7)

@tf.function
def add_times(x):
    for i in tf.range(x):
        num.assign_add(x)

add_times(5)
print(num)   # 32  (7 + 5*5)

The order of dependencies specified in code is preserved:

a = tf.Variable(1.0)
b = tf.Variable(2.0)

@tf.function
def f(x, y):
    a.assign(y * b)
    b.assign_add(x * a)
    return a + b

f(1, 2)

Demo: Python Side Effects in Graph Mode

Python functions are polymorphic — the same function can be called with different argument types, and tf.function traces (and retraces) separately for each distinct input type:

@tf.function
def square(a):
    print("Input a: ", a)
    return a * a

x = tf.Variable([[2, 2], [2, 2]], dtype=tf.float32)
square(x)   # traces a new graph for float32; "Input a:" IS printed

y = tf.Variable([[2, 2], [2, 2]], dtype=tf.int32)
square(y)   # a different type -> retraces a new graph; "Input a:" IS printed again

z = tf.Variable([[3, 3], [3, 3]], dtype=tf.float32)
square(z)   # reuses the float32 graph already traced; "Input a:" is NOT printed

Retrieving a concrete function for a specific input signature:

concrete_int_square_fn = square.get_concrete_function(tf.TensorSpec(shape=None, dtype=tf.int32))
concrete_float_square_fn = square.get_concrete_function(tf.TensorSpec(shape=None, dtype=tf.float32))

concrete_int_square_fn(tf.constant([[2, 2], [2, 2]], dtype=tf.int32))
concrete_float_square_fn(tf.constant([[2.1, 2.1], [2.1, 2.1]], dtype=tf.float32))

concrete_float_square_fn(tf.constant([[2, 2], [2, 2]], dtype=tf.int32))
# TypeError: the concrete function only accepts float32 input

print (a Python side effect) executes only during tracing, while tf.print (a graph operation) executes every time the graph runs:

@tf.function
def f(x):
    print("Python execution: ", x)
    tf.print("Graph execution: ", x)

f(1)   # both "Python execution" and "Graph execution" print (first trace)
f(1)   # only "Graph execution" prints (graph is reused)
f("Hello tf.function!")   # different input type -> retraces -> both print again

Mutating a Python list from within a tf.function is also a side effect and only happens once, during tracing:

arr = []

@tf.function
def f(x):
    for i in range(len(x)):
        arr.append(x[i])

f(tf.constant([10, 20, 30]))
arr   # only populated once, from the tracing pass

The TensorFlow-native equivalent — a tf.TensorArray — correctly participates in the graph on every call:

@tf.function
def f(x):
    tensor_arr = tf.TensorArray(dtype=tf.int32, size=0, dynamic_size=True)

    for i in range(len(x)):
        tensor_arr = tensor_arr.write(i, x[i])

    return tensor_arr.stack()

result_arr = f(tf.constant([10, 20, 30]))

If Python side effects must run on every call, tf.py_function provides an “exit hatch”:

external_list = []

def side_effect(x):
    print('Python side effect')
    external_list.append(x)

@tf.function
def fn_with_side_effects(x):
    tf.py_function(side_effect, inp=[x], Tout=[])

fn_with_side_effects(1)
fn_with_side_effects(2)
external_list   # [1, 2] -- side effect ran on every call

Standard Python control flow (for/while, including break and continue) is converted by AutoGraph into tf.while_loop equivalents:

@tf.function
def some_tanh_fn(x):
    while tf.reduce_sum(x) > 1:
        x = tf.tanh(x)
    return x

some_tanh_fn(tf.random.uniform([10]))

Demo: Variables in Graph Mode

Variables behave differently in eager mode versus graph mode. In eager mode, a function that instantiates a variable creates a new variable every time it is called:

def f(x):
    v = tf.Variable(1.0)
    return v + x

f(1)   # works fine every time in eager mode

The same function decorated with @tf.function raises an error on any call after the first:

@tf.function
def f(x):
    v = tf.Variable(1.0)
    return v + x

f(1)   # ValueError: tf.function-decorated function tried to
       #             create variables on non-first call

In graph mode, the variable created the first time the graph is traced is reused on every subsequent call — so variables must be created exactly once. The idiomatic pattern is to defer variable creation to a stateful class:

class F:
    def __init__(self):
        self._b = None

    @tf.function
    def __call__(self, x):
        if self._b is None:
            self._b = tf.Variable(1.0)
        return self._b + x

f = F()
f(1)   # works correctly on every call

tf.autograph.to_code reveals the graph-mode equivalent code that TensorFlow generates for a given function.

Static computation graphs offer a significant speedup for code that performs many small, granular operations, because tf.range allows the loop to be unrolled directly inside the static graph rather than iterating in Python:

@tf.function
def g(x):
    return x

# Using tf.range (graph mode unrolling): ~0.83 seconds for 2000 iterations
for i in tf.range(2000):
    g(i)

# Using plain Python range (no graph benefit): ~14 seconds for the same operation
for i in range(2000):
    g(i)

Module 3: Computing Gradients for Model Training

Gradient Descent Fundamentals

The parameters of a neural network model are the weights and biases of its neurons — the values found during training. To understand the training process, consider the simplest possible neural network: a single neuron with no activation function (a linear neuron), used to build a simple linear regression model. A set of x values is fed into this single neuron, and the model tries to fit a straight line — the machine learning model itself — that best predicts y from x.

For a regression model like this, the objective function to minimize is typically the mean squared error (MSE) between predicted and actual y values — this is what is minimized in order to find the best-fit regression line.

If you plot the weight w and bias b of the single neuron along the x- and y-axes, and the resulting MSE along the z-axis for every possible combination of w and b, you get a surface of MSE values. The best-fit model corresponds to the point on this surface where MSE has its smallest value. Training starts at some initial point on this surface (random initial values for w and b) and descends the surface using gradient descent to reach the point of minimum MSE. This same principle extends from a single neuron to networks with any number of neurons arranged across any number of layers — the model parameters always start with random initial values, and training converges on the best values using gradient descent optimization.

Forward and Backward Passes

Training a neural network alternates between two passes:

• Forward pass — data is fed into the network using its current weights and biases, producing a prediction y_predicted.
• Backward pass — the prediction is compared against the true label from the training data to compute an error/loss. This error is fed into an optimizer, which calculates gradients and uses them to update the model’s parameters, working backward from the last layer to the first.

Once the backward pass updates the parameters, another forward pass is made using the new values, and the cycle repeats. This is why training a neural network requires two passes: a forward pass to obtain a prediction and loss, and a backward pass to update parameters using gradients calculated from that loss.

flowchart TD
    A["Initialize model parameters<br/>(random weights and biases)"] --> B["Forward pass:<br/>compute y_predicted"]
    B --> C["Compute loss:<br/>compare y_predicted to y_actual"]
    C --> D["Backward pass:<br/>GradientTape / optimizer computes gradients"]
    D --> E["Optimizer updates<br/>weights and biases"]
    E --> F{"More epochs<br/>remaining?"}
    F -->|Yes| B
    F -->|No| G["Trained model"]

Calculating Gradients: Partial Derivatives and the Gradient Vector

The loss function (denoted here as ℒ) captures the difference between a model’s predicted output and the actual training label for a single instance, e.g., in a regression model this is the mean squared error.

A gradient is simply a vector of partial derivatives. For every parameter in the model, you calculate the partial derivative of the loss with respect to that parameter, producing the list (vector) of partial derivatives that make up the gradient. The partial derivative of the loss with respect to a parameter W is computed by holding every other parameter and the input constant, and asking: by how much does the loss change when W changes by an infinitesimal amount? This measures how sensitive the loss is to changes in W.

For a single neuron with parameters W1 and B1, the gradient contains the partial derivative of the loss with respect to W1 and the partial derivative of the loss with respect to B1. Gradient descent uses these values to find the parameter values where the loss is smallest. The same principle extends to arbitrarily large networks, where the gradient vector becomes correspondingly large.

There are three general approaches to calculating the partial derivatives that make up a gradient:

Because automatic differentiation scales well while remaining relatively easy to implement, all major neural network frameworks — TensorFlow, PyTorch, and others — rely on it to calculate gradients. In TensorFlow, the library used to calculate gradients for the backward pass (backpropagation) is the GradientTape.

Reverse-Mode Automatic Differentiation

Gradients calculated during training apply to a specific time instance (iteration) t. They are multiplied by the learning rate and used to compute the parameter values for the next time instance, t + 1 — moving every parameter in the direction of a reducing gradient, toward the smallest value of loss.

The learning rate is a number between 0 and 1 that determines the size of the step taken in the direction of the reducing gradient:

• A larger learning rate can converge faster, but risks parameters “jumping around” rather than smoothly descending to the minimum loss.
• A smaller learning rate converges more slowly and typically needs many more training epochs.

This technique — using gradients calculated at time t to compute parameters for time t + 1 — is called reverse-mode automatic differentiation, and it requires exactly two passes through the network: a forward pass to get a prediction, and a backward pass to update parameters. The backward pass is needed only during training. In TensorFlow 2.0, tape.gradient is the method used to calculate these gradients; much of the underlying mechanics of automatic differentiation is hidden from view when working with the high-level Keras APIs.

Demo: GradientTape for Gradient Calculations

The GradientTape records every operation that occurs in a with block (the forward pass) so that gradients can subsequently be computed with respect to any of the recorded variables:

import numpy as np
import matplotlib.pyplot as plt
import tensorflow as tf

x = tf.Variable(4.0)

with tf.GradientTape() as tape:
    y = x**2   # forward pass: y = x^2

dy_dx = tape.gradient(y, x)
dy_dx   # 8.0  (derivative of x^2 is 2x, evaluated at x=4)

Gradients can also be computed with respect to tensors of arbitrary shape:

w = tf.Variable(tf.random.normal((4, 2)))
b = tf.Variable(tf.ones(2, dtype=tf.float32))
x = tf.Variable([[10., 20., 30., 40.]], dtype=tf.float32)

with tf.GradientTape(persistent=True) as tape:
    y = tf.matmul(x, w) + b
    loss = tf.reduce_mean(y**2)

[dl_dw, dl_db] = tape.gradient(loss, [w, b])
# dl_dw has the same shape as w; dl_db has the same shape as b

By default, a GradientTape’s resources are released as soon as tape.gradient is called once. To call tape.gradient more than once against the same recorded computation, instantiate the tape with persistent=True.

When Keras layers are used, the GradientTape automatically records every operation in the forward pass and can compute gradients with respect to all of a layer’s trainable parameters:

layer = tf.keras.layers.Dense(2, activation='relu')
x = tf.constant([[10., 20., 30.]])

with tf.GradientTape() as tape:
    y = layer(x)
    loss = tf.reduce_sum(y**2)

grad = tape.gradient(loss, layer.trainable_variables)

for var, g in zip(layer.trainable_variables, grad):
    print(f'{var.name}, shape: {g.shape}')

Demo: Understanding GradientTape Behavior

GradientTape watches only trainable variables by default — plain tensors, constants, and variables explicitly marked trainable=False are not automatically tracked, so no gradient can be computed with respect to them:

x1 = tf.Variable(5.0)                   # trainable by default
x2 = tf.Variable(5.0, trainable=False)  # explicitly not trainable
x3 = tf.add(x1, x2)                     # a Tensor, not a Variable
x4 = tf.constant(5.0)                   # a constant

with tf.GradientTape() as tape:
    y = (x1**2) + (x2**2) + (x3**2) + (x4**2)

grad = tape.gradient(y, [x1, x2, x3, x4])
# grad == [<value>, None, None, None] -- only x1 has a gradient

tape.watch() can be used to explicitly track a tensor or a non-trainable value:

x1 = tf.constant(5.0)
x2 = tf.Variable(3.0)

with tf.GradientTape() as tape:
    tape.watch(x1)
    y = (x1**2) + (x2**2)

[dy_dx1, dy_dx2] = tape.gradient(y, [x1, x2])
# both gradients now have values

Setting watch_accessed_variables=False disables automatic tracking of variables entirely, so only values explicitly passed to tape.watch() are tracked:

with tf.GradientTape(watch_accessed_variables=False) as tape:
    tape.watch(x1)
    y = (x1**2) + (x2**2)

[dy_dx1, dy_dx2] = tape.gradient(y, [x1, x2])
# dy_dx2 is None -- x2 was never explicitly watched

GradientTape only tracks operations that are actually performed — for conditional branches, only the branch actually taken contributes to the recorded computation, and the gradient will be None for any variable that was not part of the executed branch:

x = tf.constant(1.0)
x1 = tf.Variable(5.0)
x2 = tf.Variable(3.0)

with tf.GradientTape(persistent=True) as tape:
    tape.watch(x)
    if x > 0.0:
        result = x1**2
    else:
        result = x2**2

dx1, dx2 = tape.gradient(result, [x1, x2])
# x > 0, so result = x1**2 -> dx1 has a value, dx2 is None

If a variable is not part of the computation at all, its gradient is None:

x = tf.Variable(2.)
y = tf.Variable(3.)

with tf.GradientTape() as tape:
    z = y * y                 # z depends only on y
    dy_dx = tape.gradient(z, x)

print(dy_dx)   # None -- x was never part of the computation

Demo: Simple Linear Regression with Manual Gradient Updates

This demo builds and trains a linear regression model completely manually — the weight and bias are hand-crafted variables, and the GradientTape is used directly to calculate gradients and update them.

An artificial dataset is generated first:

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import tensorflow as tf

W_true = 2
b_true = 0.5

x = np.linspace(0, 3, 130)
y = W_true * x + b_true + np.random.randn(*x.shape) * 0.5   # add noise so it doesn't fit perfectly

A minimal linear model, defined as a plain Python class with trainable weight and bias variables:

class LinearModel:
    def __init__(self):
        self.weight = tf.Variable(np.random.randn(), name="W")
        self.bias = tf.Variable(np.random.randn(), name="b")

    def __call__(self, x):
        return self.weight * x + self.bias

The loss function (mean squared error):

def loss(y, y_pred):
    return tf.reduce_mean(tf.square(y - y_pred))

The manual training step — forward pass, loss calculation, gradient calculation, and manual parameter update via assign_sub:

def train(linear_model, x, y, lr=0.01):
    with tf.GradientTape() as tape:
        y_pred = linear_model(x)
        current_loss = loss(y, y_pred)

    d_weight, d_bias = tape.gradient(current_loss,
                                      [linear_model.weight, linear_model.bias])

    linear_model.weight.assign_sub(lr * d_weight)
    linear_model.bias.assign_sub(lr * d_bias)

The training loop, tracking weight/bias history across epochs:

linear_model = LinearModel()
weights, biases = [], []

epochs = 100
lr = 0.15

for epoch_count in range(epochs):
    weights.append(linear_model.weight.numpy())
    biases.append(linear_model.bias.numpy())

    real_loss = loss(y, linear_model(x))

    train(linear_model, x, y, lr=0.12)

    print(f"Epoch count {epoch_count}: Loss value: {real_loss.numpy()}")

Results after different numbers of training epochs (true values were W_true = 2, b_true = 0.5):

The weight and bias visibly converge closer to their true values as training runs for longer, and the fitted regression line increasingly matches the scatter of the original data.

Demo: Simple Regression Using a Sequential Model

Rather than hand-crafting weights and biases, the built-in Keras Sequential API can accomplish the same regression with a single dense layer and a configured optimizer:

from tensorflow import keras
from tensorflow.keras import layers

x = pd.DataFrame(x, columns=['x'])
y = pd.DataFrame(y, columns=['y'])

model = keras.Sequential([layers.Dense(1, input_shape=(1,), activation='linear')])

optimizer = tf.keras.optimizers.SGD(learning_rate=0.001)

model.compile(loss='mse', metrics=['mse'], optimizer=optimizer)

model.fit(x, y, epochs=100)

y_pred = model.predict(x)

model.compile configures the model’s learning parameters (loss function, metrics, optimizer); model.fit performs training, with all gradient calculation and parameter updates handled automatically by the configured optimizer; model.predict produces predictions from the trained model. The fitted line closely tracks the original scattered data, just as it did with the manual GradientTape implementation.

Module 4: Using the Sequential API in Keras

Introducing the Sequential API

Keras offers two core data structures for designing neural network models: layers (composed of neurons, the active learning units) and models, which bring layers together into architectures that are trained and used for prediction. Keras offers several building blocks:

• Sequential models — a simple, linear stack of layers (tf.keras.Sequential).
• Functional API — used to build more complex model topologies.
• Model subclassing — full, granular control over the model.
• Custom layers — full control over a layer’s transformation logic.

A sequential model is a linear stack of layers where the output of one layer feeds directly into the next, until the final output is produced. All of the APIs for building one live in the tf.keras.Sequential namespace. The general steps to build and use any Keras model — sequential, functional, or subclassed — are the same:

1. Instantiate the model (a container that will hold the layers).
2. Add layers — the first layer must specify the shape of its input (the dimensionality of a single training record); subsequent layer shapes are inferred automatically by the TensorFlow 2.0 backend.
3. Compile the model with model.compile, specifying the optimizer, loss function, and (optionally) metrics to track — this ties the model to the TensorFlow 2.0 backend.
4. Train the model with model.fit, feeding in data in batches over a number of epochs.
5. Predict with model.predict on new/test data once training is complete.

There are built-in Keras layer objects for almost every common neural network layer type: dense (fully connected), convolutional, pooling, recurrent, embedding, locally connected, and more.

flowchart TD
    K["tf.keras high-level API"] --> S["Sequential API"]
    K --> Fn["Functional API"]
    K --> Sub["Model Subclassing"]
    K --> CL["Custom Layers"]
    S --> S1["Simple linear stack of layers"]
    Fn --> F1["Multiple inputs/outputs,<br/>shared layers,<br/>non-sequential data flows"]
    Sub --> Sub1["Full control:<br/>define __init__ and call()"]
    CL --> CL1["Custom transformation logic<br/>per layer"]

Demo: Exploring and Processing the Life Expectancy Dataset

The life_expectancy dataset contains information about countries across a span of years, used to predict life expectancy from features such as GDP, population, years of schooling, and immunization/disease rates.

import os, datetime
import numpy as np
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt

from sklearn.metrics import r2_score
from sklearn.preprocessing import StandardScaler

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers

data = pd.read_csv('datasets/life_expectancy.csv')
data.shape       # ~2900 records
data.isna().sum()   # many missing fields across several columns

Missing values are filled using the mean value per country rather than a simple global mean, since life expectancy strongly depends on the specific country:

countries = data['Country'].unique()

na_cols = ['Life expectancy ', 'Adult Mortality', 'Alcohol', 'Hepatitis B',
           ' BMI ', 'Polio', 'Total expenditure', 'Diphtheria ', 'GDP',
           ' thinness  1-19 years', ' thinness 5-9 years', 'Population',
           'Income composition of resources']

for col in na_cols:
    for country in countries:
        data.loc[data['Country'] == country, col] = data.loc[data['Country'] == country, col]\
            .fillna(data[data['Country'] == country][col].mean())

data = data.dropna()   # ~2100 records remain after this final cleanup

Exploratory visualizations include a boxplot of life expectancy overall, a comparison of life expectancy by Status (Developing vs. Developed), a comparison of health expenditure by Status, and a correlation heat map:

sns.boxplot('Status', 'Life expectancy ', data=data)   # developed countries: notably higher life expectancy
sns.boxplot('Status', 'Total expenditure', data=data)  # developed countries: notably higher health expenditure

data_corr = data[['Life expectancy ', 'Adult Mortality', 'Schooling',
                  'Total expenditure', 'Diphtheria ', 'GDP', 'Population']].corr()

sns.heatmap(data_corr, annot=True)
# Adult Mortality correlates -0.66 with life expectancy (negative)
# Schooling correlates +0.75 with life expectancy (positive)

Feature engineering: dropping the Country column (it carries no information beyond what other columns already encode), one-hot encoding the categorical Status feature, and standardizing all numeric features:

features = data.drop('Life expectancy ', axis=1)
target = data[['Life expectancy ']]

features = features.drop('Country', axis=1)

categorical_features = features['Status'].copy()
categorical_features = pd.get_dummies(categorical_features)

numeric_features = features.drop(['Status'], axis=1)

standardScaler = StandardScaler()
numeric_features = pd.DataFrame(standardScaler.fit_transform(numeric_features),
                                 columns=numeric_features.columns,
                                 index=numeric_features.index)

processed_features = pd.concat([numeric_features, categorical_features], axis=1, sort=False)
processed_features.shape   # 21 columns

from sklearn.model_selection import train_test_split

x_train, x_test, y_train, y_test = train_test_split(processed_features, target,
                                                      test_size=0.2, random_state=1)

Standardization is a column-wise operation: subtract the mean of a column from every value in it, then divide by the standard deviation, expressing values as z-scores (standard deviations from the mean). Neural network models tend to train far more robustly on standardized features, since raw features can have wildly different scales.

Demo: Building and Training a Sequential Model

A single-layer regression model, built using model.add:

def build_single_layer_model():
    model = tf.keras.Sequential()

    model.add(tf.keras.layers.Dense(32,
                                     input_shape=(x_train.shape[1],),
                                     activation='sigmoid'))
    model.add(tf.keras.layers.Dense(1))

    optimizer = tf.keras.optimizers.Adam(learning_rate=0.01)

    model.compile(loss='mse', metrics=['mae', 'mse'], optimizer=optimizer)

    return model

model = build_single_layer_model()
model.summary()

tf.keras.utils.plot_model(model)   # visual layer diagram (requires pydot + graphviz)

Adam (Adaptive Moment Estimation) is a popular optimizer that is computationally efficient and memory-light, which is why it’s frequently used in practice. Training and evaluation:

num_epochs = 100

training_history = model.fit(x_train, y_train,
                              epochs=num_epochs,
                              validation_split=0.2,
                              verbose=True)

model.evaluate(x_test, y_test)

y_pred = model.predict(x_test)
r2_score(y_test, y_pred)   # 0.93

For regression models, the R² score — how much of the variance in the underlying data has been captured by the model — is a more informative metric than raw loss values; a higher R² is generally better. With a single hidden layer and 100 training epochs, this model achieves an R² of 0.93 on the test set, with individual predictions tracking closely against actual life expectancy values.

Demo: Visualizing Training with TensorBoard

A deeper, multi-layer sequential model, built using the list-based constructor form:

model = keras.Sequential([
    layers.Dense(32, input_shape=(x_train.shape[1],), activation='relu'),
    layers.Dense(16, activation='relu'),
    layers.Dense(4, activation='relu'),
    layers.Dense(1)
])

optimizer = tf.keras.optimizers.Adam(learning_rate=0.001)
model.compile(loss='mse', metrics=['mae', 'mse'], optimizer=optimizer)

tf.keras.utils.plot_model(model, show_shapes=True)

plot_model(model, show_shapes=True) prints the shape of the data as it flows through each layer: an unknown batch size (?) paired with the number of neurons at each layer — (?, 32), (?, 16), (?, 4), and finally (?, 1) for the single predicted life-expectancy value per record.

A TensorBoard callback logs training events for visualization:

logdir = os.path.join("seq_logs", datetime.datetime.now().strftime("%Y%m%d-%H%M%S"))
tensorboard_callback = keras.callbacks.TensorBoard(logdir, histogram_freq=1)

training_history = model.fit(x_train, y_train,
                              validation_split=0.2,
                              epochs=500,
                              batch_size=100,
                              callbacks=[tensorboard_callback])

A callback in TensorFlow is a function used to customize model behavior during training, evaluation, or prediction — here, the TensorBoard callback logs events for later visualization:

%load_ext tensorboard
%tensorboard --logdir seq_logs --port 6500

TensorBoard exposes several tabs once training completes:

• Scalars — tracked metrics (epoch_loss, mae, mse) plotted across epochs for both training and validation data; individual series can be toggled on/off via checkboxes.
• Graphs — a visual representation of the neural network’s layers; clicking a layer highlights how data flows into it, and a Trace inputs toggle highlights the full path from input to any selected layer.
• Distributions — how model parameters (weights/kernels and biases) for each layer evolve and converge over the course of training.
• Histograms — a distribution of weights and biases for each layer, plotted per training epoch.

model.evaluate(x_test, y_test)

y_pred = model.predict(x_test)
r2_score(y_test, y_pred)   # 0.93

Demo: Comparing Optimizers and Activation Functions

Three sequential models with the same overall structure but different optimizers and activation functions are compared:

SGD optimizer, ReLU activation:

def build_model_with_sgd():
    model = keras.Sequential([
        layers.Dense(32, input_shape=(x_train.shape[1],), activation='relu'),
        layers.Dense(16, activation='relu'),
        layers.Dense(4, activation='relu'),
        layers.Dense(1)
    ])

    optimizer = tf.keras.optimizers.SGD(learning_rate=0.001)
    model.compile(loss='mse', metrics=['mae', 'mse'], optimizer=optimizer)

    return model

model_sgd = build_model_with_sgd()
training_history = model_sgd.fit(x_train, y_train, validation_split=0.2,
                                  epochs=100, batch_size=100)

model_sgd.evaluate(x_test, y_test)
y_pred = model_sgd.predict(x_test)
r2_score(y_test, y_pred)   # 0.74

The plain stochastic gradient descent (SGD) optimizer is a basic momentum-based optimizer and performs noticeably worse than Adam did — the R² of 0.74 is lower, possibly because of fewer training epochs (100 vs. 500) and/or the choice of optimizer.

RMSprop optimizer, ELU activation, a simpler/smaller architecture:

def build_model_with_rmsprop():
    model = keras.Sequential([
        layers.Dense(16, input_shape=(x_train.shape[1],), activation='elu'),
        layers.Dense(8, activation='elu'),
        layers.Dense(4, activation='elu'),
        layers.Dense(1)
    ])

    optimizer = tf.keras.optimizers.RMSprop(learning_rate=0.001)
    model.compile(loss='mse', metrics=['mae', 'mse'], optimizer=optimizer)

    return model

model_rmsprop = build_model_with_rmsprop()
training_history = model_rmsprop.fit(x_train, y_train, validation_split=0.2,
                                      epochs=100, batch_size=100)

model_rmsprop.evaluate(x_test, y_test)
y_pred = model_rmsprop.predict(x_test)
r2_score(y_test, y_pred)   # 0.83

ELU (exponential linear unit) has a shape similar to ReLU but helps mitigate the issue of saturating neurons (neurons stuck outside their active region, whose output stops changing during training). RMSprop behaves like gradient descent with momentum, normalizing gradients using the magnitude of recent gradients, and has proven to be a robust optimizer in practice.

The RMSprop optimizer with ELU activation produced a better model than plain SGD with ReLU for the same number of (100) training epochs, illustrating that both the optimizer choice and the activation function are meaningful parts of neural network model design.

Module 5: Using the Functional API and Model Subclassing in Keras

The Functional API and Model Subclassing

The Sequential API is well suited to simple, straightforward feed-forward models. When a model’s topology grows more complex, the Functional API is used instead. It supports:

• Models with multiple inputs.
• Models with multiple outputs.
• Shared layers, where a layer is reused across multiple parts of the network.
• Non-sequential data flows, where a layer might connect to a layer further down the network rather than strictly to the next one.

The Functional API is more functional in nature than the Sequential API’s object-oriented style: every layer is a callable that is invoked directly on its input tensor, and models built this way are themselves callable on any tensor, just like individual layers. The overall workflow (build model → compile → fit → evaluate → predict) is identical to the Sequential API — only the way layers are wired together differs.

For maximum control over a model’s architecture, model subclassing derives a custom class from tf.keras.Model, defining the forward pass imperatively inside a call method. Such custom models train exactly like any other Keras model and can encapsulate any mix of built-in and custom layers. The tf.keras.Model base class provides the helper functions and utilities that make this straightforward.

For even finer control over how a single layer transforms its input, custom layers can be built by deriving from tf.keras.layers.Layer.

Demo: Exploring the Heart Disease Dataset

The heart disease dataset (UCI machine learning repository) contains 303 patient records used to predict whether heart disease is present (target = 1) or absent (target = 0), based on features such as age, sex, chest pain type (cp), cholesterol (chol), and more.

from datetime import datetime
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns

from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import accuracy_score, precision_score, recall_score

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers

df = pd.read_csv('datasets/heart.csv')
df.shape         # 303 records, no null values
df.describe().T

Exploratory visualizations by gender, age, and cholesterol:

sns.countplot('sex', hue='target', data=df)
# Females (0): notably more diagnosed with heart disease
# Males (1):   notably more WITHOUT heart disease

sns.countplot('age', hue='target', data=df)
# Ages 51-54: markedly higher occurrence of heart disease

plt.scatter(df['age'], df['chol'], s=200)
# Older patients trend toward slightly higher cholesterol

All categorical features in this dataset are already numerically encoded, so only the numeric features require standardization:

features = df.drop('target', axis=1)
target = df[['target']]

categorical_features = features[['sex', 'fbs', 'exang', 'cp', 'ca', 'slope', 'thal', 'restecg']].copy()
numeric_features = features[['age', 'trestbps', 'chol', 'thalach', 'oldpeak']].copy()

standardScaler = StandardScaler()
numeric_features = pd.DataFrame(standardScaler.fit_transform(numeric_features),
                                 columns=numeric_features.columns,
                                 index=numeric_features.index)

processed_features = pd.concat([numeric_features, categorical_features], axis=1, sort=False)

x_train, x_test, y_train, y_test = train_test_split(processed_features, target,
                                                      test_size=0.2, random_state=1)

x_train, x_val, y_train, y_val = train_test_split(x_train, y_train,
                                                    test_size=0.15, random_state=10)
# Final split: 205 training records, 37 validation records, 61 test records

Demo: Building a Binary Classifier with the Functional API

def build_model():
    inputs = tf.keras.Input(shape=(x_train.shape[1],))

    dense_layer1 = layers.Dense(12, activation='relu')
    x = dense_layer1(inputs)

    dropout_layer = layers.Dropout(0.3)
    x = dropout_layer(x)

    dense_layer2 = layers.Dense(8, activation='relu')
    x = dense_layer2(x)

    predictions_layer = layers.Dense(1, activation='sigmoid')
    predictions = predictions_layer(x)

    model = tf.keras.Model(inputs=inputs, outputs=predictions)

    model.summary()

    model.compile(optimizer=tf.keras.optimizers.Adam(0.001),
                  loss=tf.keras.losses.BinaryCrossentropy(),
                  metrics=['accuracy',
                           tf.keras.metrics.Precision(0.5),
                           tf.keras.metrics.Recall(0.5)])
    return model

model = build_model()
keras.utils.plot_model(model, show_shapes=True)

flowchart LR
    In["Input: 13 features"] --> D1["Dense(12), relu"]
    D1 --> Drop["Dropout(30%)"]
    Drop --> D2["Dense(8), relu"]
    D2 --> Out["Dense(1), sigmoid<br/>(probability of heart disease)"]

Notice how every layer is invoked directly as a callable on the tensor produced by the previous layer (x = dense_layer1(inputs)). The Dropout layer randomly disables 30% of the neurons in the preceding dense layer during each training epoch, forcing the remaining neurons to learn more robustly from the data and mitigating overfitting. BinaryCrossentropy is the correct loss function for this binary classifier, and accuracy, precision, and recall are tracked as metrics.

tf.data.Dataset pipelines feed batched, shuffled data into the model:

dataset_train = tf.data.Dataset.from_tensor_slices((x_train.values, y_train.values))
dataset_train = dataset_train.batch(16)
dataset_train.shuffle(128)

dataset_val = tf.data.Dataset.from_tensor_slices((x_val.values, y_val.values))
dataset_val = dataset_val.batch(16)

num_epochs = 100

training_history = model.fit(dataset_train, epochs=num_epochs, validation_data=dataset_val)

model.fit accepts tf.data.Dataset objects directly, in addition to plain NumPy arrays/DataFrames. Evaluation and prediction, including converting predicted probabilities to hard class labels using a 0.5 threshold:

score = model.evaluate(x_test, y_test)
score_df = pd.Series(score, index=model.metrics_names)
# accuracy ~0.70, precision ~0.68, recall ~0.77

y_pred = model.predict(x_test)   # raw probability scores

y_pred = np.where(y_pred >= 0.5, 1, y_pred)
y_pred = np.where(y_pred < 0.5, 0, y_pred)

pred_results = pd.DataFrame({'y_test': y_test.values.flatten(),
                              'y_pred': y_pred.flatten().astype('int32')},
                             index=range(len(y_pred)))

pd.crosstab(pred_results.y_pred, pred_results.y_test)   # confusion matrix

accuracy_score(y_test, y_pred)
precision_score(y_test, y_pred)
recall_score(y_test, y_pred)

Demo: Exploring the Wine Dataset

The built-in scikit-learn wine dataset contains 178 records with 13 numeric predictive attributes (alcohol content, color intensity, hue, and so on), each assigned to one of 3 wine quality classes (0, 1, 2).

import pandas as pd
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt

from sklearn import datasets
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import accuracy_score

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras import Model
from tensorflow.keras.utils import to_categorical

wine_data = datasets.load_wine()

data = pd.DataFrame(data=wine_data['data'], columns=wine_data['feature_names'])
data['target'] = wine_data['target']

data.shape                    # 178 records, 14 columns (13 features + target)
data['target'].value_counts() # fairly evenly divided across the 3 classes

Exploratory visualizations:

sns.distplot(data['alcohol'], kde=1)   # most wines: 12-14% alcohol content

sns.boxplot('target', 'alcohol', data=data)     # class 1 wines: notably lower alcohol content overall
sns.boxplot('target', 'magnesium', data=data)

Because this is a multi-class classification problem, the target is one-hot encoded, and all features are standardized:

features = data.drop('target', axis=1)
target = data[['target']]

target = to_categorical(target, 3)   # one-hot encode: required for CategoricalCrossentropy loss

standardScaler = StandardScaler()
processed_features = pd.DataFrame(standardScaler.fit_transform(features),
                                   columns=features.columns,
                                   index=features.index)

x_train, x_test, y_train, y_test = train_test_split(processed_features, target,
                                                      test_size=0.2, random_state=1)

Demo: Multi-Class Classification with Model Subclassing

The wine classifier is implemented using model subclassing, deriving directly from tf.keras.Model and defining the layers in __init__ and the forward pass in call:

class WineClassificationModel(Model):

    def __init__(self, input_shape):
        super(WineClassificationModel, self).__init__()

        self.d1 = layers.Dense(128, activation='relu', input_shape=[input_shape])
        self.d2 = layers.Dense(64, activation='relu')
        self.d3 = layers.Dense(3, activation='softmax')

    def call(self, x):
        x = self.d1(x)
        x = self.d2(x)
        x = self.d3(x)
        return x

model = WineClassificationModel(x_train.shape[1])

model.compile(optimizer=keras.optimizers.SGD(lr=0.001),
              loss=tf.keras.losses.CategoricalCrossentropy(),
              metrics=['accuracy'])

flowchart LR
    In["Input: 13 features"] --> D1["Dense(128), relu"]
    D1 --> D2["Dense(64), relu"]
    D2 --> D3["Dense(3), softmax"]

The call method is invoked during the forward pass through the model — the base tf.keras.Model class supplies the rest of the required plumbing (compile, fit, evaluate, predict) once call is implemented. The output layer uses softmax activation, the standard choice for multi-class classification: it produces a probability score per class, and the class with the highest score is the model’s prediction.

num_epochs = 100

training_history = model.fit(x_train.values, y_train,
                              validation_split=0.2,
                              epochs=num_epochs,
                              batch_size=48)

score = model.evaluate(x_test, y_test)
score_df = pd.Series(score, index=model.metrics_names)   # accuracy ~0.97

y_pred = model.predict(x_test)   # a 3-column array of probability scores per record

y_pred = np.where(y_pred >= 0.5, 1, y_pred)
y_pred = np.where(y_pred < 0.5, 0, y_pred)   # threshold into a one-hot-style prediction

accuracy_score(y_test, y_pred)   # ~0.94

The accuracy computed via accuracy_score (0.94) differs slightly from the accuracy Keras itself reported during evaluate (0.97). This difference stems from the explicit 0.5 threshold used to binarize the probability scores — Keras’s own accuracy metric compares the raw predicted class (argmax of the probabilities) against the true class rather than applying a fixed threshold per output unit.

Summary

This course walked through TensorFlow 2.0 end to end, from its fundamental data structures to building and training three distinct kinds of Keras models.

Key principles covered:

• Tensors are immutable, multidimensional arrays; variables are mutable containers backed by tensors and are the recommended way to represent trainable, persistent state.
• TensorFlow 2.0 executes eagerly by default (dynamic computation graphs), unlike TensorFlow 1.x’s build-then-run static graphs — but TensorFlow 2.0 still supports static graphs, generated on demand from Python functions via the tf.function decorator (AutoGraph), for deployment-time performance.
• Training a neural network is a repeated cycle of a forward pass (compute a prediction and its loss) and a backward pass (compute gradients and update parameters) — gradients are vectors of partial derivatives of the loss with respect to every trainable parameter, computed efficiently via reverse-mode automatic differentiation using the GradientTape.
• Keras offers three progressively more flexible ways to assemble a model: the Sequential API for simple layer stacks, the Functional API for complex, non-linear topologies with multiple inputs/outputs or shared layers, and model subclassing for full control over the forward pass.
• The choice of optimizer (SGD, Adam, RMSprop) and activation function (ReLU, sigmoid, softmax, ELU) meaningfully affects model performance and is part of the model design process, not an afterthought.

Quick-reference checklist for building a TensorFlow 2.0 / Keras model:

• Prepare and standardize features; encode categorical variables (one-hot for nominal categories, one-hot for multi-class targets).
• Split data into training, validation, and test sets.
• Choose the right Keras API for the model’s topology (Sequential vs. Functional vs. Subclassing).
• Choose an appropriate output activation: linear for regression, sigmoid for binary classification, softmax for multi-class classification.
• Choose a matching loss function: MSE for regression, BinaryCrossentropy for binary classification, CategoricalCrossentropy for multi-class classification (with one-hot targets).
• Choose an optimizer (Adam is a strong general-purpose default) and a learning rate.
• Train with model.fit, monitor with a TensorBoard callback where useful, and evaluate with model.evaluate / model.predict plus a task-appropriate metric (R² for regression, accuracy/precision/recall for classification).

Search Terms

tensorflow · 2.0 · deep · neural · networks · machine · data · science · model · api · exploring · gradient · graph · graphs · sequential · computation · dataset · functional · keras · mode · static · subclassing · activation · dynamic