Type CrossDeviceOps
Namespace tensorflow.distribute
Parent PythonObjectContainer
Interfaces ICrossDeviceOps
Base class for cross-device reduction and broadcasting algorithms.
Methods
- batch_reduce
- batch_reduce
- batch_reduce_dyn
- batch_reduce_implementation
- batch_reduce_implementation
- batch_reduce_implementation_dyn
- broadcast
- broadcast
- broadcast
- broadcast_dyn
- broadcast_implementation
- broadcast_implementation
- broadcast_implementation_dyn
- reduce
- reduce
- reduce
- reduce
- reduce
- reduce
- reduce
- reduce
- reduce
- reduce
- reduce
- reduce
- reduce
- reduce
- reduce
- reduce
- reduce
- reduce
- reduce_dyn
- reduce_implementation
- reduce_implementation_dyn
Properties
Public instance methods
IList<object> batch_reduce(ReduceOp reduce_op, object value_destination_pairs)
Reduce PerReplica objects in a batch. Reduce each first element in `value_destination_pairs` to each second
element which indicates the destinations.
Parameters
-
ReduceOpreduce_op - Indicates how per_replica_value will be reduced. Accepted
values are
tf.distribute.ReduceOp.SUM,tf.distribute.ReduceOp.MEAN. -
objectvalue_destination_pairs - a list or a tuple of tuples of PerReplica objects (or tensors with device set if there is one device) and destinations.
Returns
-
IList<object> - a list of Mirrored objects.
IList<object> batch_reduce(object reduce_op, object value_destination_pairs)
Reduce PerReplica objects in a batch. Reduce each first element in `value_destination_pairs` to each second
element which indicates the destinations.
Parameters
-
objectreduce_op - Indicates how per_replica_value will be reduced. Accepted
values are
tf.distribute.ReduceOp.SUM,tf.distribute.ReduceOp.MEAN. -
objectvalue_destination_pairs - a list or a tuple of tuples of PerReplica objects (or tensors with device set if there is one device) and destinations.
Returns
-
IList<object> - a list of Mirrored objects.
object batch_reduce_dyn(object reduce_op, object value_destination_pairs)
Reduce PerReplica objects in a batch. Reduce each first element in `value_destination_pairs` to each second
element which indicates the destinations.
Parameters
-
objectreduce_op - Indicates how per_replica_value will be reduced. Accepted
values are
tf.distribute.ReduceOp.SUM,tf.distribute.ReduceOp.MEAN. -
objectvalue_destination_pairs - a list or a tuple of tuples of PerReplica objects (or tensors with device set if there is one device) and destinations.
Returns
-
object - a list of Mirrored objects.
IList<object> batch_reduce_implementation(object reduce_op, IEnumerable<ValueTuple<PerReplica, object>> value_destination_pairs)
Implementation of reduce PerReplica objects in a batch. Reduce each first element in `value_destination_pairs` to each second
element which indicates the destinations.
Parameters
-
objectreduce_op - Indicates how per_replica_value will be reduced. Accepted
values are
tf.distribute.ReduceOp.SUM,tf.distribute.ReduceOp.MEAN. -
IEnumerable<ValueTuple<PerReplica, object>>value_destination_pairs - a list or a tuple of tuples of PerReplica objects (or tensors with device set if there is one device) and destinations.
Returns
-
IList<object> - a list of Mirrored objects.
IList<object> batch_reduce_implementation(object reduce_op, object value_destination_pairs)
Implementation of reduce PerReplica objects in a batch. Reduce each first element in `value_destination_pairs` to each second
element which indicates the destinations.
Parameters
-
objectreduce_op - Indicates how per_replica_value will be reduced. Accepted
values are
tf.distribute.ReduceOp.SUM,tf.distribute.ReduceOp.MEAN. -
objectvalue_destination_pairs - a list or a tuple of tuples of PerReplica objects (or tensors with device set if there is one device) and destinations.
Returns
-
IList<object> - a list of Mirrored objects.
object batch_reduce_implementation_dyn(object reduce_op, object value_destination_pairs)
Implementation of reduce PerReplica objects in a batch. Reduce each first element in `value_destination_pairs` to each second
element which indicates the destinations.
Parameters
-
objectreduce_op - Indicates how per_replica_value will be reduced. Accepted
values are
tf.distribute.ReduceOp.SUM,tf.distribute.ReduceOp.MEAN. -
objectvalue_destination_pairs - a list or a tuple of tuples of PerReplica objects (or tensors with device set if there is one device) and destinations.
Returns
-
object - a list of Mirrored objects.
object broadcast(ValueTuple<double, int> tensor, object destinations)
Broadcast the `tensor` to destinations.
Parameters
-
ValueTuple<double, int>tensor - the tensor to broadcast.
-
objectdestinations - the broadcast destinations.
Returns
-
object - a Mirrored object.
object broadcast(RaggedTensor tensor, object destinations)
Broadcast the `tensor` to destinations.
Parameters
-
RaggedTensortensor - the tensor to broadcast.
-
objectdestinations - the broadcast destinations.
Returns
-
object - a Mirrored object.
object broadcast(IGraphNodeBase tensor, object destinations)
Broadcast the `tensor` to destinations.
Parameters
-
IGraphNodeBasetensor - the tensor to broadcast.
-
objectdestinations - the broadcast destinations.
Returns
-
object - a Mirrored object.
object broadcast_dyn(object tensor, object destinations)
Broadcast the `tensor` to destinations.
Parameters
-
objecttensor - the tensor to broadcast.
-
objectdestinations - the broadcast destinations.
Returns
-
object - a Mirrored object.
object broadcast_implementation(ValueTuple<double, int> tensor, object destinations)
Implementation of broadcast the `tensor` to destinations.
Parameters
-
ValueTuple<double, int>tensor - the tensor to broadcast.
-
objectdestinations - the broadcast destinations.
Returns
-
object - a Mirrored object.
object broadcast_implementation(IGraphNodeBase tensor, object destinations)
Implementation of broadcast the `tensor` to destinations.
Parameters
-
IGraphNodeBasetensor - the tensor to broadcast.
-
objectdestinations - the broadcast destinations.
Returns
-
object - a Mirrored object.
object broadcast_implementation_dyn(object tensor, object destinations)
Implementation of broadcast the `tensor` to destinations.
Parameters
-
objecttensor - the tensor to broadcast.
-
objectdestinations - the broadcast destinations.
Returns
-
object - a Mirrored object.
object reduce(ReduceOp reduce_op, IEnumerable<object> per_replica_value, string destinations)
Reduce `per_replica_value` to `destinations`. It runs the reduction operation defined by `reduce_op` and put the
result on `destinations`.
Parameters
-
ReduceOpreduce_op - Indicates how per_replica_value will be reduced. Accepted
values are
tf.distribute.ReduceOp.SUM,tf.distribute.ReduceOp.MEAN. -
IEnumerable<object>per_replica_value - a PerReplica object or a tensor with device set.
-
stringdestinations - the reduction destinations.
Returns
-
object - a Mirrored object.
object reduce(string reduce_op, IGraphNodeBase per_replica_value, string destinations)
Reduce `per_replica_value` to `destinations`. It runs the reduction operation defined by `reduce_op` and put the
result on `destinations`.
Parameters
-
stringreduce_op - Indicates how per_replica_value will be reduced. Accepted
values are
tf.distribute.ReduceOp.SUM,tf.distribute.ReduceOp.MEAN. -
IGraphNodeBaseper_replica_value - a PerReplica object or a tensor with device set.
-
stringdestinations - the reduction destinations.
Returns
-
object - a Mirrored object.
object reduce(string reduce_op, PerReplica per_replica_value, string destinations)
Reduce `per_replica_value` to `destinations`. It runs the reduction operation defined by `reduce_op` and put the
result on `destinations`.
Parameters
-
stringreduce_op - Indicates how per_replica_value will be reduced. Accepted
values are
tf.distribute.ReduceOp.SUM,tf.distribute.ReduceOp.MEAN. -
PerReplicaper_replica_value - a PerReplica object or a tensor with device set.
-
stringdestinations - the reduction destinations.
Returns
-
object - a Mirrored object.
object reduce(string reduce_op, ValueTuple<PythonClassContainer, PythonClassContainer> per_replica_value, string destinations)
Reduce `per_replica_value` to `destinations`. It runs the reduction operation defined by `reduce_op` and put the
result on `destinations`.
Parameters
-
stringreduce_op - Indicates how per_replica_value will be reduced. Accepted
values are
tf.distribute.ReduceOp.SUM,tf.distribute.ReduceOp.MEAN. -
ValueTuple<PythonClassContainer, PythonClassContainer>per_replica_value - a PerReplica object or a tensor with device set.
-
stringdestinations - the reduction destinations.
Returns
-
object - a Mirrored object.
object reduce(string reduce_op, IEnumerable<object> per_replica_value, string destinations)
Reduce `per_replica_value` to `destinations`. It runs the reduction operation defined by `reduce_op` and put the
result on `destinations`.
Parameters
-
stringreduce_op - Indicates how per_replica_value will be reduced. Accepted
values are
tf.distribute.ReduceOp.SUM,tf.distribute.ReduceOp.MEAN. -
IEnumerable<object>per_replica_value - a PerReplica object or a tensor with device set.
-
stringdestinations - the reduction destinations.
Returns
-
object - a Mirrored object.
object reduce(string reduce_op, IDictionary<object, object> per_replica_value, string destinations)
Reduce `per_replica_value` to `destinations`. It runs the reduction operation defined by `reduce_op` and put the
result on `destinations`.
Parameters
-
stringreduce_op - Indicates how per_replica_value will be reduced. Accepted
values are
tf.distribute.ReduceOp.SUM,tf.distribute.ReduceOp.MEAN. -
IDictionary<object, object>per_replica_value - a PerReplica object or a tensor with device set.
-
stringdestinations - the reduction destinations.
Returns
-
object - a Mirrored object.
object reduce(object reduce_op, object per_replica_value, string destinations)
Reduce `per_replica_value` to `destinations`. It runs the reduction operation defined by `reduce_op` and put the
result on `destinations`.
Parameters
-
objectreduce_op - Indicates how per_replica_value will be reduced. Accepted
values are
tf.distribute.ReduceOp.SUM,tf.distribute.ReduceOp.MEAN. -
objectper_replica_value - a PerReplica object or a tensor with device set.
-
stringdestinations - the reduction destinations.
Returns
-
object - a Mirrored object.
object reduce(object reduce_op, IGraphNodeBase per_replica_value, string destinations)
Reduce `per_replica_value` to `destinations`. It runs the reduction operation defined by `reduce_op` and put the
result on `destinations`.
Parameters
-
objectreduce_op - Indicates how per_replica_value will be reduced. Accepted
values are
tf.distribute.ReduceOp.SUM,tf.distribute.ReduceOp.MEAN. -
IGraphNodeBaseper_replica_value - a PerReplica object or a tensor with device set.
-
stringdestinations - the reduction destinations.
Returns
-
object - a Mirrored object.
object reduce(object reduce_op, PerReplica per_replica_value, string destinations)
Reduce `per_replica_value` to `destinations`. It runs the reduction operation defined by `reduce_op` and put the
result on `destinations`.
Parameters
-
objectreduce_op - Indicates how per_replica_value will be reduced. Accepted
values are
tf.distribute.ReduceOp.SUM,tf.distribute.ReduceOp.MEAN. -
PerReplicaper_replica_value - a PerReplica object or a tensor with device set.
-
stringdestinations - the reduction destinations.
Returns
-
object - a Mirrored object.
object reduce(object reduce_op, ValueTuple<PythonClassContainer, PythonClassContainer> per_replica_value, string destinations)
Reduce `per_replica_value` to `destinations`. It runs the reduction operation defined by `reduce_op` and put the
result on `destinations`.
Parameters
-
objectreduce_op - Indicates how per_replica_value will be reduced. Accepted
values are
tf.distribute.ReduceOp.SUM,tf.distribute.ReduceOp.MEAN. -
ValueTuple<PythonClassContainer, PythonClassContainer>per_replica_value - a PerReplica object or a tensor with device set.
-
stringdestinations - the reduction destinations.
Returns
-
object - a Mirrored object.
object reduce(object reduce_op, IEnumerable<object> per_replica_value, string destinations)
Reduce `per_replica_value` to `destinations`. It runs the reduction operation defined by `reduce_op` and put the
result on `destinations`.
Parameters
-
objectreduce_op - Indicates how per_replica_value will be reduced. Accepted
values are
tf.distribute.ReduceOp.SUM,tf.distribute.ReduceOp.MEAN. -
IEnumerable<object>per_replica_value - a PerReplica object or a tensor with device set.
-
stringdestinations - the reduction destinations.
Returns
-
object - a Mirrored object.
object reduce(object reduce_op, IDictionary<object, object> per_replica_value, string destinations)
Reduce `per_replica_value` to `destinations`. It runs the reduction operation defined by `reduce_op` and put the
result on `destinations`.
Parameters
-
objectreduce_op - Indicates how per_replica_value will be reduced. Accepted
values are
tf.distribute.ReduceOp.SUM,tf.distribute.ReduceOp.MEAN. -
IDictionary<object, object>per_replica_value - a PerReplica object or a tensor with device set.
-
stringdestinations - the reduction destinations.
Returns
-
object - a Mirrored object.
object reduce(ReduceOp reduce_op, object per_replica_value, string destinations)
Reduce `per_replica_value` to `destinations`. It runs the reduction operation defined by `reduce_op` and put the
result on `destinations`.
Parameters
-
ReduceOpreduce_op - Indicates how per_replica_value will be reduced. Accepted
values are
tf.distribute.ReduceOp.SUM,tf.distribute.ReduceOp.MEAN. -
objectper_replica_value - a PerReplica object or a tensor with device set.
-
stringdestinations - the reduction destinations.
Returns
-
object - a Mirrored object.
object reduce(ReduceOp reduce_op, IGraphNodeBase per_replica_value, string destinations)
Reduce `per_replica_value` to `destinations`. It runs the reduction operation defined by `reduce_op` and put the
result on `destinations`.
Parameters
-
ReduceOpreduce_op - Indicates how per_replica_value will be reduced. Accepted
values are
tf.distribute.ReduceOp.SUM,tf.distribute.ReduceOp.MEAN. -
IGraphNodeBaseper_replica_value - a PerReplica object or a tensor with device set.
-
stringdestinations - the reduction destinations.
Returns
-
object - a Mirrored object.
object reduce(ReduceOp reduce_op, PerReplica per_replica_value, string destinations)
Reduce `per_replica_value` to `destinations`. It runs the reduction operation defined by `reduce_op` and put the
result on `destinations`.
Parameters
-
ReduceOpreduce_op - Indicates how per_replica_value will be reduced. Accepted
values are
tf.distribute.ReduceOp.SUM,tf.distribute.ReduceOp.MEAN. -
PerReplicaper_replica_value - a PerReplica object or a tensor with device set.
-
stringdestinations - the reduction destinations.
Returns
-
object - a Mirrored object.
object reduce(string reduce_op, object per_replica_value, string destinations)
Reduce `per_replica_value` to `destinations`. It runs the reduction operation defined by `reduce_op` and put the
result on `destinations`.
Parameters
-
stringreduce_op - Indicates how per_replica_value will be reduced. Accepted
values are
tf.distribute.ReduceOp.SUM,tf.distribute.ReduceOp.MEAN. -
objectper_replica_value - a PerReplica object or a tensor with device set.
-
stringdestinations - the reduction destinations.
Returns
-
object - a Mirrored object.
object reduce(ReduceOp reduce_op, ValueTuple<PythonClassContainer, PythonClassContainer> per_replica_value, string destinations)
Reduce `per_replica_value` to `destinations`. It runs the reduction operation defined by `reduce_op` and put the
result on `destinations`.
Parameters
-
ReduceOpreduce_op - Indicates how per_replica_value will be reduced. Accepted
values are
tf.distribute.ReduceOp.SUM,tf.distribute.ReduceOp.MEAN. -
ValueTuple<PythonClassContainer, PythonClassContainer>per_replica_value - a PerReplica object or a tensor with device set.
-
stringdestinations - the reduction destinations.
Returns
-
object - a Mirrored object.
object reduce(ReduceOp reduce_op, IDictionary<object, object> per_replica_value, string destinations)
Reduce `per_replica_value` to `destinations`. It runs the reduction operation defined by `reduce_op` and put the
result on `destinations`.
Parameters
-
ReduceOpreduce_op - Indicates how per_replica_value will be reduced. Accepted
values are
tf.distribute.ReduceOp.SUM,tf.distribute.ReduceOp.MEAN. -
IDictionary<object, object>per_replica_value - a PerReplica object or a tensor with device set.
-
stringdestinations - the reduction destinations.
Returns
-
object - a Mirrored object.
object reduce_dyn(object reduce_op, object per_replica_value, object destinations)
Reduce `value` across replicas. Given a per-replica value returned by `experimental_run_v2`, say a
per-example loss, the batch will be divided across all the replicas. This
function allows you to aggregate across replicas and optionally also across
batch elements. For example, if you have a global batch size of 8 and 2
replicas, values for examples `[0, 1, 2, 3]` will be on replica 0 and
`[4, 5, 6, 7]` will be on replica 1. By default, `reduce` will just
aggregate across replicas, returning `[0+4, 1+5, 2+6, 3+7]`. This is useful
when each replica is computing a scalar or some other value that doesn't
have a "batch" dimension (like a gradient). More often you will want to
aggregate across the global batch, which you can get by specifying the batch
dimension as the `axis`, typically `axis=0`. In this case it would return a
scalar `0+1+2+3+4+5+6+7`. If there is a last partial batch, you will need to specify an axis so
that the resulting shape is consistent across replicas. So if the last
batch has size 6 and it is divided into [0, 1, 2, 3] and [4, 5], you
would get a shape mismatch unless you specify `axis=0`. If you specify
tf.distribute.ReduceOp.MEAN, using `axis=0` will use the correct
denominator of 6. Contrast this with computing `reduce_mean` to get a
scalar value on each replica and this function to average those means,
which will weigh some values `1/8` and others `1/4`.
Parameters
-
objectreduce_op - A
tf.distribute.ReduceOpvalue specifying how values should be combined. -
objectper_replica_value -
objectdestinations
Returns
-
object - A `Tensor`.
object reduce_implementation(object reduce_op, PerReplica per_replica_value, object destinations)
The implementation of reduce of `per_replica_value` to `destinations`. It runs the reduction operation defined by `reduce_op` and put the
result on `destinations`.
Parameters
-
objectreduce_op - Indicates how per_replica_value will be reduced. Accepted
values are
tf.distribute.ReduceOp.SUM,tf.distribute.ReduceOp.MEAN. -
PerReplicaper_replica_value - a PerReplica object or a tensor with device set.
-
objectdestinations - the reduction destinations.
Returns
-
object - a Mirrored object.
object reduce_implementation_dyn(object reduce_op, object per_replica_value, object destinations)
The implementation of reduce of `per_replica_value` to `destinations`. It runs the reduction operation defined by `reduce_op` and put the
result on `destinations`.
Parameters
-
objectreduce_op - Indicates how per_replica_value will be reduced. Accepted
values are
tf.distribute.ReduceOp.SUM,tf.distribute.ReduceOp.MEAN. -
objectper_replica_value - a PerReplica object or a tensor with device set.
-
objectdestinations - the reduction destinations.
Returns
-
object - a Mirrored object.