LostTech.TensorFlow : API Documentation

Type data

Namespace tensorflow.contrib.data

Methods

Properties

Public static methods

object assert_element_shape(ValueTuple<TensorShape, object> expected_shapes)

object assert_element_shape_dyn(object expected_shapes)

object batch_and_drop_remainder(object batch_size)

object batch_and_drop_remainder_dyn(object batch_size)

object bucket_by_sequence_length(object element_length_func, object bucket_boundaries, object bucket_batch_sizes, object padded_shapes, object padding_values, bool pad_to_bucket_boundary, bool no_padding)

object bucket_by_sequence_length_dyn(object element_length_func, object bucket_boundaries, object bucket_batch_sizes, object padded_shapes, object padding_values, ImplicitContainer<T> pad_to_bucket_boundary, ImplicitContainer<T> no_padding)

DatasetV1Adapter choose_from_datasets(object datasets, object choice_dataset)

object copy_to_device(object target_device, string source_device)

object copy_to_device_dyn(object target_device, ImplicitContainer<T> source_device)

object dense_to_sparse_batch(object batch_size, object row_shape)

object dense_to_sparse_batch_dyn(object batch_size, object row_shape)

A transformation that batches ragged elements into tf.SparseTensors.

Like `Dataset.padded_batch()`, this transformation combines multiple consecutive elements of the dataset, which might have different shapes, into a single element. The resulting element has three components (`indices`, `values`, and `dense_shape`), which comprise a tf.SparseTensor that represents the same data. The `row_shape` represents the dense shape of each row in the resulting tf.SparseTensor, to which the effective batch size is prepended.
Parameters
object batch_size
A tf.int64 scalar tf.Tensor, representing the number of consecutive elements of this dataset to combine in a single batch.
object row_shape
A tf.TensorShape or tf.int64 vector tensor-like object representing the equivalent dense shape of a row in the resulting tf.SparseTensor. Each element of this dataset must have the same rank as `row_shape`, and must have size less than or equal to `row_shape` in each dimension.
Returns
object
A `Dataset` transformation function, which can be passed to tf.data.Dataset.apply.
Show Example 
# NOTE: The following examples use `{... }` to represent the
            # contents of a dataset.
            a = { ['a', 'b', 'c'], ['a', 'b'], ['a', 'b', 'c', 'd'] } 
 a.apply(tf.data.experimental.dense_to_sparse_batch(
    batch_size=2, row_shape=[6])) ==
{
    ([[0, 0], [0, 1], [0, 2], [1, 0], [1, 1]],  # indices
     ['a', 'b', 'c', 'a', 'b'],                 # values
     [2, 6]),                                   # dense_shape
    ([[0, 0], [0, 1], [0, 2], [0, 3]],
     ['a', 'b', 'c', 'd'],
     [1, 6])
}

object enumerate_dataset(int start)

A transformation that enumerates the elements of a dataset. (deprecated)

Warning: THIS FUNCTION IS DEPRECATED. It will be removed in a future version. Instructions for updating: Use `tf.data.Dataset.enumerate()

It is similar to python's `enumerate`.
Parameters
int start
A tf.int64 scalar tf.Tensor, representing the start value for enumeration.
Returns
object
A `Dataset` transformation function, which can be passed to tf.data.Dataset.apply.
Show Example 
# NOTE: The following examples use `{... }` to represent the
            # contents of a dataset.
            a = { 1, 2, 3 }
            b = { (7, 8), (9, 10) } 
 # The nested structure of the `datasets` argument determines the
# structure of elements in the resulting dataset.
a.apply(tf.data.experimental.enumerate_dataset(start=5))
=> { (5, 1), (6, 2), (7, 3) }
b.apply(tf.data.experimental.enumerate_dataset())
=> { (0, (7, 8)), (1, (9, 10)) }

object enumerate_dataset_dyn(ImplicitContainer<T> start)

object get_single_element(object dataset)

object get_single_element_dyn(object dataset)

Returns the single element in `dataset` as a nested structure of tensors.

This function enables you to use a tf.data.Dataset in a stateless "tensor-in tensor-out" expression, without creating a `tf.compat.v1.data.Iterator`. This can be useful when your preprocessing transformations are expressed as a `Dataset`, and you want to use the transformation at serving time.
Parameters
object dataset
A tf.data.Dataset object containing a single element.
Returns
object
A nested structure of tf.Tensor objects, corresponding to the single element of `dataset`.
Show Example 
input_batch = tf.compat.v1.placeholder(tf.string, shape=[BATCH_SIZE]) 
 def preprocessing_fn(input_str):
  #...
  return image, label 
 dataset = (tf.data.Dataset.from_tensor_slices(input_batch)
          .map(preprocessing_fn, num_parallel_calls=BATCH_SIZE)
          .batch(BATCH_SIZE)) 
 image_batch, label_batch = tf.data.experimental.get_single_element(dataset)

object group_by_reducer(object key_func, object reducer)

object group_by_reducer_dyn(object key_func, object reducer)

A transformation that groups elements and performs a reduction.

This transformation maps element of a dataset to a key using `key_func` and groups the elements by key. The `reducer` is used to process each group; its `init_func` is used to initialize state for each group when it is created, the `reduce_func` is used to update the state every time an element is mapped to the matching group, and the `finalize_func` is used to map the final state to an output value.
Parameters
object key_func
A function mapping a nested structure of tensors (having shapes and types defined by `self.output_shapes` and `self.output_types`) to a scalar tf.int64 tensor.
object reducer
An instance of `Reducer`, which captures the reduction logic using the `init_func`, `reduce_func`, and `finalize_func` functions.
Returns
object
A `Dataset` transformation function, which can be passed to tf.data.Dataset.apply.

object group_by_window(object key_func, object reduce_func, object window_size, object window_size_func)

object group_by_window_dyn(object key_func, object reduce_func, object window_size, object window_size_func)

A transformation that groups windows of elements by key and reduces them.

This transformation maps each consecutive element in a dataset to a key using `key_func` and groups the elements by key. It then applies `reduce_func` to at most `window_size_func(key)` elements matching the same key. All except the final window for each key will contain `window_size_func(key)` elements; the final window may be smaller.

You may provide either a constant `window_size` or a window size determined by the key through `window_size_func`.
Parameters
object key_func
A function mapping a nested structure of tensors (having shapes and types defined by `self.output_shapes` and `self.output_types`) to a scalar tf.int64 tensor.
object reduce_func
A function mapping a key and a dataset of up to `window_size` consecutive elements matching that key to another dataset.
object window_size
A tf.int64 scalar tf.Tensor, representing the number of consecutive elements matching the same key to combine in a single batch, which will be passed to `reduce_func`. Mutually exclusive with `window_size_func`.
object window_size_func
A function mapping a key to a tf.int64 scalar tf.Tensor, representing the number of consecutive elements matching the same key to combine in a single batch, which will be passed to `reduce_func`. Mutually exclusive with `window_size`.
Returns
object
A `Dataset` transformation function, which can be passed to tf.data.Dataset.apply.

object ignore_errors()

Creates a `Dataset` from another `Dataset` and silently ignores any errors.

Use this transformation to produce a dataset that contains the same elements as the input, but silently drops any elements that caused an error. For example:
Returns
object
A `Dataset` transformation function, which can be passed to tf.data.Dataset.apply.
Show Example 
dataset = tf.data.Dataset.from_tensor_slices([1., 2., 0., 4.]) 
 # Computing `tf.debugging.check_numerics(1. / 0.)` will raise an
InvalidArgumentError.
dataset = dataset.map(lambda x: tf.debugging.check_numerics(1. / x, "error")) 
 # Using `ignore_errors()` will drop the element that causes an error.
dataset =
    dataset.apply(tf.data.experimental.ignore_errors())  # ==> {1., 0.5, 0.2}

object ignore_errors_dyn()

Creates a `Dataset` from another `Dataset` and silently ignores any errors.

Use this transformation to produce a dataset that contains the same elements as the input, but silently drops any elements that caused an error. For example:
Returns
object
A `Dataset` transformation function, which can be passed to tf.data.Dataset.apply.
Show Example 
dataset = tf.data.Dataset.from_tensor_slices([1., 2., 0., 4.]) 
 # Computing `tf.debugging.check_numerics(1. / 0.)` will raise an
InvalidArgumentError.
dataset = dataset.map(lambda x: tf.debugging.check_numerics(1. / x, "error")) 
 # Using `ignore_errors()` will drop the element that causes an error.
dataset =
    dataset.apply(tf.data.experimental.ignore_errors())  # ==> {1., 0.5, 0.2}

DatasetV1Adapter make_batched_features_dataset(object file_pattern, object batch_size, object features, ImplicitContainer<T> reader, object label_key, object reader_args, object num_epochs, bool shuffle, int shuffle_buffer_size, object shuffle_seed, object prefetch_buffer_size, object reader_num_threads, object parser_num_threads, bool sloppy_ordering, bool drop_final_batch)

DatasetV1Adapter make_csv_dataset(object file_pattern, object batch_size, object column_names, object column_defaults, object label_name, object select_columns, string field_delim, bool use_quote_delim, string na_value, bool header, object num_epochs, bool shuffle, int shuffle_buffer_size, object shuffle_seed, object prefetch_buffer_size, object num_parallel_reads, bool sloppy, int num_rows_for_inference, object compression_type)

object make_csv_dataset_dyn(object file_pattern, object batch_size, object column_names, object column_defaults, object label_name, object select_columns, ImplicitContainer<T> field_delim, ImplicitContainer<T> use_quote_delim, ImplicitContainer<T> na_value, ImplicitContainer<T> header, object num_epochs, ImplicitContainer<T> shuffle, ImplicitContainer<T> shuffle_buffer_size, object shuffle_seed, object prefetch_buffer_size, object num_parallel_reads, ImplicitContainer<T> sloppy, ImplicitContainer<T> num_rows_for_inference, object compression_type)

_Saveable make_saveable_from_iterator(object iterator)

object make_saveable_from_iterator_dyn(object iterator)

Returns a SaveableObject for saving/restoring iterator state using Saver.
Parameters
object iterator
Iterator.
Returns
object
A SaveableObject for saving/restoring iterator state using Saver.

object map_and_batch(object map_func, object batch_size, Nullable<int> num_parallel_batches, bool drop_remainder, object num_parallel_calls)

object map_and_batch_dyn(object map_func, object batch_size, object num_parallel_batches, ImplicitContainer<T> drop_remainder, object num_parallel_calls)

object padded_batch_and_drop_remainder(object batch_size, object padded_shapes, object padding_values)

object padded_batch_and_drop_remainder_dyn(object batch_size, object padded_shapes, object padding_values)

object parallel_interleave(object map_func, object cycle_length, int block_length, bool sloppy, object buffer_output_elements, object prefetch_input_elements)

object parallel_interleave_dyn(object map_func, object cycle_length, ImplicitContainer<T> block_length, ImplicitContainer<T> sloppy, object buffer_output_elements, object prefetch_input_elements)

object parse_example_dataset(object features, int num_parallel_calls)

object parse_example_dataset_dyn(object features, ImplicitContainer<T> num_parallel_calls)

object prefetch_to_device(object device, object buffer_size)

object prefetch_to_device_dyn(object device, object buffer_size)

A transformation that prefetches dataset values to the given `device`.

NOTE: Although the transformation creates a tf.data.Dataset, the transformation must be the final `Dataset` in the input pipeline.
Parameters
object device
A string. The name of a device to which elements will be prefetched.
object buffer_size
(Optional.) The number of elements to buffer on `device`. Defaults to an automatically chosen value.
Returns
object
A `Dataset` transformation function, which can be passed to tf.data.Dataset.apply.

IList<object> read_batch_features(object file_pattern, object batch_size, object features, ImplicitContainer<T> reader, object reader_args, bool randomize_input, object num_epochs, int capacity)

object read_batch_features_dyn(object file_pattern, object batch_size, object features, ImplicitContainer<T> reader, object reader_args, ImplicitContainer<T> randomize_input, object num_epochs, ImplicitContainer<T> capacity)

object reduce_dataset(DatasetV1Adapter dataset, Reducer reducer)

object reduce_dataset(Dataset dataset, Reducer reducer)

object reduce_dataset_dyn(object dataset, object reducer)

object rejection_resample(object class_func, object target_dist, object initial_dist, object seed)

object rejection_resample_dyn(object class_func, object target_dist, object initial_dist, object seed)

A transformation that resamples a dataset to achieve a target distribution.

**NOTE** Resampling is performed via rejection sampling; some fraction of the input values will be dropped.
Parameters
object class_func
A function mapping an element of the input dataset to a scalar tf.int32 tensor. Values should be in `[0, num_classes)`.
object target_dist
A floating point type tensor, shaped `[num_classes]`.
object initial_dist
(Optional.) A floating point type tensor, shaped `[num_classes]`. If not provided, the true class distribution is estimated live in a streaming fashion.
object seed
(Optional.) Python integer seed for the resampler.
Returns
object
A `Dataset` transformation function, which can be passed to tf.data.Dataset.apply.

DatasetV1Adapter sample_from_datasets(object datasets, object weights, object seed)

object scan(object initial_state, object scan_func)

object scan_dyn(object initial_state, object scan_func)

A transformation that scans a function across an input dataset.

This transformation is a stateful relative of tf.data.Dataset.map. In addition to mapping `scan_func` across the elements of the input dataset, `scan()` accumulates one or more state tensors, whose initial values are `initial_state`.
Parameters
object initial_state
A nested structure of tensors, representing the initial state of the accumulator.
object scan_func
A function that maps `(old_state, input_element)` to `(new_state, output_element). It must take two arguments and return a pair of nested structures of tensors. The `new_state` must match the structure of `initial_state`.
Returns
object
A `Dataset` transformation function, which can be passed to tf.data.Dataset.apply.

object shuffle_and_repeat(object buffer_size, object count, object seed)

object shuffle_and_repeat_dyn(object buffer_size, object count, object seed)

Shuffles and repeats a Dataset returning a new permutation for each epoch. (deprecated)

Warning: THIS FUNCTION IS DEPRECATED. It will be removed in a future version. Instructions for updating: Use `tf.data.Dataset.shuffle(buffer_size, seed)` followed by `tf.data.Dataset.repeat(count)`. Static tf.data optimizations will take care of using the fused implementation.

`dataset.apply(tf.data.experimental.shuffle_and_repeat(buffer_size, count))`

is equivalent to

`dataset.shuffle(buffer_size, reshuffle_each_iteration=True).repeat(count)`

The difference is that the latter dataset is not serializable. So, if you need to checkpoint an input pipeline with reshuffling you must use this implementation.
Parameters
object buffer_size
A tf.int64 scalar tf.Tensor, representing the maximum number elements that will be buffered when prefetching.
object count
(Optional.) A tf.int64 scalar tf.Tensor, representing the number of times the dataset should be repeated. The default behavior (if `count` is `None` or `-1`) is for the dataset be repeated indefinitely.
object seed
(Optional.) A tf.int64 scalar tf.Tensor, representing the random seed that will be used to create the distribution. See `tf.compat.v1.set_random_seed` for behavior.
Returns
object
A `Dataset` transformation function, which can be passed to tf.data.Dataset.apply.

object sliding_window_batch(int window_size, int stride, int window_shift, IGraphNodeBase window_stride)

object sliding_window_batch(int window_size, int stride, int window_shift, int window_stride)

object sliding_window_batch(IGraphNodeBase window_size, IGraphNodeBase stride, IGraphNodeBase window_shift, IGraphNodeBase window_stride)

object sliding_window_batch(IGraphNodeBase window_size, IGraphNodeBase stride, IGraphNodeBase window_shift, int window_stride)

object sliding_window_batch(int window_size, int stride, IGraphNodeBase window_shift, IGraphNodeBase window_stride)

object sliding_window_batch(IGraphNodeBase window_size, IGraphNodeBase stride, int window_shift, IGraphNodeBase window_stride)

object sliding_window_batch(IGraphNodeBase window_size, IGraphNodeBase stride, int window_shift, int window_stride)

object sliding_window_batch(IGraphNodeBase window_size, int stride, IGraphNodeBase window_shift, IGraphNodeBase window_stride)

object sliding_window_batch(int window_size, int stride, IGraphNodeBase window_shift, int window_stride)

object sliding_window_batch(IGraphNodeBase window_size, int stride, int window_shift, IGraphNodeBase window_stride)

object sliding_window_batch(IGraphNodeBase window_size, int stride, IGraphNodeBase window_shift, int window_stride)

object sliding_window_batch(int window_size, IGraphNodeBase stride, IGraphNodeBase window_shift, IGraphNodeBase window_stride)

object sliding_window_batch(int window_size, IGraphNodeBase stride, IGraphNodeBase window_shift, int window_stride)

object sliding_window_batch(int window_size, IGraphNodeBase stride, int window_shift, IGraphNodeBase window_stride)

object sliding_window_batch(int window_size, IGraphNodeBase stride, int window_shift, int window_stride)

object sliding_window_batch(IGraphNodeBase window_size, int stride, int window_shift, int window_stride)

object sliding_window_batch_dyn(object window_size, object stride, object window_shift, ImplicitContainer<T> window_stride)

object sloppy_interleave(object map_func, object cycle_length, int block_length)

object sloppy_interleave_dyn(object map_func, object cycle_length, ImplicitContainer<T> block_length)

object unique()

Creates a `Dataset` from another `Dataset`, discarding duplicates.

Use this transformation to produce a dataset that contains one instance of each unique element in the input.
Returns
object
A `Dataset` transformation function, which can be passed to tf.data.Dataset.apply.
Show Example 
dataset = tf.data.Dataset.from_tensor_slices([1, 37, 2, 37, 2, 1]) 
 # Using `unique()` will drop the duplicate elements.
dataset = dataset.apply(tf.data.experimental.unique())  # ==> { 1, 37, 2 }

object unique_dyn()

Creates a `Dataset` from another `Dataset`, discarding duplicates.

Use this transformation to produce a dataset that contains one instance of each unique element in the input.
Returns
object
A `Dataset` transformation function, which can be passed to tf.data.Dataset.apply.
Show Example 
dataset = tf.data.Dataset.from_tensor_slices([1, 37, 2, 37, 2, 1]) 
 # Using `unique()` will drop the duplicate elements.
dataset = dataset.apply(tf.data.experimental.unique())  # ==> { 1, 37, 2 }

Public properties

PythonFunctionContainer assert_element_shape_fn get;

PythonFunctionContainer batch_and_drop_remainder_fn get;

PythonFunctionContainer bucket_by_sequence_length_fn get;

PythonFunctionContainer choose_from_datasets_fn get;

PythonFunctionContainer copy_to_device_fn get;

PythonFunctionContainer Counter_fn get;

PythonFunctionContainer dense_to_sparse_batch_fn get;

PythonFunctionContainer enumerate_dataset_fn get;

PythonFunctionContainer get_single_element_fn get;

PythonFunctionContainer group_by_reducer_fn get;

PythonFunctionContainer group_by_window_fn get;

PythonFunctionContainer ignore_errors_fn get;

PythonFunctionContainer make_batched_features_dataset_fn get;

PythonFunctionContainer make_csv_dataset_fn get;

PythonFunctionContainer make_saveable_from_iterator_fn get;

PythonFunctionContainer map_and_batch_fn get;

PythonFunctionContainer padded_batch_and_drop_remainder_fn get;

PythonFunctionContainer parallel_interleave_fn get;

PythonFunctionContainer parse_example_dataset_fn get;

PythonFunctionContainer prefetch_to_device_fn get;

PythonFunctionContainer read_batch_features_fn get;

PythonFunctionContainer reduce_dataset_fn get;

PythonFunctionContainer rejection_resample_fn get;

PythonFunctionContainer sample_from_datasets_fn get;

PythonFunctionContainer shuffle_and_repeat_fn get;

PythonFunctionContainer sliding_window_batch_fn get;

PythonFunctionContainer sloppy_interleave_fn get;

PythonFunctionContainer unbatch_fn get;

PythonFunctionContainer unique_fn get;