Package io.datareplication.consumer.feed

Interface FeedConsumer

public interface FeedConsumer

An interface for consuming a Feed provided by the FeedProducer.

Feed pages are paginated collections of feed entities. Each page contains a list of feed entities and an optional link to the next page. Feed pages are consumed by starting from the provided feed URL (which will generally point to the newest page in the feed) and walking backwards until the desired starting point determined by the StartFrom parameters is reached. Then the feed pages are streamed from oldest to newest until the head of the feed is reached.

This interface exposes two different ways to consume feed data:

• streamEntities(Url, StartFrom) hides the page breaks and provides a stream of fully downloaded entities in chronological order as they appear in the feed. This is usually what you want since order matters when consuming a feed. This method also provides entity-level filtering when using either StartFrom.Timestamp or StartFrom.ContentId.
• streamPages(Url, StartFrom) instead returns a stream of StreamingPage objects. This provides more control over how the pages are consumed: for example, it allows streaming individual entity bodies (in contrast to always downloading the entire body of an entity before it is emitted) and downloading and processing multiple feed pages in parallel. Note that this is usually not what you want because the order of pages and entities matters for a feed.

  Note that this method never returns partial pages. This means that where streamEntities(Url, StartFrom) might discard entities from the first page because their timestamps are older than requested, this method will return the full first page including those entities.

A feed consumer is created using a builder: call the builder() method to create a new builder with default settings, call the methods on FeedConsumer.Builder to customize the consumer, then call FeedConsumer.Builder.build() to create a new FeedConsumer instance.

Nested Class Summary

Nested Classes

Modifier and Type	Interface	Description
static class	FeedConsumer.Builder	A builder for FeedConsumer.

Method Summary

Modifier and Type	Method	Description
static FeedConsumer.Builder	builder()	Create a new FeedConsumer.Builder with default settings.
java.util.concurrent.Flow.Publisher<@NonNull Entity<@NonNull FeedEntityHeader>>	streamEntities(@NonNull Url url, @NonNull StartFrom startFrom)	Stream the entities of the feed starting from the given Url.
java.util.concurrent.Flow.Publisher<@NonNull StreamingPage<@NonNull FeedPageHeader,@NonNull FeedEntityHeader>>	streamPages(@NonNull Url url, @NonNull StartFrom startFrom)	Stream the feed pages starting from the given Url.

Method Detail

streamPages

@NonNull
java.util.concurrent.Flow.Publisher<@NonNull StreamingPage<@NonNull FeedPageHeader,@NonNull FeedEntityHeader>> streamPages(@NonNull
                                                                                                                                 @NonNull Url url,
                                                                                                                                 @NonNull
                                                                                                                                 @NonNull StartFrom startFrom)

Stream the feed pages starting from the given Url. Streaming a StreamingPage will result into receiving already consumed entities with an older last modified date of the current FeedPage again. The ContentId from StartFrom.ContentId will not respect in streamPages, use streamEntities(Url, StartFrom) instead.

Parameters:

url - the Url to start streaming from

startFrom - the StartFrom parameter

Returns:

a Flow.Publisher of StreamingPage of FeedPageHeader and FeedEntityHeader

Throws:

FeedException.FeedNotOldEnough - if the last modified date of the last page is not older and StartFrom is not StartFrom.Beginning

HttpException - in case of HTTP errors (invalid URL, HTTP error status codes, network errors/timeouts, ...)

PageFormatException - if the HTTP response is ok, but the page response is malformed in some way (usually missing or malformed HTTP Content-Type header since multipart parsing errors will only start happening when we get to StreamingPage)

streamEntities

@NonNull
java.util.concurrent.Flow.Publisher<@NonNull Entity<@NonNull FeedEntityHeader>> streamEntities(@NonNull
                                                                                               @NonNull Url url,
                                                                                               @NonNull
                                                                                               @NonNull StartFrom startFrom)

Stream the entities of the feed starting from the given Url. Streaming entities will result into skipping already consumed entities with an older last modified date of the current FeedPage. Streaming entities will respect the ContentId from StartFrom.ContentId.

Parameters:

url - the Url to start streaming from

startFrom - the StartFrom parameter

Returns:

a Flow.Publisher of Entity of FeedEntityHeader

Throws:

FeedException.FeedNotOldEnough - if the last modified date of the last page is not older and StartFrom is not StartFrom.Beginning

FeedException.ContentIdNotFound - when using StartFrom.ContentId and the specified content ID was not found with the specified timestamp

HttpException - in case of HTTP errors (invalid URL, HTTP error status codes, network errors/timeouts, ...)

PageFormatException - if the HTTP response is ok, but the page response is malformed in some way (usually missing or malformed HTTP Content-Type header since multipart parsing errors will only start happening when we get to StreamingPage)

builder

@NonNull
static FeedConsumer.Builder builder()

Create a new FeedConsumer.Builder with default settings. Use the FeedConsumer.Builder.build() method on the returned builder to create a FeedConsumer with the specified settings.

Returns:

a new builder