Page from network and database

Provide an improved user experience by making sure that your app can be used when network connections are unreliable or when the user is offline. One way to do this is to page from the network and from a local database at the same time. This way, your app loads data directly from a local database cache and only makes requests to the network when there is no more data in the database.

The Paging library provides the RemoteMediator component for this use case. RemoteMediator manages the process of loading data from the network into the local database.

This guide assumes that you are familiar with the Room persistence library and with basic usage of the Paging library.

Basic usage

Suppose you want your app to load pages of User items from an item-keyed network data source into a local cache stored in a Room database. Figure 1 illustrates how RemoteMediator and PagingSource work together to meet this use case.

A RemoteMediator implementation manages loading paged data from the network into the database, but does not load data directly into the UI. Instead, the app uses the database as a single source of truth. In other words, the app only displays data that has been cached in the database. A PagingSource implementation (for example, one generated by Room) handles loading cached data from the database into the UI.

Create Room entities

The first step is to use the Room persistence library to define a database that holds a local cache of paged data from the network data source. Start with an implementation of RoomDatabase as described in Save data in a local database using Room.

Next, define a Room entity to represent a table of list items as described in Defining data using Room entities. Give it an id field as a primary key, as well as fields for any other information that your list items contain.

@Entity(tableName = "users")
data class User(val id: String, val label: String)

@Entity(tableName = "users")
public class User {
  public String id;
  public String label;
}

@Entity(tableName = "users")
public class User {
  public String id;
  public String label;
}

You must also define a data access object (DAO) for this Room entity as described in Accessing data using Room DAOs. The DAO for the list item entity must include the following methods:

• An insertAll() method that inserts a list of items into the table.
• A method that takes the query string as a parameter and returns a PagingSource object for the list of results. This way, a Pager object can use this table as a source of paged data.
• A clearAll() method that deletes all of the table's data.

@Dao
interface UserDao {
  @Insert(onConflict = OnConflictStrategy.REPLACE)
  suspend fun insertAll(users: List<User>)

  @Query("SELECT * FROM users WHERE label LIKE :query")
  fun pagingSource(query: String): PagingSource<Int, User>

  @Query("DELETE FROM users")
  suspend fun clearAll()
}

@Dao
interface UserDao {
  @Insert(onConflict = OnConflictStrategy.REPLACE)
  void insertAll(List<User> users);

  @Query("SELECT * FROM users WHERE mLabel LIKE :query")
  PagingSource<Integer, User> pagingSource(String query);

  @Query("DELETE FROM users")
  int clearAll();
}

@Dao
interface UserDao {
  @Insert(onConflict = OnConflictStrategy.REPLACE)
  void insertAll(List<User> users);

  @Query("SELECT * FROM users WHERE mLabel LIKE :query")
  PagingSource<Integer, User> pagingSource(String query);

  @Query("DELETE FROM users")
  int clearAll();
}

Implement a RemoteMediator

RemoteMediator is similar to the PagingSource component. It includes a load() method that you must override to define loading behavior. The difference is that instead of loading paged data from a data source to a RecyclerView list, a RemoteMediator object loads paged data from a network data source and saves it to a local database.

A typical RemoteMediator implementation includes the following parameters:

• query: A query string defining which data to retrieve from the backend service.
• database: The Room database that serves as a local cache.
• networkService: An API instance for the backend service.

Create a RemoteMediator<Key, Value> implementation. The Key type and the Value type should be the same as they would be if you were defining a PagingSource against the same network data source. For more information on selecting type parameters, see Select key and value types.

@OptIn(ExperimentalPagingApi::class)
class ExampleRemoteMediator(
  private val query: String,
  private val database: RoomDb,
  private val networkService: ExampleBackendService
) : RemoteMediator<Int, User>() {
  val userDao = database.userDao()

  override suspend fun load(
    loadType: LoadType,
    state: PagingState<Int, User>
  ): MediatorResult {
    // ...
  }
}

@UseExperimental(markerClass = ExperimentalPagingApi.class)
class ExampleRemoteMediator extends RxRemoteMediator<Integer, User> {
  private String query;
  private ExampleBackendService networkService;
  private RoomDb database;
  private UserDao userDao;

  ExampleRemoteMediator(
    String query,
    ExampleBackendService networkService, RoomDb database
  ) {
    query = query;
    networkService = networkService;
    database = database;
    userDao = database.userDao();
  }

  @NotNull
  @Override
  public Single<MediatorResult> loadSingle(
    @NotNull LoadType loadType,
    @NotNull PagingState<Integer, User> state
  ) {
    ...
  }
}

class RemoteMediatorGuavaItemKeyedSample extends ExampleRemoteMediator<Integer, User> {
  private String query;
  private ExampleBackendService networkService;
  private RoomDb database;
  private UserDao userDao;
  private Executor bgExecutor;

  ExampleRemoteMediator(
    String query,
    ExampleBackendService networkService,
    RoomDb database,
    Executor bgExecutor
  ) {
    this.query = query;
    this.networkService = networkService;
    this.database = database;
    this.userDao = database.userDao();
    this.bgExecutor = bgExecutor;
  }

  @NotNull
  @Override
  public ListenableFuture<MediatorResult> loadFuture(
    @NotNull LoadType loadType,
    @NotNull PagingState<Integer, User> state
  ) {
    ...
  }
}

The load() method is responsible for updating the backing dataset and invalidating the PagingSource. Some libraries that support paging (like Room) will automatically handle invalidating PagingSource objects that they implement.

The load() method takes in two parameters:

• PagingState, which contains information about the pages loaded so far, the most recently accessed index, and the PagingConfig object that you used to initialize the paging stream.
• LoadType, which indicates the type of the load: REFRESH, APPEND, or PREPEND.

The return value of the load() method is a MediatorResult object. MediatorResult can either be MediatorResult.Error (which includes the error description) or MediatorResult.Success (which includes a signal stating whether or not there is more data to load).

The load() method must perform the following steps:

1. Determine which page to load from the network depending on the load type and the data that has been loaded so far.
2. Trigger the network request.
3. Perform actions depending on the outcome of the load operation:
   • If the load is successful and the received list of items is not empty, then store the list items in the database and return MediatorResult.Success(endOfPaginationReached = false).
   • If the load is successful and the received list of items is empty, then return MediatorResult.Success(endOfPaginationReached = true).
   • If the request causes an error, then return MediatorResult.Error.

override suspend fun load(
  loadType: LoadType,
  state: PagingState<Int, User>
): MediatorResult {
  return try {
    // The network load method takes an optional after=<user.id>
    // parameter. For every page after the first, pass the last user
    // ID to let it continue from where it left off. For REFRESH,
    // pass null to load the first page.
    val loadKey = when (loadType) {
      LoadType.REFRESH -> null
      // In this example, you never need to prepend, since REFRESH
      // will always load the first page in the list. Immediately
      // return, reporting end of pagination.
      LoadType.PREPEND ->
        return MediatorResult.Success(endOfPaginationReached = true)
      LoadType.APPEND -> {
        val lastItem = state.lastItemOrNull()

        // You must explicitly check if the last item is null when
        // appending, since passing null to networkService is only
        // valid for initial load. If lastItem is null it means no
        // items were loaded after the initial REFRESH and there are
        // no more items to load.
        if (lastItem == null) {
          return MediatorResult.Success(
            endOfPaginationReached = true
          )
        }

        lastItem.id
      }
    }

    // Suspending network load via Retrofit. This doesn't need to be
    // wrapped in a withContext(Dispatcher.IO) { ... } block since
    // Retrofit's Coroutine CallAdapter dispatches on a worker
    // thread.
    val response = networkService.searchUsers(
      query = query, after = loadKey
    )

    database.withTransaction {
      if (loadType == LoadType.REFRESH) {
        userDao.deleteByQuery(query)
      }

      // Insert new users into database, which invalidates the
      // current PagingData, allowing Paging to present the updates
      // in the DB.
      userDao.insertAll(response.users)
    }

    MediatorResult.Success(
      endOfPaginationReached = response.nextKey == null
    )
  } catch (e: IOException) {
    MediatorResult.Error(e)
  } catch (e: HttpException) {
    MediatorResult.Error(e)
  }
}

@NotNull
@Override
public Single<MediatorResult> loadSingle(
  @NotNull LoadType loadType,
  @NotNull PagingState<Integer, User> state
) {
  // The network load method takes an optional after=<user.id> parameter. For
  // every page after the first, pass the last user ID to let it continue from
  // where it left off. For REFRESH, pass null to load the first page.
  String loadKey = null;
  switch (loadType) {
    case REFRESH:
      break;
    case PREPEND:
      // In this example, you never need to prepend, since REFRESH will always
      // load the first page in the list. Immediately return, reporting end of
      // pagination.
      return Single.just(new MediatorResult.Success(true));
    case APPEND:
      User lastItem = state.lastItemOrNull();

      // You must explicitly check if the last item is null when appending,
      // since passing null to networkService is only valid for initial load.
      // If lastItem is null it means no items were loaded after the initial
      // REFRESH and there are no more items to load.
      if (lastItem == null) {
        return Single.just(new MediatorResult.Success(true));
      }

      loadKey = lastItem.getId();
      break;
  }

  return networkService.searchUsers(query, loadKey)
    .subscribeOn(Schedulers.io())
    .map((Function<SearchUserResponse, MediatorResult>) response -> {
      database.runInTransaction(() -> {
        if (loadType == LoadType.REFRESH) {
          userDao.deleteByQuery(query);
        }

        // Insert new users into database, which invalidates the current
        // PagingData, allowing Paging to present the updates in the DB.
        userDao.insertAll(response.getUsers());
      });

      return new MediatorResult.Success(response.getNextKey() == null);
    })
    .onErrorResumeNext(e -> {
      if (e instanceof IOException || e instanceof HttpException) {
        return Single.just(new MediatorResult.Error(e));
      }

      return Single.error(e);
    });
}

@NotNull
@Override
public ListenableFuture<MediatorResult> loadFuture(
  @NotNull LoadType loadType,
  @NotNull PagingState<Integer, User> state
) {
  // The network load method takes an optional after=<user.id> parameter. For
  // every page after the first, pass the last user ID to let it continue from
  // where it left off. For REFRESH, pass null to load the first page.
  String loadKey = null;
  switch (loadType) {
    case REFRESH:
      break;
    case PREPEND:
      // In this example, you never need to prepend, since REFRESH will always
      // load the first page in the list. Immediately return, reporting end of
      // pagination.
      return Futures.immediateFuture(new MediatorResult.Success(true));
    case APPEND:
      User lastItem = state.lastItemOrNull();

      // You must explicitly check if the last item is null when appending,
      // since passing null to networkService is only valid for initial load.
      // If lastItem is null it means no items were loaded after the initial
      // REFRESH and there are no more items to load.
      if (lastItem == null) {
        return Futures.immediateFuture(new MediatorResult.Success(true));
      }

      loadKey = lastItem.getId();
      break;
  }

  ListenableFuture<MediatorResult> networkResult = Futures.transform(
    networkService.searchUsers(query, loadKey),
    response -> {
      database.runInTransaction(() -> {
        if (loadType == LoadType.REFRESH) {
          userDao.deleteByQuery(query);
        }

        // Insert new users into database, which invalidates the current
        // PagingData, allowing Paging to present the updates in the DB.
        userDao.insertAll(response.getUsers());
      });

      return new MediatorResult.Success(response.getNextKey() == null);
    }, bgExecutor);

  ListenableFuture<MediatorResult> ioCatchingNetworkResult =
    Futures.catching(
      networkResult,
      IOException.class,
      MediatorResult.Error::new,
      bgExecutor
    );

  return Futures.catching(
    ioCatchingNetworkResult,
    HttpException.class,
    MediatorResult.Error::new,
    bgExecutor
  );
}

Create a Pager

Finally, you must create a Pager instance to set up the stream of paged data. This is similar to creating a Pager from a simple network data source, but there are two things you must do differently:

• Instead of passing a PagingSource constructor directly, you must provide the query method that returns a PagingSource object from the DAO.
• You must provide an instance of your RemoteMediator implementation as the remoteMediator parameter.

val userDao = database.userDao()
val pager = Pager(
  config = PagingConfig(pageSize = 50)
  remoteMediator = ExampleRemoteMediator(query, database, networkService)
) {
  userDao.pagingSource(query)
}

UserDao userDao = database.userDao();
Pager<Integer, User> pager = Pager(
  new PagingConfig(/* pageSize = */ 20),
  null, // initialKey,
  new ExampleRemoteMediator(query, database, networkService)
  () -> userDao.pagingSource(query));

UserDao userDao = database.userDao();
Pager<Integer, User> pager = Pager(
  new PagingConfig(/* pageSize = */ 20),
  null, // initialKey
  new ExampleRemoteMediator(query, database, networkService, bgExecutor),
  () -> userDao.pagingSource(query));

Manage remote keys

Remote keys are keys that a RemoteMediator implementation uses to tell the backend service which data to load next. In the simplest case, each item of paged data includes a remote key that you can easily reference. However, if the remote keys do not correspond to individual items, then you must store them separately and manage them in your load() method.

This section describes how to collect, store, and update remote keys that are not stored in individual items.

Add remote key table

When remote keys are not directly associated with list items, it is best to store them in a separate table in the local database. Define a Room entity that represents a table of remote keys:

@Entity(tableName = "remote_keys")
data class RemoteKey(val label: String, val nextKey: String?)

@Entity(tableName = "remote_keys")
public class RemoteKey {
  public String label;
  public String nextKey;
}

@Entity(tableName = "remote_keys")
public class RemoteKey {
  public String label;
  public String nextKey;
}

You must also define a DAO for the RemoteKey entity:

@Dao
interface RemoteKeyDao {
  @Insert(onConflict = OnConflictStrategy.REPLACE)
  suspend fun insertOrReplace(remoteKey: RemoteKey)

  @Query("SELECT * FROM remote_keys WHERE label = :query")
  suspend fun remoteKeyByQuery(query: String): RemoteKey

  @Query("DELETE FROM remote_keys WHERE label = :query")
  suspend fun deleteByQuery(query: String)
}

@Dao
interface RemoteKeyDao {
  @Insert(onConflict = OnConflictStrategy.REPLACE)
  void insertOrReplace(RemoteKey remoteKey);

  @Query("SELECT * FROM remote_keys WHERE label = :query")
  Single<RemoteKey> remoteKeyByQuerySingle(String query);

  @Query("DELETE FROM remote_keys WHERE label = :query")
  void deleteByQuery(String query);
}

@Dao
interface RemoteKeyDao {
  @Insert(onConflict = OnConflictStrategy.REPLACE)
  void insertOrReplace(RemoteKey remoteKey);

  @Query("SELECT * FROM remote_keys WHERE label = :query")
  ListenableFuture<RemoteKey> remoteKeyByQueryFuture(String query);

  @Query("DELETE FROM remote_keys WHERE label = :query")
  void deleteByQuery(String query);
}

Loading with remote keys

When your load() method needs to manage remote keys, you must define it differently in the following ways as compared to basic usage of RemoteMediator:

• Include an additional property that holds a reference to the DAO for your remote key table.
• Determine which key to load next by querying the remote key table instead of using PagingState.
• Insert or store the returned remote key from the network data source in addition to the paged data itself.

@OptIn(ExperimentalPagingApi::class)
class ExampleRemoteMediator(
  private val query: String,
  private val database: RoomDb,
  private val networkService: ExampleBackendService
) : RemoteMediator<Int, User>() {
  val userDao = database.userDao()
  val remoteKeyDao = database.remoteKeyDao()

  override suspend fun load(
    loadType: LoadType,
    state: PagingState<Int, User>
  ): MediatorResult {
    return try {
      // The network load method takes an optional String
      // parameter. For every page after the first, pass the String
      // token returned from the previous page to let it continue
      // from where it left off. For REFRESH, pass null to load the
      // first page.
      val loadKey = when (loadType) {
        LoadType.REFRESH -> null
        // In this example, you never need to prepend, since REFRESH
        // will always load the first page in the list. Immediately
        // return, reporting end of pagination.
        LoadType.PREPEND -> return MediatorResult.Success(
          endOfPaginationReached = true
        )
        // Query remoteKeyDao for the next RemoteKey.
        LoadType.APPEND -> {
          val remoteKey = database.withTransaction {
            remoteKeyDao.remoteKeyByQuery(query)
          }

          // You must explicitly check if the page key is null when
          // appending, since null is only valid for initial load.
          // If you receive null for APPEND, that means you have
          // reached the end of pagination and there are no more
          // items to load.
          if (remoteKey.nextKey == null) {
            return MediatorResult.Success(
              endOfPaginationReached = true
            )
          }

          remoteKey.nextKey
        }
      }

      // Suspending network load via Retrofit. This doesn't need to
      // be wrapped in a withContext(Dispatcher.IO) { ... } block
      // since Retrofit's Coroutine CallAdapter dispatches on a
      // worker thread.
      val response = networkService.searchUsers(query, loadKey)

      // Store loaded data, and next key in transaction, so that
      // they're always consistent.
      database.withTransaction {
        if (loadType == LoadType.REFRESH) {
          remoteKeyDao.deleteByQuery(query)
          userDao.deleteByQuery(query)
        }

        // Update RemoteKey for this query.
        remoteKeyDao.insertOrReplace(
          RemoteKey(query, response.nextKey)
        )

        // Insert new users into database, which invalidates the
        // current PagingData, allowing Paging to present the updates
        // in the DB.
        userDao.insertAll(response.users)
      }

      MediatorResult.Success(
        endOfPaginationReached = response.nextKey == null
      )
    } catch (e: IOException) {
      MediatorResult.Error(e)
    } catch (e: HttpException) {
      MediatorResult.Error(e)
    }
  }
}

@NotNull
@Override
public Single<MediatorResult> loadSingle(
  @NotNull LoadType loadType,
  @NotNull PagingState<Integer, User> state
) {
  // The network load method takes an optional String parameter. For every page
  // after the first, pass the String token returned from the previous page to
  // let it continue from where it left off. For REFRESH, pass null to load the
  // first page.
  Single<RemoteKey> remoteKeySingle = null;
  switch (loadType) {
    case REFRESH:
      // Initial load should use null as the page key, so you can return null
      // directly.
      remoteKeySingle = Single.just(new RemoteKey(mQuery, null));
      break;
    case PREPEND:
      // In this example, you never need to prepend, since REFRESH will always
      // load the first page in the list. Immediately return, reporting end of
      // pagination.
      return Single.just(new MediatorResult.Success(true));
    case APPEND:
      // Query remoteKeyDao for the next RemoteKey.
      remoteKeySingle = mRemoteKeyDao.remoteKeyByQuerySingle(mQuery);
      break;
  }

  return remoteKeySingle
    .subscribeOn(Schedulers.io())
    .flatMap((Function<RemoteKey, Single<MediatorResult>>) remoteKey -> {
      // You must explicitly check if the page key is null when appending,
      // since null is only valid for initial load. If you receive null
      // for APPEND, that means you have reached the end of pagination and
      // there are no more items to load.
      if (loadType != REFRESH && remoteKey.getNextKey() == null) {
        return Single.just(new MediatorResult.Success(true));
      }

      return networkService.searchUsers(query, remoteKey.getNextKey())
        .map(response -> {
          database.runInTransaction(() -> {
            if (loadType == LoadType.REFRESH) {
              userDao.deleteByQuery(query);
              remoteKeyDao.deleteByQuery(query);
            }

            // Update RemoteKey for this query.
            remoteKeyDao.insertOrReplace(new RemoteKey(query, response.getNextKey()));

            // Insert new users into database, which invalidates the current
            // PagingData, allowing Paging to present the updates in the DB.
            userDao.insertAll(response.getUsers());
          });

          return new MediatorResult.Success(response.getNextKey() == null);
        });
    })
    .onErrorResumeNext(e -> {
      if (e instanceof IOException || e instanceof HttpException) {
        return Single.just(new MediatorResult.Error(e));
      }

      return Single.error(e);
    });
}

@NotNull
@Override
public ListenableFuture<MediatorResult> loadFuture(
  @NotNull LoadType loadType,
  @NotNull PagingState<Integer, User> state
) {
  // The network load method takes an optional after=<user.id> parameter. For
  // every page after the first, pass the last user ID to let it continue from
  // where it left off. For REFRESH, pass null to load the first page.
  ResolvableFuture<RemoteKey> remoteKeyFuture = ResolvableFuture.create();
  switch (loadType) {
    case REFRESH:
      remoteKeyFuture.set(new RemoteKey(query, null));
      break;
    case PREPEND:
      // In this example, you never need to prepend, since REFRESH will always
      // load the first page in the list. Immediately return, reporting end of
      // pagination.
      return Futures.immediateFuture(new MediatorResult.Success(true));
    case APPEND:
      User lastItem = state.lastItemOrNull();

      // You must explicitly check if the last item is null when appending,
      // since passing null to networkService is only valid for initial load.
      // If lastItem is null it means no items were loaded after the initial
      // REFRESH and there are no more items to load.
      if (lastItem == null) {
        return Futures.immediateFuture(new MediatorResult.Success(true));
      }

      // Query remoteKeyDao for the next RemoteKey.
      remoteKeyFuture.setFuture(
        remoteKeyDao.remoteKeyByQueryFuture(query));
      break;
  }

  return Futures.transformAsync(remoteKeyFuture, remoteKey -> {
    // You must explicitly check if the page key is null when appending,
    // since null is only valid for initial load. If you receive null
    // for APPEND, that means you have reached the end of pagination and
    // there are no more items to load.
    if (loadType != LoadType.REFRESH && remoteKey.getNextKey() == null) {
      return Futures.immediateFuture(new MediatorResult.Success(true));
    }

    ListenableFuture<MediatorResult> networkResult = Futures.transform(
      networkService.searchUsers(query, remoteKey.getNextKey()),
      response -> {
        database.runInTransaction(() -> {
        if (loadType == LoadType.REFRESH) {
          userDao.deleteByQuery(query);
          remoteKeyDao.deleteByQuery(query);
        }

        // Update RemoteKey for this query.
        remoteKeyDao.insertOrReplace(new RemoteKey(query, response.getNextKey()));

        // Insert new users into database, which invalidates the current
        // PagingData, allowing Paging to present the updates in the DB.
        userDao.insertAll(response.getUsers());
      });

      return new MediatorResult.Success(response.getNextKey() == null);
    }, bgExecutor);

    ListenableFuture<MediatorResult> ioCatchingNetworkResult =
      Futures.catching(
        networkResult,
        IOException.class,
        MediatorResult.Error::new,
        bgExecutor
      );

    return Futures.catching(
      ioCatchingNetworkResult,
      HttpException.class,
      MediatorResult.Error::new,
      bgExecutor
    );
  }, bgExecutor);
}

Refresh in place

If your app only needs to support network refreshes from the top of the list as in the previous examples, then your RemoteMediator does not need to define prepend load behavior.

However, if your app needs to support loading incrementally from the network into the local database, then you must provide support for resuming pagination starting at the anchor, the user's scroll position. Room's PagingSource implementation handles this for you, but if you're not using Room you can do this by overriding PagingSource.getRefreshKey().

// Item-keyed.
override fun getRefreshKey(state: PagingState): String? {
  return state.anchorPosition?.let { anchorPosition ->
    state.getClosestItemToPosition(anchorPosition)?.id
  }
}

// Positional.
override fun getRefreshKey(state: PagingState): Int? {
  return state.anchorPosition
}

// Item-keyed.
@Nullable
@Override
String getRefreshKey(state: PagingState) {
  Integer anchorPosition = state.anchorPosition;
  if (anchorPosition == null) {
    return null;
  }

  return state.getClosestItemToPosition(anchorPosition);
}

// Positional.
@Nullable
@Override
Integer getRefreshKey(state: PagingState) {
  return state.anchorPosition;
}

// Item-keyed.
@Nullable
@Override
String getRefreshKey(state: PagingState) {
  Integer anchorPosition = state.anchorPosition;
  if (anchorPosition == null) {
    return null;
  }

  return state.getClosestItemToPosition(anchorPosition);
}

// Positional.
@Nullable
@Override
Integer getRefreshKey(state: PagingState) {
  return state.anchorPosition;
}

Figure 2 illustrates the process of loading data first from the local database, and then from the network once the database is out of data.

Additional resources

To learn more about the Paging library, see the following additional resources:

Codelabs

• Android Paging codelab

Samples

• Android Architecture Components Paging with Database and Network sample