Offline networking support with CoreData and NSOperation

In this post, I propose a pattern for allowing apps to transmit data through unstable network connections. I’ll be taking advantage of the modern architecture present on the iOS Platform, as well as the popular AFNetworking (or AlamoFire). To follow along, you’ll need some knowledge of iOS Native Development, NSOperation API, CoreData, and Networking.

The pattern I’ll be discussing could be seen as an application of the Command pattern documented in Erich Gamma’s Design Patterns, Elements of Reusable Object-Oriented Software, with the aid of the persistence layer provided by CoreData. It can be easily migrated to platforms such as Android, provided we have a similar APIs that support a higher abstraction layer for the management of threading, operations, and database access for persistence.

Let’s start by visualizing a scenario in which we’d need to transmit data over an unstable network:

1. The user triggers a UI control that requires a network call to start a process on the backend (e.g.,  uploading a photo).
2. The data (parameters for the network call) are wrapped in a typical network request to instruct the network API to start the communication.
3. The request is initiated with the platform API (or by using an API wrapping the core network subsystem such as the mentioned frameworks); if necessary, the required hardware establishes a connection with the access point for data transmission.
4. The request either fail or succeeds to reach the server; the response is captured and processed with the required business logic within the app.
5. Once processed, the response data triggers a state change in the app. This change is reflected in the UI, informing the user as to whether the communication succeeded or failed; if the request succeeded, the UI also needs to reflect execution of the business logic associated with the initial request.

Any modern app should be prepared to deal with edge cases where communication between the app and the server is not guaranteed. However, it’s also important to decouple the business logic, UI changes, and data management associated with network request errors and retries, as failure to do so dramatically increases the complexity of the code. The pattern described below is one approach to doing so.

Asynchronous Operations

Our first step is to decouple network communications and encapsulate them in tiny units called operations. This is a straightforward concept from the command pattern that will prove to be useful in the following sections.

Operations are an implementation of NSOperation. The pattern we will use is the asynchronous operation, taking advantage of the API provided by the iOS platform in the Foundation framework. Wrapping network requests in operations is essential to understanding networking requirements for the app, and to the method we’ll use to inform the App of the results of requests.

To simplify the implementation of these operations, we can create an abstract operation that will be the base class for our custom operations. The abstract operation will have the management required by NSOperation in order to behave as expected by the API. Here is a simple implementation of this AbstractAsyncOperation:

[code language=”swift”]
class AbstractAsyncOperation : Operation{
private var hasFinished : Bool = false
private var startedExecution : Bool = false

/**
Sets whether the operation is executing
– Parameter status: Whether the operation is still executing
*/
func updateExecutionStatus( _ status: Bool ){
self .willChangeValue(forKey: “isExecuting”)
startedExecution = status
self.didChangeValue(forKey: “isExecuting”)
}

/**
Sets whether this operation has finished
– Parameter finished: Whether the operation has finished
*/
func updateFinishedStatus( _ status: Bool){
self.willChangeValue(forKey: “isFinished”)
hasFinished = status
self.didChangeValue(forKey: “isFinished”)
}

// MARK: – Overrides
override var isAsynchronous: Bool { return true }
override var isFinished: Bool { return hasFinished }
override var isExecuting: Bool { return startedExecution }

// MARK: – Override cancellation to update the command

override func cancel() {
if(!isExecuting){
finish()
}
super.cancel()
}

// MARK: – Methods

/// The base implementation for the abstract operation
/// This is required by the NSOperation API to start execution of the asynchronous operation
override func start() {
updateExecutionStatus(true)
guard !isCancelled
else{
finish()
return
}

// Implement the actual functionality for the operation
doOperation()
}

/**
This is the empty implementation for this operation and should be overriden by subclasses.
*/
public func doOperation(){

}

/**
Marks the operation as finished and sets the command to the status given

This method allows to mark the operation as finished by setting the execution state to false
and the finished property to true.
*/
public func finish( ){
updateExecutionStatus(false)
updateFinishedStatus(true)
}

}
[/code]

Notice that we’re only implementing the basic states required to comply with the NSOperation API and the semantics of an asynchronous operation. It will be the responsibility of child classes to call the cancel and finish methods when appropriate to avoid saturation of the operation queue.  The later is very important: failure to call the finish method for an operation (i.e. informing the operation queue the operation has finished) will result in the operation remaining in the queue indefinitely, thus filling and stalling the queue.

I won’t cover the basics or behavior of OperationQueue (NSOperationQueue) here; if you need a refresher, I recommend reviewing Apple’s documentation.

By implementing our base abstract class, we’ve eased the overall process of complying with Apple’s API. Now, to wrap a network operation in this abstract operation, we need to:

• Extend AbstractAsyncOperation
• Implement doOperation
• Check for isCancelled
• Call finish() when appropriate

You may have noticed that our abstract operation doesn’t contain any reference to networking, retries, or persistence, with the exception of the transient states required by the NSOperation API.

We already have a way to encapsulate network operations within an abstraction of an operation (an asynchronous one). Now we need a smart queue that can adapt itself whenever networking conditions change.

Take a look at the figure above. We need the operation queue to react to different networking conditions, and to app state conditions, to decide whether we can process further operations that require network connectivity or foreground CPU processing time. Recall that if you don’t implement background modes or tasks on iOS, your app will stall, and cease to consume any CPU whatsoever. Usually, the background mode in an iOS application suspends the operation queue and finishes the network connections.

Here’s one implementation of an Operation Queue that can react to such changes:

[code language=”swift”]
/**
This queue allows to decide whether operations can be queued depending on the state of the app
notably the background state and the network reachability.
*/
class GlobalStateAwareQueue : OperationQueue{

/// Whether the application is active, so it can execute network operations
var isAppActive = false

/// Whether the network is reachable
var isNetworkActive = false

/// Whether the queue can queue operations
var canQueue : Bool {
return isAppActive && isNetworkActive
}

/**
An operation that will do a sanity check whenever the operation queue finishes its operations
*/
fileprivate weak var sanityOperation : Operation?
}
[/code]

The code above ensures that the queue is informed of both network condition changes and app lifecycle changes. Note that the only thing we added to the base OperationQueue is an awareness of convenience states. Note that we have a flag for isNetworkActive, which we can use along the Reachability API as demonstrated in Apple’s sample implementation. When either the app or the network is inactive, we know there’s no sense in queueing more network-related operations, since they’re going to fail anyway.

The app state changes are managed internally by the Operation Queue, but there are some cases in which a simple queue suspension won’t handle all of the edge cases required in common apps. The isAppActive flag is more flexible, and can manage such cases at a higher level.

At this point, we can handle most use cases with just the Operation Queue API and some Network Helper implementation. Let’s now take a look at how serialization can help us achieve offline networking without scattering this handling in view controllers and other helper classes.

CoreData Integration

In this figure, we’re introducing a database scheme (achieved with CoreData Model Editor included in Xcode) to present a model for serializing network commands/requests .

You can see we have several properties that might make sense, depending on our particular business case. However, we can create a minimalist implementation with just the properties date, retries, status, type, and uuid. Every other property is a convenience property for storing relationships with your own model.

Here’s a summary of the Command Entity properties:

• uuid: (String) The UUID (primary key) for the command. We could use CoreData’s own PK system, but sometimes it’s useful to have such keys generated and transmitted to the server (instead of plain integers).
• type: (Int) The type of command. This is an important property, since it allows us to create the appropriate operation related to the command. Though there are many ways to achieve this (e.g., dynamic class instantiation), it’s often easier to understand and locate a command based on a key rather than in the implementation class.
• date: (Date) The timestamp when the command entity was created.
• retries: (Int) The number of times execution of this command has been attempted in an operation queue.
• status: (Int) The current status of the command.

Why use CoreData here, rather than files or SQLite? Because of CoreData’s concurrency types.

By using a private execution queue, we’re not only leaving the main queue for the important task of UI rendering and updating, but we’re also guaranteeing ourselves a consistent state whenever we access the underlying database through the serial queue. This feature is extremely important, since we need to have a consistent state whenever we try to wrap a network request in an operation.

The following code snippet illustrates the allocation of the CoreData stack:

[code language=”swift”]
/**
Extension to UIApplication to have the CoreData stack ready
*/
extension UIApplication{

/// The name for the coredata model used in the app
static let coreDataModel = “Model”

/// The name for the file used to persist the data for core data
static let coreDataFile = “CommandsQueue”

/// Access to the persistentContainer property
@objc public var persistentContainer : NSPersistentContainer? {
get{
return objc_getAssociatedObject(self, &kCoreDataPersistentContainerKey) as? NSPersistentContainer
}
}

/// Access to the root MOC
@objc public var globalMOC : NSManagedObjectContext?{
get{
return objc_getAssociatedObject(self, &kRootMOC) as? NSManagedObjectContext
}
}

// MARK : – Initialization

/**
Initializes the coredata stack according the doc for iOS 10.0

This call is synchronized. The callback closure will be called when the persistence store
for the stack is initialized or fails. The lock mechanism is released before calling the closure, so it’s safe to make changes
and call this method again.

A global MOC is initialized with its own private queue.

• Parameter callback: The closure to be called when the stack is initialized.
  */
  @objc func initCoreData( _ callback : (()->Void)?){

// Just to be safe, make this call
appInternalCoreDataLock.lock()

guard persistentContainer == nil
else {
appInternalCoreDataLock.unlock()
return
}

let container = AppPersistentContainer(name: UIApplication.coreDataModel)
container.loadPersistentStores { (description, error) in
if error == nil {
objc_setAssociatedObject(self, &kCoreDataPersistentContainerKey, container, .OBJC_ASSOCIATION_RETAIN)
objc_setAssociatedObject(self, &kRootMOC, container.newBackgroundContext() , .OBJC_ASSOCIATION_RETAIN_NONATOMIC)

// Unlock until we have finished initializing core data
appInternalCoreDataLock.unlock()
if callback != nil{
DispatchQueue.main.async(execute: callback!)
}
}
}
}
[/code]

Now that we have a way of initializing the CoreData stack and our model for serializing networking requests, we need some helper states to reflect our Command Entity’s state of execution. To implement these helper states, we’ll use the Command subclass CoreData provides when generating the Managed Object Classes (MOCs):

[code language=”swift”]
public class CommandMO: NSManagedObject {

/// The name of the core data entity
@objc static public let entityName = “Command”

/// The maximum number of retries for a command. For the purpose of this example we’ll use a hard coded ceiling
@objc static public let maximumRetries = 5

/// The command types
enum types : Int16 {

/// your commands enumerated here…
….
}

/// The status for a command
enum status : Int16 {
/// Command just created and not yet queued to be sent
case new = 0
/// Command queued to be sent
case queued = 1
/// Command sent to the server
case sent = 2
/// Command received a successful response
case success = 4
/// Command sent to the server and returned with error
case error = 3
/// Command retried too many times
case tooManyTimes = 10
}

/**
Validates whether this command has been set too many times to error
– Note: Execute only in moc’s queue
*/
public func validateErrorStatus(){
if let internalStatus = CommandMO.status(rawValue: self.status),
internalStatus == .error{
retries = retries + 1
if retries >= CommandMO.maximumRetries{
self.status = CommandMO.status.tooManyTimes.rawValue
}
}
}
}
[/code]

As you can see, the type property allows you to easily define any number of commands reflecting network requests.

Besides command types, it’s important to notice the command states. These command states allow us to track the execution status of a network operation, including whether it’s completed, experienced an error, or is awaiting retry. Modifications to the Managed Object should be done on the MOCs execution queue to avoid corruption.

The following figure can help us understand how commands are transitioned to a particular state:

• • When we create a command, it is in the new state.
  • When the command has an operation associated and is queued in an operation queue, it transitions to a queued state.
  • When the operation starts, execution could be cancelled, or could experience a data error (either of which would transition the command state to error), or could start the network request (which would transition the command state to sent).
  • When the network executes the completion handler (either by an error or a successful response) we transition the command state to either error or success.
  • Whenever we transition the command to the error state, we verify the number of retries. If we reach a threshold of attempts, we transition the command state to tooManyTimes.

Queue Persistence Integration

We have the following pending tasks to wire our base classes that model our problem:

• Command-to-Operation Mapping
• Operation Queueing
• Queue adaptation to app state changes.

For the first two, we need to create both entities (Command and Operation objects) and communicate this to the Operation Queue instance. To achieve this I prefer the DAO pattern to decouple database code from some other logic. After we are done with the entity creation we communicate to some factory object that will use the type property from the Command entity to figure out which operation to instantiate. Once we have the appropriate operation we can queue it in the Operation Queue instance we have in the app (either by the Singleton pattern of by a dynamic property in a Singleton Extension ).

Recall that Core Data Entities should be manipulated in the MOCs queue. To cross thread boundaries and avoid using the main thread for database access, we’ll use the uuid property we defined when creating the Command entity. Here’s an implementation of the notification whenever we create a new command entity:

[code language=”swift”]
/**
DAO to create commands to be persisted in the app’s database.
*/
class CommandDAO: NSObject {

// MARK: – Private Utility Methods

private static func saveContextAndPost( _ moc : NSManagedObjectContext , command : CommandMO , description : String? = nil){
do{
try moc.save()
DispatchQueue.main.async {
NotificationCenter.default.post(name: .newCommand, object: nil, userInfo: [UserInfoCommandKeys.command.rawValue:command.uuid!])
}
}
catch{
print(“(description ?? “While saving the context”) : (error)”)
}
}
….
// MARK: – Command creation methods

[/code]

When we confirm the serialization of the command, we post the message so the queue manager will handle the message and will build up the CommandMO entity. Here’s the handler for this notification, calling the factory method that handles the operation creation:

[code language=”swift”]
/// Handler when a new command is created
@objc func onNewCommandCreated( _ notification : Notification){
guard let userInfo = notification.userInfo,
let commandUUID = userInfo[UserInfoCommandKeys.command.rawValue] as? String,
let moc = UIApplication.shared.globalMOC
else { return }

moc.perform {
[weak self] in
self?.enqueue(uuid: commandUUID)
}
}
[/code]

Why are we executing the enqueue method in the MOCs operation queue? Since we need to update the CommandMO entity when the operation is actually queued, and to recover the entity’s properties, we need to do it in the MOCs queue. Why are we jumping between queues? Decoupling — we don’t want the DAO to also be in charge of queueing operations.

The enqueue method is very important; it’s the entry point for adding any operation to the GlobalStateAwareQueue:

[code language=”swift”]
/**
Queues a command in the upload queue to be executed

This method should be executed on the private’s moc queue since it modifies mos
*/
private func enqueue( uuid : String ) {
guard let queue = uploadQueue,
let moc = globalMOC,
queue.canQueue,
let command = CommandDAO.fetchCommand(uuid: uuid),
let commandOperation = commandOperationFactory(command: command)
else { return }

// The operation is unique, we should queue it
queue.addOperation(commandOperation)

// Check if we have a sanity operation already in place
if let sanityOperation = queue.sanityOperation{
sanityOperation.addDependency(commandOperation)
}
else{

// Create a new sanity operation
let sanityOperation = BlockOperation{
[weak self] in
print(“Checking for pending commands…”)
self?.enqueuePendingCommands()
}

// Setup the dependency and queue the operation
queue.sanityOperation = sanityOperation
sanityOperation.addDependency(commandOperation)
queue.addOperation(sanityOperation)
}

command.status = CommandMO.status.queued.rawValue
do{
try moc.save()
}
catch{
print(“There was an error updating the context (error)”)
}
}
[/code]

The logic of this method can be summarized as follows:

• Get entity references (queue, MOC, and command)
• Create the operation (via the factory method)
• Queue the operation
• Provide the sanity operation (see below) if required

What’s the sanity operation? Here’s the trick to this approach — we use the addDependency API provided in NSOperation to know whether the operations have finished (or whether the queue is empty, since we’re using this queue exclusively). This allows the current operation to review a database containing all the commands that need to be added to the operation queue. The addDependency API acts like a barrier, so it’s a very useful pattern to continue with work that may have been left behind (for example, due to network conditions).

You might have noted the enqueuePendingCommands method in the preceding code. This method has a fairly straightforward implementation:

[code language=”swift”]
/**
Fetches all the pending commands in the database in order to have them queued
*/
private func enqueuePendingCommands(){
guard let queue = uploadQueue,
queue.canQueue && !queue.isSuspended else {
print(“Couldn’t enqueue pending commands since either there’s no queue or is cancelled “)
return
}
CommandDAO.fetchPendingCommands { [weak self] (fetched) in
guard let commands = fetched else { return }

// Clear the sanity operation, a new one will be created if needed.
queue.sanityOperation = nil
for command in commands{
self?.enqueue(uuid: command.uuid!)
}
}
}
[/code]

Notice that we only query the database, and in the completion handler we enqueue the commands. The completion handler for fetchPendingCommands is executed in the MOCs queue so it won’t corrupt the MOCs data.

We have the mechanism to add commands (operations) to the operation queue, and since these operations are subclasses of AbstractAsyncOperation, we only need to integrate the appropriate calls to load the information stored in the CommandMO entities. Following are the changes to AbstractAsyncOperation to support these CommandMO entities:

[code language=”swift”]
/**
An asynchronouse generic operation
*/
class AbstractAsyncOperation : Operation{
…
/// The command that should be sent to the server
var commandUUID : String? = nil
…
public func finish( status : CommandMO.status? = nil){
if status != nil{
persist(status: status!)
}
…
}

public func persist( status : CommandMO.status ){
guard let moc = UIApplication.shared.globalMOC,
let uuid = commandUUID else { return }
moc.perform{
guard let command = CommandDAO.fetchCommand(uuid: uuid) else { return }
command.status = status.rawValue
command.validateErrorStatus()
do …
/// Save the moc, handle error
}
}
…
[/code]

Serializing the execution status in the base abstract operation is a great way to keep track of execution status of network requests (or any other operation we might want to serialize in a database).

Networking integration

The following figure generalizes a network operation and the use of this API to keep track of its status:

Note that the sent status is updated once we’ve actually sent the network request. Since we’re using the MOCs serial queue to update the status of the command, this status will be updated in an ordered fashion, which avoids inconsistencies in the command state.

We’ve seen how we can use a Core Data Entity to generate an operation that will ultimately trigger a network request and handle possible errors. The piece we’re missing is informing the operation queue of state changes, as well as the actual handling we will use to stop further processing.

First, let’s implement the case in which we notice changes in reachability. As mentioned before, we can make good use of the API exposed in AFNetworking for instance. This framework has a callback or handler that informs the app of such changes:

[code language=”swift”]
func onReachabilityChanged( status : AFNetworkReachabilityStatus){

guard let queue = uploadQueue else { return }

if status == .notReachable || status == .unknown{
queue.isNetworkActive = false
queue.cancelAllOperations()
}
else{
queue.isNetworkActive = true
enqueuePendingCommands()
}
}
[/code]

As you can see from the code snippet, we have an entry point to inform the queue that network conditions have changed. Current execution operations (that are perhaps stuck in the networking layer from iOS) would fail by a timeout or some other condition (e.g., a reset), which would cause the following operations to transition to the execution state. Instead of waiting for all queued operations to fail, however, we cancel all the pending operations and wait until we have again conditions to transfer data. The operations are stored in the database so that when we invoke the enqueuePendingCommands, the processing of pending network requests will resume with all the operations that haven’t been processed successfully or haven’t reached the maximum number or tries.

Limitations

There are some drawbacks to this approach, including:

• Interactivity: Since we’re serializing the network requests, there’s no guarantee objects waiting for a response will be present when processing occurs. Imagine, for instance, that we’re requesting a state from the server and we wrap the request in an operation and the corresponding command. If the network isn’t available, the Operation is not instantiated, and there’s no way for the caller to get a reference (or even to pass the caller reference) to the actual operation instance.  Network operations of this kind are not suitable for serialization, since the UI is waiting for some kind of update for the network request.
• Network request order: The scheme presented doesn’t take in account network request order. To implement such a scheme, we would need to adapt the command entity to support chained dependencies. Furthermore, we would need to modify the enqueue method to avoid to enqueue commands that have non-successful dependencies.
• Background modes: This approach does not attempt to acquire background CPU time to finish processing network requests.
• Large data files: This approach isn’t suitable for uploading large files; for that, you’d need to use system services.

I hope that you find this information useful in your next network integration for an iOS App. One of my favorite features of this approach is the use of Operation Queues and the way in which we can wrap (and thus isolate/decouple) generic functions into operations. Dependency is one of the most powerful tools in this API,  and it’s worth taking the time to understand it as you work to develop your next App Store hit.