Developer Chronicles on Azure, Cloud, IoT, Modern Javascript and .NET

Azure Cosmos DB – TTL (Time to Live) – Reference Usecase

October 9, 2018 .NET, .NET Core, .NET Framework, Analytics, Architecture, Azure, Azure, Azure Cosmos DB, Azure Functions, Azure IoT Suite, Cloud Computing, Cold Path Analytics, CosmosDB, Emerging Technologies, Hot Path Analytics, Intelligent Cloud, Intelligent Edge, IoT Edge, IoT Hub, Microsoft, Realtime Analytics, Visual Studio 2017, VisualStudio, VS2017, Windows No comments

TTL capability within Azure Cosmos DB is a live saver, as it would take necessary steps to purge redudent data based on the configurations you may. 

Let us think in terms of an Industrial IoT scenario, devices can produce vast amounts of telemetry information, logs and user session information that is only useful until we operate on them and take action on them, to be specific up to finate period of time. Once that data becomes surplus, we need an application logic that purges these old records.

With the “Time to Live” or TTL, Microsoft Cosmos DB provides an ability to have your documents automatically purged from database storage after a certian period if time(which you configured)

  • This TTL by default can be set on a document collection level and later can be overridden on a per document basis.
  • Once the TTL is set, Cosmos DB service will automatically remove the documents when its lifetime is over.
  • Inorder to track TTL, Cosmos DB uses an offset field to check when it was last modified.  This field is identifiable as “_ts”, which exists in every document you create.  Basically it is a UNIX epoch timestamp. This field is updated everytime when the document is modified. [Ref: Picture1]

image

[Picture1]

Enabling TTL on Cosmos DB Collection:

You can enable TTL on a Cosmos DB collection simply by using Azure Portal –> Cosmos DB collection setting for existing or during creation of  a new collection)

TTL value needs to be set in seconds – if you need 90 days => 60 sec * 60 min * 24 hour * 90 days = 7776000 seconds

image

[Picture2]

Below is a one of the reference architecture in which Cosmos DB – TTL would be essentially useful and viable to any Iot business case:

image

[Picture3]

Hope that was helpful to get some understanding. For more references visit:  Cosmos DB Documentation

Azure Cosmos DB–Multi Master

October 8, 2018 .NET, .NET Core, .NET Framework, ASP.NET, Azure, Azure CLI, Azure Cosmos DB, CosmosDB, Data Consistancy, Data Integrity, Microsoft, Multi-master, Performance, Reliability, Resilliancy, Scalability, Scale Up No comments

During the Ignite 2018, Microsoft has announced the general availability of Multi-Master feature being introduced to Azure Cosmos DB to provide more control into data redundancy and elastic scalability for your data from different regions with multiple writes and read instances.

What is Multi-Master essentially?

Multi-master is a capability that provided as part of Cosmos DB, that would provide you multiple write regions and provides an option to handle conflict resolution automatically through different options provided by the platform. Most of the major scenarios you would encounter the conflict can be resolved with these simple configurations.

A sample diagram depicting a use case of load balanced web app writing to respective regional master:-

image

With multi-master, Azure Cosmos DB delivers a single digit millisecond write latency at the 99th percentile anywhere in the world, and now offers 99.999 percent write availability (in addition to 99.999 percent read availability) backed by the industry-leading SLAs.

image

Wow! That’s an amazing performance Cosmos DB guarantees to provide so that your mission-critical systems will have zero downtime, if they start using Cosmos DB.

 

How to Enabled Multi-Master support in your Cosmos DB solutions?

Currently multi-master can only be enabled for new Cosmos DB instances using “Enable Multi-Master” option in Azure Portal or through PowerShell or ARM templates or through SDK.

These options are detailed below with necessary examples:

1.) Azure Portal – Enable Multi-region writes and Enable geo-redundancy

image

2.) Azure CLI 
Set the “enable-multiple-write-locations” parameter to “true”

az cosmosdb create \
   –-name "thingx-cosmosdb-dev" \
   --resource-group "consmosify-dev" \
   --default-consistency-level "Session" \
   --enable-automatic-failover "true" \
   --locations "EastUS=0" "WestUS=1" \
   --enable-multiple-write-locations true \

3.) AzureRM PowerShell
In AzureRM PowerShell cmdlet – Set enableMultipleWriteLocations parameter to “true”

$locations = @(@{"locationName"="East US"; "failoverPriority"=0},
             @{"locationName"="West US"; "failoverPriority"=1})

$iprangefilter = ""

$consistencyPolicy = @{"defaultConsistencyLevel"="Session";
                       "maxIntervalInSeconds"= "10";
                       "maxStalenessPrefix"="200"}

$CosmosDBProperties = @{"databaseAccountOfferType"="Standard";
                        "locations"=$locations;
                        "consistencyPolicy"=$consistencyPolicy;
                        "ipRangeFilter"=$iprangefilter;
                        "enableMultipleWriteLocations"="true"}

New-AzureRmResource -ResourceType "Microsoft.DocumentDb/databaseAccounts" `
  -ApiVersion "2015-04-08" `
  -ResourceGroupName "consmosify-dev" `
  -Location "East US" `
  -Name "thingx-cosmosdb-dev" `
  -Properties $CosmosDBProperties

4.) Through CosmosDB SDK
Setting connection policy in DocumentDBClient and set UseMultipleWriteLocations to true.

ConnectionPolicy policy = new ConnectionPolicy
{
   ConnectionMode = ConnectionMode.Direct,
   ConnectionProtocol = Protocol.Tcp,
   UseMultipleWriteLocations = true,
};
policy.PreferredLocations.Add("East US");
policy.PreferredLocations.Add("West US");
policy.PreferredLocations.Add("West Europe");
policy.PreferredLocations.Add("North Europe");
policy.PreferredLocations.Add("Southeast Asia");
policy.PreferredLocations.Add("Japan East");
policy.PreferredLocations.Add("Japan West");

Azure Cosmos DB multi-master configuration is the game changes that really makes it a true global scale database with automatic conflict resolution capabilities for data synchronization and consistancy.

In my later sessions I will write examples to cover how conflict resolutions can be configured and used in realtime scenarios.

Useful Refs:

Azure Cosmos DB–Reserved Capacity

October 7, 2018 Azure, Azure Cosmos DB, Billing, CosmosDB, Enterprise Agreement, Infrastructure, Microsoft, Pay-as-you-go No comments

Azure Cosmos DB is a planet scale global document database which have been available for Azure Customers based on pay-as-you-go. Reserved Capacity is a new long term pre-paid billing commitment customers can get a discounted pricing. Azure Cosmos DB reserved capacity helps you save money by pre-paying for Azure Cosmos DB resources for a period of one year or three years.

  • Reserve capacity allows you to get a optimum discount for the throughput(RUs) provisioned for Cosmos DB resources.
  • For Ex: Databases and Containers (tables/collections/graphs).
  • It can significantly reduce your Cosmos DB costs and it enables you to save up to 65 percent on regular prices with one-year or three-year upfront commitment.
  • Reserved capacity provides a billing discount and does not affect the runtime state of your Cosmos DB resources.
  • It is available to all supported APIs (including MongoDB, Cassandra, SQL, Gremlin and Azure Tables) and all regions worldwide.

    • Scope of Reservation:

    A reservation is scoped to a single subscription or shared across all your enrolment:

    • Scoped to a Single Subscription: that means a set of cosmos db account resources such as database or containers within the selected subscription.
    • Shared/Scoped to an enrolment: Billing benefit can be shared across any subscription in the enterprise agreement or pay-as-you-go subscriptions.

    cosmos-db-reserved-capacity

    Who can opt-in for Reserved Capacity?

    This feature is currently only applicable for enrolment (Enterprise Agreement customers) or account (Pay-as-you-go customers). Users with MSDN Subscription benefits are not applicable for the Reserved Capacity benefits.

    You can buy Azure Cosmos DB reserved capacity from the Azure portal.

    1. Sign in to the Azure portal.
    2. Select All services > Reservations > Add and Select Product Type > Cosmos DB

    image

    8fafbeff-f11c-4894-98e0-e03d79d1787d

    For more details review reserved capacity documentation.

    Azure Cosmos DB – 429 Too Many Requests

    October 6, 2018 .NET, Azure, CosmosDB, Document DB, Microsoft, Performance, Reliability, Resilliancy, Scalability, Visual Studio 2017, VisualStudio, VS2017 No comments

    Recently while I was doing Performance Testing in one of the APIs interacting with Cosmos DB, I encountered a problem as Azure Cosmos DB API’s started returning Http Code 429.  Http Status Code 429 indicates that too many request been received or request rate is very large. This error would happen when we have concurrent users trying to write or read from same cosmos db collection.

    Following diagram covers the architecture of the performance test I am performing:

    image

    Based on analysis it found out to be the Throttling happening from Azure Cosmos DB, as we make requests that may use more than provisioned Request Units(RU) per second. We were using default Cosmos DB configuration for a fixed collection of 1000 RU’s per second which is sufficient enough for a 500 reads and 100 writes for a 1 kb file. You can refer more about Request Units from Azure Docs.

    image

     

     

     

    Solution(s):

    1. Now first logical step we can do is to get rid off this error by increasing the Throughput for the collection.  I am going to increase to 10000 RU/s maximum allocatable for a Storage Capacity: Fixed.   This should ideally improve the Throughput for 250 or more virtual users hitting.

    image

    2. Second logical step is to improve the code: Improve the connection parameters in the Document DB SDK –> DocumentDbClient. For this I referred to the Microsoft Docs: Performance tips for Azure Cosmos DB and .NET

    Providing optimum values to the following Properties in RetryOption class   to be passed as parameter to Connection Policy.

    image

     

    In my case I provided a value of 30 to give ultimate results:

    new RetryOptions() { MaxRetryAttemptsOnThrottledRequests = 30, MaxRetryWaitTimeInSeconds = 30  }

    That should resolve most of the 429 issues when dealing with Cosmos DB SDK

    Azure Database for MariaDB: Public Preview

    October 4, 2018 Azure Database for MariaDB, Managed Services, MariaDB, OpenSource No comments

    During Ignite 2018, Microsoft has announced the availability of Maria DB support in Azure Database services. Today it has been opened for Public Preview for all Azure customers.

    mariadbhero

    What is MariaDB?

    MariaDB is a community-developed fork of the MySQL relational database management system intended to remain free under the GNU GPL.Development is led by some of the original developers of MySQL, who forked it due to concerns over its acquisition by Oracle Corporation.Wikipedia

    Azure Database for MariaDB: Public Preview Availability

    The Azure Database for MariaDB service is now available in preview. It offers an enterprise-ready, fully managed database service that based on the Community Edition of MariaDB.

    The service features open-source compatibility, built-in high availability, dynamic scaling, and flexible pricing. Customers can lift and shift to the cloud and use languages and frameworks of their choice, leveraging the power of MariaDB running on Azure.

    To learn more about the service, view the service page, pricing, and documentation.

    You can create a MariaDB server by using the Azure portal or Azure CLI.

    More References:

    Azure Cosmos DB–Setting Up New Database using Azure CLI–Sample

    October 1, 2018 Azure, Azure Cosmos DB, Codes, CosmosDB, KnowledgeBase, Microsoft, PowerShell, Windows Azure Development No comments

    Purpose of this article is to help you with few steps of commands to provision a new Azure Cosmos DB database instance through Azure CLI or Azure Cloud Shell.

    Here is the snippet:

     <# 
   This Bash script should help you create a Azure Cosmos DB instance using Azure CLI with bare minimal configuration 
#>

export ACCOUNT_NAME="thingx-retail-store-db"
export DB_RESOURCE_GROUP="thingx-dev"
export DB_LOCATION="southcentralus"
export DB_NAME="Products" 
export DB_THROUGHPUT=1000  ## bare minimal for 500 read and 100 write configuration for a 1KB document.
export DB_COLLECTION_NAME="Groceries"

##Optional: If resource group does not exist create a new one 
az group create --name $DB_RESOURCE_GROUP --location $DB_LOCATION

##1.0 Create the Azure Cosmos DB Account 
az cosmosdb create --name $ACCOUNT_NAME --kind GlobalDocumentDB --resource-group $DB_RESOURCE_GROUP

##2.0 Create Products database in the account  
az cosmosdb database create --name $ACCOUNT_NAME --db-name $DB_NAME --resource-group $DB_RESOURCE_GROUP

##3.0 Create Groceries collection in Products database
az cosmosdb collection create --collection-name $DB_COLLECTION_NAME --partition-key-path "/productId" --throughput $DB_THROUGHPUT --name $ACCOUNT_NAME --db-name $DB_NAME --resource-group $DB_RESOURCE_GROUP

     

    For the ease of this article I used Azure Cloud Shell, that you can launch from your azure portal by clicking on the Shell icon on the top portal menu.

    image

    image

    Now you are ready to execute all the commands listed above in the sample bash script

    image

    Create New Azure Cosmos DB Account

     <span data-mce-type="bookmark" style="display: inline-block; width: 0px; overflow: hidden; line-height: 0;" class="mce_SELRES_start"></span>
az cosmosdb create --name $ACCOUNT_NAME --kind GlobalDocumentDB --resource-group $DB_RESOURCE_GROUP

    image

    Create Products Database in the account

    az cosmosdb database create --name $ACCOUNT_NAME --db-name $DB_NAME --resource-group $DB_RESOURCE_GROUP

    image

    Create Groceries collection in Products database

    az cosmosdb collection create --collection-name $DB_COLLECTION_NAME --partition-key-path "/productId" --throughput $DB_THROUGHPUT --name $ACCOUNT_NAME --db-name $DB_NAME --resource-group $DB_RESOURCE_GROUP

    image

    Now if you browse Azure Portal you can see resources created in “thingx-dev” resource group.

    image

    Upon browsing with Data Explorer you can see the Groceries collection inside Products DB.

    image

    So that’s the few silly easy steps to create Cosmos DB database from Azure CLI or Azure Cloud Shell. Hope that makes it easy for you.