Category: AWS

whooh. Did my AWS Certified Database – Specialty Certificate on 10/15/2020. That’s quite an exam!!!

Build and Deploy .Net Core WebAPI Container to Amazon EKS using CDK & cdk8s

Sep 4 2020, wrote a blog where we leveraged the development capabilities of the CDK for Kubernetes framework also known as cdk8s along with the AWS Cloud Development Kit (AWS CDK) framework to provision infrastructure through AWS CloudFormation.

cdk8s allows us to define Kubernetes apps and components using familiar languages. cdk8s is an open-source software development framework for defining Kubernetes applications and reusable abstractions using familiar programming languages and rich object-oriented APIs. cdk8s apps synthesize into standard Kubernetes manifests which can be applied to any Kubernetes cluster. cdk8s lets you define applications using Typescript, JavaScript, and Python. In this blog we will use Python.

The AWS CDK is an open source software development framework to model and provision your cloud application resources using familiar programming languages, including TypeScript, JavaScript, Python, C# and Java.

To read more: https://aws.amazon.com/blogs/developer/build-and-deploy-net-core-webapi-container-to-amazon-eks-using-cdk-cdk8s/

AWS offers many purpose built Databases. Note this is an immense wide topic. Thought of writing this for my own refresher.

AWS RDS – Relational Database:
– data is actually relation, ACID (atomic, consistent, integrity, durable) compliant
– referential integrity
– static and unchanging
– ubiquitous – easy and available in many flavors. can handle different types of workloads
– RDS read replicas for 6/6 DB engines available
ex: shopper to read history. query from read db instead of main db

Anti pattern: lot of loads can run but are not good for JSon object or if there is no well defined schema

DynamoDB – NO SQL
– fully managed, multi region
– multi primary, no SQL
– built in security backup and restore
low latency key baseeed query. fast performance handle throughput maintaining consistency
– ex: product desc

for Hot data -  
    DAX Caching - DAX - items to cache in a matter of microseconds 
    reduces response times of eventually consistent read workloads from single digit milliseconds to microseconds
    devs need not modify their app logic

    Elasticache for redis and ElasticCache for MemChached

Amazon Redshift – Dataware House
for cold data – for analytics, columnar database services help
columnar storage tech in order to improve I/O efficiency and parallelized queries across multiple nodes for fast query performance
Analytics, trend monitoring and gaming insights to know what’s ging on in the system
ex: daily report,allow pulling from datalake
use standard SQL.

- AQUA - Advanced query accelerated cache. that does substantial share of data processsing in place on the cache,
  enabling Redshift to run upto 10x faster

Other options/flavors are
financial requirements, doc storage reqruiement or mobile game specific community

Below are some more purpose built database services options available in AWS

Amazon QLDB – ledger database transparent cryptographically verifiable transaction log. Immutable
when you need to keep track of financial activity in an organization

Amazon Document DB for MongoDB for storage of JSON – fault tolerant, self healing storage that supports automatic data scaling
allows customer to scale from 10GB all the way upto 64TB per database cluster

Amazon Neptune – Graph – database option
Propert Graph and W3C’s RDF along with respective query
ex: Games – connectivity between each player
Supports Apache TinkerPop Gremlin and SPARQL

Amazon Keyspaces – AWS Managed Cassandra compatible service
if customer needs wide column key store –
automatically supports three replicas that can be distributed across different AZs

Amazon Timestream – Timeseries database. need to analyze billions/trillions of data
ex: user activity log. ex: trivago
automates rollups, retention, tiering and compression of data
ex: from device sensors constantly

---

Data patterns

-read heavy
    lot of incoming data. more query processing on that. there is a read contention/querying due to lot of records
    querying over and over again
    Option - materialized view is a database object 

-micro service
    avoid sharing data between micro services

- break monolith into small domains which could be microservice or a collection of microservices

- isolate bounded context - part of domain driven design - back your data based on functionality

- Event sourcing and CQRS (command query responsibilty seggregation)
    changig the parts of the app to define which is responsible of reading vs writing

- SAGA Pattern - sequence of events that happen in a Distributed transactions
    orchestrate/choreograph using orchestrators like step functions to coordinate activities to perform/trigger a transactional boundary to commit/rollback across distributed design
    only works when we can embrace eventual consistency

---

AWS Database Migrations

Consider a source (on prem) –> RDS / EC2 migration.

Types of migration

• like to like (homogenous) – moving between database instances using the same db tech
• like to unlike (heterogenous) – migratnig to a completely different database technology

Options available for migration

• DMS – Database management service
• third party
• partner
• backup and restore

for like to like – DMS uses replication instance and keeps the source available/online. DB engines are compatible

for like to unlike pattern – SCT (Schema conversion tool) + DMS can help
if the datastructure are not same they need to be converted to a compatible type
ex: once-off migration, source to target, consolidate databases, dev and test

Snowball edge:
if the bandwidth, infrastructure is not available, snowball edge can be used to copy the data into S3 and can be migrated

Howdy, This is my first wordpress blog on my aws instance