Wednesday, 31 August 2016

Python script to move records from CSV File to a Dynamodb table



Write a python script to move records from a csv file to a dynamo db table. 


Solution

# Script to write csv records into dynamo db table.
# For help on prerequisites and running this script, read this blog.

from __future__ import print_function # Python 2/3 compatibility
from __future__ import division #Python 2/3 compatiblity for integer division
import argparse
import boto3
import csv
import time

# command line arguments
parser = argparse.ArgumentParser(description='Write CSV records to dynamo db table. CSV Header must map to dynamo table field names.')
parser.add_argument('csvFile', help='Path to csv file location')
parser.add_argument('table', help='Dynamo db table name')
parser.add_argument('writeRate', default=5, type=int, nargs='?', help='Number of records to write in table per second (default:5)')
parser.add_argument('delimiter', default='|', nargs='?', help='Delimiter for csv records (default=|)')
parser.add_argument('region', default='us-west-2', nargs='?', help='Dynamo db region name (default=us-west-2')
args = parser.parse_args()
print(args)

# dynamodb and table initialization
endpointUrl = "https://dynamodb." + args.region + ".amazonaws.com"
dynamodb = boto3.resource('dynamodb', region_name=args.region, endpoint_url=endpointUrl)
table = dynamodb.Table(args.table)

# write records to dynamo db
with open(args.csvFile) as csv_file:
    tokens = csv.reader(csv_file, delimiter=args.delimiter)
    # read first line in file which contains dynamo db field names
    header = tokens.next();
    # read second line in file which contains dynamo db field data types
    headerFormat = tokens.next();
    # rest of file contain new records
    for token in tokens:
       item = {}
       for i,val in enumerate(token):
         if val:
           key = header[i]
           if headerFormat[i]=='int':
             val = int(val)
           item[key] = val
       print(item)
       table.put_item(Item = item)

       time.sleep(1/args.writeRate) # to accomodate max write provisioned capacity for table

Preconditions

  • Install boto3 python package on your box
  • Set up AWS credentials
  • Dynamodb field data type is either int or string. New data type will require improvements in code.
  • CSV first two lines are always header and header data type.

Script Arguments

Mandatory
  • csvFile - Path to csv file location
  • table - Dynamo db table name
Optional
  • writeRate - Number of records to write in table per second (default:5)
  • delimiter - Delimiter for csv records (default=|)
  • region - Dynamo db region name (default=us-west-2)

Sample Input

field1|field2|field3|field4|field5
string|string|string|string|int
GZ6MHDMM0IB1IR0P|A27LZ9R92G1SJ|OPEN||1

Sample Run

  • python csvToDynamodb.py abc.csv ABC
  • python csvToDynamodb.py abc.csv ABC 50 | us-east-1

Future improvements

  • Add conversion log for other dynamo db data types. If for concerned table Dynamo db fields have format other than string or int, we will have to convert CSV string value to dynamo db field type required.

Help

  • python csvToDynamodb.py -h

10 comments:

  1. The article provided by you is very nice and it is very helpful to know the more information.keep update with your blogs .I found a article related to you..once you can check it out

    python online training

    ReplyDelete
    Replies
    1. Goibibo _Customer Care Helpline number _
      7063539605
      Any problem call my agent (24*7) hours available.
      Head office_
      Contact hair_9958429949
      Customer care Helpline number _
      Any-problem call
      ___8670530538
      Online problem call my assistant

      ___

      ______
      Customer Care Helpline number _
      7063539605
      Any problem call my agent (24*7) hours available.
      Head office_
      Contact hair_9958429949
      Customer care Helpline number _
      Any-problem call
      ___8670530538
      Online problem call my assistant

      ___

      ______
      Customer Care Helpline number _
      7063539605
      Any problem call my agent (24*7) hours available.
      Head office_
      Contact hair_9958429949
      Customer care Helpline number _
      Any-problem call
      ___8670530538
      Online problem call my assistant

      ___

      ______
      Customer Care Helpline number _
      7063539605
      Any problem call my agent (24*7) hours available.
      Head office_
      Contact hair_9958429949
      Customer care Helpline number _
      Any-problem call
      ___8670530538
      Online problem call my assistant

      ___

      ______
      Customer Care Helpline number _
      7063539605
      Any problem call my agent (24*7) hours available.
      Head office_
      Contact hair_9958429949
      Customer care Helpline number _
      Any-problem call
      ___8670530538
      Online problem call my assistant

      ___

      ______
      Customer Care Helpline number _
      7063539605
      Any problem call my agent (24*7) hours available.
      Head office_
      Contact hair_9958429949
      Customer care Helpline number _
      Any-problem call
      ___8670530538
      Online problem call my assistant

      ___

      ______

      Customer Care Helpline number _
      7063539605
      Any problem call my agent (24*7) hours available.
      Head office_
      Contact hair_9958429949
      Customer care Helpline number _
      Any-problem call
      ___8670530538
      Online problem call my assistant

      ___

      ______
      Customer Care Helpline number _
      7063539605
      Any problem call my agent (24*7) hours available.
      Head office_
      Contact hair_9958429949
      Customer care Helpline number _
      Any-problem call
      ___8670530538
      Online problem call my assistant

      ___

      ______
      Customer Care Helpline number _
      7063539605
      Any problem call my agent (24*7) hours available.
      Head office_
      Contact hair_9958429949
      Customer care Helpline number _
      Any-problem call
      ___8670530538
      Online problem call my assistant

      ___

      ______
      Customer Care Helpline number _
      7063539605
      Any problem call my agent (24*7) hours available.
      Head office_
      Contact hair_9958429949
      Customer care Helpline number _
      Any-problem call
      ___8670530538
      Online problem call my assistant

      ___

      ______
      Customer Care Helpline number _
      7063539605
      Any problem call my agent (24*7) hours available.
      Head office_
      Contact hair_9958429949
      Customer care Helpline number _
      Any-problem call
      ___8670530538
      Online problem call my assistant

      ___

      ______
      Customer Care Helpline number _
      7063539605
      Any problem call my agent (24*7) hours available.
      Head office_
      Contact hair_9958429949
      Customer care Helpline number _
      Any-problem call
      ___8670530538
      Online problem call my assistant

      ___

      ______

      Delete
    2. Goibibo _Customer Care Helpline number _
      7063539605
      Any problem call my agent (24*7) hours available.
      Head office_
      Contact hair_9958429949
      Customer care Helpline number _
      Any-problem call
      ___8670530538
      Online problem call my assistant

      ___

      ______
      Customer Care Helpline number _
      7063539605
      Any problem call my agent (24*7) hours available.
      Head office_
      Contact hair_9958429949
      Customer care Helpline number _
      Any-problem call
      ___8670530538
      Online problem call my assistant

      ___

      ______
      Customer Care Helpline number _
      7063539605
      Any problem call my agent (24*7) hours available.
      Head office_
      Contact hair_9958429949
      Customer care Helpline number _
      Any-problem call
      ___8670530538
      Online problem call my assistant

      ___

      ______
      Customer Care Helpline number _
      7063539605
      Any problem call my agent (24*7) hours available.
      Head office_
      Contact hair_9958429949
      Customer care Helpline number _
      Any-problem call
      ___8670530538
      Online problem call my assistant

      ___

      ______
      Customer Care Helpline number _
      7063539605
      Any problem call my agent (24*7) hours available.
      Head office_
      Contact hair_9958429949
      Customer care Helpline number _
      Any-problem call
      ___8670530538
      Online problem call my assistant

      ___

      ______
      Customer Care Helpline number _
      7063539605
      Any problem call my agent (24*7) hours available.
      Head office_
      Contact hair_9958429949
      Customer care Helpline number _
      Any-problem call
      ___8670530538
      Online problem call my assistant

      ___

      ______

      Customer Care Helpline number _
      7063539605
      Any problem call my agent (24*7) hours available.
      Head office_
      Contact hair_9958429949
      Customer care Helpline number _
      Any-problem call
      ___8670530538
      Online problem call my assistant

      ___

      ______
      Customer Care Helpline number _
      7063539605
      Any problem call my agent (24*7) hours available.
      Head office_
      Contact hair_9958429949
      Customer care Helpline number _
      Any-problem call
      ___8670530538
      Online problem call my assistant

      ___

      ______
      Customer Care Helpline number _
      7063539605
      Any problem call my agent (24*7) hours available.
      Head office_
      Contact hair_9958429949
      Customer care Helpline number _
      Any-problem call
      ___8670530538
      Online problem call my assistant

      ___

      ______
      Customer Care Helpline number _
      7063539605
      Any problem call my agent (24*7) hours available.
      Head office_
      Contact hair_9958429949
      Customer care Helpline number _
      Any-problem call
      ___8670530538
      Online problem call my assistant

      ___

      ______
      Customer Care Helpline number _
      7063539605
      Any problem call my agent (24*7) hours available.
      Head office_
      Contact hair_9958429949
      Customer care Helpline number _
      Any-problem call
      ___8670530538
      Online problem call my assistant

      ___

      ______
      Customer Care Helpline number _
      7063539605
      Any problem call my agent (24*7) hours available.
      Head office_
      Contact hair_9958429949
      Customer care Helpline number _
      Any-problem call
      ___8670530538
      Online problem call my assistant

      ___

      ______

      Delete

  2. That is very interesting; you are a very skilled blogger. I have shared your website in my social networks! A very nice guide. I will definitely follow these tips. Thank you for sharing such detailed article.

    python online course

    ReplyDelete
  3. I got error

    us-east-1: command not found
    Traceback (most recent call last):
    File "test.py", line 27, in
    header = tokens.next();
    AttributeError: '_csv.reader' object has no attribute 'next'
    Exception ignored in: <_io.TextIOWrapper name='' mode='w' encoding='UTF-8'>
    BrokenPipeError: [Errno 32] Broken pipe

    ReplyDelete
    Replies
    1. for new version of python: change to next(tokens);

      Delete
  4. Really good information to show through this blog. I really appreciate you for all the valuable information that you are providing us through your blog. python Online course

    ReplyDelete
  5. It is really a great work and the way in which you are sharing the knowledge is excellent.
    aws training in omr | aws training in velachery | best aws training center in chennai

    ReplyDelete