R14 – Memory Quota Exceeded

We, like many other organisations, are using heroku as the deployment server for our project open event organizer server. Things are pretty simple and awesome when your project is in its beginning phase and things run pretty smoothly. But as your project grows, there comes some server problem. And one of the biggest problems as your project grows is memory. Now since various packages have a different amount of memory assigned to you in case of hosting in generic servers such as heroku, so it might result in memory quota exceeded. Recently, we faced such a problem. R14 – Memory Quota Exceeded. Took us quite some time to understand what and why and how this occurred. So let me share a few things I found about this error.

Continue reading “R14 – Memory Quota Exceeded”

Python code examples

I’ve met many weird examples of  behaviour in python language while working on Open Event project. Today I’d like to share some examples with you. I think this knowledge is necessary, if you’d like to increase a  bit your knowledge in python area.

Simple adding one element to python list:

def foo(value, x=[]):
  x.append(value)
  return x

>>> print(foo(1))
>>> print(foo(2))
>>> print(foo(3, []))
>>> print(foo(4))

OUTPUT

[1] 
[1, 2] 
[3]
[1, 2, 4]

First output is obvious, but second not exactly. Let me explain it, It happens because x(empty list) argument is only evaluated once, So on every call foo(), we modify that list, appending a value to it. Finally we have [1,2, 4] output. I recommend to avoid mutable params as default.

Another example:

Do you know which type it is?

>>> print(type([ el for el in range(10)]))
>>> print(type({ el for el in range(10)}))
>>> print(type(( el for el in range(10))))

Again first and second type are obvious <class ‘list’>, <class ‘set’>. You may  think that last one should return type tuple but it returns a generator <class ‘generator’>.

Example:

Do you think that below code returns an exception?

list= [1,2,3,4,5]
>>> print(list [8:])

If you think that above expression returns index error you’re wrong. It returns empty list [].

Example funny boolean operators

>>> 'c' == ('c' or 'b')
True
>>> 'd' == ('a' or 'd')
False
>>> 'c' == ('c' and 'b')
False 
>>> 'd' == ('a' and 'd')
True

You can think that that OR and AND operators are broken.

You have to know how python interpreter behaves while looking for OR and AND operators.

So OR Expression takes the first statement and checks if it is true. If the first statement is true, then Python returns object’s value without checking second value. If first statement is false interpreter checks second value and returns that value.

AND operator checks if first statement is false, the whole statement has to be false. So it returns first value, but if first statement is true it checks second statement and returns second value.

Below i will show you how it works

>>> 'c' == ('c' or 'b')
>>> 'c' == 'c'
True
>>> 'd' == ('a' or 'd')
>>> 'd' == 'a'
False
>>> 'c' == ('c' and 'b')
>>> 'c' == 'b'
False 
>>> 'd' == ('a' and 'd')
>>> 'd' == 'd'
True

I hope that i have explained you how the python interpreter checks OR and AND operators. So know above examples should be more understandable.

Using ftp-deploy in node.js to publish websites over FTP

In the Open Event Webapp Generator, we recently added the functionality for organisers to submit their ftp credentials and when the website is generated, it’ll automatically upload the website to the chosen ftp server (allowing creation of subdirectory internally, if the organiser so wants).

To achieve we used the very useful nodejs module ftp-deploy which is a wrapper on the popular jsftp library

The code dealing with ftp deployment in our webapp generator can be found here  –

https://github.com/fossasia/open-event-webapp/blob/development/src/backend/ftpdeploy.js

As can be seen, deploying using ftp-deploy is pretty straightforward. Primarily we need a config object

 


  var config = {
    username: ftpDetails.user, //prompted on commandline if not given
    password: ftpDetails.pass, // optional, prompted if none given
    host: ftpDetails.host,
    port: 21,
    localRoot: path.join(__dirname, '/../../dist', appFolder), //local folder containing website
    remoteRoot: ftpDetails.path, //path on ftp server to host website
    exclude: ['.git', '.idea', 'tmp/*'],
    continueOnError: true
};

You can set up some event listeners for events like uploaded uploading and upload-error

Small Docker images using Alpine Linux

Everyone likes optimization, small file sizes and such.. Won’t it be great if you are able to reduce your Docker image sizes by a factor of 2 or more. Say hello to Alpine Linux. It is a minimal Linux distro weighing just 5 MBs. It also has basic linux tools and a nice package manager APK. APK is quite stable and has a considerable amount of packages.

apk add python gcc

In this post, my main motto is how to squeeze the best out of AlpineLinux to create the smallest possible Docker image. So let’s start.

Step 1: Use AlpineLinux based images

Ok, I know that’s obvious but just for the sake of completeness of this article, I will state that prefer using Alpine based images wherever possible. Python and Redis have their official Alpine based images whereas NodeJS has good unoffical Alpine-based images. Same goes for Postgres, Ruby and other popular environments.

Step 2: Install only needed dependencies

Prefer installing select dependencies over installing a package that contains lots of them. For example, prefer installing gcc and development libraries over buildpacks. You can find listing of Alpine packages on their website.

Pro Tip – A great list of Debian v/s Alpine development packages is at alpine-buildpack-deps Docker Hub page (scroll down to Packages). It is a very complete list and you will always find the dependency you are looking for.

Step 3: Delete build dependencies after use

Build dependencies are required by components/libraries to build native extensions for the platform. Once the build is done, they are not needed. So you should delete the build-dependencies after their job is complete. Have a look at the following snippet.

RUN apk add --virtual build-dependencies gcc python-dev linux-headers musl-dev postgresql-dev \
    && pip install -r requirements.txt \
    && apk del build-dependencies

I am using --virtual to give a label to the pacakages installed on that instance and then when pip install is done, I am deleting them.

Step 4: Remove cache

Cache can take up lots of un-needed space. So always run apk add with --no-cache parameter.

RUN apk add --no-cache package1 package2

If you are using npm for manaing project dependencies and bower for managing frontend dependencies, it is recommended to clear their cache too.

RUN npm cache clean && bower cache clean

Step 5: Learn from the experts

Each and every image on Docker Hub is open source, meaning that it’s Dockerfile is freely available. Since the official images are made as efficient as possible, it’s easy to find great tricks on how to achieve optimum performance and compact size in them. So when viewing an image on DockerHub, don’t forget to peek into its Dockerfile, it helps more than you can imagine.

Conclusion

That’s all I have for now. I will keep you updated on new tips if I find any. In my personal experience, I found AlpineLinux to be worth using. I tried deploying Open Event Server on Alpine but faced some issues so ended up creating a Dockerfile using debain:jessie. But for small projects, I would recommend Alpine. On large and complex projects however, you may face issues with Alpine at times. That maybe due to lack of packages, lack of library support or some other thing. But it’s not impossible to overcome those issues so if you try hard enough, you can get your app running on Alpine.

 

{{ Repost from my personal blog http://aviaryan.in/blog/gsoc/docker-using-alpine.html }}

Bottoms sheets in android

Hey Guys I recently used Bottom sheets, so that I should write a blog about it because I don’t see a lot of developers using this in their app UI’s.

It’s a very interesting UI element. A Bottom Sheet is a sheet of material that slides up from the bottom edge of the screen. Displayed only as a result of a user-initiated action, and can be swiped up to reveal additional content. It can be a temporary modal surface or a persistent structural element of an app.

This component was introduced in the Android Design Support library 23.2. Many apps like Google Maps use the bottom sheet, in which a sliding window pops up from the bottom of the screen. Also the Google play music app uses. When we drag up the sheet we see the song details as well as the current playlist.

Usage of expanded and collapsed Bottom sheets in Android

There are 3 states of Bottom sheets :-

  • Expanded — When the sheet is completely visible.
  • Collapsed — When the sheet is partially visible.
  • Hidden — When the sheet is completely hidden.

Step 1 is we need to import the latest design support library. Put this line in your app level build.gradle file.

compile ‘com.android.support:design:23.2.0’

Then one should create a new Blank Activity (not Empty Activity) in Android Studio. It sets up the CoordinatorLayout by default.

So now there ate two layouts created by default namely activity_main.xml and content_main.xml.

<?xml version="1.0" encoding="utf-8"?>
<RelativeLayout xmlns:android="http://schemas.android.com/apk/res/android"
xmlns:app="http://schemas.android.com/apk/res-auto"
xmlns:tools="http://schemas.android.com/tools"
android:layout_width="match_parent"
android:layout_height="match_parent"
app:layout_behavior="@string/appbar_scrolling_view_behavior"
tools:context="yet.best.bottomsheets.MainActivity"
tools:showIn="@layout/activity_main">

<Button
android:id="@+id/open"
android:layout_width="wrap_content"
android:layout_height="wrap_content"
android:layout_centerHorizontal="true"
android:layout_margin="5dp"
android:text="Open Bottom Sheet" />

<Button
android:id="@+id/collapse"
android:layout_width="wrap_content"
android:layout_height="wrap_content"
android:layout_below="@+id/open"
android:layout_centerHorizontal="true"
android:layout_margin="5dp"
android:text="Collapse Bottom Sheet" />

<Button
android:id="@+id/hide"
android:layout_width="wrap_content"
android:layout_height="wrap_content"
android:layout_below="@+id/collapse"
android:layout_centerHorizontal="true"
android:layout_margin="5dp"
android:text="Hide Bottom Sheet" />

</RelativeLayout>

Notice that 3 Buttons have been created in this layout to perform different actions with the bottom sheets.

activity_main.xml

<?xml version="1.0" encoding="utf-8"?>

<android.support.design.widget.CoordinatorLayout xmlns:android="http://schemas.android.com/apk/res/android"
xmlns:app="http://schemas.android.com/apk/res-auto"
xmlns:tools="http://schemas.android.com/tools"
android:layout_width="match_parent"
android:layout_height="match_parent"
android:fitsSystemWindows="true"
tools:context="yet.best.bottomsheets.MainActivity">

<android.support.design.widget.AppBarLayout
android:layout_width="match_parent"
android:layout_height="wrap_content"
android:theme="@style/AppTheme.AppBarOverlay">

<android.support.v7.widget.Toolbar
android:id="@+id/toolbar"
android:layout_width="match_parent"
android:layout_height="?attr/actionBarSize"
android:background="?attr/colorPrimary"
app:popupTheme="@style/AppTheme.PopupOverlay" />

</android.support.design.widget.AppBarLayout>

<include layout="@layout/content_main" />

<include
android:id="@+id/bottom_sheet"
layout="@layout/bottomsheet_main" />

<android.support.design.widget.FloatingActionButton
android:id="@+id/fab"
android:layout_width="wrap_content"
android:layout_height="wrap_content"
android:layout_margin="@dimen/fab_margin"
android:src="@android:drawable/ic_dialog_email"
app:layout_anchor="@id/bottom_sheet"
app:layout_anchorGravity="top|right|end" />

</android.support.design.widget.CoordinatorLayout>

Those who aren’t familiar with the coordinator layout — basically there is a base level layout activity_main which contains the FloatingButton and within this layout including content_main.xml which will contain the rest of the layout. Notice that one also has to include bottomsheet_main.xml. This layout contains our bottom sheet layout which will be created next.

Create a new layout file called bottomsheet_main.xml

bottomsheet_main.xml

<?xml version="1.0" encoding="utf-8"?>

<RelativeLayout xmlns:android="http://schemas.android.com/apk/res/android"
xmlns:app="http://schemas.android.com/apk/res-auto"
android:layout_width="match_parent"
android:layout_height="300dp"
android:background="#d3d3d3"
app:behavior_hideable="true"
app:behavior_peekHeight="70dp"
app:layout_behavior="@string/bottom_sheet_behavior">

<TextView
android:id="@+id/heading"
android:layout_width="wrap_content"
android:layout_height="wrap_content"
android:layout_margin="7dp"
android:text="Collapsed"
android:textSize="18sp" />

<TextView
android:layout_width="wrap_content"
android:layout_height="wrap_content"
android:layout_centerInParent="true"
android:text="Main Content"
android:textSize="20sp" />

</RelativeLayout>

This layout is how our bottom sheet will actually look. You can design it as you want.

Now for the actual java code. This is the easiest part actually. Just set listeners to the 3 buttons created and perform the corresponding action with the bottom sheet.

public class MainActivity extends AppCompatActivity {

BottomSheetBehavior bottomSheetBehavior;
Button open, collapse, hide;
TextView heading;
@Override
 protected void onCreate(Bundle savedInstanceState) {
  super.onCreate(savedInstanceState);
  setContentView(R.layout.activity_main);
  Toolbar toolbar = (Toolbar) findViewById(R.id.toolbar);
  setSupportActionBar(toolbar);

View bottomSheet = findViewById(R.id.bottom_sheet);
  bottomSheetBehavior = BottomSheetBehavior.from(bottomSheet);

setup();
 }

private void setup() {
  open = (Button) findViewById(R.id.open);
  collapse = (Button) findViewById(R.id.collapse);
  hide = (Button) findViewById(R.id.hide);
  heading = (TextView) findViewById(R.id.heading);

//Handling movement of bottom sheets from buttons
  open.setOnClickListener(new View.OnClickListener() {
   @Override
   public void onClick(View view) {
    bottomSheetBehavior.setState(BottomSheetBehavior.STATE_EXPANDED);
    heading.setText("Welcome");
    heading.setTextColor(ContextCompat.getColor(MainActivity.this, R.color.colorPrimary));
   }
  });

collapse.setOnClickListener(new View.OnClickListener() {
   @Override
   public void onClick(View view) {
    bottomSheetBehavior.setState(BottomSheetBehavior.STATE_COLLAPSED);
    heading.setText("Collapsed");
    heading.setTextColor(ContextCompat.getColor(MainActivity.this, R.color.colorAccent));
   }
  });

hide.setOnClickListener(new View.OnClickListener() {
   @Override
   public void onClick(View view) {
    bottomSheetBehavior.setState(BottomSheetBehavior.STATE_HIDDEN);
   }
  });

//Handling movement of bottom sheets from sliding
  bottomSheetBehavior.setBottomSheetCallback(new BottomSheetBehavior.BottomSheetCallback() {
   @Override
   public void onStateChanged(View bottomSheet, int newState) {
    if (newState == BottomSheetBehavior.STATE_COLLAPSED) {
     heading.setText("Collapsed");
     heading.setTextColor(ContextCompat.getColor(MainActivity.this, R.color.colorAccent));
    } else if (newState == BottomSheetBehavior.STATE_EXPANDED) {
     heading.setText("Welcome");
     heading.setTextColor(ContextCompat.getColor(MainActivity.this, R.color.colorPrimary));
    }
   }

@Override
   public void onSlide(View bottomSheet, float slideOffset) {}
  });
 }
}

We just use the bottomSheetBehavior.setState() method to set the relevant state on each button click.

The bottom sheets can also be dragged by touch gestures. One can simply touch the sheet and drag them up or down. For these touch gestures to be handled one has to implement the onStateChanged() listener. This listener is fired everytime the state of the sheet changes by gestures. Whenever this triggers it checks the state of the bottom sheet and again do the desired action which user would have done by the button clicks.

As you can see, this is a pretty neat UI solution and can be implemented so easily. Go try it out for yourself. Adios!

Implementing Fee Structure for Ticketing

After implementing the ticketing system in Open Event, it was necessary to implement the control of fees for each type of currency. Thus an Admin page for controlling the percentage of fees and the maximum allowed fee for each type of currency was made:

1

Here initially on loading the system for the first time the service fees and maximum fees are 0. The Super Admin then sets the values. However if for some reason the maximum fee is still left blank then it becomes equal to the service fees.

2

The backend code for this is as follows:

if 'service_fee' in kwargs:
    ticket_service_fees = kwargs.get('service_fee')
    ticket_maximum_fees = kwargs.get('maximum_fee')
    from app.helpers.data_getter import DataGetter
    from app.helpers.data import save_to_db
    currencies = DataGetter.get_payment_currencies()
    ticket_fees = DataGetter.get_fee_settings()
    if not ticket_fees:
        for i, (currency, has_paypal, has_stripe) in enumerate(currencies):
            currency = currency.split(' ')[0]
            if float(ticket_maximum_fees[i]) == 0.0:
                ticket_maximum_fees[i] = ticket_service_fees[i]
            ticket_fee = TicketFees(currency=currency,
                                    service_fee=ticket_service_fees[i],
                                    maximum_fee=ticket_maximum_fees[i])
            save_to_db(ticket_fee, "Ticket Fees settings saved")
    else:
        i = 0
        for fee in ticket_fees:
            if float(ticket_maximum_fees[i]) == 0.0:
                ticket_maximum_fees[i] = ticket_service_fees[i]
            fee.service_fee = ticket_service_fees[i]
            fee.maximum_fee = ticket_maximum_fees[i]
            save_to_db(fee, "Fee Options Updated")
            i += 1

So it checks if the ‘service fees’ is there in the request. If Yes then it stores all the service fees in one list and all the corresponding maximum fees in another list. Then it checks if the fee settings are already stored in the database using the

ticket_fees = DataGetter.get_fee_settings()

If the organizer is setting the fees for the first time then it will return None. In that case new settings are created. If fees are already created then the settings are just modified. And in both cases if the maximum fee is 0 then the maximum fee will be set to service fee.

Thus the fee system is implemented for the tickets.

Resizing Uploaded Image (Python)

While we make websites were we need to upload images such as in event organizing server, the image for the event needs to be shown in various different sizes in different pages. But an image with high resolution might be an overkill for using at a place where we just need it to be shown as a thumbnail. So what most CMS websites do is re-size the image uploaded and store a smaller image as thumbnail. So how do we do that? Let’s find out.

Continue reading “Resizing Uploaded Image (Python)”

Accepting Stripe payments on behalf of a third-party

{ Repost from my personal blog @ https://blog.codezero.xyz/accepting-stripe-payments-on-behalf-of-a-third-party }

In Open Event, we allow the organizer of each event to link their Stripe account, so that all ticket payments go directly into their account. To make it simpler for the organizer to setup the link, we have a Connect with stripe button on the event creation form.

Clicking on the button, the organizer is greeted with a signup flow similar to Login with Facebook or any other social login. Through this process, we’re able to securely and easily obtain the credentials required to accept payments on behalf of the organizer.

For this very purpose, stripe provides us with an OAuth interface called as Stripe Connect. Stripe Connect allows us to connect and interact with other stripe accounts through an API.

We’ll be using Python’s requests library for making all the HTTP Requests to the API.
You will be needing a stripe account for this.

Registering your platform
The OAuth Flow

The OAuth flow is similar to most platforms.

  • The user is redirected to an authorization page where they login to their stripe account and authorize your app to access their account
  • The user is then redirected back to a callback URL with an Authorization code
  • The server makes a request to the Token API with the Authorization code to retrieve the access_token, refresh_token and other credentials.

Implementing the flow

Redirect the user to the Authorization URL.
https://connect.stripe.com/oauth/authorize?response_type=code&client_id=ca_8x1ebxrl8eOwOSqRTVLUJkWtcfP92YJE&scope=read_write&redirect_uri=http://localhost/stripe/callback  

The authorization url accepts the following parameters.

  1. client_id – The client ID acquired when registering your platform.required.
  2. response_type – Response type. The value is always code. required.
  3. redirect_uri – The URL to redirect the customer to after authorization.
  4. scope – Can be read_write or read_only. The default is read_only. For analytics purposes, read_only is appropriate; To perform charges on behalf of the connected user, We will need to request read_write scope instead.

The user will be taken to stripe authorization page, where the user can login to an existing account or create a new account without breaking the flow. Once the user has authorized the application, he/she is taken back to the Callback URL with the result.

Requesting the access token with the authorization code

The user is redirected back to the callback URL.

If the authorization failed, the callback URL has a query string parameter error with the error name and a parameter error_description with the description of the error.

If the authorization was a success, the callback URL has the authorization code in the code query string parameter.

import requests

data = {  
    'client_secret': 'CLIENT_SECRET',
    'code': 'AUTHORIZATION_CODE',
    'grant_type': 'authorization_code'
}

response = requests.post('https://connect.stripe.com/oauth/token', data=data)

The client_secret is also obtained when registering your platform. The codeparameter is the authorization code.

On making this request, a json response will be returned.

If the request was a success, the following response will be obtained.

{
  "token_type": "bearer",
  "stripe_publishable_key": PUBLISHABLE_KEY,
  "scope": "read_write",
  "livemode": false,
  "stripe_user_id": USER_ID,
  "refresh_token": REFRESH_TOKEN,
  "access_token": ACCESS_TOKEN
}

If the request failed for some reason, an error will be returned.

{
  "error": "invalid_grant",
  "error_description": "Authorization code does not exist: AUTHORIZATION_CODE"
}

The access_token token obtained can be used as the secret key to accept payments like discussed in Integrating Stripe in the Flask web framework.

Twitter Oauth

Oauth_logo.svg.png

What is Oauth?

It’s an open protocol which allows to secure an authorization in a simple and standard method from web, mobile and desktop applications.Facebook, Google Twitter, Github and more web services use this protocol to authenticate user. Using Oauth is very convenient, because it delegates user authentication to the service which host user account. It allows us to get resources from another web service without giving any login or password. If you have a service and want to prepare a authentication via Twitter, the best solution is to use OAuth. Recently Open Event team met a problem in an user profile page. We’d like to automatically fill information about user. Of course, to solve it we use Oauth protocol, to authenticate with Twitter After a three-steps authentication we can get name and profile picture.If you need another information from Twitter profile like recent tweets or followers’ list. You have to visit Twitter API site to see more samples of resource which you can get

How do Open event team implement communication between Orga-server and Twitter?

All services have a very similar flow. Below i will show you how it looks in our case.

Before starting you need to create your own twitter app. You can create app in Twitter apps site https://apps.twitter.com/. If  create an app you will see a CONSUMER KEY and CONSUMER SECRET KEY which shouldn’t be human-readable, so remember not to share these keys.

Below example shows how to get basic information about twitter profile

We use oauth2 python library https://github.com/joestump/python-oauth2

consumer = oauth2.Consumer(key=TwitterOAuth.get_client_id(),

                          secret=TwitterOAuth.get_client_secret())

client = oauth2.Client(consumer)

TwitterOAuth.get_client_id() CONSUMER KEY

TwitterOAuth.get_client_secret()  – CONSUMER SECRET KEY

Then we send GET request to request_token endpoint to get oauth_token

client.request('https://api.twitter.com/oauth/request_token', "GET")
Response: oauth_token=Z6eEdO8MOmk394WozF5oKyuAv855l4Mlqo7hhlSLik&
oauth_token_secret=Kd75W4OQfb2oJTV0vzGzeXftVAwgMnEK9MumzYcM&
oauth_callback_confirmed=true

Next step is to redirect user to Twitter Authentication site

twitter-oauth.png

You can see in an url a redirect_uri. So after sign in Client will get a callback from Twitter with oauth_verifier and oauth_token params

https://api.twitter.com/oauth/authenticate?oauth_token=RcYGqAAAAAAAwdbbAAABVoM1UMo&oauth_token_secret=wZfpPpCugAmdF3AuohEnvBTRxdCmllxu&oauth_callback_confirmed=true&redirect_uri=http://open-event-dev.herokuapp.com/tCallback

The last step is to get an access token. If we have an oauth_verifier and an oauth_token it’s pretty easy

def get_access_token(self, oauth_verifier, oauth_token):

   consumer = self.get_consumer()

   client = oauth2.Client(consumer)

   return client.request(

       self.TW_ACCESS_TOKEN_URI + 'oauth_verifier=' + oauth_verifier + 
       "&oauth_token=" + oauth_token, "POST")

Where TW_ACCESS_TOKEN_URI is https://api.twitter.com/oauth/access_token

Final step is to get our user information

resp, content = client.request("https://api.twitter.com/1.1/users/show.json?
                               screen_name=" + access_token["screen_name"] +
                               "&user_id=" + access_token["user_id"] , "GET")

user_info = json.loads(content)

In an user_info variable you can get a profile picture or a profile name.

Summarizing, oauth protocol is very secure and easy to use by developer. At the beginning an oauth flow can seem to be a little hard to  understand but if you spend some time trying tp understand it, everything becomes easier.  And it’s secured. because you don’t need to store a login or a password, and an access token has an expired time. This is the main feature of Oauth protocol.

Handling data in android

So this week I was working with getting some data from the sqlite database in android and someone who was a beginner in android also asked me to help him with the same. I asked him what he knew and he said that even after reading up a lot he wasn’t able to figure out what exactly to do with the data he wants to save and use in his app. I have seen that this is a problem with a lot of people starting to develop android apps. So, I decided to write a blog on how can you handle your data in your android apps.

Most of the android apps need to save data even if only to save some user preferences. So primarily there are 3 ways to save your data :

  1. In form of key values (SharedPreferences)
  2. Reading/Writing to files
  3. Writing to a database

So let’s go step by step. When we need to store just some preferences of the users like if they want notifications or what kind of theme they want : light or dark etc. So basically if we want to store a key value in the persitant storage of the app we can do that using SharedPreferences. To use sharedpreferences, we initialise the sharedpreference object like

SharedPreferences sharedPreferences = PreferenceManager.getDefaultSharedPreferences(this);

in oncreate and cache it. Then we just need to add or retrieve what we want using this cached SharedPreferences object. To Add a key value pair :

sharedPreferences.edit().putString("someKey", "someValue").apply();

Also you can put all kinds of stuff here. For example right now we added a string with key “someKey” and Value “someValue”. We can also add Booleans, Floats, Ints, Longs, StringSets etc.

To retrieve the same value we do something like this

sharedPreferences.getString("someKey", "DefaultValue");

Now if you want some logs or some values that can be exported and sent to your server, you should write them to files and maybe read some json inputs etc. as well.

Basically android has a File system similar to other platforms. All android devices have two file storage areas : “Internal” and “external” storage. The difference between them is as follows :

Internal :

  • Always available
  • Files saved here are accesible by only your app
  • When user uninstalls the app, system removes all your app’s files from internal storage

Best to use this when you want to be sure that neither the user nor the other app’s can access your files

External :

  • It’s not always available because user can mount external storage as USB storage and remove it as well
  • It’s readable by anything(Other apps, users etc.)
  • When you uninstall, system removes your app’s files from here only if you save them in the directory from getExternalFilesDir()

Now to read and write files, you need extra permissions

  • android.permission.WRITE_EXTERNAL_STORAGE
  • android.permission.READ_EXTERNAL_STORAGE

So now let’s get down to it. How do I save and read files in my app?

You first initialise the File object

File file = new File(context.getFilesDir(), filename);

This will create a file with filename in the internal storage. For external storage

first check if the storage is available, then just create a file using getExternalStoragePublicDirectory

File file = new File(Environment.getExternalStoragePublicDirectory(
            Environment.DIRECTORY_PICTURES), albumName);
    if (!file.mkdirs()) {
        Log.e(LOG_TAG, "Directory not created");
    }
    return file;

This is for writing public files.

Now we move onto the most used part in an android app which is a database.Android has a built in SQLite database package which helps us in writing databases in files with syntax similar to SQL.

You need to add 2 classes which are mandatory and another class which basically helps you get organised. So the first is a Contract. This is where you actually write statements that will be executed later on to initialise or create the tables we want. For this make an a static abstract inner class that implements BaseColums like

public static abstract class Microlocation implements BaseColumns {
    public static final String TABLE_NAME = "microlocation";

    public static final String ID = "id";

    public static final String NAME = "name";

    public static final String LATITUDE = "latitude";

    public static final String LONGITUDE = "longitude";

    public static final String FLOOR = "floor";

    public static final String[] FULL_PROJECTION = {
            ID,
            NAME,
            LATITUDE,
            LONGITUDE,
            FLOOR

    };

    public static final String CREATE_TABLE =
            "CREATE TABLE " + TABLE_NAME
                    + " ("
                    + ID + INT_TYPE + PRIMARY_KEY + COMMA_SEP
                    + NAME + TEXT_TYPE + COMMA_SEP
                    + LATITUDE + REAL_TYPE + COMMA_SEP
                    + LONGITUDE + REAL_TYPE + COMMA_SEP
                    + FLOOR + INT_TYPE
                    + " );";

    public static final String DELETE_TABLE = "DROP TABLE IF EXISTS " + TABLE_NAME;


}

Here we are making static final Strings for column names and then creating a static final String CREATE_TABLE which is basically a statement that creates the table Microlocation with the specified key, columns, data types etc.

After adding this structure for all the tables we want to have in our database, we move on to adding a DbHelper class that extends SQLiteOpenHelper which basically has two Abstract methods called onCreate(SQLiteDatabase db) and onUpgrade(SQLiteDatabase db) which are called when the database is created and database version is changed respectively. We call all our CREATE_TABLE static Strings in onCreate which in turn creates all the tables. Something like this :

@Override
public void onCreate(SQLiteDatabase db) {
    db.execSQL(DbContract.Speakers.CREATE_TABLE);
    db.execSQL(DbContract.Sponsors.CREATE_TABLE);
    db.execSQL(DbContract.Sessions.CREATE_TABLE);
    db.execSQL(DbContract.Tracks.CREATE_TABLE);
    db.execSQL(DbContract.Sessionsspeakers.CREATE_TABLE);
    db.execSQL(DbContract.Event.CREATE_TABLE);
    db.execSQL(DbContract.Microlocation.CREATE_TABLE);
    db.execSQL(DbContract.Versions.CREATE_TABLE);
    db.execSQL(DbContract.Bookmarks.CREATE_TABLE);
    db.execSQL(DbContract.EventDates.CREATE_TABLE);
}

You can also call DELETE_TABLE Strings in onUpgrade and the call onCreate again if you like but it’s not compulsory.

Now that you’re database is initialised, let’s add some records into it. For example I have to add a new Micrlocation I’d create a method in my data model where I’ll add a basic structure for the query and then format it with the values for a particular object of the model. Something, like this

public String generateSql() {
    String insertQuery = "INSERT INTO %s VALUES ('%d', %s, '%f', '%f', '%d');";
    return String.format(Locale.ENGLISH,
            insertQuery,
            DbContract.Microlocation.TABLE_NAME,
            id,
            DatabaseUtils.sqlEscapeString(StringUtils.optionalString(name)),
            latitude,
            longitude,
            floor);
}

and then I’d execute the string returned by the call

String query = model.generateSql();

by this

public void insertQuery(String query, DbHelper mDbHelper) {
    SQLiteDatabase db = mDbHelper.getWritableDatabase();
    db.beginTransaction();
    db.execSQL(query);
  
    db.setTransactionSuccessful();
    db.endTransaction();
}

Where db is just a SQLiteDatabase instance.

Now that we have records we want to retrieve them according to usage and for that we create helper methods. This is an example of the retrieving all the microlocations added to the database in ASCENDING order of NAME

public ArrayList<org.fossasia.openevent.data.Microlocation> getMicrolocationsList(SQLiteDatabase mDb) {
    String sortOrder = DbContract.Microlocation.NAME + ASCENDING;
    Cursor cursor = mDb.query(
            DbContract.Microlocation.TABLE_NAME,
            DbContract.Microlocation.FULL_PROJECTION,
            null,
            null,
            null,
            null,
            sortOrder
    );

    ArrayList<org.fossasia.openevent.data.Microlocation> microlocations = new ArrayList<>();
    org.fossasia.openevent.data.Microlocation microlocation;

    cursor.moveToFirst();
    while (!cursor.isAfterLast()) {
        microlocation = new org.fossasia.openevent.data.Microlocation(
                cursor.getInt(cursor.getColumnIndex(DbContract.Microlocation.ID)),
                cursor.getString(cursor.getColumnIndex(DbContract.Microlocation.NAME)),
                cursor.getFloat(cursor.getColumnIndex(DbContract.Microlocation.LATITUDE)),
                cursor.getFloat(cursor.getColumnIndex(DbContract.Microlocation.LONGITUDE)),
                cursor.getInt(cursor.getColumnIndex(DbContract.Microlocation.FLOOR))
        );
        microlocations.add(microlocation);
        cursor.moveToNext();
    }
    cursor.close();
    return microlocations;
}

First we create a cursor and then just iterate of the cursor to get microlocation objects and add them to an Arralist and return the Arraylist to the calling method.

So This are most of the things that are there to handling data in Android. Should be sufficient to get you started.

Sorry for the long post but the content couldn’t be made any smaller but I hope you gain something from this post. You can checkout implementations I have followed for the Open event project in the github repo https://github.com/fossasia/open-event-android. You can also write to me anytime on FB, Twitter, Email etc. and I’ll be happy to answer any queries. Adios!

References : 1) developers.android.com

2) https://github.com/fossasia/open-event-android